Programming
RAG Chatbot System Design: A Production-Ready Architecture
Architecting a multi-tenant RAG system requires careful isolation, reliable background ingestion, and optimized retrieval pipelines.
By Tushar Gautam · · 6 min read
Short answer
A production-ready RAG chatbot system requires decoupled ingestion and retrieval pipelines. Use background jobs like Celery and Redis to process scraped data into vector stores. Implement multi-tenant isolation at the database level and ensure your LLM orchestration handles context injection securely to prevent cross-tenant data leakage during query processing.
Key takeaways
- Decouple document ingestion from query serving to maintain low latency.
- Use unique tenant IDs in your vector database metadata to prevent cross-tenant data leaks.
- Implement background worker queues for scraping and embedding generation to avoid request timeouts.
- Choose a vector database that supports filtering by metadata tags for scalable multi-tenancy.
High-Level Architecture for RAG Chatbots
A high-performance RAG (Retrieval-Augmented Generation) system consists of two primary pipelines: the ingestion pipeline and the retrieval pipeline. The ingestion pipeline focuses on transforming raw, unstructured data into a structured vector format, while the retrieval pipeline handles query execution, searching those embeddings, and injecting relevant context into the LLM prompt.
In a production environment, you should use an asynchronous architecture. Never perform scraping or vector embedding generation inside your main request-response cycle. If your web server attempts to crawl a webpage, parse the content, and generate embeddings while the user waits for a response, the API will time out immediately. Instead, offload these heavy, time-consuming tasks to background workers like Celery and Redis.
This decoupling allows your API to remain responsive regardless of the document load. When a user requests to add a website to their knowledge base, the API simply pushes a task to the queue and returns an immediate success message. The background workers then pick up the task, handle the network latency associated with scraping, and perform the computation-heavy embedding process offline, ensuring the user experience is not degraded by backend processing delays.
How to Handle Multi-Tenant Data Isolation
In a multi-tenant SaaS environment, the most critical design constraint is preventing a user from accessing or retrieving another user's private data. You must enforce strict isolation by including a unique tenant identifier in every query sent to your vector store. Without this, a query could return relevant but highly sensitive content from the wrong account.
When designing your vector store schema, ensure every document includes a metadata field, such as tenant_id. During the retrieval phase, your search query must include a mandatory filter for this ID. Relying on application-level filtering, where you retrieve all results and then drop the ones not belonging to the user, is extremely dangerous and prone to accidental data leakage if the filtering logic ever fails.
Database-level filtering provides the necessary security layer. By delegating the filter to the vector database engine, you ensure that only documents authorized for a specific tenant are even considered in the similarity search. This method keeps your data secure while maintaining performance, as the database engine can index the tenant_id field for faster querying.
| Isolation Level | Security | Performance | Complexity |
|---|---|---|---|
| Dedicated Index | Highest | Lower | High |
| Metadata Filtering | High | High | Low |
| App-level Filter | Low | High | Low |
The Ingestion Pipeline: Scraped Sites to Vector Embeddings
The ingestion process should be modular. Start by scraping the target data using a reliable client, then clean the HTML to remove headers, footers, and scripts that do not add value to the retrieval process. Once cleaned, split the content into manageable chunks. Using smaller chunks ensures better granularity during search, while overlapping chunks maintain context across boundaries, preventing a topic from being cut off mid-sentence during a split.
Once processed, these chunks must be converted into numerical vector embeddings. This transformation happens using a model that understands the semantic meaning of your text. Send these chunks to the embedding model in small batches to respect API rate limits. Store the resulting vectors alongside your metadata in your vector database, ensuring that every vector is correctly tagged with its original source URL and its owner's tenant_id.
This entire workflow is best handled by background tasks. Generating embeddings for large document sets is computationally expensive and can take significant time. If you attempt this during an HTTP request, you invite high latency and potential failure points that will disrupt the user experience.
from celery import shared_task
@shared_task
def ingest_document_task(tenant_id, document_url):
content = scrape_site(document_url)
chunks = split_text(content)
vectors = generate_embeddings(chunks)
for v in vectors:
db.vectors.insert_one({
'tenant_id': tenant_id,
'vector': v,
'metadata': {'url': document_url}
})Query Time: Optimizing Retrieval for Relevance
Retrieval performance depends on both the vector similarity search and the reranking process. A simple vector search often returns documents that are mathematically similar in the vector space but lack specific contextual relevance to the user's actual prompt. A vector search provides speed, but it may surface noise if the query is ambiguous.
To improve response quality, retrieve a larger pool of potential matches first. By fetching, for example, 20 results instead of 5, you increase the likelihood that the truly relevant documents are included in the set. After this initial retrieval, use a reranking model—a smaller, faster, and more specialized model—to evaluate these 20 chunks specifically against the user query.
The reranker assigns a probability score to each chunk, allowing you to select only the most relevant content before passing it to the primary LLM generation model. This two-step process balances the speed of vector search with the high-precision evaluation of reranking, ensuring your chatbot output is accurate and based on the most pertinent pieces of data from your document store.
Scaling Background Tasks for Large Document Stores
As your user base grows, you will encounter bottlenecks in your task queue. A single-worker setup will eventually fail under the weight of concurrent ingestion requests. Ensure your Redis instance is optimized for high-throughput messaging by adjusting memory settings and persistence strategies. Monitor your Celery worker health closely to track memory usage and task processing time.
For massive document ingestion, consider using a distributed task queue with multiple worker groups. Separate your queues by function, creating a 'high priority' queue for user-facing tasks—like chatbot response generation—and a 'low priority' queue for background-heavy tasks like bulk PDF processing or full-site re-indexing. By dedicating specific worker nodes to these queues, you guarantee that an ingestion backlog does not interfere with the chatbot's latency.
Finally, implement back-off and retry logic in your worker tasks. Scraping and embedding APIs occasionally fail due to network blips. If a task fails, retrying it with an exponential delay prevents the system from crashing while still ensuring the data eventually makes it into your vector store. Read more in my guide on /blog/flask-celery-redis-example.
Gotchas and Trade-offs: When RAG Fails
RAG is not a perfect solution. It performs poorly when the retrieved data is noisy, poorly formatted, or irrelevant. If your scraper pulls in too much boilerplate text, navigation menus, or unrelated footer content, the quality of your chatbot's responses will degrade significantly because the LLM is being distracted by junk context.
Another trade-off is the cost associated with embedding storage and API calls. Generating embeddings for massive datasets costs money, and storing thousands of vectors in a high-performance database adds to your monthly infrastructure bill. Be prepared to implement a clear storage policy, perhaps limiting the number of tokens or total number of documents per tenant, to maintain profitability as your user base grows.
Finally, keep in mind that the LLM has a finite context window. Even if you retrieve perfect documents, injecting too much data into the prompt can cause the model to ignore important details or increase your token consumption unnecessarily. Aim to balance the amount of retrieved context with the model's capabilities to maintain both quality and cost-efficiency.
Frequently asked questions
Why is multi-tenant isolation important for RAG?
Without metadata-level isolation, a query might retrieve data from other tenants, causing critical privacy breaches. Always filter by a unique tenant identifier at the database query level to ensure data segregation.
Should I use a managed vector database or host my own?
Managed services simplify backups, scaling, and maintenance. If you are starting, managed services allow you to focus on building features rather than managing infrastructure. For high-scale requirements, self-hosting provides more control over costs and latency.
How often should I update the vector store?
Update your index whenever the source data changes. If your chatbot relies on live website data, implement a re-indexing strategy using webhooks or periodic background jobs to keep the vector embeddings current with your content.
Sources & further reading
About the author
Tushar Gautam is a full-stack & ai product engineer in India. He builds AI SaaS products such as an AI video ad generator, a RAG chatbot platform and a Razorpay-powered report service, mostly with Next.js, Flask and the Gemini API. See the projects, GitHub or LinkedIn.
This article was drafted with the help of AI (Google Gemini) from official documentation and published by Tushar Gautam. Spotted an error? Let me know.