How to Build a RAG Chatbot: A Production Architecture Guide

Why RAG is the architecture of choice for enterprise chatbots

Retrieval-Augmented Generation (RAG) bridges the gap between static model knowledge and your private data. Unlike fine-tuning, which modifies the model weights, RAG provides relevant context to the model at inference time, reducing hallucinations and allowing for real-time data updates. By injecting retrieved data directly into the prompt, you ensure that the AI remains grounded in your specific documents.

A production RAG system requires a clean pipeline: document ingestion, text chunking, embedding generation, and finally, semantic retrieval. Without this structure, your model will either be blind to your data or overwhelmed by irrelevant information. This flow is necessary because standard LLMs cannot natively query your internal databases or private file storage during their initial training phase.

When moving beyond a prototype, the focus shifts to maintaining performance. You must account for how your system fetches data quickly and how that data is formatted before hitting the LLM. If your retrieval process is inefficient or if the data ingested is messy, the LLM will struggle to synthesize an accurate answer, leading to the exact inaccuracies RAG aims to prevent.

How to handle document ingestion and vectorization

Document ingestion is the first failure point in many RAG systems. You should never process files directly in your main request-response cycle. Instead, trigger a background task to handle reading, cleaning, and chunking. When a user uploads a large PDF, synchronous processing would likely cause your web server to time out, resulting in a poor user experience and potential request termination by your load balancer.

Chunking strategy is critical. If your chunks are too small, the model lacks context; if they are too large, you hit token limits and introduce noise. For most text-based applications, overlapping windows of 500-1000 characters perform consistently. This overlap is vital because it preserves the continuity of thought that might be severed if a sentence or concept is split arbitrarily across two distinct vector storage entries.

During the cleaning phase, strip out unnecessary artifacts like HTML tags, excessive whitespace, or formatting characters that do not add semantic value. These artifacts consume tokens and add noise to the vector representation, which can lead to lower quality similarity matches during the query phase. Once cleaned, the text is passed to an embedding model to generate the vector representation that will eventually reside in your database.

from celery import Celery

celery_app = Celery('tasks', broker='redis://localhost:6379/0')

@celery_app.task
def process_document(file_path, user_id):
    text = extract_text_from_file(file_path)
    chunks = chunk_text(text, size=1000, overlap=200)
    for chunk in chunks:
        vector = generate_embedding(chunk)
        save_to_vector_db(vector, chunk, user_id)

Which vector store should you use for RAG?

Choosing between a dedicated vector store and a relational database depends on your scale and current infrastructure. For many developers, using the pgvector extension in PostgreSQL is sufficient and significantly reduces operational overhead. By keeping vectors in the same database as your application metadata, you avoid the complexity of managing an additional infrastructure component or worrying about synchronization issues between two separate data systems.

If your application scales to millions of vectors or requires complex filtering at high concurrency, dedicated stores like Pinecone or Weaviate offer optimized query paths. These dedicated systems use specialized data structures like HNSW (Hierarchical Navigable Small World) graphs or inverted file indexes to provide faster search results than a standard table scan. However, start with what you know to keep the system simple.

Operational complexity is a hidden cost. Managed services provide convenience but introduce external dependencies and costs that grow with your data. If you are already using PostgreSQL, explore the pgvector extension first, as it offers a solid balance between features and ease of use for early-stage development while still allowing for growth as your indexing needs become more demanding.

How to connect the LLM to your vector store

Once your data is vectorized, your retrieval function must perform a similarity search based on the user's query. Use cosine similarity to find the most relevant chunks in your database. This metric is the standard for comparing the orientation of vectors, allowing you to identify which pieces of information are mathematically closest to the semantic meaning of the user's input.

The resulting text context must be injected into your prompt template. Ensure you clearly define the role of the bot in your system instruction to prevent the model from answering questions outside the retrieved context. For example, explicitly telling the model to ignore prior knowledge and rely solely on the provided snippets minimizes the risk of generating inaccurate information about topics it was trained on but that are not present in your specific data.

Formatting is also essential. When you pass retrieved chunks to the LLM, use clear separators to demarcate where the retrieved information ends and where the user's question begins. This separation helps the model differentiate between the context it must analyze and the instructions it must follow, resulting in a more focused and accurate final output.

def get_bot_response(user_query, user_id):
    query_vector = generate_embedding(user_query)
    context = db.query_similar(query_vector, user_id, limit=3)
    
    prompt = f"Use the following context to answer the question: {context}\nQuestion: {user_query}"
    return call_llm(prompt)

Ensuring production scalability with background workers

A production RAG chatbot must be responsive. If you perform vector generation synchronously during the chat, the user will experience long delays. Use a dedicated worker queue to handle document uploads, scraping, and embedding updates. This creates a clean separation between the user-facing chat API and the resource-heavy ingestion pipeline, ensuring that the chat interface remains fluid even when large documents are being processed.

By offloading these tasks, your Flask or FastAPI server remains lightweight and available to handle chat requests. Ensure your database connections are pooled correctly to prevent exhaustion during spikes in traffic. If your workers are constantly opening and closing connections to the database, you risk reaching the connection limits of your database instance, which would bring your entire system to a halt.

Monitor your background worker queues closely. When a backlog of documents grows, user experience may decline as their new data remains unavailable for search. Implementing a priority queue or scaling your number of worker nodes dynamically based on the queue size can prevent these bottlenecks and ensure that new user information is searchable as quickly as possible.

Gotchas and performance trade-offs

Be wary of over-indexing. Every document you store incurs a cost in embedding API usage and database storage. Implement a cleanup routine to remove stale data or vectors associated with deleted user accounts. Over time, failing to prune your vector database will result in degraded performance and unnecessary cloud costs, as the system must sort through irrelevant or obsolete data to find meaningful results.

Security is often overlooked in RAG systems. Ensure your query retrieval is scoped to the user_id or org_id at the database level. Failing to do this can lead to cross-tenant data leakage, where one user retrieves sensitive information from another's documents. Always include an ID filter in your SQL queries when performing similarity searches to ensure that the retrieved context is restricted to the data that the specific user has permission to see.

Finally, consider the latency of your embedding model. While higher-dimensional models might offer better accuracy, they also take longer to compute and store. Test multiple embedding models to find one that fits your specific accuracy requirements without adding excessive latency to your ingestion pipeline. Optimization here often yields the highest return in terms of system responsiveness.

Frequently asked questions

Sources & further reading

• ai.google.dev/gemini-api/docs/file-search
• ai.google.dev/gemini-api/docs/embeddings