RAG Architecture — Designing Context-Aware AI Systems
Introduction
You ship a customer-facing chatbot backed by GPT-4. It answers 80% of questions brilliantly. Then a user asks about a product SKU you added last week, a policy change from this morning, or a support ticket filed an hour ago. The model hallucinates confidently. The problem isn’t the model — it’s that the model’s weights are frozen in time and isolated from your operational data. Retrieval-Augmented Generation (RAG) is the architectural pattern that closes that gap: it lets a language model reason over your data, now, without retraining.
The gap between a “system design interview” and a “production system” is massive. “Hands On Distributed System Design Course” newsletter exists to bridge that gap. If you are working for any company no one is going to teach you how system is design or built. Fun part developers rarely document everything. you need to dig through to understand the system. Learn from scratch. Access Private Git Repo Now.
Core Concept: What RAG Actually Does
A language model is a frozen function. Its weights encode general world knowledge up to a training cutoff; it cannot access your database, your docs, your tickets. RAG separates the retrieval problem from the reasoning problem.
The pattern works in two phases:
Indexing phase (offline): Source documents — PDFs, markdown files, database rows, Confluence pages — are chunked into segments, typically 256–512 tokens. Each chunk is embedded into a high-dimensional vector using a model like text-embedding-3-small or bge-large-en. These vectors are stored in a vector database (Pinecone, Weaviate, pgvector, Qdrant). This phase runs continuously as data changes.
Query phase (online): When a user submits a query, that query is embedded using the same embedding model. The vector store performs approximate nearest-neighbor (ANN) search — cosine or dot-product similarity — returning the top-k most semantically similar chunks, typically k=3–10. Those chunks are injected into the LLM prompt as context. The model reasons over retrieved evidence rather than relying solely on its weights.
The critical architectural insight: the LLM is doing grounded reading comprehension, not recall. This is fundamentally different from how models are typically deployed.
Chunking strategy is load-bearing. If chunks are too large, you waste context tokens and dilute relevance. Too small, and you lose surrounding context needed to answer coherently. Semantic chunking — splitting at paragraph or section boundaries rather than fixed token counts — outperforms naive sliding windows. Hierarchical chunking (summary chunk + child detail chunk) handles both coarse-grained and fine-grained queries.
Embedding model selection affects recall ceiling. Retrieval quality is bounded by how well the embedding model captures semantic similarity for your domain. General-purpose models underperform on specialized corpora (legal, medical, code). Fine-tuning an embedding model on domain-specific positive/negative pairs — contrastive training — measurably improves recall. This step is skipped in most initial deployments and becomes the dominant bottleneck at scale.
Reranking separates relevant from merely similar. First-pass ANN retrieval optimizes for speed using approximate algorithms (HNSW, IVF). A reranker — typically a cross-encoder model like Cohere Rerank or a fine-tuned BERT — takes the top-50 ANN results and re-scores them with full cross-attention between query and document. Reranking improves precision substantially and adds 20–100ms latency. At production scale, you run the reranker only when the primary retrieval confidence is below a threshold.
Hybrid search outperforms pure vector search for most production workloads. Combining dense vector similarity with sparse BM25 retrieval (keyword matching) via reciprocal rank fusion (RRF) handles cases where exact-match terms — product IDs, error codes, names — would be missed by semantic search alone. Elasticsearch and OpenSearch support this natively; Weaviate and Qdrant have hybrid modes.
