A language model is a closed-book student. It sat one enormous exam, memorised everything it could into its weights, and now answers from memory alone — confidently, fluently, and sometimes completely wrong, because it cannot tell the difference between remembering and inventing. Retrieval-Augmented Generation turns that into an open-book exam : before answering, the model is handed the three pages of your documentation that actually contain the answer. That single change fixes freshness, fixes attribution, and dramatically reduces invention — and it turns out that almost all the engineering, and almost all the failure, lives in the unglamorous half : finding the right three pages. This post builds the whole pipeline from first principles, with the geometry, the ranking formulas, and the honest limits.
1. Why a model needs a library
A Transformer stores what it learned in its weights. That parametric knowledge has three structural problems, and none of them is a bug you can train away :
- It is frozen. The weights encode the world as of the training cut-off. Yesterday’s incident report, this morning’s price change, your company’s internal wiki — none of it is in there, and none of it ever will be.
- It is unattributable. The model cannot point at where it learned something, because the fact is not stored anywhere in particular : it is smeared across billions of parameters. “Cite your source” is not a request a purely parametric model can honour.
- It is confidently lossy. Training compresses an enormous corpus into a fixed number of weights. Compression loses detail, and when the model reconstructs a half-remembered detail it produces something fluent rather than something true. That is the mechanism behind hallucination.
The instinctive fix — fine-tune on your documents — addresses none of them properly. Fine-tuning is expensive, must be redone whenever the documents change, still gives no citations, and teaches style far more reliably than it teaches facts.
The other fix is embarrassingly simple : do not make the model remember, let it look things up. Put the documents in a searchable store ; when a question arrives, find the passages most likely to contain the answer ; paste them into the prompt ; ask the model to answer from those passages and cite them. That is Retrieval-Augmented Generation, introduced under that name by Lewis et al. (Lewis et al., 2020).
In plain words : the retriever is a librarian who fetches a handful of pages, and the generator is a reader who answers using only those pages. Every property people like about RAG — freshness, citations, less invention — comes from the fact that the answer is conditioned on text the system can point at.
Notice the asymmetry in the figure : the top lane runs once (or whenever documents change), the bottom lane runs on every single question. That is what makes the design practical — the expensive work of reading the whole corpus is amortised into an index, and a query only touches $k$ passages.
2. Turning meaning into geometry : embeddings
For the librarian to work, “relevant to this question” must become something a computer can compute. Keyword matching is not enough : a user asking “how do I reset my password ?” should be handed a page titled “Recovering account access”, which shares not a single content word with the question.
The tool for that is the embedding : a function $\phi$ mapping a piece of text to a vector in $\mathbb{R}^d$ (typically $d$ between $384$ and $3072$), trained so that texts with similar meaning land close together. Modern embedders are Transformer encoders trained with a contrastive objective — pairs that mean the same thing are pulled together, unrelated pairs pushed apart (Reimers & Gurevych, 2019). Training them specifically on question–passage pairs, rather than on generic sentence similarity, is what made dense retrieval competitive with keyword search in the first place (Karpukhin et al., 2020). The result is a space where direction encodes meaning.
Which is why the similarity measure is the cosine of the angle between two vectors, not their distance :
Why the angle and not the raw distance ? Because length is noise here. A long chunk that repeats a term produces a longer vector than a short chunk that says the same thing once ; ranking by dot product alone would systematically prefer the verbose one. Dividing by the norms throws that away and keeps only the direction — the meaning.
In practice every vector is normalised to unit length once, at index time. That is not just an optimisation, it makes two apparently different questions the same question :
Proof. Expanding the squared norm, $$ \lVert u - v \rVert^{2} = (u-v)\cdot(u-v) = \lVert u \rVert^{2} + \lVert v \rVert^{2} - 2\,u \cdot v = 1 + 1 - 2\cos\theta. $$ The right-hand side is a strictly decreasing function of $\cos\theta$, so the two orderings are reverses of one another and select the same top-$k$.
$\square$
This is a genuinely useful fact : it means a vector database that only knows how to find Euclidean nearest neighbours is a perfectly good cosine search engine, provided you normalise on the way in. It also means the dot product alone suffices, since $u \cdot v = \cos\theta$ for unit vectors — and a dot product against a whole index is one matrix multiplication.
3. Chunking : the boring step that decides everything
You cannot embed a 400-page manual as one vector. A single vector has a fixed budget of meaning ; average an entire manual into it and you get a vector that means “this is a manual” and nothing else. So documents are cut into chunks, and each chunk is embedded separately.
This is where most RAG systems are silently won or lost, because chunking sets a hard ceiling on what retrieval can ever do :
- Chunks too large : the embedding is diluted (many topics averaged into one direction), and each retrieved chunk burns context on mostly irrelevant text.
- Chunks too small : the chunk no longer carries enough context to be interpretable. A chunk reading “It must be renewed every 90 days.” is useless — the reader cannot tell what “it” is.
The standard mitigation is overlap : consecutive chunks share a tail, so a sentence near a boundary appears in both and keeps its neighbourhood. Overlap is not free, and the cost is easy to quantify. With chunk size $c$ and overlap $o < c$, consecutive chunks start every $c - o$ characters, so a document of length $L$ produces
\[\begin{equation} n \;=\; \left\lceil \frac{L - o}{\,c - o\,} \right\rceil \quad\text{chunks, storing}\quad \frac{n\,c}{L} \;\approx\; \frac{c}{c - o} \quad\text{times the original text.} \label{eq:chunks} \end{equation}\]In plain words : a 50 % overlap ($o = c/2$) doubles the size of your index, and a 75 % overlap quadruples it. That is the trade you are making — index size and query cost against the risk of cutting an answer in half.
4. Finding the neighbours fast
Now the search problem. Given the query vector $q$ and an index of $N$ chunk vectors, we want the $k$ largest cosines. Exact search computes all $N$ dot products :
\[\begin{equation} \text{cost} \;=\; \mathcal{O}(N d) \quad\text{per query.} \label{eq:exact} \end{equation}\]For $N = 10^{4}$ that is nothing — a single NumPy matrix multiply, microseconds, and you should absolutely do it. For $N = 10^{8}$ chunks and $d = 1024$ it is $10^{11}$ multiply-adds per question, which is not a search engine, it is a batch job.
So production systems give up exactness. Approximate nearest neighbour (ANN) search accepts a small probability of missing a true neighbour in exchange for orders of magnitude in speed — the very same bargain struck by Bloom filters (a small false-positive rate buys a tiny memory footprint) and HyperLogLog (a few percent of counting error buys kilobytes instead of gigabytes). Approximation is the standard currency for buying scale.
The dominant method is HNSW, Hierarchical Navigable Small World graphs (Malkov & Yashunin, 2020). The idea is a skip list in disguise :
- Build a graph where each vector is a node linked to some of its nearest neighbours.
- Stack several such graphs in layers. A node appears in layer $\ell$ with probability decaying geometrically, so the top layer holds a handful of nodes and the bottom layer holds all $N$.
- To search, enter at the top layer and walk greedily — repeatedly step to the neighbour closest to the query — until no neighbour improves. Then drop one layer and repeat, starting from where you landed.
In plain words : the top layers are the motorway network, with a few nodes and very long hops that cross the whole space in a couple of moves. The bottom layer is the local street map. You take the motorway to get roughly there, then exit and navigate street by street. Each layer contributes a constant expected number of hops, and there are $\mathcal{O}(\log N)$ layers, so a query costs
\[\begin{equation} \mathcal{O}(d \log N) \quad\text{instead of}\quad \mathcal{O}(N d). \label{eq:hnsw} \end{equation}\]At $N = 10^{8}$ that is the difference between a hundred million comparisons and a few hundred. The price is recall below 1 : the greedy walk can get stuck in a local minimum and miss a true neighbour. Every ANN index exposes a knob (efSearch in HNSW, nprobe in IVF-style indexes (Johnson et al., 2021)) that trades latency for recall, and that knob is one of the few in a RAG system with an honest, measurable meaning.
5. Where dense retrieval fails : lexical search and hybrid
Embeddings are excellent at meaning and surprisingly bad at strings. Ask for error code ORA-01555 or part number MX-4471-B, and the embedder — which never saw that token during training — maps it to a vague direction near “technical identifier”. Semantically similar, lexically useless : you will get the page about a different error code.
The classical answer is still the right one. BM25 ranks documents by weighted term overlap (Robertson & Zaragoza, 2009) :
\[\begin{equation} \operatorname{BM25}(q,d) \;=\; \sum_{t \in q} \operatorname{IDF}(t)\; \cdot\; \frac{f(t,d)\,(k_1 + 1)}{f(t,d) + k_1\Big(1 - b + b\,\dfrac{\lvert d \rvert}{\text{avgdl}}\Big)}, \label{eq:bm25} \end{equation}\]where $f(t,d)$ is how often term $t$ occurs in document $d$, $\lvert d \rvert$ is the document length, $\text{avgdl}$ the mean length, and $k_1 \approx 1.5$, $b \approx 0.75$ are the usual constants. Two ideas are worth extracting from that formula, because both are good engineering :
- Saturation. As $f(t,d) \to \infty$ the fraction tends to $k_1 + 1$, a finite ceiling. A document mentioning a term twenty times is not twenty times more relevant than one mentioning it once — the tenth occurrence tells you almost nothing new. $k_1$ controls how fast the credit saturates.
- Length normalisation. The $b \frac{\lvert d \rvert}{\text{avgdl}}$ term penalises long documents, which would otherwise win purely by containing more words. It is the same instinct as dividing by $\lVert v \rVert$ in the cosine.
And $\operatorname{IDF}(t) = \ln!\left(\frac{N - n_t + 0.5}{n_t + 0.5} + 1\right)$, with $n_t$ the number of documents containing $t$, makes rare terms count for much more than common ones — which is exactly why BM25 nails the part number that the embedder smeared away.
So run both retrievers and merge. The catch is that a BM25 score of $14.2$ and a cosine of $0.83$ live on incomparable scales, and normalising them is a fiddly, corpus-dependent mess. The robust trick is to throw the scores away and keep only the ranks — Reciprocal Rank Fusion (Cormack et al., 2009) :
\[\begin{equation} \operatorname{RRF}(d) \;=\; \sum_{r \in \text{rankers}} \frac{1}{\kappa + \operatorname{rank}_r(d)}, \qquad \kappa \approx 60. \label{eq:rrf} \end{equation}\]In plain words : each ranker votes, a first place is worth more than a second, and nobody has to agree on what a “score” means. The constant $\kappa$ flattens the top of the curve so that the difference between rank 1 and rank 2 does not overwhelm the fact that a document was ranked well by both systems.
def rrf(rankings, kappa=60):
"""Fuse ranked ID lists. Scores are never compared."""
fused = {}
for ranking in rankings:
for r, doc in enumerate(ranking, start=1):
fused[doc] = fused.get(doc, 0) + 1 / (kappa + r)
return sorted(fused, key=fused.get, reverse=True)
Hybrid retrieval is, in my experience, the single highest-return change in a mediocre RAG system — well ahead of swapping the generator for a bigger one.
6. Reranking : cheap recall, then expensive precision
There is a structural reason retrieval is not very precise, and understanding it explains the standard architecture.
The retriever uses a bi-encoder : the question and the chunk are encoded separately, into $\phi(q)$ and $\phi(c)$, and compared by a cosine. That separation is what makes indexing possible — every $\phi(c)$ is computed once, offline, and a query only needs one new embedding. But it also means the encoder must summarise a chunk into a single vector without knowing what will be asked. Information is necessarily lost.
A cross-encoder does the opposite : it feeds the pair $(q, c)$ through a Transformer together, so every token of the question can attend to every token of the chunk before a single relevance score comes out (Nogueira & Cho, 2019). Far more accurate — and unusable as a search index, because nothing can be precomputed : ranking $N$ chunks needs $N$ forward passes per query.
Hence the two-stage design that every serious system converges on :
\[\begin{equation} \underbrace{N = 10^{6}\;\text{chunks}}_{\text{corpus}} \;\xrightarrow[\ \mathcal{O}(d\log N)\ ]{\text{bi-encoder + ANN}}\; \underbrace{50\;\text{candidates}}_{\text{high recall}} \;\xrightarrow[\ 50\ \text{forward passes}\ ]{\text{cross-encoder}}\; \underbrace{5\;\text{passages}}_{\text{high precision}} \label{eq:twostage} \end{equation}\]In plain words : cast a wide, cheap net to make sure the answer is somewhere in the catch, then pay for a careful reading of fifty candidates to pick the best five. The first stage is optimised for recall (do not lose the right chunk), the second for precision (put it first). Reranking fifty candidates is a fixed, affordable cost that does not grow with the corpus.
7. The generation half
The retrieved passages now go into the prompt. This half is shorter, but two things matter.
The instruction must be grounding, not decoration. The generator’s default behaviour is to answer from parametric memory ; it has to be told, explicitly, to prefer the context, to cite, and — critically — that refusing is an acceptable answer. Without that last clause the model will always produce something.
Structure the context. Number the passages and ask for the numbers back. Citations are not a nicety : they are what makes the output checkable, and a claim carrying a chunk number can be verified mechanically against the chunk.
import numpy as np
import anthropic
client = anthropic.Anthropic()
SYSTEM = (
"Answer using only the numbered context passages. "
"Cite the passage number for every claim, like [3]. "
"If the context does not contain the answer, say so "
"plainly instead of guessing."
)
def top_k(query_vec, index, k=5):
"""index : (N, d) matrix of unit-norm chunk vectors."""
scores = index @ query_vec # cosine, rows are unit
best = np.argpartition(-scores, k)[:k]
return best[np.argsort(-scores[best])]
def answer(question, chunks, hits):
context = "\n\n".join(f"[{i}] {chunks[i]}" for i in hits)
prompt = f"<context>\n{context}\n</context>\n\n{question}"
response = client.messages.create(
model="claude-opus-5",
max_tokens=16000,
system=SYSTEM,
messages=[{"role": "user", "content": prompt}],
)
return "".join(
b.text for b in response.content if b.type == "text"
)
That is a complete, working RAG loop in about twenty lines. Everything else in this post — chunking strategy, hybrid search, ANN indexes, reranking — is about making the hits on line three of answer contain the right passages. Which brings us to the only thing that tells you whether they do.
8. Measuring it, and the ceiling you cannot cross
RAG systems fail in two completely different ways, and lumping them together is why so many are tuned by superstition. Always evaluate the two stages separately.
Retrieval metrics need a set of questions with known relevant chunks.
\[\begin{equation} \operatorname{Recall}@k = \frac{\lvert R(q) \cap \text{relevant}(q) \rvert}{\lvert \text{relevant}(q) \rvert}, \qquad \operatorname{MRR} = \frac{1}{\lvert Q \rvert}\sum_{q \in Q} \frac{1}{\operatorname{rank}_q}, \label{eq:recall} \end{equation}\]where $\operatorname{rank}_q$ is the position of the first relevant chunk. Recall@$k$ asks “did we get it at all ?”, MRR asks “how near the top ?”. When relevance has degrees, nDCG (Järvelin & Kekäläinen, 2002) weights each hit by its usefulness and discounts it by its position :
\[\begin{equation} \operatorname{DCG}@k = \sum_{i=1}^{k} \frac{2^{\text{rel}_i} - 1}{\log_2(i+1)}, \qquad \operatorname{nDCG}@k = \frac{\operatorname{DCG}@k}{\operatorname{IDCG}@k}. \label{eq:ndcg} \end{equation}\]Generation metrics are about faithfulness : is every claim in the answer supported by a retrieved passage, and does the answer address the question ? Both are usually scored by a second model acting as a judge, against the passages actually shown.
Now the result that should govern where you spend your time :
In plain words : if the right chunk is only retrieved 70 % of the time, your system cannot be right more than 70 % of the time — no matter which model you plug in, how you word the prompt, or how much you pay per token. The generator can only ever lose accuracy relative to this bound, never gain it.1
This gives a diagnostic that costs nothing and settles most arguments. When an answer is wrong, look at what was retrieved before touching anything else :
- The right chunk was not retrieved → the fault is in chunking, embedding, or search. Changing the prompt is theatre.
- The right chunk was retrieved and the answer is still wrong → now it is a generation problem : prompt, context ordering, or model.
9. Where RAG breaks
RAG is a retrieval system with a language model attached, and it inherits every limitation of retrieval.
It cannot aggregate. “How many of our customers are on the enterprise plan ?” requires scanning the whole corpus ; retrieval returns $k$ passages. The system will confidently count the ones it happened to see and report a number that is not merely wrong but plausibly wrong. RAG is not a database, and questions of the form count / sum / compare all need a different tool — usually generated SQL, or a MapReduce-style pass over the full corpus.
It cannot chain. “Who manages the person who wrote the incident report on the payments outage ?” needs two hops : find the report’s author, then look up their manager. One retrieval round keyed on the original question will rarely surface both facts, because the second query cannot be formed until the first is answered. Multi-hop demands an agentic loop that retrieves, reads, and retrieves again.
Long context is not free attention. Stuffing fifty passages in “just in case” degrades accuracy : models recover information near the beginning and the end of a long context far more reliably than in the middle, the lost in the middle effect (Liu et al., 2024). More context is not more knowledge, and the reranker’s job is precisely to keep the number of passages small and their order meaningful.
The index goes stale silently. Embeddings are computed once. Change a document and the index still serves the old vector, with no error and no warning — the system confidently answers from a version of the truth that no longer exists. Re-indexing is an operational obligation, not an optimisation.
Changing the embedding model invalidates everything. Vectors from two different embedders are not comparable — they are coordinates in unrelated spaces. Upgrading the embedder means re-embedding the entire corpus, which for a large index is the most expensive routine operation in the whole system.
10. Conclusion
Retrieval-Augmented Generation is best understood not as a clever prompting trick but as an information retrieval system that happens to end in a language model. That framing tells you where the difficulty is, and it is not where most teams look : it is in cutting documents into units that mean something, in mapping meaning to geometry well enough that a cosine ranks correctly, in combining semantic and lexical search because neither alone is sufficient, and in spending a little compute on a careful second read.
The retrieval ceiling is the sentence to remember. Your system cannot be more correct than its retrieval is complete, so the first question about any wrong answer is never “which model should we try ?” but “was the right passage even in the prompt ?” Answer that honestly, with measurements, and the rest of the engineering follows. Answer it by intuition, and you will spend months tuning the half that was never broken.
References
- Cormack, G. V., Clarke, C. L. A., & Buettcher, S. (2009). Reciprocal Rank Fusion Outperforms Condorcet and Individual Rank Learning Methods. Proceedings of the 32nd International ACM SIGIR Conference on Research and Development in Information Retrieval, 758–759.
@inproceedings{Cormack2009, author = {Cormack, Gordon V. and Clarke, Charles L. A. and Buettcher, Stefan}, title = {Reciprocal Rank Fusion Outperforms Condorcet and Individual Rank Learning Methods}, booktitle = {Proceedings of the 32nd International ACM SIGIR Conference on Research and Development in Information Retrieval}, pages = {758--759}, year = {2009} } - Johnson, J., Douze, M., & Jégou, H. (2021). Billion-Scale Similarity Search with GPUs. IEEE Transactions on Big Data, 7(3), 535–547.
@article{Johnson2021, author = {Johnson, Jeff and Douze, Matthijs and J{\'e}gou, Herv{\'e}}, title = {Billion-Scale Similarity Search with GPUs}, journal = {IEEE Transactions on Big Data}, volume = {7}, number = {3}, pages = {535--547}, year = {2021} } - Järvelin, K., & Kekäläinen, J. (2002). Cumulated Gain-Based Evaluation of IR Techniques. ACM Transactions on Information Systems, 20(4), 422–446.
@article{Jarvelin2002, author = {J{\"a}rvelin, Kalervo and Kek{\"a}l{\"a}inen, Jaana}, title = {Cumulated Gain-Based Evaluation of IR Techniques}, journal = {ACM Transactions on Information Systems}, volume = {20}, number = {4}, pages = {422--446}, year = {2002} } - Karpukhin, V., Oğuz, B., Min, S., Lewis, P., Wu, L., Edunov, S., Chen, D., & Yih, W.-tau. (2020). Dense Passage Retrieval for Open-Domain Question Answering. Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), 6769–6781.
@inproceedings{Karpukhin2020, author = {Karpukhin, Vladimir and O{\u{g}}uz, Barlas and Min, Sewon and Lewis, Patrick and Wu, Ledell and Edunov, Sergey and Chen, Danqi and Yih, Wen-tau}, title = {Dense Passage Retrieval for Open-Domain Question Answering}, booktitle = {Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)}, pages = {6769--6781}, year = {2020} } - Lewis, P., Perez, E., Piktus, A., Petroni, F., Karpukhin, V., Goyal, N., Küttler, H., Lewis, M., Yih, W.-tau, Rocktäschel, T., Riedel, S., & Kiela, D. (2020). Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks. Advances in Neural Information Processing Systems 33 (NeurIPS), 9459–9474.
@inproceedings{Lewis2020, author = {Lewis, Patrick and Perez, Ethan and Piktus, Aleksandra and Petroni, Fabio and Karpukhin, Vladimir and Goyal, Naman and K{\"u}ttler, Heinrich and Lewis, Mike and Yih, Wen-tau and Rockt{\"a}schel, Tim and Riedel, Sebastian and Kiela, Douwe}, title = {Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks}, booktitle = {Advances in Neural Information Processing Systems 33 (NeurIPS)}, pages = {9459--9474}, year = {2020} } - Liu, N. F., Lin, K., Hewitt, J., Paranjape, A., Bevilacqua, M., Petroni, F., & Liang, P. (2024). Lost in the Middle: How Language Models Use Long Contexts. Transactions of the Association for Computational Linguistics, 12, 157–173.
@article{Liu2024, author = {Liu, Nelson F. and Lin, Kevin and Hewitt, John and Paranjape, Ashwin and Bevilacqua, Michele and Petroni, Fabio and Liang, Percy}, title = {Lost in the Middle: How Language Models Use Long Contexts}, journal = {Transactions of the Association for Computational Linguistics}, volume = {12}, pages = {157--173}, year = {2024} } - Malkov, Y. A., & Yashunin, D. A. (2020). Efficient and Robust Approximate Nearest Neighbor Search Using Hierarchical Navigable Small World Graphs. IEEE Transactions on Pattern Analysis and Machine Intelligence, 42(4), 824–836.
@article{Malkov2020, author = {Malkov, Yury A. and Yashunin, Dmitry A.}, title = {Efficient and Robust Approximate Nearest Neighbor Search Using Hierarchical Navigable Small World Graphs}, journal = {IEEE Transactions on Pattern Analysis and Machine Intelligence}, volume = {42}, number = {4}, pages = {824--836}, year = {2020} } - Nogueira, R., & Cho, K. (2019). Passage Re-ranking with BERT. ArXiv Preprint ArXiv:1901.04085.
@article{Nogueira2019, author = {Nogueira, Rodrigo and Cho, Kyunghyun}, title = {Passage Re-ranking with BERT}, journal = {arXiv preprint arXiv:1901.04085}, year = {2019} } - Reimers, N., & Gurevych, I. (2019). Sentence-BERT: Sentence Embeddings Using Siamese BERT-Networks. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing (EMNLP), 3982–3992.
@inproceedings{Reimers2019, author = {Reimers, Nils and Gurevych, Iryna}, title = {Sentence-BERT: Sentence Embeddings Using Siamese BERT-Networks}, booktitle = {Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing (EMNLP)}, pages = {3982--3992}, year = {2019} } - Robertson, S., & Zaragoza, H. (2009). The Probabilistic Relevance Framework: BM25 and Beyond. Foundations and Trends in Information Retrieval, 3(4), 333–389.
@article{Robertson2009, author = {Robertson, Stephen and Zaragoza, Hugo}, title = {The Probabilistic Relevance Framework: BM25 and Beyond}, journal = {Foundations and Trends in Information Retrieval}, volume = {3}, number = {4}, pages = {333--389}, year = {2009} }
-
The bound is an upper limit on grounded correctness. A model can of course answer correctly from its own parametric memory when retrieval fails — but then the citation is missing or fabricated, and you have no way to distinguish that lucky case from a hallucination. Counting it as a success measures the wrong thing. ↩