Every RAG tutorial follows the same arc: chunk some documents, embed them, stuff the top-k results into a prompt, and watch the model produce a surprisingly good answer. That demo takes an afternoon. Getting the same system to hold up under real users, real document churn, and real cost constraints takes a lot longer — and almost none of that work is covered in the tutorials.
Here are five lessons that consistently mattered more than the embedding model I picked.
1. Chunking strategy beats embedding model choice
Teams spend days benchmarking embedding models and minutes on chunking. That ratio is backwards. Fixed-size chunking with naive overlap reliably produces retrieval that's technically relevant and practically useless — a paragraph gets split mid-thought, and the model retrieves half an argument. Structure-aware chunking (splitting on headings, list boundaries, and semantic sections instead of raw character counts) fixed more retrieval-quality complaints than any model swap did.
2. Retrieval needs a relevance floor, not just top-k
Always returning the top-k results means always returning something — even when nothing in the index is actually relevant to the query. That's how you get confident, fluent, wrong answers. Adding a similarity-score threshold, below which the system says "I don't have enough information" instead of guessing, did more for trust than any prompt-engineering tweak.
3. Evaluate retrieval and generation separately
A bad answer can come from two very different failures: the retriever pulled the wrong context, or the generator ignored good context. Debugging both at once wastes time. Keeping a small, hand-labeled eval set that scores retrieval precision/recall independently from answer quality made it obvious, within minutes, which half of the pipeline actually regressed after a change.
- Log retrieved chunk IDs alongside every generated answer, not just the final text.
- Track retrieval precision/recall on a fixed eval set on every pipeline change.
- Score answer faithfulness against the retrieved context, not against "correctness" in the abstract.
4. Re-indexing strategy is a production concern, not an afterthought
Source documents change. A pipeline that only handles the initial bulk index and has no story for incremental updates, deletions, or versioning will quietly serve stale answers for weeks before anyone notices. Building the diff-and-reindex path on day one — even a simple one — is far cheaper than retrofitting it after the index has drifted from reality.
5. Cost scales with context window, not with cleverness
It's tempting to solve marginal retrieval-quality issues by just stuffing more chunks into the prompt. That works, right up until token costs and latency make the feature economically unworkable at scale. The higher-leverage fix is almost always tighter retrieval — fewer, better chunks — over a bigger context window.
None of this is exotic. It's the unglamorous 80% of RAG work that doesn't show up in a weekend demo, but is exactly what separates a proof-of-concept from something you can put in front of real users.