Retrieval-augmented generation looks deceptively simple: index your documents, retrieve relevant passages, let a language model answer with them. A weekend of work gets you a demo that answers most questions impressively. The distance between that demo and a system your team can rely on is engineering in four places the demo skips.

1. Retrieval is where accuracy is won or lost

When a grounded assistant answers wrongly, the cause is usually not the language model — it is that the right passage never reached it. Real corpora are hostile to naive indexing: tables that lose meaning when flattened, standards where one clause modifies another fifty pages away, drawings, multilingual archives, near-duplicate versions of the same manual. Chunking strategy, hybrid search, and metadata filtering tuned to your document types matter more than which model answers.

2. Citations must be enforced, not encouraged

An answer without a source is an opinion. The system should attach the exact passages behind every claim, and — harder but more important — refuse when the corpus does not contain an answer. "I don't know, and here is where I looked" is a feature. A confident fabrication in a compliance or engineering context is how these systems lose trust permanently, and trust does not come back.

3. Evaluation before rollout, not after complaints

Before anything goes live, collect the questions your team actually asks — a hundred real ones beat a thousand synthetic ones — with reference answers from your experts. Measure retrieval hit rate and answer groundedness against that set. This becomes the gate for going live and the regression test for every later change. Teams that skip this learn their system's failure modes from their angriest users.

4. The unglamorous parts decide adoption

Access control that respects who may see which documents. Logging that lets you audit any answer after the fact. A pipeline for adding, updating, and retiring documents without an engineer. Cost and latency measured before you scale, not discovered on the invoice. None of this demos well; all of it decides whether the system is still in use a year later.

The honest take

We build these systems for clients, and we build our own language-AI products — so we say this from both sides of the table: if a vendor shows you a RAG demo without an evaluation set, you are looking at the easy 20%. Ask them how the system knows when it doesn't know. The answer tells you everything.