RAG in production: what changes after the demo
A RAG demo takes an afternoon: load some documents, embed them, ask a question, get a good answer. The production system is a different project. The model is the smallest part of it. Most of the work is in what gets into the index, what comes out of it, who is allowed to see it, and how you know the answers are still right next month.
In short
Eight areas decide whether RAG works in production: ingestion that keeps document structure and metadata; retrieval that combines vectors with keyword search and a reranker; freshness, so changed and deleted documents leave the index; permissions applied before retrieval, not in the prompt; evaluation on a fixed set of questions with known sources; citations and refusals, so the system answers only from what it found; security against instructions hidden in documents; and observability for every stage. Choosing the architecture is covered in a separate article; this one is about running it.
1. Ingestion: what goes into the index
Split documents by their structure, not by a fixed number of characters. A chunk that starts in the middle of a table or cuts a procedure in half retrieves badly and answers worse. Keep headings with the text under them, and keep tables whole.
Store metadata with every chunk: the source, the version, the date it changed, and who may read it. Retrieval quality, freshness, and permissions all depend on it later.
The embedding model is part of the index. Change the model and every vector must be recomputed. Store the model name with the vectors so a partial reindex is visible.
2. Retrieval: what comes out
Pure vector search is weak on exact terms: product codes, part numbers, names, and abbreviations. Hybrid retrieval adds keyword search alongside the vectors and merges the results. A reranker then reorders the top candidates by how well each answers the question.
Tune how many chunks reach the model on your own questions, not by default. Too few and the answer misses context; too many and the relevant passage drowns in noise and the cost grows.
3. Freshness: when documents change
Reindex incrementally when a document changes, and make sure deletions reach the index. A document removed from the source but still present in the index will keep answering questions.
Check deletions by counting, not by the response. Some vector stores report a delete by filter as completed even when the filter matched nothing. Count the points before and after; that is the only proof that old content is gone.
4. Permissions: before retrieval, not in the prompt
If users may see different documents, the filter belongs in the retrieval query, using the permissions stored with each chunk. Telling the model not to reveal what the user may not see is not access control: once a chunk is in the context, it can make its way into the answer.
5. Evaluation: how you know it still works
Build a fixed set of real questions, each with the documents that should answer it. Measure two things separately: whether retrieval found the right sources, and whether the answer remains faithful to them. Run the set on every change to chunking, embeddings, retrieval settings, or the prompt.
In production, add an LLM-as-judge on a small sample of real answers and monitor user feedback. A drop in either is usually the first sign that the index or the questions have changed.
6. Citations and refusals
Instruct the system to answer only from the retrieved context, cite the source of each claim, and say it does not know when nothing relevant was found. Track the refusal rate: a rising rate points to gaps in the index, while a rate near zero often means the system is guessing.
7. Security: instructions hidden in documents
Retrieved text is data, not instructions. A document, an email, or a web page in the index can contain text written to steer the model, and RAG will deliver it straight into the context. Mark retrieved content as untrusted in the prompt, limit which tools the model can call, and remove personal data before it is indexed or logged.
8. Observability: every stage separately
Trace retrieval, reranking, and generation as separate steps, with latency and the number of chunks and tokens at each stage. When an answer is wrong, the trace shows whether retrieval missed the source or the model ignored it, and the two require different fixes.
Frequently asked questions
What are the best practices for RAG in production?
Structure-aware chunking with metadata, hybrid retrieval with a reranker, incremental reindexing that also removes deleted documents, permissions applied in the retrieval query, evaluation on a fixed question set, citations with refusals when nothing relevant is found, protection against instructions inside documents, and tracing at every stage.
Why does a RAG system that worked in the demo fail in production?
Real documents are messier than the demo set, they change and get deleted, different users may see different documents, and no one measures quality after launch. Each of these needs its own mechanism, not a better model.
How do you evaluate a RAG system?
With a fixed set of real questions, each linked to the documents that should answer it. Retrieval is measured by whether the right sources were found, and the answer by whether it remains faithful to them. The set runs on every change, and a sample of production answers is checked continuously.
Where do permissions belong in RAG?
In the retrieval query, as a filter on permissions stored with each chunk. Instructions in the prompt are not access control.