How to take a RAG system from prototype to production.
A chat interface and a vector database can make a convincing demo. Production requires evidence that the right people get grounded answers, at an acceptable cost, when the data and model inevitably change.
A production RAG system is not a prompt wrapped around search. It is a controlled information system with a language model at the final step.
Define success before choosing the stack
Start with the decision or task the user is trying to complete. Then build a small set of representative questions with expected evidence, acceptable answers, and known failure cases. Without this set, teams tune retrieval by intuition and cannot tell whether a change improved the product.
Track answer quality alongside groundedness, citation correctness, response time, and cost. Different workflows weight these differently: a support assistant may value speed, while a regulated review workflow may prioritise traceability and abstention.
Treat retrieval as a system
Chunk size is only one lever. Production retrieval depends on document parsing, metadata, filters, query rewriting, lexical and semantic search, reranking, and the context finally passed to the model.
Inspect retrieval separately from answer generation. If the correct evidence never enters the context, prompt changes cannot recover it. Keep a labelled retrieval set so changes to parsing, embeddings, or ranking can be tested before release.
Preserve permissions end to end
A useful answer can still be a serious failure if it reveals information the user should not see. Carry access rules from the source into the index and enforce them during retrieval, not after the answer has been generated.
Test permission boundaries explicitly. Include users with overlapping roles, revoked access, stale documents, and content that must never appear in logs or traces.
Build evaluation into delivery
Use offline evaluation for repeatable release checks and sampled production review for real-world drift. The test set should include ordinary questions, ambiguous requests, missing information, adversarial phrasing, and examples where the correct behaviour is to say that the evidence is insufficient.
Review failures by category. A retrieval miss, an unsupported synthesis, and a formatting defect need different fixes. One blended score hides that distinction.
Design the failure path
Decide what the system should do when search returns weak evidence, a tool fails, the request is outside scope, or the user asks for restricted information. A safe refusal, a clarifying question, or a handoff is often a better product outcome than a fluent guess.
Keep citations tied to the claims they support. Give users enough source context to verify an answer without making them reconstruct the entire reasoning process.
Make cost and quality observable
Log the retrieval path, selected sources, model version, latency, token usage, and user feedback with appropriate privacy controls. These traces make regressions diagnosable and help teams understand the cost of each workflow.
Set release thresholds and operating budgets before traffic grows. Cache safe repeated work, route simple requests to smaller models when evaluation supports it, and keep model replacement possible rather than binding product logic to one provider.