RAG architecture: build, evaluate and operate it in production

A RAG system does not become reliable merely because it finds passages close to a question. It becomes operational when the team can explain which source was used, why it was ranked, whether it authorised the answer and how quality is measured after every change.

RAG, or retrieval-augmented generation, connects a generative model to external memory queried at request time. The foundational NeurIPS paper published in 2020 already highlighted two benefits: updating knowledge without retraining the whole model and retaining inspectable provenance. Delivering that promise in production requires more than a vector database and a prompt.

The architecture at a glance

This separation matters because a poor answer may originate in a missing document, bad chunking, incomplete retrieval, inaccurate ranking or unfaithful generation. A single aggregate score does not identify the component to repair.

Confirm that RAG is the right tool

RAG fits when an answer depends on an identifiable, changing and inspectable corpus: product documentation, internal procedures, contracts, a catalogue or a knowledge base. It is especially useful when users need visible evidence and knowledge changes faster than a model.

It is not automatically appropriate for transactional or deterministic requests. Live stock should often come from an API or structured SQL query. A critical rule may need a deterministic engine. A small stable corpus may be served by conventional search. One architecture can route between these paths rather than forcing every request into a vector index.

Before prototyping, write ten sample questions with the authoritative source, acceptable answer, required permissions and expected behaviour when information is missing. This brief prevents an impressive demonstration from being mistaken for a verifiable product.

Build a traceable chain from source to citation

Every document needs a stable identifier, version, owner, validity date, language and access level. Each chunk inherits this information and references a location in the source: page, section, paragraph or line where the format permits.

The index is not the system of record. It is a rebuildable retrieval projection. The file, business database or document repository remains responsible for the content. A correction must invalidate the previous version and trigger reindexing without leaving stale chunks accessible.

At minimum, the ingestion pipeline should record the document fingerprint, parser version, chunking strategy, embedding model, indexing date and validation outcome. This metadata makes comparison between system versions and rollback possible.

Our data architecture for AI guide covers governance, pipelines and source ownership. Here, the goal is to turn that foundation into a measurable answer chain.

Chunk around structure, not a magic number

A chunk that is too short loses definitions and conditions that give a sentence meaning. A chunk that is too long dilutes the signal, increases model context and may combine several subjects. A fixed character or token count is a starting point, not a universal rule.

Preserve document units first: title and section, contractual clause, procedure, FAQ entry, code block or table row. Add overlap only when it prevents an idea from being split, because systematic overlap increases duplicate results.

Test at least two strategies on the same questions. Measure whether the evidence passage appears near the top, then inspect context volume and citations. The right granularity for an FAQ is not the right one for a 400-page manual.

Our RAG document-quality guide examines cleaning, versioning and metadata before indexing.

Choose between lexical, vector and hybrid search

Lexical search is effective for rare words, references, acronyms, product names and error messages. Vector search connects different wording with similar meaning. Neither wins for every query.

Hybrid search runs both paths and fuses their rankings. Azure AI Search and Elasticsearch document Reciprocal Rank Fusion, or RRF: incompatible BM25 and vector-similarity scores are not added directly; each result's position in the original lists produces a shared ranking.

Start with simple baselines: lexical alone, vector alone and then hybrid on the same test questions. Retain the least complex option that delivers a reproducible improvement. Hybrid search does not compensate for inconsistent terminology or missing filters.

An extension such as pgvector supports an initial PostgreSQL implementation with exact search and approximate HNSW or IVFFlat indexes. A specialised database becomes relevant when volume, distribution, filtering, latency or operations demand it. The decision must include backup, restoration, monitoring and team skills, not only a query benchmark.

Filter before ranking and rerank only when needed

Structured filters restrict retrieval to authorised and relevant documents: tenant, product, language, country, validity window or confidentiality level. Permissions must be applied before text enters model context. Removing a citation from the interface after generation does not repair a leak.

Initial retrieval should favour recall: the final ranking needs a reasonable candidate set containing the right evidence. A reranker then evaluates each query-passage pair more precisely and reorders the candidates. It can improve the top positions, but adds latency and cost.

Do not run a reranker against the entire index. Retrieve a few dozen candidates, remove duplicates, rerank them and keep only the passages useful to the model. Measure the MRR or nDCG improvement alongside p95 cost. If the top results were already correct, this component may not be justified.

Generate grounded, cited answers that can abstain

The prompt must clearly separate instructions, the user question and retrieved excerpts. It should forbid following instructions contained in documents and require an answer limited to supplied evidence. Every passage needs a citation identifier that the application can resolve to the actual source.

A correct citation is more than a document attached to an answer. The cited passage must support the specific claim, be available to the user and match the version consulted. Validate citations for important claims rather than only at whole-answer level.

Design three outcomes: supported answer, clarification request and insufficient evidence. Forcing an answer every time mechanically increases fabrication. Deliberately unanswerable questions therefore belong in the test set.

Measure retrieval before generation

A useful evaluation set connects each question to one or more expected passages. For every pipeline version, record the ordered result list and calculate several metrics:

• Recall@k: share of questions where at least one expected passage appears in the first k results;
• Precision@k: share of retrieved results that are relevant;
• MRR: rewards an earlier first relevant result;
• nDCG: accounts for ordering and multiple relevance levels.

Recall is critical when missing the source makes successful generation impossible. Precision matters when context is constrained or irrelevant passages distract the model. MRR and nDCG distinguish systems that find the same documents in a different order.

Segment results by question type, language, source and access level. An average can hide excellent FAQ results and complete failure on tables or newly updated documents.

Evaluate the answer separately from its evidence

Once retrieval is satisfactory, measure at least four dimensions:

1. faithfulness: is every factual claim supported by context?
2. relevance: does the answer address the question?
3. completeness: does it contain everything required for the task?
4. citation quality: do references support the correct claims?

RAGAS helped formalise automated measurements of context relevance, faithfulness and answer relevance. These scores accelerate comparisons, but an LLM judge remains an instrument that needs calibration. Compare its decisions with a human-annotated sample, keep explanations and monitor disagreement.

Product scoring must reflect the consequence of error. A low-risk internal assistant may allow a one-click correction. Legal, health or financial decisions require stricter criteria, human review and thresholds. Our guide to evaluating a chatbot before production adds robustness and behavioural tests.

Build a test set that resembles production

Start with real questions, anonymised where necessary, then add cases that expose every stage: synonyms, exact references, spelling errors, multiple languages, tables, long documents, conflicting versions and questions requiring several passages.

Also include:

• questions with no answer in the corpus;
• users with different permissions;
• a deleted or expired document;
• a malicious instruction embedded in a source;
• an ambiguous question requiring clarification;
• an update whose freshness must become visible quickly.

Version questions, expected evidence, annotations and thresholds with pipeline code. Add production incidents to the regression set. Every change to model, embedding, chunking, filtering or prompt should replay the same suite.

Security: treat documents as untrusted input

OWASP includes vector and embedding weaknesses among risks for LLM applications. A corpus may contain poisoned data, prompt-injection instructions, personal information or chunks exposed to the wrong tenant.

Separate indexes where isolation requires it, apply ACLs during retrieval, validate allowed sources and log retrieved identifiers. Scan files before ingestion, restrict formats and prevent documents from modifying system instructions. Explicitly test cross-user and cross-organisation leakage.

Evaluation logs may themselves contain sensitive questions, excerpts and answers. Define retention, access and anonymisation. Observability must not become another uncontrolled copy of the corpus.

Observe latency, cost and drift in production

Trace query rewriting, embedding, retrieval, reranking and generation separately. Record candidate count, final passage count, context tokens and citations. Aggregate latency cannot identify a regression.

At minimum, track p50 and p95, cost per useful answer, abstention rate, uncited answers, source opening, human escalations, ingestion errors and age of the newest document. Regularly sample answers for human review.

Deploy changes behind the evaluation suite and, when traffic permits, a canary or controlled test. Keep the previous index and pipeline version until stability is established. Our data-pipeline observability guide details freshness and operational controls.

The production readiness gate

Exact thresholds depend on risk and the current process. The goal is not 100 per cent on one metric, but an explicit accepted level and a defined consequence when the system falls below it.

A progressive implementation plan

The first iteration can use one well-understood source and fifty annotated questions. Build a lexical baseline, add vector retrieval and measure hybrid search. Introduce reranking only when results demonstrate an ordering problem.

The second iteration adds verifiable citations, abstention, permission tests and end-to-end traces. The third opens to a limited group, collects real failures and measures time saved or tasks resolved. Additional sources and formats come after this loop is stable.

A serious RAG architecture is not the one with the most components. It is the one that can say, for every answer: this is the evidence, these are the permissions, this is the pipeline version and this is the test that allows us to serve it.