Your RAG answer is wrong. Where should you look first?

The model is the last link in a longer chain

An assistant gives a convincing answer that contradicts an internal document. It is tempting to replace the model. A better first question is whether the correct, current passage ever reached it. Without that evidence, even a strong generation model is working with the wrong material.

Retrieval-augmented generation, or RAG, supplies retrieved information alongside a question when asking a model to respond. Microsoft documents retrieval, augmentation and generation as separate parts of this pattern. Its architecture guidance also treats preparation and evaluation as substantial engineering work. The diagnostic sequence below is our suggested way to investigate failures.

If the information is missing, inspect ingestion

Start with the source of truth. Confirm that the required document is available to the application, successfully parsed and represented in the searchable data. Inspect document dates and replacement rules. An old policy and its replacement can both look relevant to a search engine.

Check a few actual passages rather than relying on an ingestion success counter. Tables, headings and scanned pages can lose meaning during extraction. Record the document identifier, version and section location so that a returned passage can be traced back to a human-readable source.

Treat deletion as a data lifecycle event too. Removing a source document should trigger a review of any indexed representation and cached content derived from it.

If the information exists, inspect retrieval

Run the failing question against retrieval without asking the model to generate an answer. Review the passages returned and ask whether they contain enough evidence to answer the actual question. A passage can be topically similar without containing the required fact.

Microsoft’s retrieval guidance discusses combining search methods and reranking candidates, with relevance and latency trade-offs. Our recommendation is to change one retrieval assumption at a time and compare results against the same questions. More returned text is not automatically better context.

Also test authorization explicitly. Retrieval must respect the caller’s access before restricted content is supplied to the model. Instructions asking the model not to reveal a document are not a replacement for enforcing the document’s permissions.

If the evidence is right, inspect the answer contract

Give the generation step a clear task: answer from the supplied evidence, cite the supporting material and acknowledge when that material cannot support an answer. Then test whether the response follows that contract.

Check each important claim against its citation. A citation that points to a real document can still fail to support the sentence beside it. Include questions whose answers are absent, ambiguous or contradicted by an obsolete document. A system that always answers may be less useful than one that can identify its own evidence gap.

Keep retrieved text separate from trusted application instructions. Content retrieved from a document may include instructions or malicious text; it should not gain authority simply because it arrived through search.

Build a failure notebook before tuning

Choose a small, representative set of questions and record the expected source, the returned evidence, the answer and the reason it passed or failed. Include examples from real user tasks when you have permission to use them, and remove sensitive material from shared test artifacts.

• Missing evidence: the expected passage never became searchable.
• Retrieval miss: the passage exists but was not returned.
• Authorization failure: evidence was supplied to the wrong caller.
• Unsupported answer: the evidence was available, but the response added or distorted a claim.

A useful release decision balances several signals

Our suggested release review compares the failure categories alongside response time and cost per useful interaction. Test proposed changes against the earlier questions as well as new ones so an apparent fix does not hide a regression.

Do not promise that RAG eliminates hallucinations. A defensible improvement is more specific: the updated system retrieves the expected evidence more often, refuses unsupported questions appropriately and meets the application’s operating constraints. That gives the team something concrete to investigate when the next answer goes wrong.