A confident answer. The wrong evidence.
When an AI answer sounds right but misses the point, look at what the retriever handed it. Four failure modes explain much of the gap.
A broken response is easy to notice. A fluent answer based on the wrong document is much harder.
That is the uncomfortable part of retrieval-augmented generation. The model can follow every instruction and still let a user down, because the information it received was wrong, incomplete, or out of date.
Before rewriting the prompt, follow the evidence.
Four places to look
A thought cut in half. The useful answer crosses a chunk boundary. One passage contains the vocabulary in the question; its neighbour contains the explanation. Structure-aware chunking, overlap, or retrieving adjacent passages can reconnect them.
The almost-identical document. Two versions of a policy differ by one important clause. Semantic similarity alone does not establish which version applies. Use dates, document versions, and other metadata to narrow the search.
An answer that does not exist. Nearest-neighbour search can return plausible passages even when none answers the question. Assess relevance as well as rank, and give the system a clear way to acknowledge missing evidence.
The right passage for the wrong person. A relevant document may still be outside a user’s permissions. Enforce access controls in the retrieval query, before restricted content reaches the model.
Make the invisible measurable
Evaluate retrieval separately from the final answer. Start with questions whose source documents have been identified by someone who knows the material. Measure whether the relevant passages appear in the retrieved results.
If the evidence never reaches the model, a prompt change cannot make it use that evidence.
Record enough retrieval context to investigate failures: document identifiers, versions, relevance scores, and the answer they informed. Apply your data-access and retention rules to those records.
Make citations openable. A source link helps users check an answer and gives the team a concrete place to investigate when something feels wrong.
Start upstream
Prompt changes are quick to try, which makes them an easy place to spend a week.
The more useful first question is smaller: did the system find the information it needed? Establish that before polishing the answer. A more elegant explanation cannot rescue the wrong evidence.
Explore our AI engineering work or tell us about your retrieval problem.