RAG at work: why document access does not guarantee a good answer

A service engineer asks an assistant what to check after replacing a module. The answer is fluent, contains three steps and links to a genuine manual. Unfortunately, the manual concerns revision A while the device on the bench is revision B. In another attempt, the assistant selects the right manual but omits a bulletin that changes one step. Both answers can sound convincing and contain a correct quotation.

Calling these failures hallucinations is not enough. We need to establish whether the system understood the scope, had access to the right source, retrieved it, retained the relevant exception and used the evidence correctly. Only then can the team decide whether to improve documentation, indexing, permissions, context preparation or answer generation.

We will examine a service assistant using a small fictional corpus. Every document, procedure identifier and case is teaching material. None describes an actual repair or BBest24 deployment. This is not a model benchmark. It is a worked acceptance-test example that can later be adapted to a particular system.

What RAG adds and what it does not decide

Retrieval-augmented generation, or RAG, combines answer generation with retrieval of external material. Instead of relying only on knowledge stored in model parameters, the system supplies document passages relevant to a query. Lewis and colleagues describe such a combination of a generative model and retrieved external memory. Their research results do not guarantee the quality of any assistant connected to a folder of PDFs.

A retrieved passage can be true but inapplicable to the user’s device. It can be current but inaccessible to the user’s role. It can also contain a general rule without an exception removed during document chunking. The unit of evaluation is therefore a correctly handled task in a particular context, not similarity between the answer and a source passage.

Our task is to identify the appropriate checks after module replacement in a specified revision. A correct result may be a list of steps, a request for the revision or a refusal because sufficient authorised evidence is unavailable. The system receives no credit for answering at any cost. Nor does it receive credit for refusing every question, because that would fail the job it was built to perform.

Five documents are enough to expose the problem

The corpus covers two product revisions and five documents. The labels PAIR, Z, T-A and T-B identify fictional activities, not equipment instructions. Check Z includes the requirement to stop and escalate if it fails.

The service role may use current A1, B2 and S07 within their appropriate scope. It cannot access INT. The internal role can also access that note. B1 remains in the corpus as a historical-document example but cannot serve as the current instruction for a present-day procedure question.

A correct revision-B answer must combine B2 and S07: register the module, perform Z, and proceed to T-B only after Z passes. B2 alone is insufficient. A date in a filename does not replace an explicit supplements or supersedes relationship. A newer file may concern another device, while an older manual can remain applicable to revision A.

The table defines the minimum corpus needed to follow scope, version and access decisions. It is not a complete service-documentation set.

Define the expected answer before evaluating the system

For “What checks follow module replacement in revision B?”, the case record needs more than a sample paragraph. Record the user’s role, revision, required sources B2 and S07, the three semantic units PAIR, Z and T-B, and their required order. Separately identify the critical condition: a failed Z must not lead to a recommendation to continue.

A question without a revision requires clarification. The assistant should explain that the procedure depends on the device version and request information that identifies it. It should not combine A1 and B2 into one universal procedure. The missing distinction is part of the task, not an inconvenience to conceal.

For revision C, which the corpus does not cover, the expected result is an explanation that the available evidence cannot support a procedure, with a route to appropriate support. When service asks for INT, its contents must not be disclosed. A document’s title or existence can also be sensitive, so refusal wording must respect the chosen metadata-disclosure policy.

In a real project, the person responsible for the procedure and its applicability must approve these expectations. Here, they define a fictional example rather than report domain approval. That distinction matters: an automatically generated reference answer does not become true merely because it is used to grade another model.

Measure successive points where information can be lost

We draw four visible pipeline stages but evaluate five aspects: scope, access, retrieval, answer and citations. This prevents a correct answer obtained through an access violation from passing, and prevents successful retrieval from concealing an omitted condition later in the pipeline.

First, the system establishes the query’s scope. Revision may come from an explicit user statement or an authorised lookup of device identity. It should not be inferred from whichever document happens to have the highest similarity score. Conflicting information needs resolution before an instruction is produced.

Permissions then restrict what context can reach the model. Checking only when documents are imported is insufficient because user roles can change afterwards. The OWASP RAG Security Cheat Sheet discusses propagating access metadata to chunks and enforcing access during retrieval. In our example, INT must never enter the context of a service user, even if the final answer happens not to cite it.

Only within that authorised scope do we assess whether the material needed for the task was found. Revision B requires B2 and S07. We then evaluate the answer and whether its references support their associated claims. Each stage needs its own result, document identities and a way to attribute failure.

Retrieval relevance is not answer completeness

If two documents are required and only B2 is retrieved, coverage of required sources is 1 / 2 = 50% under this simple definition. If B2 is the only returned document, precision against the required document set is 1 / 1 = 100%. Both figures are true. The result still cannot support a complete answer.

These metrics require a defined unit. We might count documents, chunks or specific pieces of information. Three chunks from the same page are not three independent pieces of evidence that the bulletin was retrieved. A practical test records required identities and locations within documents rather than only expected keywords.

A similar trap appears after generation. “Register the module and run T-B” contains two claims supported by B2. It can therefore receive full support for what it actually says. Yet it omits Z and the stopping condition. Against three required semantic units, it covers only 2 / 3, approximately 66.7%. In a service task, that is a material error that a good context-faithfulness score must not hide.

The RAGAs paper from EACL 2024 proposes separate dimensions for evaluating context and generation. Such tools can help detect regressions, but their scores must be connected to the user’s decision. A metric name does not replace a definition of its denominator, grading method or acceptance criterion.

A map of ten deliberately different cases

We constructed ten observations by hand. Two satisfy expectations and eight deliberately illustrate failures. They are not samples from a model run, so they cannot establish a field error rate or compare AI providers. Their purpose is to check whether an evaluation method distinguishes different failure mechanisms.

The map marks the first stage that fails its case record. Later stages are not evaluated in this illustration. An access violation is therefore not diluted by later credit for polished prose. A full audit may record all coexisting defects while retaining the originating cause and independent acceptance blockers.

The map is produced by comparing the agreed labels and evidence lists in the table. Such a check does not interpret unrestricted language like a domain expert or prove that an arbitrary paraphrase is semantically correct. This illustrates grading rules; it is not a production RAG judge.

Chunking is a substantive design decision

A common indexing error begins with mechanically splitting text into similarly sized passages. The heading “revision B only” remains in the previous chunk, a table loses its column headings, and an exception moves to the next page. Retrieval finds a sentence about T-B, but the model never sees the condition governing its use.

For our corpus, a chunk should preserve document identity, revision, status, access role and relationships to other documents. S07 must be discoverable through B2 even if the query does not contain the word bulletin. That does not require sending the entire documentation set with every question. It requires controlled context expansion to include dependencies needed to resolve the task.

Test a chunking change using the same questions while holding other configuration constant. Check whether the missing condition returns to context, how much irrelevant material is added and how response time changes. More context can help, but it can also increase cost and competing instructions. Character count alone does not measure knowledge completeness.

Extraction output also needs versioning. A new PDF parser can arrange a table differently even when the source file is unchanged. Reproducing a regression requires the document version, extractor, chunking rules, index and retrieval configuration. Recording only the model name leaves many possible causes invisible.

Updating a source is a coordinated knowledge change

Suppose S07 is approved while the index still contains only B2. The system does not need to invent any false information to give an outdated answer. It can faithfully summarise the context it receives. Responsibility for freshness therefore cannot rest entirely with model prompts.

Document publication should define when a new version becomes available for answers, how superseded chunks are marked and what happens to cached results. For a critical change, temporarily limiting answers may be preferable to combining a new document status with an old index. An identity for a consistent corpus release helps establish which knowledge set produced an answer.

Revoking access creates a similar problem. An earlier answer must not reach a person who has lost permission merely because the text remains cached. Cache keys and renewed authorisation must account for access scope. Q09 deliberately tests this path because correct retrieval filtering does not automatically protect every other place where answers are stored.

If two approved documents give contradictory advice, retrieval ranking does not determine their authority. An explicit precedence relationship or a decision by the documentation owner is needed. An additional test can supply two current bulletins without saying which supersedes the other. The expected outcome is to identify the conflict and escalate, not arbitrarily choose the newer date. This case is outside the ten mapped observations; it illustrates extending the set when another risk class emerges. The correction may be to organise the documentation rather than change the model.

A citation is a claim-to-evidence relationship

A clickable reference is valuable only when the reader can verify a specific claim. A quotation from A1 does not support a revision-B procedure, even if both documents use the word module. A document title alone does not demonstrate that an exception was included. Answers should connect steps to the appropriate passage and source version.

Citation evaluation should separate address validity, the user’s access to the destination, applicable scope and actual support for the statement. Some checks are deterministic: the identifier exists, belongs to the authorised corpus and has not been withdrawn. Others require interpretation, such as whether the cited paragraph supports a mandatory Z check and whether its condition was preserved.

References should not be attached after generation merely by searching for similar words. That process can decorate an unsupported statement with a genuine document. Preserve the link to evidence actually used in context, then separately check whether the generated claims are supported by it.

A source does not gain authority to issue instructions

A document may contain a sentence telling the model to skip a check, reveal another note or disregard answer rules. In Q10, it is an illustrative instruction to omit Z. It does not become part of an approved procedure just because it appears inside a retrieved passage.

OWASP LLM01:2025 describes indirect prompt injection through external content and notes that RAG alone does not eliminate the risk. Separating application instructions from source data helps but is not a mathematical guarantee. Permissions and tool access must be enforced outside the model’s discretionary decisions.

The assistant in this example only provides information. If it could initiate service actions, answer evaluation would not be sufficient to authorise an operation. Separate execution, confirmation and outcome-observation rules would be needed. A pilot should not expand from knowledge retrieval to device control merely because an initial demonstration looks convincing.

Use an automatic judge without handing it acceptance authority

A model grading answers can accelerate review of a large case set, but it needs an explicit rubric. Asking whether an answer is good is too broad. Better dimensions include the correct revision, required checks, absence of contradictory recommendations, justified refusal and support for individual claims.

First compare automatic grades with expert assessment on a representative subset. Pay particular attention to a judge accepting an answer with an omitted exception or rejecting an appropriate short refusal. Disagreements should refine the criteria rather than automatically favour whichever system provides more detailed prose.

Separate tuning cases from acceptance cases. If each failure leads to adding the answer to that exact test into the prompt, improvement may simply reflect adaptation to known examples. Retain new questions from the same failure classes, alternative wording and cases following documentation changes. Report results by category as well: a high average can conceal a systematic problem affecting one revision.

Conditions for entering a pilot

The acceptance record should identify the corpus release, model and retrieval configuration, case set and owner of expected answers. In this example, every access violation and use of a withdrawn procedure is a blocker rather than a deduction from an average. Finding none in a finite test set does not prove that they cannot occur elsewhere.

Alongside correctness, measure the share of questions the system can answer, whether refusals are justified, time to a useful answer and handling cost. Record retrieval-service failures and timeouts too. An infrastructure failure should not appear to the user as a substantive absence of documentation. Otherwise, more refusals may be misinterpreted as greater model caution.

A practical review ends with three questions for each failed case: what information was missing, where was it lost and which test will verify the correction? Q02 needs S07 to be retrieved; Q06 needs the already available bulletin to be used correctly; Q04 needs protected context stopped before it reaches the model. Replacing the model may help at one of these points, but it cannot substitute for identifying the cause.

Sources

• Lewis and colleagues: Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks, NeurIPS 2020, arXiv v4 dated 12 April 2021; combining retrieval with generation.
• Es and colleagues: RAGAs: Automated Evaluation of Retrieval Augmented Generation, EACL 2024, Demonstrations, pp. 150-158; separate dimensions for context and answer evaluation. We cite the paper, not the current library interface.
• OWASP: RAG Security Cheat Sheet, access control across the knowledge pipeline and testing disclosure paths.
• OWASP LLM01:2025: Prompt Injection, indirect instructions in retrieved sources and the limits of RAG as a defence.

Sources checked on 21 October 2026. Documents A1, B1, B2, S07 and INT and observations Q01-Q10 are fictional. No real model outputs were used to claim quality results; the executable tests reproduce the example’s rules and annotations.