A Failure Taxonomy for Source-Grounded Question Answering

A citation can be accurate and still sit beneath a defective answer.

Consider a question that asks about two organizations. The retrieval system finds strong evidence about the first and nothing useful about the second. The language model summarizes the first correctly, attaches a valid citation, and stops. Nothing in the response is obviously fabricated. Yet the answer is incomplete, and its polished surface may conceal the missing evidence more effectively than an uncited answer would.

This is why source-grounded question answering cannot be evaluated as a binary condition. An answer is not trustworthy merely because documents were retrieved, citations were displayed, or individual sentences resemble the supplied text. Grounding is an evidentiary chain. It begins with the source universe, passes through document preparation, retrieval and evidence selection, continues through synthesis and attribution, and ends with a decision about whether the system should answer at all.

The central failure occurs wherever that chain breaks.

A useful taxonomy must therefore do more than label an output “hallucinated.” It must identify where relevant evidence disappeared, where irrelevant evidence entered, where meaning changed, where attribution failed, and where the system should have stopped rather than producing an answer. That diagnosis determines what must be fixed. A retrieval defect cannot be repaired by rewriting the generation prompt. A weak source collection cannot be rescued by better citations. And a fluent refusal is not safe if the system had sufficient evidence but failed to use it.

“Grounded” is not one property

Retrieval-augmented generation systems usually contain several distinguishable operations. Documents are selected and divided into units. A retriever searches those units. Some systems rerank or filter the results. A language model then constructs an answer from the surviving context. The final interface may add citations, confidence language, or links to source passages.

Research evaluation frameworks increasingly reflect this modularity. RAGAs distinguishes the quality of retrieved context from the model’s use of that context and the quality of the generated response. ARES evaluates context relevance, answer faithfulness, and answer relevance as separate dimensions. RAGChecker likewise uses diagnostic measures for retrieval and generation rather than reducing the pipeline to one final score. (ACL Anthology)

These distinctions matter because correctness and grounding can diverge. A model may give a correct answer from its internal knowledge even though the retrieved evidence does not support it. It may faithfully reproduce a source that is outdated or wrong. It may cite a relevant document that does not entail the specific claim. It may answer one part of a compound question and omit the rest. It may also produce a complete and well-cited response to a question that the available source collection was never authoritative enough to answer.

A single “answer accuracy” score compresses these cases into the same result. A failure taxonomy keeps them separate.

Source-universe failure: the answer was constrained before the question arrived

The earliest failure can occur before retrieval. The available collection may omit a controlling document, contain an obsolete version, favor one viewpoint, mix authoritative and informal sources without distinction, or lack the material needed to answer the anticipated questions.

This category is broader than the internal mechanics of a conventional RAG pipeline. A 2026 EACL study by Kin Kwan Leung and colleagues developed a detailed taxonomy spanning chunking, retrieval, reranking, and generation, but explicitly excluded faults in the underlying corpus and other anomalies outside the pipeline. The authors also cautioned that their taxonomy was not exhaustive and would change as architectures acquire additional stages. (arXiv)

For public-interest systems, corpus failure cannot remain outside the evaluation boundary. A benefits-information assistant that retrieves perfectly from an outdated manual is still wrong. A university research assistant built only on abstracts may miss qualifications contained in methods and appendices. A policy tool that treats a press release as equivalent to a statute may return textually faithful but institutionally indefensible answers.

The relevant test is not merely whether the source collection contains information related to the query. It is whether the collection has adequate coverage, authority, currency and provenance for the claim the system is being asked to make.

Source-grounded evaluation should therefore record the corpus version, inclusion criteria, excluded source classes, update schedule and authority hierarchy. When sources conflict, the system should preserve the conflict rather than silently selecting whichever passage ranks highest.

Segmentation failure: the evidence was present but broken apart

Documents must usually be divided into retrievable units. That engineering choice can destroy meaning.

Segments that are too small can separate a fact from its subject, qualification, date or exception. Segments that are too large can bury the relevant sentence among unrelated material. Arbitrary boundaries can split a definition from the paragraph that limits it or detach a pronoun from the entity it refers to.

Leung and colleagues classify these conditions as overchunking, underchunking and context mismatch. In one of their examples, a passage describing why the Philippines is prone to typhoons was divided so that the country’s name appeared in one chunk while a key explanatory sentence referred only to “the country” in another. The evidence existed in the document, but the segmented representation weakened the connection needed for retrieval. (arXiv)

The visible symptom may look like a search failure or an incomplete answer. The cause lies earlier. Increasing the number of retrieved passages may compensate in some cases, but it does not repair document structure. A responsible evaluation must inspect whether each necessary concept survives segmentation with enough context to remain interpretable.

This is one reason production logs should preserve document-to-chunk lineage. Without it, evaluators can see what was retrieved but cannot determine what the retrieval system was given a fair chance to find.

Retrieval failure: the evidence existed but never reached the model

Retrieval fails when relevant material is absent from the selected context, when loosely related passages crowd out better evidence, or when lexical similarity is mistaken for the user’s actual intent.

A compound question exposes this weakness quickly. If a query asks for a comparison between two entities, a retriever may return several strong passages about one and none about the other. The resulting context appears relevant because every passage matches part of the question. It is still insufficient.

The 2026 EACL taxonomy distinguishes missed retrieval, low relevance and semantic drift. Its examples include multi-entity questions for which retrieved passages covered only one entity, as well as questions where repeated but incidental terms caused irrelevant passages to rank highly. (arXiv)

Retrieval quality must therefore be measured against the information requirements of the question, not merely against topical similarity. For a multi-part question, each part needs evidence coverage. For a comparison, evidence must exist for every side. For a causal question, retrieving a passage that states the outcome without the mechanism is not enough. For a question involving a date, jurisdiction or version, those constraints must survive query interpretation.

This makes retrieval evaluation partly a question-analysis problem. Before measuring recall, the evaluator needs to know what must be recalled.

Evidence-selection failure: the right material was found and then discarded

Many systems retrieve a broad candidate set and then use a reranker or filter to decide what enters the model’s context window. This stage creates its own failure modes.

Relevant evidence can be ranked below an arbitrary cutoff. Irrelevant passages can be promoted because they share persuasive language, familiar entities or superficial semantic similarity with the query. The first condition reduces recall; the second fills the generation context with noise.

Leung and colleagues describe an example in which the correct passage was retrieved but ranked below the threshold and never sent to the generator. They also show how an irrelevant passage mentioning New York led a model to name the New York Stock Exchange even though the supplied documents did not answer the question. (arXiv)

A final-answer review cannot reliably distinguish these cases from retrieval or generation failures. The evaluator needs the pre-rerank candidate set, reranking scores, cutoff policy and final context. If those intermediate records are unavailable, the system becomes less observable precisely as more automation is added.

That is the article’s crucial turn: a more elaborate pipeline can produce a better answer while making the causes of its failures harder to locate. Additional components are useful only when their decisions remain inspectable.

Synthesis failure: the evidence arrived, but the answer changed it

The generator can fail even when the relevant evidence is present.

It may introduce unsupported content, rely on internal knowledge instead of the supplied documents, omit critical parts of the answer, merge facts belonging to different people or events, answer a related question rather than the one asked, reverse chronology, or mishandle quantities.

RAGTruth demonstrates that retrieval does not eliminate unsupported generation. The dataset contains nearly 18,000 naturally generated RAG responses from multiple language models, manually annotated at both response and word level for hallucination-related behavior. Its purpose is grounded in the observation that models can still produce claims that are unsupported by or contradictory to retrieved content. (ACL Anthology)

The FRAMES benchmark reaches the same problem from a different direction. Its questions require factual retrieval and reasoning across multiple sources. In the authors’ reported baseline experiments, accuracy rose from 0.40 without retrieval to 0.66 with their multi-step retrieval pipeline. Retrieval produced a substantial improvement, but one-third of the questions remained incorrect in that experimental setting. (ACL Anthology)

That result should not be read as a general performance estimate for every RAG system. It demonstrates a narrower point: access to relevant documents can improve question answering without resolving the synthesis problem.

A synthesis evaluation must distinguish whether every material claim follows from the supplied evidence, whether the answer covers the requested scope, and whether the model preserved relationships among entities, dates, conditions and quantities. A model that cites the correct paragraph while reversing its meaning has not produced a grounded answer.

Attribution failure: the citation is real, but the claim is not supported

Citations improve inspectability only when they are correctly attached to claims and the cited material actually supports those claims.

The ALCE benchmark treats answer correctness and citation quality as separate evaluation dimensions. It was designed for end-to-end systems that retrieve evidence and generate long-form answers with citations, recognizing that fluent text, correct content and adequate attribution are related but non-identical properties. (ACL Anthology)

Attribution can fail in several ways. A citation may point to a passage that discusses the same topic without entailing the claim. A paragraph containing several factual statements may cite a source supporting only one. A response may cite every sentence while omitting evidence that contradicts its conclusion. A citation may support a number but not the comparison or causal interpretation built around it. The answer may also rely on an authoritative source for a claim outside that source’s jurisdiction or scope.

Citation presence is therefore an interface feature. Citation correctness is an evidentiary property.

A rigorous review should work at the claim level. Each externally verifiable assertion should be mapped to the exact source passage that supports it. The evaluator should then ask whether the passage entails the claim, whether another source materially contradicts it, whether the source is appropriate for the claim, and whether uncited connective language introduces a conclusion that the evidence does not establish.

Answerability failure: the system answered when the evidence required restraint

A source-grounded system needs a defined condition for not answering.

Abstention is appropriate when the source collection does not contain the answer, when retrieved evidence is insufficient, when the question is materially ambiguous, or when available sources conflict in a way the system cannot responsibly resolve.

The EACL taxonomy identifies abstention failure as a distinct generation error. One example involved a model inferring that no fundraising occurred because the retrieved documents did not mention fundraising. Absence of evidence was converted into evidence of absence. (arXiv)

A refusal can also be defective. A system may abstain even though sufficient evidence was retrieved, perhaps because segmentation obscured it or the generator failed to recognize it. Evaluation must therefore test both sides of the boundary: whether the system refrains when it lacks support and whether it answers when the source record is adequate.

The correct response is sometimes neither an answer nor a refusal. It may be a clarification request, a statement of unresolved conflict, a description of the missing evidence, or a referral to an authorized human decision-maker.

This boundary matters most when users may mistake informational assistance for institutional authority. A cited response about a law, public benefit, medical issue or safety condition does not acquire legal, administrative or professional authority from the presence of sources.

Governance failure: the answer cannot be responsibly reviewed or used

Technical grounding is necessary but insufficient. A system can produce a well-supported answer while failing institutionally.

The underlying sources may be inaccessible to the user. The interface may hide uncertainty behind a generic confidence indicator. Logs may omit the model, prompt, corpus and retrieval versions needed to reproduce the answer. Corrections may not propagate to prior outputs. Human reviewers may be formally present but lack the time, evidence or authority needed to challenge the system.

NIST’s Generative AI Profile is a voluntary companion to the AI Risk Management Framework intended to help organizations incorporate trustworthiness considerations into the design, development, use and evaluation of generative AI systems. That lifecycle framing is important here: source grounding cannot be evaluated once and assumed to persist while models, indexes, corpora and interfaces continue to change. (nist.gov)

For a public-interest institution, the final question is not simply whether the model generated a correct response. It is whether the answer can be inspected, challenged, corrected and prevented from exceeding its intended authority.

Automatic evaluation helps, but it cannot close the case

Automated evaluators are useful for regression testing, large-scale monitoring and identifying clusters that deserve investigation. They should not be mistaken for neutral arbiters.

The 2026 RAG error-classification study offers unusually direct evidence of this limitation. Its automated system reached 57.8 percent agreement with human annotators when identifying the pipeline stage responsible for an error and 40.3 percent accuracy when classifying the detailed error type. The authors concluded that error classification remained difficult because failures propagate across stages and the intermediate evidence is complex. (arXiv)

The same study found that error distributions changed across datasets. Fabricated content was relatively uncommon in one analysis, while retrieval, chunking or misinterpretation failures were more prominent. The authors explicitly warned that their taxonomy was not exhaustive and that their evaluation focused on single-turn textual queries rather than multi-turn or multimodal systems. (arXiv)

These limitations do not weaken the case for taxonomies. They clarify how taxonomies should be used: as structured aids for diagnosis, not as claims that every failure has one objectively discoverable label.

Some failures have multiple causes. A poorly segmented passage may reduce retrieval recall, which may encourage the model to rely on internal knowledge, which may produce a claim accompanied by a citation to a merely related passage. Forcing that chain into one category erases the causal sequence.

The better practice is to identify the earliest observable break and record the downstream effects.

What a reproducible evaluation should preserve

A credible source-grounded QA evaluation begins by freezing the conditions under which the answer was produced. The corpus, document versions, chunking method, retrieval configuration, reranking policy, prompt, model version and generation settings should be recorded. Without those conditions, another evaluator cannot reproduce the evidence path.

The question should then be decomposed into its required information units. Evaluators should determine whether the source universe contains an answer, whether segmentation preserves the necessary context, whether retrieval and reranking deliver adequate evidence, whether every material claim follows from that evidence, whether the citations support the claims to which they are attached, and whether the system responds appropriately when support is absent.

The test set should include compound questions, conflicting documents, stale sources, ambiguous wording, unanswerable questions, temporal comparisons, numerical reasoning and cases in which a highly similar passage is nevertheless irrelevant. Aggregate accuracy should be accompanied by stage-specific findings and representative failures. Automated judgments should be calibrated against human review, with higher scrutiny when outputs could influence rights, services, health, finances or physical safety.

The intended result is not a universal declaration that a model is safe. It is a documented account of how a specific system behaved under specified conditions, which failures were observed, how those failures were classified, and what evidence remains missing.

That boundary is consistent with TAIRC’s research portfolio, which places evaluation before deployment, treats governance as part of research design, and limits the organization to analytical, evaluative and openly disseminated work rather than certification, operational deployment or autonomous decision-making.

The citation is the end of the chain, not proof that the chain survived

Source-grounded question answering is often described as a way to make language-model outputs more factual and verifiable. That description is directionally reasonable but incomplete.

Retrieval changes the shape of the reliability problem. It creates an evidence path that can be examined, but it also creates more places where evidence can be lost, distorted, promoted, suppressed or attached to the wrong claim.

The most useful failure taxonomy therefore follows the evidence. It begins with the source universe, moves through segmentation, retrieval and selection, examines synthesis and attribution, and ends with answerability and institutional use.

The governing question is not, “Does the answer have sources?”

It is: “Can every material claim be traced through an intact, appropriate and reviewable evidence path—and does the system stop when that path breaks?”

A citation is only the visible end of that path. It is not proof that the rest of it survived.

Sources and verification

TAIRC’s organizational boundaries, research commitments and intended public-interest orientation were verified against the TAIRC Research Portfolio and the detailed TAIRC Research Categories and Topics document.

The lifecycle risk-management context was verified against the National Institute of Standards and Technology’s Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile, NIST AI 600-1, published July 26, 2024, with its NIST record updated April 8, 2026. (nist.gov)

The pipeline-stage taxonomy, examples, empirical classification results and stated limitations were verified against Kin Kwan Leung and colleagues, Classifying and Addressing the Diversity of Errors in Retrieval-Augmented Generation Systems, published in the main proceedings of EACL 2026. (ACL Anthology)

The evidence that unsupported and contradictory statements can persist in retrieval-augmented outputs was verified against Cheng Niu and colleagues, RAGTruth: A Hallucination Corpus for Developing Trustworthy Retrieval-Augmented Language Models, ACL 2024. (ACL Anthology)

The multi-source retrieval and reasoning results were verified against Satyapriya Krishna and colleagues, Fact, Fetch, and Reason: A Unified Evaluation of Retrieval-Augmented Generation, NAACL 2025. (ACL Anthology)

The distinction among answer correctness, citation quality and fluency was verified against Tianyu Gao and colleagues, Enabling Large Language Models to Generate Text with Citations, EMNLP 2023. (ACL Anthology)

The component-level evaluation dimensions discussed in the article were verified against Jon Saad-Falcon and colleagues, ARES: An Automated Evaluation Framework for Retrieval-Augmented Generation Systems, NAACL 2024; Shahul Es and colleagues, RAGAs: Automated Evaluation of Retrieval Augmented Generation, EACL 2024; and Dongyu Ru and colleagues, RAGChecker: A Fine-Grained Framework for Diagnosing Retrieval-Augmented Generation, presented through Amazon Science and NeurIPS 2024. (ACL Anthology)