A Reliability Test for Ambiguous Instructions

An AI system should neither guess through missing requirements nor interrupt every routine task. Reliability begins when its behaviour changes with the consequence of being wrong.

“Review this application and tell me whether it qualifies.”

The request sounds direct. It is not. Which programme? Which jurisdiction? Which version of the governing rules? Does “qualifies” mean that the application is complete, that the applicant appears eligible, or that an authorised official should approve it? Has the system received all relevant facts? Is it permitted to make that determination?

A language model can conceal every missing element behind one fluent answer. It can also avoid the problem by asking a generic question—“Can you provide more information?”—that does little to resolve the uncertainty. Neither behaviour demonstrates reliability.

The relevant test is more demanding. Can the system recognise when an instruction permits materially different interpretations, identify the missing information that changes the outcome, and choose an appropriate response without assuming authority it does not have?

That is the core of a reliability test for ambiguous instructions.

The wrong test rewards a lucky guess

A conventional accuracy test asks whether the final answer matches an expected answer. That works poorly when the instruction itself does not identify a single defensible answer.

Suppose an evaluator asks a model to “summarise the new policy for staff.” The model may need to know which policy, which staff, how much detail is required, whether the summary is informational or operational, and which document version controls. A response can happen to resemble what the evaluator intended while still being produced through an unjustified assumption.

Passing such an answer rewards guessing.

NIST’s AI Risk Management Framework defines reliability in relation to required performance under specified conditions. It also states that realistic test sets should represent expected conditions of use and that some failures warrant greater weight because they can cause greater harm. Robustness includes behaviour in unexpected circumstances, not merely repetition of performance observed in a narrow laboratory setting.

Ambiguity therefore cannot be treated as incidental noise around the “real” task. It is part of the condition under which the system is being asked to perform.

A useful test does not ask only, “Did the model produce the intended answer?” It asks, “What did the model do when the information required to justify that answer was missing?”

Ambiguity must be specified before it can be measured

The word ambiguous covers several different failures of specification.

An instruction may contain a term with multiple meanings. It may omit the person, place, period, object, jurisdiction, source, output format, or decision standard needed to complete the task. It may contain mutually inconsistent requirements. It may refer to an unfamiliar entity that the system silently substitutes with a more familiar one. It may be linguistically clear while remaining institutionally incomplete because the requester has not stated who holds authority for the decision.

The CLAMBER benchmark, published at ACL 2024, separated ambiguous queries into epistemic misalignment, linguistic ambiguity, and missing output information, with finer categories for unfamiliar entities, contradictions, lexical and semantic ambiguity, and missing who, when, where, or what elements. Its roughly 12,000 examples were designed to test both ambiguity recognition and the generation of clarifying questions.

That taxonomy is useful, but public institutions need another dimension: consequence.

“Draft a shorter version of this paragraph” may be mildly ambiguous if no target length is given. “Determine whether this person qualifies” is different. Missing jurisdiction, evidence, authority, or controlling criteria can change both the answer and the legitimacy of producing one.

The same linguistic uncertainty can therefore require different behaviour in different settings. Reliability is not a fixed tendency to answer, ask, or refuse. It is a calibrated response to the information gap, the likely consequence of an incorrect assumption, and the authority attached to the task.

Build matched cases, not isolated prompts

A defensible evaluation begins with matched cases.

For each scenario, the evaluator creates an ambiguous instruction, a resolved version that supplies the missing information, and a clear control that should be answerable without further clarification. The three versions should preserve the same underlying task. The variable being tested is the presence or absence of information that materially changes the response.

Consider a hypothetical administrative example.

The ambiguous version asks: “Tell me whether this notice gives the applicant enough time to respond.”

The resolved version identifies the jurisdiction, proceeding, notice date, service method, governing rule, and question being asked.

The clear control asks the model to calculate a stated number of calendar days between two dates, without asking it to interpret legal sufficiency.

The ambiguous case tests whether the system detects a consequential information gap. The resolved case tests whether it can use the clarification correctly. The control tests whether an excessive tendency to seek clarification has damaged ordinary task completion.

This matched design prevents two misleading conclusions. A system should not receive credit merely because its unstated assumption happened to match the evaluator’s intended interpretation. It should not receive credit for caution merely because it refuses or questions every instruction.

The test set should also vary the surface wording while preserving the same information structure. A model that reacts appropriately to “Which programme are you referring to?” but not to an equivalent omission expressed differently has learned a phrase pattern, not a reliable decision rule. This is a robustness control rather than a separate measure of general intelligence.

Score the decision before the answer

The first scored event should be the model’s response decision.

The evaluator should record whether the system answers directly, answers while disclosing an assumption, presents multiple interpretations, asks a clarifying question, limits the scope of its response, or defers to an authorised human process. None of these actions is inherently correct. Correctness depends on the case.

A low-consequence, reversible task may permit the model to proceed under an explicit assumption. A materially outcome-changing omission should normally trigger a targeted question. An instruction that asks the system to exercise authority it does not possess should trigger a boundary statement rather than an attempted decision.

The next measure is ambiguity detection: did the system recognise that the instruction lacked information needed for a justified response?

Detection alone is insufficient. The question must address the decisive gap. CLAMBER’s evaluation of the model versions included in its 2024 study found clarifying questions that targeted the wrong issue, remained under-specified, added unnecessary detail, or failed to clarify at all. The authors also reported that chain-of-thought and few-shot prompting produced only marginal improvements in ambiguity identification in their experiments and could increase overconfidence in some smaller models. Those findings are tied to the tested models, prompts, and benchmark; they should not be treated as a permanent description of every current system.

A clarification such as “Can you be more specific?” is therefore not evidence of success unless the user can respond with information that resolves the relevant uncertainty. In the application example, “Which programme and jurisdiction should govern the assessment?” is useful. “Can you provide more context?” transfers the diagnostic work back to the user.

The test should then examine post-clarification completion. Once the missing information is supplied, does the system incorporate it correctly, or does it continue with its original assumption? Does it answer the question now presented? Does it preserve the limits of the source material and the institution’s authority?

A system that asks an excellent question and then ignores the answer has not resolved the ambiguity.

The strongest-looking metric is still a proxy

At this point, an evaluator may be tempted to reduce the test to a clarification rate: the percentage of ambiguous prompts that caused the model to ask a question.

That metric is easy to calculate. It is also easy to optimise badly.

Natural language frequently omits information that people can reasonably infer from shared context. Research presented in the 2025 NAACL Findings paper Clarify When Necessary treats the choice to clarify as a trade-off between improvement in the final task and the interaction cost imposed on the user. The authors argue that the decision cannot be reduced to a binary ambiguity label because one interpretation may be dominant and because acceptable caution depends on the user and the setting.

A system that asks a question after every possible omission may post a strong clarification rate while becoming unusable. Worse, ritual clarification can create an appearance of care without improving the answer.

The more relevant measure is clarification utility: did asking the question materially improve the probability of a correct, properly bounded response?

That requires comparison. The evaluator should examine what the system would have produced without clarification, what it produced after receiving the answer, and whether the clarification changed the result in the direction justified by the evidence.

The test must also record false clarification: clear instructions that the system unnecessarily interrupts. Research on Alignment with Perceived Ambiguity, published at EMNLP 2024, explicitly measured the trade-off between identifying ambiguous queries and preserving correct responses to unambiguous ones. Its proposed training method outperformed the evaluated baselines on the study’s question-answering datasets, showing that better ambiguity handling is technically possible. The authors also limited their conclusions: the work focused mainly on short-form, single-query question answering using the model families tested in the study, not extended administrative reasoning or every contemporary model.

Contrary evidence matters here. AmbiBench, introduced in research on six ambiguously specified classification tasks, found that the combination of model scale and training with human-feedback data allowed the evaluated models to approach or exceed human participant accuracy across those tasks. That result shows that models can infer intended tasks under some constrained forms of ambiguity. It does not show that silent inference is safe across open-ended institutional work.

The lesson is narrower and more useful: asking is not always superior to inferring, and inferring is not always superior to asking. The evaluation must determine whether the system distinguishes between the two under the conditions that matter.

Failures should be weighted by consequence

A single aggregate score can obscure the result an institution most needs to see.

Failing to request a preferred document length is not equivalent to silently choosing an eligibility rule. Asking an unnecessary question in a brainstorming task is not equivalent to inventing a missing fact in a benefits, legal, safety, or personnel context.

The evaluator should therefore assign the expected behaviour and potential consequence before running the test. That determination should be made by people familiar with the actual workflow, affected users, governing documents, and authority boundaries—not reconstructed after seeing the model’s output.

The scoring record should distinguish harmless inefficiency from unjustified assumption, incorrect action, concealed uncertainty, source distortion, and attempted exercise of reserved human authority. A result can then report both frequency and severity rather than allowing numerous trivial successes to cancel a small number of consequential failures.

This approach follows NIST’s emphasis on context-specific measurement, representative conditions, documented limitations, and prioritisation of failures by potential harm. NIST’s Generative AI Profile also cautions that anecdotal demonstrations, human examinations, prompt-engineering exercises, and laboratory benchmarks may not establish validity or reliability in real deployment contexts. Prompt sensitivity and heterogeneous conditions can widen the gap between benchmark performance and actual use.

The outcome of an ambiguity test is therefore evidence about a defined system, configuration, task family, and period. It is not a declaration that the underlying model is generally reliable.

The evidence record is part of the test

An independent reviewer should be able to reconstruct why every item was included, what behaviour was expected, what happened, and how the result was scored.

That requires preserving the exact instruction, the intended interpretations, the missing information, the consequence classification, the expected response behaviour, and the rubric used by human evaluators. The record should identify the model and system version, evaluation date, system-level instructions, retrieval sources, enabled tools, relevant generation settings, and any preprocessing applied to the prompt or output.

It should also retain the actual outputs, including failed and inconvenient ones. Reporting only representative successes converts an evaluation into a demonstration.

Where human raters decide whether a question was useful or whether an assumption was material, the evaluation should document the rater guidance, domain qualifications, disagreements, and adjudication process. A reviewer needs to know whether apparent model inconsistency may instead reflect an unstable scoring rule.

The U.S. Government Accountability Office’s AI Accountability Framework organises oversight around governance, data, performance, and monitoring. It provides questions and evidence procedures for agencies, auditors, and third-party assessors rather than treating a headline performance result as sufficient accountability. NIST similarly recommends documenting model assumptions, limitations, evaluation data, user instructions, and relevant lifecycle changes.

For an ambiguity evaluation, the minimum credible record is not the final score. It is the chain connecting the instruction, the missing information, the expected decision, the observed behaviour, the evidence used to judge it, and the limits of the conclusion.

What the test can establish

A well-designed test can establish whether a specified system, under documented conditions, tends to recognise selected forms of ambiguity; whether it asks questions that resolve material uncertainty; whether it proceeds appropriately when clarification is unnecessary; whether it uses the clarification it receives; and whether its behaviour changes when the potential consequence changes.

It can expose systematic patterns. A system may detect lexical ambiguity but miss contradictions. It may ask useful questions about dates yet invent jurisdiction. It may preserve authority boundaries in explicit eligibility requests but cross them when the same request is framed as a summary. Those patterns are more actionable than a single “ambiguity score.”

The test cannot establish universal safety, future reliability, legal compliance, or suitability for deployment. It cannot substitute for evaluation of factual accuracy, source fidelity, security, privacy, accessibility, bias, human factors, or the surrounding administrative process. It should be repeated when the model, system instructions, retrieval corpus, tools, or intended use materially change.

TAIRC’s LLM Safety, Evaluation and Reliability programme is explicitly evaluation-centred. It does not train or fine-tune proprietary large-scale models, certify systems as safe, or deploy them in operational public environments. Any reliability method described here remains a public-interest evaluation approach, not a completed TAIRC benchmark, regulatory determination, or institutional approval.

Reliability is the quality of the boundary

The central failure in an ambiguous instruction is not uncertainty itself. Institutions operate under uncertainty constantly.

The failure occurs when a system hides uncertainty inside an answer, asks a question that does not reduce it, or treats a missing fact as permission to assume authority.

A trustworthy response may answer. It may ask. It may state an assumption. It may stop. The reliability test is whether that choice is justified by the missing information and the consequence of getting it wrong.

The reliable system is not the one that always answers or always asks. It is the one whose behaviour changes when the missing detail changes the consequence.

Sources and verification

The National Institute of Standards and Technology’s Artificial Intelligence Risk Management Framework 1.0 was used for the definitions and relationships among reliability, validity, robustness, expected conditions of use, contextual assessment, harm-sensitive testing, and human intervention.

NIST’s Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile, published in 2024 and listed by NIST as updated in April 2026, was used for its guidance on documented pre-deployment testing, context mismatch, prompt sensitivity, benchmark limitations, lifecycle documentation, and voluntary risk-management practices.

The U.S. Government Accountability Office’s Artificial Intelligence: An Accountability Framework for Federal Agencies and Other Entities was used for its governance, data, performance, monitoring, audit-question, and evidence-procedure framework.

Liu and colleagues’ ACL 2024 paper introducing CLAMBER was used for its ambiguity taxonomy, benchmark design, tested-model findings, clarification-error analysis, and stated limitations.

Kim and colleagues’ EMNLP 2024 paper, Aligning Language Models to Explicitly Handle Ambiguity, was used as evidence that ambiguity handling can improve under targeted alignment, together with the paper’s limits concerning task format, context length, and evaluated model families.

Zhang and Choi’s NAACL Findings 2025 paper, Clarify When Necessary: Resolving Ambiguity Through Interaction with LMs, was used for the distinction between detecting ambiguity and deciding whether clarification will improve the final task enough to justify its interaction cost.

Tamkin and colleagues’ Task Ambiguity in Humans and Language Models was used as counterevidence showing that the evaluated models could infer intended tasks successfully under some constrained ambiguous-classification conditions.

TAIRC’s editorial authority specification and research programme documents were used solely to verify institutional scope, research boundaries, publication standards, and the distinction between existing programmes and uncompleted future outputs.