The Missing Metadata in Most Language-Model Claims

A language-model claim often arrives as a sentence with the laboratory removed.

“Model A is more accurate.” “Model B hallucinates less.” “This system can summarize policy.” “The assistant works across 50 languages.”

Each statement may be based on a real evaluation. Yet none is interpretable until the reader can reconstruct what was evaluated, under which conditions, against which data, with which scoring method, and for what intended use.

That is the central reporting failure. The public record usually preserves the conclusion and discards the conditions that gave the conclusion meaning.

For researchers, this blocks replication. For procurement teams, it creates false comparability. For journalists, it turns a qualified experimental result into a clean headline. For public institutions, it can convert an incomplete claim into a decision about a system that people may be expected to trust.

A language-model claim should therefore be treated as incomplete unless it binds the result to a specific model or system, a defined task and dataset, a documented evaluation procedure, an identified scoring method, an account of uncertainty, and a clear decision context.

Metadata is not administrative decoration around the evidence. It is part of the evidence.

A claim is a compressed experiment

The sentence “Model A scored 87 percent” looks self-contained. It is not.

Somewhere behind it was a model snapshot, an evaluation date, a benchmark version, a sample, a prompt template, a chat format, a decoding configuration, a scoring script, and a rule for handling outputs that did not fit the expected format. There may also have been retrieval, tool access, content filters, retries, human graders, or another model acting as a judge.

Remove those details and the score becomes portable in the worst sense. It can travel into a report, a leaderboard, a sales presentation, or a funding proposal while leaving behind the circumstances that limit what it proves.

The model-card proposal introduced by Margaret Mitchell and her coauthors was designed to resist this collapse. It called for documentation of intended uses, evaluation procedures, performance characteristics, relevant groups and conditions, and contexts in which a model may be unsuitable. The basic idea remains sound: performance must be reported with enough context for a reader to understand where the result applies and where it may fail.

But language-model claims now operate across more layers than a standalone model card can reliably capture. A deployed assistant may combine a foundation model with a system prompt, retrieval index, policy layer, external tools, memory, output filters, and human review. A claim about the “model” may actually describe that entire assembly.

The first task is to identify which one.

The first missing field is identity

A model name is rarely a sufficient identifier.

Open models may have base, instruction-tuned, quantized, merged, or fine-tuned variants. A local evaluation may use a particular checkpoint revision while a public claim names only the model family. Even when the weights remain stable, the tokenizer, serving stack, context window, or chat template may differ.

A usable claim record should state the exact model identifier available to the evaluator. For an open model, that may include the repository, revision, checkpoint hash, quantization method, tokenizer version, and inference backend. For an API-hosted model, it should include the provider’s model identifier, evaluation date, API version where available, and any evidence that the system was pinned to a dated snapshot.

When a precise snapshot cannot be obtained, that absence is material metadata. The report should say that the evaluated system may change and that later attempts may not reproduce the same behavior.

This is not pedantry. A result cannot be attached to a stable object when the object has not been stably identified.

The evaluated model may not be the system people use

The phrase “the model produced this answer” often hides a larger system.

A policy assistant may retrieve documents before generating a response. A customer-service tool may insert account context, constrain the answer through business rules, and pass the output through a safety filter. A research assistant may call a search tool, summarize retrieved pages, and attach citations selected by another component. An internal government tool may include a review queue that catches errors before anyone acts.

Those additions can improve performance. They can also introduce new failure modes.

Retrieval may supply outdated material. A system prompt may suppress uncertainty. A tool may fail silently. A filter may remove a correct answer or permit a harmful one. Human review may exist on paper but be skipped under workload pressure.

NIST’s Generative AI Profile distinguishes risks arising at model, system, application, and use-case levels. Its suggested actions address context of use, upstream data sources, retrieval-augmented generation, content moderation, business rules, evaluation data, assumptions, limitations, data provenance, and other system details that can affect performance. The profile is voluntary guidance, not a certification or a universal legal requirement, but its layered framing captures a basic truth: two applications using the same foundation model may require very different evidence.

A claim should name the layer it describes.

“The base model answered correctly” is different from “the retrieval-assisted application produced a source-supported response.” “The model refused the prompt” is different from “the surrounding safety system blocked the output.” “The assistant achieved 90 percent citation accuracy” is incomplete if the citation generator, retrieval index, and verification process are not identified.

The model is one component. The claim may belong to the system.

Procedure choices leave fingerprints

Evaluation is full of choices that appear small until they alter the result.

The evaluator chooses the instruction wording, the number and format of examples, the chat template, the system instruction, whether reasoning is requested, the temperature, the token limit, the stopping rule, the number of samples, and the retry policy. For tool-using systems, the evaluator also sets the tool budget, time limit, available actions, and success criteria.

Mature evaluation frameworks expose these controls because they belong to the experiment. EleutherAI’s Language Model Evaluation Harness supports configurable prompt design, post-processing, answer extraction, multiple generations, few-shot settings, chat templates, system instructions, and different inference backends. Stanford’s HELM framework emphasizes transparent, reproducible evaluation and provides prompt-level visibility for its leaderboards.

Research has shown why a single prompt should not be treated as neutral. A 2024 study published in Transactions of the Association for Computational Linguistics evaluated 20 language models across 39 tasks and approximately 6.5 million instances using instruction paraphrases. The researchers found that different instruction templates could change absolute performance and relative rankings. Later research has complicated the picture by finding that some apparent prompt sensitivity may be introduced by rigid scoring methods rather than by the model alone. That disagreement strengthens the reporting requirement: both the prompt and the scoring procedure need to be visible.

A claim such as “Model A outperforms Model B” should therefore disclose whether both systems received the same effective instruction, whether their native chat templates were used, whether the evaluation was repeated across prompt variants, and whether the scoring method recognized semantically equivalent answers.

Without that record, a reported advantage may belong to the setup.

A score can survive after its meaning has changed

Evaluation data carries its own hidden decisions.

A benchmark name may conceal multiple versions, splits, translations, filtered subsets, or community implementations. A reported average may come from the full dataset, a convenience sample, or a private test set that no outside reader can inspect. Examples may have been removed because they failed to parse. A multilingual result may combine strong performance in high-resource languages with severe failures elsewhere.

Then comes scoring.

Exact match, pass rate, pairwise preference, rubric-based human review, and model-based judging answer different questions. Each can be appropriate. None is interchangeable.

A model judge introduces its own version, prompt, sampling settings, rubric, positional effects, and possible conflicts of interest. Human evaluation introduces rater qualifications, instructions, adjudication rules, and agreement measures.

The scoring parser matters too. A system may know the answer but fail to place it in the expected slot. Another may produce a fluent explanation that earns credit from a model judge despite containing a factual defect. An evaluation can therefore measure task performance, format compliance, grader preference, or some mixture of all three.

The claim must identify which.

NIST’s Generative AI Profile calls for scientific-integrity and testing considerations to be documented, including experimental design, data collection and selection, representativeness, suitability, construct validation, evaluation methods, and risks that cannot be measured. The principle extends beyond the framework itself. A metric has meaning only through the construct it is supposed to represent.

“Accuracy” is not a complete description of a measure.

Uncertainty is metadata too

Language-model outputs can vary across runs, yet many claims are reported as if one number settled the question.

The record should show the sample size, number of runs, aggregation method, variance or confidence interval where appropriate, and treatment of failures or missing outputs. If only one deterministic run was performed, the report should say so. If the evaluation used a sample rather than the full test set, the sampling rule should be visible.

Aggregate scores should be accompanied by the distribution that matters for the intended use.

A public-information assistant may have a respectable average while failing consistently on eligibility exceptions, translated questions, or documents with conflicting dates. A summarization system may perform well on ordinary material and collapse when the source contains tables, amendments, or cross-references.

The relevant question is not simply how often the system succeeded.

It is where the failures concentrate, how severe they are, whether a user can detect them, and what happens next.

Missing metadata transfers authority

The usual defense of sparse reporting is practical. Full documentation takes time. Some information is proprietary. A public audience may not read technical details. Security-sensitive information cannot always be released.

All of that can be true.

The deeper problem is that omission is not neutral. When the method disappears, the reader is asked to substitute trust in the speaker. The model developer, evaluator, university, benchmark maintainer, or institution becomes the reason to believe the claim.

Missing metadata transfers authority from the record to the reputation of whoever published it.

That transfer is especially risky when the claim travels beyond its original audience. A research paper may assume that readers understand the benchmark. A leaderboard may assume familiarity with its evaluation harness. A company announcement may compress a technical appendix into one comparative sentence. By the time the claim reaches a procurement memo, the caveats may be gone.

A June 2026 preprint proposing “Evaluation Cards” addresses this interpretive gap by combining benchmark metadata, evaluation-run data, and model metadata into a unified record. Its authors analyzed 5,816 models, 635 benchmarks, and more than 101,000 reported results. They found that 96.5 percent of the analyzed model-benchmark-metric records lacked at least one field in the proposal’s minimal reproducibility sub-schema. Maximum-token settings were absent from 95.6 percent of those records, while temperature was absent from 93.9 percent. The framework separates reproducibility, reporting completeness, provenance, and comparability rather than compressing them into a single judgment.

The work is a preprint, its schema is not an adopted standard, and the authors acknowledge limits in the corpus, source coverage, and completeness measures. Still, it captures the reporting problem with unusual clarity: the evaluation ecosystem is rich in results and poor in interpretive records.

The minimum record for a public claim

A responsible claim record does not need to publish every internal artifact. It does need to preserve the fields that could change the reader’s interpretation.

The record should identify the claimant, evaluator, evaluation date, and relationship between the evaluator and the system developer. First-party testing is useful, but it does not occupy the same evidentiary position as independent replication.

The evaluated object must be identified precisely enough for another qualified party to know what was tested. That includes the model snapshot or checkpoint where possible, along with any application components that materially shaped the output.

The task and intended use must be defined. “Summarization” is too broad. Summarizing meeting notes, scientific papers, statutes, medical records, and public-benefit instructions creates different evidence requirements and different consequences of error.

The data record should identify the benchmark or dataset, version, split, language, sampling method, exclusions, preprocessing, and known contamination concerns. Private data may remain protected, but its composition and relevance should still be described at a level that permits scrutiny.

The procedure should preserve the prompt or prompt family, chat template, few-shot examples, system instruction, tool access, retrieval configuration, generation settings, run count, seed where meaningful, retry logic, and evaluation-harness version.

The scoring path must be traceable. That means recording the metric, implementation, parser, judge model or human rubric, grader instructions, adjudication process, and uncertainty calculation.

The result beneath the average also belongs in the record. Subgroup performance, failure categories, excluded cases, variance, justified confidence intervals, and representative errors give the aggregate its boundary.

The report should state what the claim does not support. An evaluation on general knowledge does not establish policy accuracy. English-language performance does not establish multilingual reliability. A laboratory test does not establish safe operation under real workload, adversarial use, changing source documents, or weak human oversight.

Finally, the record needs an owner, version, change history, and reevaluation trigger. A claim tied to a changing system should not remain permanently active.

This record is longer than a leaderboard row.

That is the point.

What should remain private

Transparency does not require publishing secrets, personal data, exploitable vulnerabilities, confidential test items, or proprietary implementation details.

A mature reporting system separates public disclosure from controlled evidence.

The public record should still reveal that restricted evidence exists, who may review it, why access is limited, which parts of the claim depend on it, and what independent assurance has been performed. “Confidential” should describe an access condition, not end the inquiry.

An institution may withhold an adversarial prompt set to prevent gaming while publishing the threat categories, test methodology, evaluator qualifications, scoring rules, aggregate results, limitations, and process for authorized review. A provider may protect a system prompt while disclosing its functional role, major policy constraints, version history, and whether changes trigger reevaluation.

The boundary should be reasoned and documented. Otherwise secrecy becomes another form of missing metadata.

A practical test: could the omission change the decision?

The cleanest way to decide whether a field belongs in a claim record is to apply a deletion test.

Imagine the field is removed. Could a reasonable reader reach a different conclusion about the system’s capability, risk, comparability, reproducibility, or fitness for use?

If the answer is yes, the field is material.

Would the claim look different if the reader knew the result came from a single run? If the benchmark was translated automatically? If one model used retrieval while its comparator did not? If the evaluator removed malformed answers? If the “independent” test was funded and designed by the developer? If the judge was another model from the same provider? If the tested snapshot is no longer available?

Those are not side notes. They are decision variables.

The deletion test also prevents metadata reporting from becoming an indiscriminate dump. The goal is not maximum volume. It is sufficient interpretability. A field belongs because it can change the claim’s meaning or the action taken from it.

The claim should expire when the system changes

A language-model claim is dated evidence about a moving target.

The model may be updated. The retrieval corpus may change. A benchmark may be revised. The system prompt may be edited. A new safety filter may reduce one failure while creating another. User behavior may reveal problems that pre-deployment testing did not capture.

The claim record should therefore define what makes the result stale.

A new model snapshot should trigger review. So should a material change in the prompt, tool set, retrieval data, scoring method, user population, decision role, or consequence of error. Serious incidents and repeated complaints should reopen the evidence even when the technical configuration appears unchanged.

Versioning does not weaken a claim. It keeps the claim attached to the conditions under which it was earned.

A language-model claim without metadata may still be true. The problem is that no one outside the original evaluation can tell how far the truth travels.

The next time a report says a model is safer, more accurate, more capable, cheaper, or ready for institutional use, the first question should not be whether the number is impressive.

It should be whether the laboratory is still attached.

Because a result without its conditions is not yet public evidence. It is a conclusion asking to be trusted.