Model Cards, System Cards, and Evaluation Reports: Three Different Jobs

An AI review team receives a model card, a system card, and a technical evaluation report from a prospective supplier. The package is extensive. It describes intended uses, safety measures, benchmark results, limitations, and deployment controls.

Yet the documents may still leave the central decision unanswered: Which model was actually tested? Which system configuration will be deployed? Do the reported results apply to that configuration? Could an independent evaluator reproduce the claim?

This is not a paperwork problem. It is an evidence problem.

A model card describes a model. A system card describes an assembled AI system and the conditions under which it is intended to operate. An evaluation report documents a specific inquiry into a model or system: what was tested, how it was tested, what happened, and how much confidence the evidence warrants.

The three artifacts can contain overlapping material, but they perform different accountability jobs. Treating them as interchangeable creates a category error. It allows claims about one object—the model, the deployed system, or the evaluation—to be supported by evidence about another.

The defensible approach is not to demand three documents for ceremonial completeness. It is to preserve three distinct units of analysis and connect them through explicit, versioned references.

The model card’s job: describe the model as an object of use

The model-card framework was introduced to accompany trained machine-learning models with structured information about their intended uses, performance characteristics, evaluation conditions, relevant groups, limitations, and caveats. The original proposal emphasized details such as the model’s version, type, intended and out-of-scope uses, evaluation data, metrics, quantitative analyses, ethical considerations, and known limitations. (arXiv⁠)

Its unit of analysis is the model.

That distinction matters because a model may be reused in many systems. The same language model could support a document-search application, a customer-service assistant, a writing tool, or a workflow that invokes external software. The model’s general properties may remain relevant across those uses, but the risks, interfaces, users, safeguards, and consequences can change substantially.

A useful model card should therefore help a prospective user answer questions about model selection and reuse. What version is being described? What inputs and outputs does it accept? What purposes did its developer anticipate? What uses were excluded? Which populations, languages, domains, or operating conditions were evaluated? Which limitations are known? Which important questions remain untested?

A model card cannot, by itself, establish how an application built around that model will behave. It may say little about retrieval components, system prompts, routing logic, tool permissions, user-interface constraints, moderation layers, logging, human review, rate limits, or incident-response procedures. Those elements can alter both performance and risk without changing the underlying model.

The presence of a model card also does not establish that the card is complete. A 2024 preprint examining 32,111 Hugging Face model-documentation records found highly uneven informativeness; sections concerning evaluation, limitations, and environmental impact were among the least frequently completed. The study demonstrates a practical limitation of documentation regimes: adopting a familiar format does not guarantee that decision-relevant information has been supplied. (arXiv⁠)

A model card is therefore a structured disclosure, not a certification. It records what its author is prepared to state about a model. Its reliability still depends on the evidence cited, the specificity of the versioning, the completeness of the disclosures, and the independence of any supporting evaluations.

The system card’s job: describe the assembled system and its deployment logic

A system card changes the object under examination.

The U.S. National Telecommunications and Information Administration distinguishes system cards from model cards by describing system cards as disclosures about entire AI systems, often composed of several models or other components working together. According to NTIA, system-level documentation can explain how input is processed and account for deployment context and real-world interaction in ways that an individual model card may not. (NTIA⁠)

The system card’s job is to make the assembled configuration inspectable.

That configuration may include one or more models, a router, retrieval infrastructure, external tools, policy filters, memory, user authentication, monitoring, human escalation, or application-specific instructions. The system card should identify which components are included, how they interact, what decisions they influence, who is expected to use the system, what safeguards apply, and which deployment conditions fall outside the documented scope.

OpenAI’s August 2025 GPT-5 system card provides a concrete example of why the system level can differ from the model level. The described GPT-5 product was a routed system involving a faster model, a reasoning model, a real-time router, and smaller fallback variants. The card specified that its primary evaluation focus covered particular models within that larger configuration. (OpenAI⁠)

Anthropic uses the term in a somewhat different but overlapping way, describing its system cards as documents covering model capabilities, safety evaluations, and responsible-deployment decisions. Its public index lists successive cards tied to dated model releases. (Anthropic⁠)

These examples reveal a limitation as well as a useful practice. “System card” is not yet a perfectly standardized label. One organization may use it for an application-level architecture; another may use it for a model family, its evaluations, safeguards, and release decision. NTIA has found substantial variation in the breadth and depth of model cards, system cards, and associated technical reports, making comparisons difficult and leaving readers to distinguish genuine unknowns from omitted information. (NTIA⁠)

The label therefore cannot carry the evidentiary burden. The document must state what it considers the system.

A credible system card should make its boundary visible. It should name the included model versions, major components, supported interfaces, deployment environment, intended users, prohibited or unsupported uses, human-oversight arrangements, monitoring mechanisms, known failure paths, and conditions that would require reassessment. It should also identify which conclusions come from direct testing, which come from design assumptions, and which remain unresolved.

A system card is especially important when an organization must decide whether an application is ready for controlled use. That decision cannot be made from model-level characteristics alone. NIST’s AI Risk Management Framework emphasizes documenting intended purposes, deployment settings, users, assumptions, limitations, requirements, and potential impacts as part of understanding an AI system’s context. (NIST AI Resource Center⁠)

Still, a system card is not independent assurance merely because it discusses safety. It commonly remains a self-authored account of the developer’s system, evidence, and deployment reasoning. It can disclose the basis of a decision. It does not automatically prove that the decision was correct.

The evaluation report’s job: preserve the evidence behind a claim

An evaluation report is organized around neither a model nor a system in the abstract. Its unit of analysis is an inquiry.

The report should state the question being examined, the object tested, the protocol used, the operating conditions, the evidence observed, the uncertainty surrounding the result, and the limitations on interpretation.

NTIA describes claim validation as one purpose of AI evaluation: determining whether a system performs as claimed within its stated limitations. It also recognizes evaluations that apply criteria independent of the developer’s own claims. Those inquiries may be narrow, such as testing task accuracy, or broader, such as examining governance, architecture, bias, privacy, robustness, or other risk dimensions. (NTIA⁠)

The evaluation report’s job is therefore to expose the path from question to conclusion.

A result such as “the model achieved 82 percent accuracy” is incomplete without the dataset version, sampling rules, prompt or input format, scoring method, model version, system configuration, decoding settings, exclusions, comparison baseline, uncertainty analysis, and date of testing. Depending on the evaluation, additional details may include annotator instructions, inter-rater agreement, software versions, tool access, random seeds, stopping conditions, and procedures for handling invalid outputs.

NIST’s AI Risk Management Framework Playbook specifically calls for documentation of test sets, metrics, tools, processes, and materials used during testing, evaluation, validation, and verification. It links this documentation to repeatability, consistency, and risk-management decisions. (NIST AI Resource Center⁠)

Stanford’s HELM Capabilities project demonstrates the practical value of this specificity. Its published methodology identifies scenarios, prompt templates, model versions, metrics, and prompt-level records intended to support reproducibility. The researchers also found that results reported by providers, official benchmark sources, and third-party leaderboards could vary substantially or conflict, illustrating why a score without its evaluation conditions is weak evidence. (Stanford CRFM⁠)

NIST’s own evaluation publications show that reports can address different objects and methods. NIST AI 700-1, released in June 2025, documented a pilot study of text generation and AI-generated-text discrimination. NIST AI 700-2, released in November 2025, described scenario-based evaluation involving human testers interacting with AI applications to examine real-world risks and impacts. (NIST AI Resource Center⁠)

Neither report format is universally sufficient. The appropriate design follows the claim being tested.

An evaluation report can also be rigorous and still have limited decision value. A study may be reproducible but poorly matched to the intended deployment. A benchmark may measure a convenient proxy rather than the outcome that matters. An average may conceal severe failures in a subgroup or operating condition. A test may become stale after a model, prompt, retrieval corpus, interface, or policy layer changes.

Reproducibility is necessary evidence of procedural clarity. It is not proof that the procedure asked the right question.

The documents are separated by responsibility, not length

The difference among these artifacts is sometimes described as a difference in detail: a model card is short, a system card is broader, and an evaluation report is technical.

That description is unreliable.

A model card may be long. A system card may contain hundreds of pages of evaluation results. An evaluation report may cover an entire application. Organizations may combine all three into one publication or distribute them across several web pages and appendices.

The decisive question is not how many pages exist. It is whether each claim has a defined home.

A claim about a model’s intended use belongs in a model-level record. A claim about the configuration presented to users belongs in a system-level record. A claim that the model or system achieved a result belongs in an evaluation record that preserves the method and evidence.

One publication can serve as the reader-facing entry point, but the underlying records should remain separable. Otherwise, a revision to one layer can silently invalidate another.

Consider a system that replaces its underlying model while leaving its interface unchanged. The system card may still look familiar, but previous evaluation results may no longer apply. Consider a system that keeps the same model but adds retrieval, browsing, or tool execution. The model card may remain accurate, while the system’s risk profile changes. Consider an evaluation that changes its prompt template or scoring model. The subject of the evaluation may be unchanged, but the results may no longer be comparable.

Versioning is therefore part of the evidence, not an administrative afterthought.

The model card should identify the exact model release. The system card should identify the model releases and other components included in the documented configuration. The evaluation report should reference the precise model or system version tested and preserve the protocol that produced the result. Each artifact should carry a date, an owner, a revision history, and a statement describing what changed.

When one layer changes materially, the documentation chain should show which other records require review.

A practical decision rule

The correct artifact follows the object of the decision.

When the decision concerns whether a particular model is appropriate for reuse, the model card is the primary disclosure. When the decision concerns whether an assembled application should be acquired, deployed, or permitted to perform a defined function, the system card becomes necessary. When the decision concerns whether a performance, safety, fairness, robustness, or reliability claim is supported, the evaluation report carries the evidentiary burden.

Consequential decisions generally require all three.

Before accepting a claim, a reviewer should be able to determine what object the claim describes; which version was examined; which deployment configuration was involved; what question the evaluation asked; what data, prompts, metrics, tools, and scoring procedures were used; what uncertainty or adverse findings were observed; who produced the evidence; whether any review was independent; and what future change would invalidate the conclusion.

When those answers all point vaguely to the same polished document, the evidence chain remains incomplete.

Why one comprehensive document is not enough

There is a reasonable counterargument: requiring separate artifacts can create duplication, maintenance costs, and reader confusion. A single comprehensive system report may be easier to publish and easier to navigate.

That approach can work as an interface. It is weaker as a record-management architecture unless the combined document preserves the three distinct layers.

A comprehensive report should allow a reviewer to isolate the model disclosure, the system configuration, and each evaluation protocol. It should distinguish inherited model limitations from application-specific limitations. It should connect conclusions to named evaluation records rather than placing results in an undifferentiated narrative. It should make clear when evidence applies only to a particular version or environment.

Disclosure also has legitimate limits. Security concerns, privacy obligations, intellectual-property constraints, or the risk of facilitating misuse may prevent every detail from being published openly. NTIA recognizes that some AI-system information may appropriately be shared only with trusted parties or government bodies rather than the general public. (NTIA⁠)

Restricted disclosure, however, should not be confused with nonexistent evidence. A public artifact can identify the category of information withheld, explain the reason for the restriction, state who can review it, and distinguish an undisclosed finding from an evaluation that was never performed.

The same principle applies to independent review. A developer-authored evaluation may be useful and technically sound, but authorship and independence should remain visible. A self-assessment, an external audit, an academic replication, and a regulator’s examination do not provide identical forms of assurance.

Transparency begins when the records connect

Model cards, system cards, and evaluation reports are often discussed as transparency products. Their deeper value is traceability.

A model card without system context can invite readers to generalize model-level evidence to an application that was never evaluated. A system card without accessible evaluation records can turn safety conclusions into assertions. An evaluation report without exact model and system identifiers can produce findings that are impossible to apply.

The strongest documentation architecture separates the three jobs and then reconnects them.

The model card defines the model being offered. The system card defines the configuration being used. The evaluation report defines the evidence supporting a particular claim. Cross-references show whether the evidence still applies.

This approach does not eliminate uncertainty. It makes uncertainty locatable. It allows a reviewer to see whether a missing answer reflects an acknowledged limitation, a restricted disclosure, an obsolete test, an unexamined deployment condition, or an unsupported claim.

TAIRC’s Open LLM Transparency & Evaluation Frameworks program is explicitly limited to analytical, evaluative, and documentation-focused work. It does not train proprietary large-scale language models, deploy commercial systems, or provide certification, endorsement, or regulatory approval. Its published program description also distinguishes intended future outputs from completed research artifacts. (TAIRC⁠)

That boundary reflects the central lesson of documentation itself: describe the evidence that exists, identify the evidence that does not, and never let the name of an artifact imply more assurance than its contents can support.

A model card is a map of a model. A system card is a map of an assembled system. An evaluation report is the record of a test.

None is a verdict.

Sources and verification

Margaret Mitchell, Simone Wu, Andrew Zaldivar, Parker Barnes, Lucy Vasserman, Ben Hutchinson, Elena Spitzer, Inioluwa Deborah Raji, and Timnit Gebru, “Model Cards for Model Reporting,” FAccT 2019. Used to verify the original purpose, scope, and proposed contents of model cards. (arXiv⁠)

Weixin Liang and co-authors, “What’s Documented in AI? Systematic Analysis of 32K AI Model Cards,” arXiv preprint, February 2024. Used to verify evidence concerning uneven completeness in a large corpus of Hugging Face model documentation. (arXiv⁠)

National Telecommunications and Information Administration, “AI System Disclosures,” March 27, 2024. Used to verify the government’s distinction between model cards and system cards, observed variation among disclosure artifacts, comparability limitations, and the possibility of tiered access to sensitive information. (NTIA⁠)

National Telecommunications and Information Administration, “Purpose of Evaluation,” March 27, 2024. Used to verify the roles of claim-validation and criteria-based AI-system evaluations. (NTIA⁠)

National Institute of Standards and Technology, AI Risk Management Framework Core and AI RMF Playbook, Measure function. Used to verify documentation expectations concerning deployment context, intended purpose, limitations, test sets, metrics, tools, methods, repeatability, and risk decisions. (NIST AI Resource Center⁠)

Jialiang Xu, Yifan Mai, and Percy Liang, “HELM Capabilities: Evaluating LMs Capability by Capability,” Stanford Center for Research on Foundation Models, March 20, 2025. Used to verify the named example concerning versioned models, scenario-specific prompts, prompt-level transparency, reproducibility, and conflicting reported benchmark results. (Stanford CRFM⁠)

National Institute of Standards and Technology, NIST AI 700-1 and NIST AI 700-2 evaluation-report summaries, June and November 2025. Used to verify the scope and publication dates of the GenAI text-evaluation and ARIA pilot reports. (NIST AI Resource Center⁠)

OpenAI, “GPT-5 System Card,” August 7, 2025. Used to verify the example of a routed AI system containing multiple model variants and a system card with a defined evaluation scope. (OpenAI⁠)

Anthropic, “Model System Cards.” Used to verify Anthropic’s stated purpose for system cards and its release-specific documentation practice through June 2026. (Anthropic⁠)

The AI Research Center, “Open LLM Transparency & Evaluation Frameworks.” Used to verify TAIRC’s program scope, explicit institutional boundaries, and the status of expected outputs. (TAIRC⁠)

All cited web sources were opened and checked on July 19, 2026.