Reporting Cost, Latency, and Failure Together
A language model can appear inexpensive because its report excludes retries. It can appear fast because unsuccessful requests terminate quickly. It can appear reliable because timed-out requests are removed before quality is calculated.
Each statement may be numerically correct. Together, they can describe a system that is expensive, slow, and unreliable for the people expected to use it.
That is the central reporting failure: cost, latency, and failure are often measured as separate properties even though institutions experience them as one outcome. A response has operational value only when it is produced within an acceptable time, meets the task’s quality and safety requirements, and does so at a cost the institution can sustain.
A credible evaluation should therefore report all three dimensions against the same workload and the same denominator. It should disclose the complete distribution, show which requests failed and why, and calculate the cost of acceptable results rather than the cost of generating output.
The governing question is not, “How cheap is this model?” or “How fast is it?” It is:
How much does it cost to obtain a response that arrives in time and is acceptable for the intended use?
Three dashboards can describe three different systems
Cost reports commonly use tokens, requests, accelerator-hours, or API charges as their denominator. Latency reports may include only completed requests. Quality reports may score only responses that reached the evaluator. Failures that occur before scoring—timeouts, truncated generations, rejected calls, malformed outputs, unavailable services—can disappear between these accounting systems.
The result is denominator drift.
Suppose an evaluation sends a fixed collection of requests. The cost calculation counts every API call, including retries. The latency calculation counts only requests that returned a response. The quality calculation excludes malformed responses because the automatic grader could not parse them. The three reported figures no longer describe the same population of attempts.
This is not a minor statistical defect. It can reverse an institutional decision.
A system with a low price per request may become expensive when repeated calls, fallbacks, validation, and human correction are included. A system with low average latency may have an unacceptable tail of delayed requests. A system with high accuracy on completed outputs may fail too often to support a real workflow.
NIST’s AI Risk Management Framework Playbook treats measurement as context-dependent. It recommends defining acceptable performance limits, documenting the reasons for selected metrics, reporting errors and negative impacts, and evaluating performance under conditions similar to the intended deployment setting. Its guidance explicitly places validity measures, such as false-positive and false-negative rates, alongside efficiency measures such as prediction latency. It also warns that test-set estimates alone do not directly measure real-world impacts.
The reporting implication is straightforward: performance dimensions should not be detached from the conditions and consequences that give them meaning.
Begin with one unit of work
Cost, latency, and failure can be compared only after the evaluation defines what counts as one attempted unit of work.
For a language-model study, that unit might be one user request under a specified prompt template, model version, context length, generation configuration, workload distribution, concurrency level, and execution environment. The report should state whether the measurement begins when the user submits the request, when the application sends the model call, or when the model service begins processing it. It should also state when the measurement ends.
Those boundaries affect every result.
A report measuring only the model provider’s response interval does not measure the user’s complete wait. Retrieval, document processing, routing, safety checks, retries, post-processing, and application rendering may all add delay. Likewise, an API invoice does not necessarily represent the full cost of the workflow. A self-hosted evaluation may have infrastructure, engineering, monitoring, and idle-capacity costs that a token-price comparison omits.
No single cost boundary is universally correct. The boundary must match the decision.
A researcher comparing model APIs may reasonably report direct inference charges. A university planning an internal service may need to include hosting and operational support. A public institution evaluating a document-assistance workflow may also need to account for human review when review is required to make an output usable.
The mistake is not choosing a narrow boundary. The mistake is presenting a narrow boundary as total cost.
Cost per output is not cost per usable output
The most familiar efficiency measures—cost per token and cost per request—remain useful. They help explain billing behavior and resource consumption. They do not reveal whether the purchased output succeeded.
A more decision-relevant measure is:
Cost per acceptable completion = total scoped cost divided by the number of acceptable completions.
An acceptable completion must be defined before the results are inspected. At minimum, it should meet the evaluation’s task-quality threshold, remain within its latency requirement, and avoid any failure condition the institution has classified as disqualifying.
This measure changes how retries are treated. A retry is no longer invisible overhead. Its cost remains in the numerator, while only the eventual acceptable result contributes to the denominator. If several attempts are required to produce one usable response, the report shows that consequence.
Human review requires the same discipline. If human validation is mandatory for every output, the report may present model-only cost and workflow cost separately. If reviewers intervene only after certain failures, the evaluation should disclose the intervention rate and the rule that triggered review. Combining these costs without explanation obscures the technical result; excluding them while claiming end-to-end efficiency obscures the operational result.
Research on model routing illustrates why joint measurement matters. FrugalGPT evaluated cascades that selected among language models to improve the relationship between cost and accuracy. RouteLLM similarly studied routers that choose between stronger and weaker models and reported more than twofold cost reductions in some evaluated settings without substantial quality loss. These results do not establish a universal saving: they depend on the tested models, tasks, prices, routing methods, and quality definitions. They do show that cost cannot be interpreted independently of the acceptance criterion the system is designed to preserve.
A cost comparison without a shared quality threshold is a price comparison, not an efficiency evaluation.
Latency is a distribution
Average latency compresses a workload into one value. Users encounter individual requests.
A small number of long delays can dominate the experience of an interactive or multi-stage system even when the mean remains attractive. Google’s research on large online services described how tail latency becomes increasingly consequential as systems grow in scale and complexity: a composite operation can be delayed by one unusually slow component.
A transparent report should therefore present latency as a distribution. The median can describe the typical request. Higher percentiles can show how the slower portion of the workload behaves. The appropriate percentile and threshold depend on the intended use; there is no universal latency objective that applies to every task.
Language-model serving introduces several distinct delays. Time to first token measures how long a user waits before generation begins. Time per output token describes the pace of generation after that point. End-to-end latency measures the interval until the response is complete. DistServe, presented at USENIX OSDI 2024, treated time to first token and time per output token as separate service objectives and defined serving performance in terms of requests completed within both constraints. Its reported improvements were specific to the models, applications, hardware configurations, and latency requirements in the study.
The current MLPerf Inference framework also avoids treating speed as unconstrained output production. Its datacenter benchmarks pair scenario-specific performance metrics with datasets, quality targets, latency constraints, and compliance rules. The benchmark structure reflects a fundamental principle: throughput or latency has meaning only when the submitted system continues to satisfy the benchmark’s required quality conditions.
Latency reports should also separate successful and unsuccessful requests. Google’s Site Reliability Engineering guidance gives a useful example: a server error may be returned extremely quickly, causing an aggregate latency figure to look better even though the request failed. The same guidance defines errors broadly enough to include explicit failures, incorrect content returned with a nominal success code, and responses that violate a stated service objective.
A fast failure remains a failure. A slow failure is worse. Neither should improve the system’s apparent responsiveness.
Failure is an outcome, not a status code
Language-model failures do not fit into one error rate.
Some are technical: the service is unavailable, the request is rejected, the response is truncated, or the output cannot be parsed. Some violate a time requirement. Some produce fluent output that fails the task. Others cross a safety, policy, privacy, or authority boundary even when the answer is factually plausible.
The report does not need to collapse these conditions into a universal taxonomy. It does need to define them before measurement and preserve them in the results.
This distinction prevents a nominally successful response from being counted as acceptable merely because the application received text. It also prevents every undesirable response from being treated as equivalent. A harmless formatting error, an unsupported factual answer, and an output that could cause a consequential institutional error require different interpretation and different controls.
Stanford’s Holistic Evaluation of Language Models was designed around a related concern. Rather than treating accuracy as the sole dimension of performance, HELM evaluated multiple measures, including calibration, robustness, fairness, bias, toxicity, and efficiency, within shared scenarios. The purpose was to expose trade-offs that a single metric would conceal. The framework also documented incomplete coverage rather than implying that its measurements exhausted the evaluation space.
The lesson is broader than any particular benchmark: a response can succeed under one measure and fail under another. Reporting should retain that structure.
The denominator is a governance decision
At first glance, combining cost, latency, and failure appears to be a measurement problem. The deeper issue is institutional authority.
What counts as acceptable depends on who will use the response, what they may do with it, and what happens when the response is wrong.
A delayed output may be tolerable in an offline literature review and disqualifying in an interactive service. A partially correct answer may help a researcher generate questions and remain unacceptable in a public-information interface. An unsupported statement may be a correctable defect in an internal draft and a serious failure when presented as official guidance.
The same model behavior can therefore produce different evaluation outcomes without either evaluation being dishonest. The difference must come from a declared context of use, not from changing the acceptance rule after seeing the results.
This is where reporting becomes governance. The acceptance criteria determine which harms, delays, and expenses the institution is willing to absorb. The evaluator should not bury those judgments inside code or convert them into a neutral-looking composite score.
A report should name the decision owner, intended users, foreseeable affected groups, permitted use, prohibited use, review requirement, escalation path, and consequence of failure whenever those factors materially shape the measurement. NIST’s guidance similarly states that the purpose, audience, needs, operators, social setting, and intended context influence what should be measured.
The arithmetic begins only after those choices are visible.
A minimum joint reporting record
TAIRC proposes that cost, latency, and failure be reported through one traceable evaluation record rather than three disconnected summaries.
The record should begin with the total number of attempted units of work. Every attempt should remain accounted for, including requests that did not return a scorable response.
It should then report the number and proportion of acceptable completions under a predeclared rule. Failure results should remain disaggregated by the categories relevant to the use case, such as service errors, timeouts, malformed outputs, task-quality failures, and policy or safety failures. Categories may overlap when one request violates more than one condition; the report should explain how overlaps are counted.
Latency should be reported for the full attempt population where measurement is possible, with separate distributions for acceptable completions and failed attempts. For streaming systems, the report should distinguish initial response delay, generation pace, and completion time when those measures matter to the user experience.
Cost should be reported within a named scope. At minimum, the record should show total scoped cost, cost per attempted unit, and cost per acceptable completion. Where retries, routing, fallbacks, retrieval, or human review are part of the evaluated workflow, the report should state whether each component is included.
The evaluation should preserve the conditions required to interpret the results: model and system versions, evaluation dates, prompt and generation settings, workload composition, hardware or service environment, concurrency, sample size, scorer, acceptance thresholds, and known differences between the test setting and intended use.
Raw component measures should remain available even when the report presents a joint measure. Readers must be able to see whether an improvement came from lower cost, shorter delay, fewer failures, a changed workload, or a relaxed acceptance rule.
This framework has not been validated by TAIRC as a completed benchmark or certification method. It is a proposed reporting discipline derived from established risk-management guidance, multi-metric evaluation research, reliability engineering, and current inference-benchmarking practice.
Why one composite score is still not enough
Combining cost, latency, and failure does not require reducing them to one ranking.
A weighted score can be useful when an institution has already established defensible priorities. It can also conceal those priorities. Small changes in the weights may reorder the models, and a strong result in one dimension may compensate mathematically for a failure that should have been disqualifying.
Hard acceptance constraints should be applied before optimization. A system that violates a critical safety condition should not recover its ranking through lower cost. A system that routinely misses the workflow’s deadline should not appear preferable because it produces inexpensive tokens.
After disqualifying conditions are applied, the remaining options can be shown as trade-offs. One configuration may be cheaper but slower. Another may reduce failures at greater expense. A third may dominate both under the tested workload. Presenting that frontier is more honest than declaring a universal winner.
Even “goodput”—the number of requests completed within stated service objectives—requires scrutiny. A research preprint revisiting service-level and goodput metrics showed that certain formulations could reward counterintuitive behavior, including delaying token delivery to improve a tail measure or abandoning requests that were likely to miss the objective. The paper proposed a revised framework, but its larger contribution is cautionary: once a metric becomes a target, evaluators must test whether the metric can improve while user outcomes deteriorate.
No summary statistic eliminates the need to inspect the underlying records.
What a responsible comparison can conclude
A joint report can support bounded conclusions.
It can show that one tested configuration produced more acceptable completions per dollar under a specified workload. It can show that another configuration reduced tail latency while holding the same quality threshold. It can reveal that apparent savings disappeared after retries and review were included. It can identify the failure categories responsible for poor operational performance.
It cannot prove that the result will transfer unchanged to another population, task, provider version, hardware environment, traffic pattern, or institutional setting. Model behavior, service conditions, prices, and software stacks can change. The report should state the evaluation date and preserve enough information for repetition.
This limitation is a reason for better reporting, not a reason to abandon measurement.
Cost, latency, and failure belong together because each constrains the meaning of the others. Cheap output has little value when it repeatedly fails. Accurate output may be unusable when it arrives too late. Fast output can be dangerous when speed hides incorrect or prohibited results.
The most useful metric is therefore not the price of generating an answer. It is the cost of obtaining an acceptable one under conditions that resemble the decision the institution actually faces.
The cheapest answer is not the one with the lowest invoice. It is the one that arrives in time, survives scrutiny, and does not have to be done again.
TAIRC’s Open LLM Transparency & Evaluation Frameworks program is limited to analytical, evaluative, and documentation-focused research. It does not train proprietary large-scale models, optimize commercial systems, certify model safety, endorse vendors, grant regulatory approval, or represent proposed reporting methods as completed empirical findings.
Sources and verification
The AI Research Center, TAIRC Research Portfolio and TAIRC Research Categories & Topics. These organizational records were used to verify TAIRC’s mission, the scope of its Open LLM Transparency & Evaluation Frameworks program, its planned-output language, and its prohibitions on proprietary model training, commercial optimization, production deployment, certification, endorsement, and regulatory claims.
National Institute of Standards and Technology, AI Risk Management Framework Playbook: Measure. Used to verify guidance on context-dependent measurement, acceptable performance limits, metric documentation, errors and impacts, evaluation under deployment-like conditions, and joint consideration of validity and efficiency measures.
Percy Liang and colleagues, Holistic Evaluation of Language Models, Transactions on Machine Learning Research, and Stanford Center for Research on Foundation Models’ HELM documentation. Used to verify HELM’s multi-metric design and its inclusion of accuracy, calibration, robustness, fairness, bias, toxicity, and efficiency.
Google, Site Reliability Engineering: Monitoring Distributed Systems. Used to verify the distinction between successful-request and failed-request latency and the treatment of explicit, implicit, and service-objective failures.
Jeffrey Dean and Luiz André Barroso, The Tail at Scale. Used to verify the importance of tail latency in large, complex, and multi-component systems.
MLCommons, MLPerf Inference: Datacenter. Used to verify that MLPerf performance measurements are tied to defined scenarios, datasets, quality targets, latency constraints, and compliance rules.
Yinmin Zhong and colleagues, DistServe: Disaggregating Prefill and Decoding for Goodput-Optimized Large Language Model Serving, USENIX OSDI 2024. Used to verify the separation of time to first token and time per output token and the study’s use of request rates constrained by both latency objectives.
Lingjiao Chen, Matei Zaharia, and James Zou, FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performance. Used to verify research on model cascades that jointly optimize cost and task performance.
Isaac Ong and colleagues, RouteLLM: Learning to Route LLMs with Preference Data. Used to verify research on routing between stronger and weaker models and the reported cost-quality results under the study’s evaluated conditions.
Zhihao Wang and colleagues, Revisiting Service Level Objectives and System Level Metrics for Large Language Model Serving. Used as contradictory evidence showing that some service-level and goodput formulations can reward delayed token delivery or abandoned requests. The source was treated as a research preprint rather than an established standard.