Version Drift: Why Yesterday’s Evaluation May Not Describe Today’s Model
A model endpoint can keep a familiar name while the system behind that name changes.
Google’s Gemini documentation states that its “latest” aliases are replaced when a new release becomes available. OpenAI provides dated snapshots so developers can bind an application to a particular model version. Anthropic distinguishes pinned model identifiers from older convenience aliases that resolve to dated releases. The providers use different naming systems, but the implication is the same: a model name is not always a permanent technical identity.
That distinction matters whenever an institution relies on an evaluation conducted last month, last quarter, or even yesterday.
A procurement team may have approved a language model after testing its factual accuracy. A research group may have published a comparison of several systems. A public agency may have reviewed a model’s ability to summarize policy documents without omitting material qualifications. Those conclusions describe the system that was tested under the conditions that existed at the time.
They do not automatically describe the system responding now.
An evaluation is a dated observation, not a permanent property certificate.
The Model Name Is Not the Evaluated Object
Evaluation reports often identify the tested system with a product-level label: GPT, Claude, Gemini, Llama, or another model family. That may be enough for casual discussion. It is not enough for reproducible institutional evidence.
The actual object being evaluated is a configuration.
That configuration includes the exact model release, the API or interface used to access it, the system instructions, the prompt template, generation settings, available tools, retrieval sources, safety filters, output-processing logic, evaluation dataset, scoring method, and date of execution. A change in any material component can alter the result even when the visible model name stays the same.
This is why two institutions can claim to have evaluated “the same model” and still produce results that are not genuinely comparable. One may have called a dated API snapshot with a fixed system prompt. Another may have used a consumer chat interface that receives continuous product updates. One may have enabled web search or retrieval. Another may have tested the base model without tools. One may have scored exact matches. Another may have relied on a language model to judge whether answers were acceptable.
The shared brand name conceals different systems.
A fair evaluation therefore begins with a harder question than “Which model did you test?”
It asks: “What exact combination of model, instructions, data, tools, controls, and scoring procedures produced this result?”
Four Changes Can Hide Behind One Stable Label
Version drift is often described as a model changing over time. That definition is too narrow for institutional use.
The first form is model drift. The provider releases different weights, modifies post-training, changes safety behavior, or replaces one model snapshot with another.
The second is configuration drift. The underlying model may remain fixed while the system prompt, reasoning setting, generation parameters, context-management rules, moderation layer, or tool permissions change.
The third is pipeline drift. A retrieval index is rebuilt. A document corpus is updated. A reranker changes. A citation parser is replaced. A previously unavailable search tool is enabled. The model is technically the same, but the system receiving and shaping its answers is not.
The fourth is evaluation drift. The test set changes, a benchmark ages, a scoring script is patched, the instructions given to human reviewers are revised, or an automated judge is replaced. The measured score moves even though the system being measured may not have changed.
These cases require different responses. A provider update may require full regression testing. A corrected scoring bug may require recalculating previous results. A changed retrieval corpus may call for source-quality testing rather than a general model benchmark. A change in the user population may require new representative test cases.
Calling all of them “the model got better” or “the model got worse” hides the evidence needed to determine what happened.
Longitudinal Evidence Shows the Problem, Not Its Entire Cause
Researchers have documented meaningful changes in model behavior across releases.
A longitudinal study by Lingjiao Chen, Matei Zaharia, and James Zou compared March 2023 and June 2023 versions of GPT-3.5 and GPT-4 across tasks involving mathematics, code generation, sensitive questions, multi-hop reasoning, medical examination questions, and visual reasoning. The authors reported substantial differences across versions and argued for continuous monitoring of language-model services. The peer-reviewed version was published in the Harvard Data Science Review in 2024.
The study is useful because it shows why the phrase “GPT-4 performance” can be incomplete. The March and June systems did not behave identically.
It does not, by itself, establish a universal rate of deterioration. Nor can an external evaluation fully attribute every observed change to altered model weights, safety policies, prompt sensitivity, infrastructure, or another hidden component. The researchers could compare outputs, but they did not have complete visibility into the provider’s internal system.
That limitation is part of the lesson.
External evaluators should report observed change without inventing a causal explanation. “Performance declined on this task under these conditions” is supportable. “The provider damaged the model’s reasoning ability” may not be.
The distinction between observation and attribution is where credible evaluation separates itself from speculation.
A Pinned Version Still Does Not Guarantee Identical Answers
Version pinning reduces one source of uncertainty. It does not eliminate all of them.
OpenAI’s documentation describes model snapshots as a way to hold performance and behavior more consistent, while its reproducibility guidance says deterministic output remains a best-effort property rather than a guarantee. Anthropic similarly states that identical requests may produce different outputs even when temperature is set to zero.
That means a single successful response is weak evidence.
A serious evaluation uses repeated trials where variability matters. It records the full request configuration. It preserves raw outputs rather than retaining only aggregate scores. It distinguishes a systematic behavioral change from ordinary sampling variation.
Suppose a summarization system omits an important limitation in one run out of one hundred. That may indicate a low-frequency failure under a stable configuration. If the same omission appears in forty runs after an update, the evidence points toward a material behavioral change. Without repeated measurements and archived outputs, the institution cannot tell the difference.
Pinning identifies the system more precisely. Replication establishes how that system behaves.
Both are required.
Every Evaluation Has a Half-Life
An evaluation does not become invalid merely because time has passed. Its usefulness decays when the evaluated system and the current system stop being demonstrably equivalent.
This suggests a more useful concept than a universal expiration date: evaluation half-life.
A tightly controlled extraction workflow using a pinned model snapshot, fixed prompt, fixed schema, versioned code, stable data source, and deterministic validator may retain evidentiary value for a substantial period. A consumer-facing assistant using a moving model alias, live search, frequently updated safety policies, changing retrieval sources, and undocumented system instructions may require much more frequent testing.
The half-life should also depend on consequence.
An outdated evaluation of a brainstorming assistant may create inconvenience. An outdated evaluation of a system used to summarize regulatory obligations, explain public services, or support a safety-sensitive workflow may expose people to materially incorrect information.
The calendar alone cannot decide when to retest. The decision should reflect system volatility, evidence quality, use context, affected population, and severity of failure.
The deeper problem is therefore less about change than traceability.
Models will change. Retrieval corpora will change. policies will change. User needs will change.
The institutional failure occurs when nobody can reconstruct which change made the previous evidence obsolete.
The Evaluation Identity Record
Every published evaluation should include an identity record detailed enough for another reviewer to understand what was actually tested.
The record should name the provider, exact model identifier, snapshot or release date, and whether the identifier was pinned or represented a moving alias. If an alias was used, the resolved model version should be captured when the platform exposes it.
It should preserve the system and developer instructions, or provide cryptographic hashes when the instructions cannot be published. It should identify the user-prompt template, few-shot examples, generation settings, reasoning or effort configuration, tool permissions, safety middleware, moderation rules, and any provider-specific parameters that could affect behavior.
For retrieval-augmented systems, the record should identify the source corpus, document versions, ingestion date, chunking process, embedding model, retrieval configuration, reranker, citation method, and rules governing when external search is permitted.
The evaluation side needs the same discipline. The benchmark version, individual test-case identifiers, sampling method, expected outputs, scoring code, evaluator instructions, judge-model version, human-adjudication procedure, execution environment, and repository commit should all be recorded.
Raw outputs should be retained when privacy, licensing, security, and data-governance requirements permit. Aggregate scores cannot reveal whether an update caused a broad decline, a narrow regression, a new refusal pattern, a formatting failure, or a change affecting one language or user group.
This identity record does not certify that a model is safe. It does not prove that an evaluation is complete. It establishes the narrower claim that the reported result is attached to an inspectable system and method.
Without that attachment, a score floats free of the conditions that produced it.
When Yesterday’s Evaluation Can Still Be Used
An earlier evaluation remains relevant when the institution can establish continuity.
The current model should match the evaluated snapshot, or the provider and evaluator should supply evidence that the change is immaterial to the tested use. The instructions, tools, retrieval pipeline, safety controls, output processors, evaluation data, scoring method, and deployment context should remain substantially equivalent. Any known updates should be reviewed for their possible effect on the measured capability or risk.
Even then, the conclusion must stay within its original scope.
A test showing strong English-language summarization does not establish multilingual reliability. A benchmark conducted without retrieval does not describe a retrieval-enabled system. An evaluation of short, clean prompts does not establish performance on incomplete records or adversarial inputs. A model that passed a general factuality test has not therefore been validated for legal, medical, financial, or safety-critical guidance.
Old evidence may remain useful. Its boundaries do not expand with age.
Retesting Should Be Triggered by Change, Not Ritual
Institutions need recurring evaluation, but an arbitrary monthly rerun can become paperwork without insight.
A stronger process combines scheduled monitoring with change-triggered testing.
A new model snapshot, alias reassignment, system-prompt revision, retrieval-corpus rebuild, tool addition, policy update, scoring change, provider migration, or unexplained production shift should trigger a defined evaluation response. The response may range from a focused regression test to a complete re-evaluation, depending on the affected component and the consequences of failure.
The test suite should contain a frozen anchor set and a changing challenge set.
The anchor set detects whether known behavior has moved. Its prompts, expected outcomes, scoring rules, and review instructions remain stable across versions. The challenge set captures new risks, emerging use patterns, fresh documents, changing language, and failures observed after deployment.
Using only the anchor set turns evaluation into a historical replay. Using only fresh tests destroys comparability.
The result should not be reduced immediately to one average. Reviewers should examine which cases changed, in which direction, for whom, and under what conditions. An update that raises the overall score can still produce an unacceptable regression in a narrow but consequential category.
Improvement can be drift.
Monitoring Is Part of the Evidence
The National Institute of Standards and Technology’s AI Risk Management Framework materials treat monitoring as a lifecycle responsibility. NIST notes that system performance and trustworthiness may shift after deployment, recommends regular evaluation and post-deployment testing, and calls for continuous monitoring of third-party pretrained models and connected system components.
This changes what a credible evaluation program looks like.
The work is not complete when a benchmark report is published. The institution needs a change log, an inventory of model dependencies, defined retesting triggers, preserved baselines, escalation thresholds, rollback or suspension procedures, and a way to communicate when prior findings no longer apply.
A public report should also carry a status.
The result may be current for a named snapshot. It may be provisionally applicable to a successor pending focused regression testing. It may have expired because the evaluated endpoint was retired. It may remain historically valuable while no longer supporting present deployment decisions.
Silence is the worst status. It allows an old score to keep circulating after its object has disappeared.
The Publication Rule
Any institution publishing model evaluations should follow one rule:
Never report a model score without reporting the identity and date of the system that earned it.
The report should also state what would cause the conclusion to be reconsidered. That may include a new snapshot, a changed alias, a revised prompt, a new safety layer, a retrieval update, an altered judge, or evidence of changed performance in the intended setting.
This is not a demand for perfect visibility into proprietary systems. External evaluators will rarely receive that.
It is a demand to document what can be known, mark what cannot be known, and avoid presenting temporary evidence as timeless truth.
Yesterday’s evaluation may have been rigorous. It may still be useful. It may even remain the best evidence available.
But it describes yesterday’s system until continuity with today’s system has been demonstrated.
A benchmark score without a version record is a timestamp with the date torn off.