How to Evaluate a New AI Model Release: A Checklist

A new model lands on a Tuesday, the announcement has a table where the new row is bold, and by Wednesday someone asks whether you can switch by Friday. The answer is almost never "yes" or "no". It is a sequence of cheap checks that either kill the idea early or earn it a gradual rollout.

EduGeek explains that sequence in eight steps, from reading the announcement skeptically to keeping an exit. The concrete examples are three releases from the last few weeks: Anthropic's Claude Opus 5.5 (2026-09-22), OpenAI's GPT-6 Sol and Luna (2026-09-22), and Google's Gemini 4 Argon (generally available 2026-10-13). For a roundup of what shipped, see AI Updates: October 2026.

# Step 1: Read the announcement like a reviewer

A launch post is vendor-authored, so the footnotes are the interesting part. Ask who ran the number, at what setting, against what, and on which version or subset of the benchmark.

Anthropic's Claude Opus 5.5 post is a useful specimen because it discloses a lot. Its comparison rows for OpenAI models are labelled as reported by OpenAI, and one benchmark (OSWorld 2.1) is marked "partial". It also says Terminal-Bench 4.0 has a standard error of about 2.6 points for Opus 5.5, that production safeguards were on during testing and may have lowered some scores, and that on one benchmark Zapier counted safeguard interventions as failures. Anthropic adds that margins are a less reliable guide at this capability level and that the model often suspects it is being evaluated. Every one of those caveats changes how much a one-point lead means.

Anthropic's Sonnet 5.5 post notes that its Terminal-Bench and CursorBench charts use GPT-5.6 Sol because GPT-6 Sol results had not been publicly reported on those benchmarks, and that three third-party GPT-6 Sol scores in its table may not yet reflect a recent bug fix (Anthropic does not expect a major impact on two of them). That matches OpenAI's changelog entry of 2026-09-25, which says an image-encoding bug had hurt GPT-6 Sol and Luna image understanding and advises teams to rerun evaluations. Numbers published in launch week can be stale within days.

Do the scores respond to effort? The same Sonnet 5.5 post notes that Opus 5.5's Terminal-Bench score is at Xhigh effort, and footnotes that Sonnet 5.5's FrontierCode score is lower at Max effort than at Xhigh, which Anthropic attributes to a code-review skill spawning subagents. A model with a tunable effort level has a curve, not a score; test the point you will run.

## Contamination and the eval-to-real-task gap

Benchmark contamination means evaluation items leaked into training data, so a score measures recall as much as skill. A 2024 survey of benchmark data contamination reviews how this happens and which alternative evaluation designs reduce it, and LiveBench is one such design, drawing questions from recently released sources and refreshing them regularly. You rarely get to audit a vendor's training data, so assume older public benchmarks are less trustworthy than fresh or private ones.

Even a clean benchmark measures a task someone else chose; your prompts, documents and tools differ. A model can lead a public coding benchmark and still lose on your repository.

## Noise is real

A single point difference on a few hundred items is often noise. Evan Miller's Adding Error Bars to Evals argues that evals are experiments and should be analysed as such: treat questions as a sample, report confidence intervals, and use proper formulas when comparing two models. Apply that thinking to vendor tables too. If a table has no error bars, assume they are wider than the gap.

# Step 2: Read the migration notes, not just the benchmarks

The most expensive surprise in a model upgrade is usually an API contract change, not a quality change. Read the migration section first. Recent examples from vendor documentation:

Notice the pattern: sampling controls are disappearing, and one vendor documents them as ignored while another documents 400 errors. A silent ignore is worse for evaluation, because your temperature-0 assumption quietly stops holding. Add a test that asserts which parameters your code sends and what the response reports back.

# Step 3: Build an eval on your own workload

A public benchmark is a prior; your own eval is the likelihood. Build it before the release, not after, and keep it stable so every candidate faces the same exam. The LLM evaluation guide covers construction in depth; the essentials for release triage are:

1. •Sample real traffic. Pull a few hundred anonymised production requests, stratified by task type, language and length. Include the hard and the boring.
2. •Add the failures. Every production incident and every complaint becomes a permanent test case.
3. •Define pass/fail per task. Prefer programmatic checks (schema validity, exact fields, unit tests, retrieval hit) over a model grading another model. If you use an LLM judge, calibrate it against human labels and keep the judge fixed across candidates.
4. •Run at the setting you will deploy. Same effort or thinking level, same maximum output tokens, same system prompt, same tools.
5. •Compare paired, with an interval. Score baseline and candidate on identical items and bootstrap the difference.

import random

def paired_bootstrap(baseline, candidate, n=10_000, seed=7):
    """baseline, candidate: lists of 0/1 results on the SAME tasks."""
    rng = random.Random(seed)
    pairs = list(zip(baseline, candidate))
    diffs = []
    for _ in range(n):
        sample = [rng.choice(pairs) for _ in pairs]
        diffs.append(sum(c - b for b, c in sample) / len(sample))
    diffs.sort()
    return diffs[int(0.025 * n)], diffs[int(0.975 * n)]

lo, hi = paired_bootstrap(base_results, cand_results)
print(f"pass-rate delta, 95% interval: [{lo:+.3f}, {hi:+.3f}]")

If the interval spans zero, you have not shown a difference, however the launch post reads. Slice by task type too: better on average but worse on your highest-value slice is not an upgrade.

Prompts are part of the system under test: give each candidate a short, equal tuning budget rather than comparing a polished baseline against a raw candidate.

# Step 4: Price a successful task, not a token

Per-token prices are inputs; the quantity you pay is cost per successful task:

cost_per_success = (input_tokens * p_in + cached_tokens * p_cache + output_tokens * p_out) / pass_rate

Check these before trusting the sticker price:

• →Thinking tokens bill as output. Google's guide says Gemini 4 Argon's output price includes thinking tokens, so a lower effort setting changes cost materially. Measure tokens per task at the setting you will run.
• →Tokenizers differ across generations. Anthropic's Haiku 5.5 post says the model uses a new tokenizer and slightly more tokens per task than Haiku 4.5, and that its cost comparison accounts for this. Your own logs will tell you whether that holds for your text.
• →Introductory prices expire. Per Google's guide, Argon's input and output prices double on 2026-12-01 (from $2 and $10 to $4 and $20 per million tokens). Model cost at the post-promotion price.
• →Caching changes the maths. Cached input is a fraction of base input on several models. Measure your real hit rate, not the theoretical one, as with any cache.
• →Billing rules change. Anthropic's release notes record that from 2026-09-24 certain refusals before any output are billed again.

For the wider discipline, see cloud cost optimisation and FinOps basics.

## Latency, context and rate limits

Measure time to first token and total completion time on your prompts, at your concurrency, at different times of day. Vendor speed claims describe a specific tier or mode; OpenAI, for example, lists separate fast and ultrafast modes. Then check three things a benchmark table omits:

• →Effective context. A 1M-token window (Opus 5.5 and Argon both list one) tells you the limit, not how well the model uses the middle of it. Run needle and multi-document tests at your real prompt lengths.
• →Output budget. Gemini's guide advises raising max_output_tokens because thinking shares the budget, and a too-low cap truncates answers that look like quality failures.
• →Rate limits and error semantics. OpenAI split its overload errors on 2026-09-02 into slow_down (429) and server_is_overloaded (503). Your retry and fallback logic needs to know the difference.

# Step 5: Test tools and structured output

Most production failures in LLM features are not wrong prose; they are malformed JSON, a skipped tool call or an argument of the wrong type. Test these explicitly:

• →Schema conformance rate. Over a few hundred requests, what fraction parses and validates without repair? Track retries per success.
• →Tool selection and arguments. Check that the right tool is called with valid arguments, including parallel calls and the case where no tool should be called.
• →Forced tool use. If your code relied on tool_choice forcing a call, the Claude migration notes above mean you need tool_choice: auto plus strict tool use or structured outputs to keep tool inputs schema-conformant. Re-test the no-tool path.
• →Multi-step stability. Run agent loops to completion many times and compare failure modes (stuck loops, repeated calls, premature stops).
• →Prompt-injection resistance. Put adversarial documents in tool results. Vendor claims on this are self-reported.

# Step 6: Read the safety and data-handling terms

Engineers often skip this step and legal finds it later. Read the terms per model:

• →Retention. Anthropic's release notes say Claude Fable 5.1 and Mythos 5.1 require 30-day data retention, while its Opus 5.5 and Sonnet 5.5 posts say zero data retention is available. Two models from one vendor, two different terms.
• →Residency. OpenAI's changelog says Ultrafast mode for GPT-6 Astra (2026-09-29) was US data residency only, and that Ultrafast for GPT-6.1 Sol (2026-10-08) added US and EU residency. If you have EU customers, region support is a launch gate.
• →State that sticks to an account. Anthropic notes that Sonnet 5.5 thinking blocks work only in the account that produced them, which affects failover design.
• →Vendor-side monitoring. OpenAI's documentation for misalignment monitoring describes asynchronous review of agent reasoning and actions, with HTTP 403 misalignment_policy_violation for blocked requests. Know what is reviewed, and handle the block as a first-class outcome.
• →Licence terms for open weights. Meta's Muse Spark Open model card requires attribution and a separate agreement for model-as-a-service offerings above a revenue threshold. MIT-licensed weights carry no such clause.
• →Acceptable use. Anthropic's 2026 usage policy update takes effect 2026-11-12; recheck that your use case still fits.

A model that wins on quality but fails a retention or residency requirement is a no.

# Step 7: Roll out behind a gate, with a way back

An eval proves the model can do your tasks. A rollout proves it does them under production load with real users. Sequence both.

1. •Offline eval gate. The candidate must meet pre-agreed thresholds on the workload eval (overall and on critical slices) and on cost and latency budgets. Put this in CI so a prompt or model change cannot merge without it.
2. •Shadow traffic. Send a copy of production requests to the candidate and discard the responses (or store them for comparison). You learn real latency, error rates, token counts and, with automated checks, output quality, with zero user exposure. Mind data-handling terms and side effects: shadow requests to a tool-using agent must not execute real actions.
3. •Canary. Google's SRE workbook defines a canary as "a partial and time-limited deployment of a change in a service and its evaluation" and advises comparing the canary population against the control using a small set of metrics (no more than about a dozen), favouring user-facing signals such as HTTP return codes and latency. Adapt that: route a small percentage of traffic, compare task success, retries, latency and cost per success against the control, and define the rollback trigger before you start.
4. •Ramp and soak. Increase in stages and watch peak load and rare inputs.

Pin versions and keep a rollback target. Prefer dated snapshot IDs over moving aliases. Google's changelog records gemini-3.7-flash requests being routed to gemini-3.8-flash on 2026-10-08, and DeepSeek's pricing page says its legacy model names are served by the newer model. An alias that moves is a deploy you did not make. Also remember that rollback targets expire: Anthropic's notes record Claude Opus 4.1 as retired on 2026-08-05, with requests now returning an error, and Sonnet 4.5 retires on 2026-11-30. Check the old model's retirement date before you promise a rollback.

Make the switch a config change. Keep model ID, effort level and prompt version behind a flag, so reverting takes seconds.

# Step 8: Plan for lock-in and fallback

Every model is a dependency with a price list, terms and a retirement date. Lock-in comes less from the model call than from what accumulates around it.

• →Server-side state. Gemini's guide suggests previous_interaction_id to keep conversation state on the server, and OpenAI's Agents API offers durable sessions. Both are convenient and both make switching vendors harder. Keep your canonical conversation record yourself.
• →Prompt and parameter dialects. Different vendors reject or ignore different parameters, as the table in Step 2 shows. Isolate vendor specifics in a thin adapter and keep prompts in versioned files with per-model variants.
• →Tool schemas and agent runtimes. Keep tool definitions in a neutral format, and treat vendor agent platforms as replaceable. Google's changelog shows its Antigravity agent preview changing built-in tool names and parameters between versions.
• →Fallback routing. Define a secondary model (another size from the same vendor, or another vendor) and route to it on specific errors, such as 503 overloads or a policy block you can safely retry elsewhere. Apply the Step 3 and Step 5 tests to the fallback too. An untested fallback is a second outage waiting for a trigger.

# Key takeaways

• →Read launch posts for their footnotes: effort settings, subsets, error bars, comparison choices and later bug fixes matter more than the bold row.
• →Early numbers go stale (OpenAI's 2026-09-25 image fix), and some of the newest Claude, GPT and Gemini models reject, ignore or remove parameters your code may rely on. Read migration notes first.
• →Build a private, stable workload eval and compare candidates with paired intervals, not single-point deltas.
• →Price cost per successful task at post-promotion prices, with real cache hit rates and thinking tokens, and check retention, residency and licence terms per model.
• →Ship through shadow traffic and a canary with rollback criteria defined up front, and pin snapshot IDs.
• →Keep conversation state, prompts and tool schemas portable, and test your fallback as hard as your primary.

# Further reading

• →Evan Miller, Adding Error Bars to Evals: A Statistical Approach to Language Model Evaluations
• →Xu et al., Benchmark Data Contamination of Large Language Models: A Survey
• →White et al., LiveBench: A Challenging, Contamination-Limited LLM Benchmark
• →Google, SRE Workbook, Chapter 16: Canarying Releases
• →Google, What's new in Gemini 4 Argon, an example of migration notes worth reading closely