How to Evaluate LLM Applications: A Practical Guide to Evals

Every team building on a large language model (LLM) starts the same way: change the prompt, try five inputs, decide it feels better, ship. That works until the third change quietly breaks something the first change fixed, and nobody can say when. EduGeek explains evals as the thing that replaces this loop: a repeatable measurement of how your application behaves on the inputs you care about, run on every change.

This guide covers how to define success, build a dataset, choose between code checks and model judges, evaluate RAG and agents, handle non-determinism with proper statistics, gate releases in continuous integration (CI), and avoid the pitfalls that make eval numbers lie. It ends with a small, runnable Python harness.

# Why vibes do not scale

Informal testing fails for structural reasons, not for lack of discipline:

• →Small, biased samples. You test the inputs you can think of, which are the easy ones.
• →No memory. A fix for one case silently regresses another, and without a fixed set you will not notice.
• →Output variance. Models are stochastic. One good response proves little about the rate of good responses.
• →No comparison. Swapping a model or prompt needs a before and after on identical inputs, not two separate impressions.

An eval is a fixed set of inputs, a way to score outputs, and an aggregate you can compare across versions. Everything else is detail.

# Define success per task first

"Is the answer good?" is not a metric. Write down, per task, what a pass means, in terms a second person could apply identically.

Prefer several narrow binary criteria to one holistic 1-to-10 score. "Mentions the cancellation deadline" and "contains no invented policy" are far more reliable to grade, and far more diagnostic when they fail, than "quality: 7".

# Build a golden dataset from real traffic

The best test inputs are the ones users actually send. Sample from production logs (with consent and redaction of personal data), and stratify so rare but important categories are not drowned out by the common ones. Then deliberately add:

• →Known failures: every production bug becomes a permanent test case.
• →Adversarial and edge inputs: empty input, very long input, other languages, prompt-injection attempts.
• →Should-refuse and should-abstain cases: questions the system must decline or answer with "I don't know".

Label with the people who own the domain, write down the labeling guidelines, and have two people label a subset to measure how often they disagree. If humans disagree often, the task definition is vague and no automated grader will fix that.

Version the dataset like code. Keep a development split you iterate against and a held-out split you consult rarely, so you can tell improvement from overfitting. Start small: a few dozen well-chosen cases will surface big problems, and you grow the set as production teaches you. How many you eventually need depends on how small an effect you want to detect, which the statistics section makes concrete.

# Deterministic checks first, judges second

Use code wherever code can decide. Deterministic graders are free, fast, perfectly repeatable and impossible to flatter: JSON schema validity, exact or normalized match, regular expressions, required and forbidden strings, numeric tolerance, unit tests for generated code, "did the tool call have the right arguments".

Use an LLM-as-judge for the remainder, such as tone, helpfulness and faithfulness to a source, where criteria are real but not computable. Judges are useful and biased in known ways. The MT-Bench paper reported position, verbosity and self-enhancement biases, alongside limited reasoning ability on hard questions.

The "Large Language Models are not Fair Evaluators" paper showed the order effect directly, and the self-preference paper linked below ties the preference to a model's ability to recognize its own text.

Calibrate every judge against humans before you trust it. Have people label a few dozen to a few hundred outputs, run the judge on the same ones, and compute agreement beyond chance (Cohen's kappa) plus the judge's true-positive and true-negative rates. A judge that passes everything has high agreement on an easy set and is worthless for catching failures, which is why the error rates on each class matter more than overall accuracy. Re-check when you change the judge model or prompt.

Practical judge design: one criterion per call, a short rubric with concrete pass and fail examples, a required brief justification before the verdict, and a reference answer where one exists.

# Offline and online evals

Offline evals run your fixed dataset against a candidate before release. They are repeatable and safe, and they catch regressions, but they only cover inputs you thought of.

Online evals observe live traffic: sample a fraction of real responses and score them with judges or code checks, track guardrail trigger rates, and gather user signals such as thumbs, retries, edits and escalations to a human. Online signals are noisy and lagging but they show what you missed. Run shadow comparisons or staged rollouts for risky changes. The loop that matters: online monitoring finds a failure, you add it to the golden set, and the offline eval prevents its return. The same discipline underpins how to evaluate a new AI model release before switching providers.

# RAG-specific metrics

A RAG system has two components that fail differently, so score them separately. The RAG guide explains the pipeline; the metrics below map onto it, and the Ragas paper proposes a reference-free suite (faithfulness, answer relevance, context relevance) along similar lines. Context recall and precision as defined below need labeled references.

• →Context recall: of the facts needed to answer, what share appears in the retrieved context? Low recall means retrieval missed evidence. Measured against labeled gold passages or gold facts.
• →Context precision: of what was retrieved, how much is relevant, and are relevant chunks ranked near the top? Low precision means noisy context that distracts the generator.
• →Faithfulness (groundedness): is every claim in the answer supported by the retrieved context? Typically judged claim by claim. Low faithfulness means the model is inventing or leaning on its parametric memory.
• →Answer correctness and relevance: does the answer actually resolve the question, compared with a reference?

Read them as a diagnostic grid. Low context recall with high faithfulness means a retrieval problem; high recall with low faithfulness means a generation problem. Add an abstention metric: when the corpus lacks the answer, does the system say so?

# Evaluating agents and tool use

Agents add multi-step trajectories, so final-answer grading alone hides problems. Evaluate three layers:

1. •Outcome: did the task end in the correct state? Prefer checking the world (database row, created ticket, file contents) over reading the agent's claim about it.
2. •Tool use: right tool chosen, arguments valid and correct, no forbidden or destructive calls. Mostly deterministic.
3. •Trajectory efficiency: step count, repeated calls, loops, tokens and wall-clock time. An agent that succeeds in forty steps where ten suffice is a cost and latency bug.

Run agent evals in a sandbox with stubbed or recorded external services so they are repeatable, assert on side effects, and cap steps so a looping agent fails fast instead of burning budget. Include cases where the right behavior is to stop and ask or refuse.

# Non-determinism and confidence intervals

A single run's pass rate is an estimate with uncertainty, and ignoring that produces fake wins. Two sources of noise exist: sampling noise (the same case can pass or fail on different runs) and case noise (your dataset is a finite sample of all possible inputs).

Handle the first by running each case several times and averaging, and by lowering sampling temperature where the task allows, while remembering production uses its real setting. Handle the second with an interval. For a pass rate p over n independent cases, the standard error is sqrt(p(1 - p) / n). At p = 0.8 and n = 100 that is 0.04, so a 95% interval is roughly plus or minus 0.078. A "two point improvement" on 100 cases is noise.

The sample size needed is derivable. Comparing two independent systems at about 80% pass rate, the standard error of the difference is sqrt(2 x 0.16 / n). To get a 95% interval half-width of 0.02 you need n of about 3,000.

Because both versions run on the same cases, a paired comparison, which resamples per-case differences, is usually tighter than that unpaired figure, by an amount that depends on how strongly the two versions' per-case results correlate, and it is what the harness below uses. The honest reporting format is a difference with its interval, not two bare averages.

# Gate regressions in CI

Wire the offline eval into the pipeline so quality is enforced, not remembered. A workable gate:

• →Run on every change to prompts, model choice, retrieval settings or tools, not only code.
• →Fail when the lower bound of the paired quality difference falls below an agreed tolerance, rather than when a raw average dips. This is a non-inferiority test: it passes only if the data can rule out a drop larger than the tolerance, so a set too small for its interval to fit inside the tolerance will block even an unchanged system.
• →Gate separately on critical slices, such as safety, refusals and your highest-value categories, so an overall gain cannot hide a slice that collapsed. Slices are small, so their intervals are wide, and every extra gate adds a chance of a spurious failure; size slices accordingly.
• →Keep a fast tier (small, mostly deterministic, runs on every commit) and a slow tier (full set with judges, runs before merge or nightly). Run both from your normal CI pipeline.
• →Store reports as artifacts so any regression can be diffed against a past run.

# Cost and latency are metrics too

A change that raises quality by one point and triples cost per request may be a bad trade; one that cuts cost by half at no measurable quality loss is a win you can only claim with data. Record, per run, tokens in and out, cost per call, and latency percentiles. Track the p95, not only the mean, because users feel the tail. Report quality, cost and latency together, and set budgets as gate conditions: for example, p95 may not regress beyond a stated ratio. Include judge costs in your own eval budget; they are real.

# Common pitfalls

• →Test-set leakage. Dataset examples pasted into prompts as few-shot examples, or into fine-tuning data, make the eval measure memorization. Keep a held-out split sealed.
• →Overfitting prompts to the eval. After many rounds of tuning against the same cases you are optimizing for those cases. Refresh the set from new traffic and watch the gap between dev and held-out scores.
• →Contaminated public benchmarks. Public test sets may appear in a model's training data. Trust your private, task-specific set above leaderboards.
• →Unvalidated judges. An uncalibrated judge is a random number generator with confident prose.
• →Averages that hide slices. Report per-category results.
• →Ignoring the abstain and refuse cases, which rewards systems that answer everything.
• →Treating the eval as finished. Traffic drifts; the dataset must too.

# A small eval harness

This standard-library Python harness ties the pieces together: deterministic graders before a judge, repeated sampling, per-case scores, p95 latency and cost, a paired-bootstrap confidence interval on the quality difference, Cohen's kappa for judge calibration, and a CI gate. run_app() and llm_judge() are stubs with a seeded random generator so it runs anywhere; replace them with your application and your judge call.

In the demo the candidate is deliberately worse, so the process exits with status 1, which is exactly what a CI job should do on a regression. With only 60 cases and a 0.02 tolerance, this stubbed gate also fails a candidate of identical quality most of the time (47 of 50 seeds in a test run), because the interval is wider than the tolerance. Real gates need enough cases for the interval to fit inside the tolerance.

import json, random, statistics, sys
from dataclasses import dataclass


@dataclass(frozen=True)
class Case:
    id: str
    prompt: str
    must_contain: tuple = ()
    tags: tuple = ()


# ---- Stubs: replace with your application and your judge model ----
def run_app(case, rng, quality):
    """STUB for the system under test. Returns (output, latency_s, cost_usd)."""
    ok = rng.random() < quality
    text = " ".join(case.must_contain) if ok else "I am not sure."
    return text, rng.uniform(0.4, 2.0), 0.002


def llm_judge(prompt, answer):
    """STUB for an LLM-as-judge call. Returns a 0/1 verdict on helpfulness."""
    return int(len(answer.split()) >= 2)


# ---- Graders: deterministic first, judge only for what code cannot check ----
def grade(case, output):
    if any(s.lower() not in output.lower() for s in case.must_contain):
        return 0
    return llm_judge(case.prompt, output)


# ---- Harness ----
def evaluate(cases, quality, n_samples=5, seed=0):
    rng = random.Random(seed)
    per_case, latencies, cost = {}, [], 0.0
    for case in cases:
        scores = []
        for _ in range(n_samples):  # non-determinism: sample repeatedly
            out, lat, usd = run_app(case, rng, quality)
            scores.append(grade(case, out))
            latencies.append(lat)
            cost += usd
        per_case[case.id] = sum(scores) / n_samples
    latencies.sort()
    return {"per_case": per_case,
            "p95_latency_s": latencies[int(0.95 * (len(latencies) - 1))],
            "cost_per_call_usd": cost / (len(cases) * n_samples)}


def paired_bootstrap_diff(base, cand, iters=2000, seed=1):
    """95% CI for mean(cand - base), resampling cases (the unit of variation)."""
    rng = random.Random(seed)
    diffs = [cand[k] - base[k] for k in base]
    means = sorted(statistics.fmean(rng.choices(diffs, k=len(diffs))) for _ in range(iters))
    return statistics.fmean(diffs), means[int(0.025 * iters)], means[int(0.975 * iters) - 1]


def cohens_kappa(human, judge):
    """Judge-vs-human agreement beyond chance, for binary labels."""
    n = len(human)
    observed = sum(h == j for h, j in zip(human, judge)) / n
    p_h, p_j = sum(human) / n, sum(judge) / n
    expected = p_h * p_j + (1 - p_h) * (1 - p_j)
    return (observed - expected) / (1 - expected) if expected < 1 else 1.0


def gate(base_report, cand_report, max_drop=0.02, max_p95_ratio=1.25):
    # Non-inferiority: fails unless the CI rules out a drop larger than max_drop.
    mean, lo, hi = paired_bootstrap_diff(base_report["per_case"], cand_report["per_case"])
    failures = []
    if lo < -max_drop:
        failures.append(f"quality: diff {mean:+.3f}, 95% CI [{lo:+.3f}, {hi:+.3f}]")
    if cand_report["p95_latency_s"] > max_p95_ratio * base_report["p95_latency_s"]:
        failures.append("latency: p95 regressed beyond allowed ratio")
    return failures


if __name__ == "__main__":
    cases = [Case(f"c{i}", f"question {i}", must_contain=(f"answer{i}", "ok")) for i in range(60)]
    base = evaluate(cases, quality=0.85)
    cand = evaluate(cases, quality=0.70, seed=7)
    print(json.dumps({k: v for k, v in cand.items() if k != "per_case"}))
    print("kappa:", round(cohens_kappa([1, 0, 1, 1, 0, 1, 0, 0], [1, 0, 1, 0, 0, 1, 1, 0]), 2))
    problems = gate(base, cand)
    for p in problems:
        print("FAIL", p)
    sys.exit(1 if problems else 0)

# Key takeaways

• →An eval is fixed inputs, a scoring method and a comparable aggregate; vibes cannot detect regressions.
• →Define success per task as narrow, binary, independently checkable criteria.
• →Build the golden set from real traffic, add every production failure, and keep a sealed held-out split.
• →Use code checks wherever possible; calibrate any LLM judge against human labels and mitigate position, verbosity and self-preference bias.
• →For RAG, separate retrieval metrics (context recall and precision) from generation metrics (faithfulness). For agents, check end state, tool calls and efficiency, not just the final text.
• →Report differences with confidence intervals, use paired comparisons, and gate CI on the lower bound.
• →Treat cost and latency as first-class metrics, and guard against leakage and prompt overfitting.

# Further reading

• →Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena (Zheng et al.), the LLM-as-judge study and its documented biases.
• →Large Language Models are not Fair Evaluators (Wang et al.), on the order effect in model judges.
• →LLM Evaluators Recognize and Favor Their Own Generations (Panickssery et al.), on self-preference in model judges.
• →Ragas: Automated Evaluation of Retrieval Augmented Generation, a reference-free evaluation suite for RAG.
• →Holistic Evaluation of Language Models (Liang et al.), a broad multi-metric evaluation framework.