Chapter 12Testing, evaluation, and trace discipline

Testing an agentic system is a different problem from testing a deterministic system. The system under test is partially probabilistic; the failure modes are statistical; the assertions are over distributions of behavior rather than over single outputs. The standard testing techniques — unit tests, integration tests, end-to-end tests — still apply to the deterministic substrate. They are necessary; they are not sufficient.

This chapter develops what is added on top: the vocabulary of ground truth and the unit of evaluation, trace discipline as the system of record, evaluation as a continuous process, and a set of test patterns that are tractable to apply against probabilistic components. The argument is that the architecture from Chapters 5–10 makes these patterns possible — bounded, governed, observable systems can be tested in ways that unconstrained agents cannot. The broader evaluation literature — benchmarks across planning, tool use, memory, and domain tasks, and the gaps in cost-efficiency, safety, and stability that motivate the architecture here — is surveyed in the academic literature (Survey on Evaluation of LLM-based Agents, 2025); see Chapter 21.

The chapter takes a position: in 2026, testing has shifted from a phase to a property. The relevant question is not “did the system pass the test suite before release”, though it should, but “is the system continuously evaluating itself against the envelopes its architecture defines?” The shift is forced by the nature of probabilistic components: their behavior can change between releases without code changes (model upgrades), within releases under load (sampling variability), and across time even with the same model and prompt (drift in input distributions). A pre-release test pass certifies the system at one instant; a continuous evaluation certifies it across instants.

Three senses of ground truth

The phrase “ground truth” carries three different meanings in agentic engineering, and using them interchangeably obscures which team owns a defect and where to look for it, so the chapter fixes the vocabulary before the techniques.

Grounding concerns the evidence and state presented to the agent: whether the sources it reasons over are authoritative, current, correctly scoped, and semantically coherent. It is an input problem, and the book develops it where the inputs live — the memory architecture and read-path measurements of Chapter 7, the ingestion pipeline of Chapter 8, and the semantic layer and entity resolution of Chapter 14. Evaluation ground truth concerns the evidence used to decide whether a trial of the system succeeded. It is an output problem, and it is this chapter’s subject. Training ground truth concerns the supervision or reward signal used to change model parameters. It belongs to teams that train or post-train models, and it is outside this book’s architectural scope: the systems this book describes consume models; they do not train them.

The distinction matters because strength in one sense proves nothing about another. Perfect retrieval does not prove task success — an agent can be handed exactly the right evidence and still act on it wrongly, and only evaluation ground truth detects that. The converse also holds: a well-built evaluation suite running over an agent whose retrieval layer feeds it stale or contradictory data will measure, accurately, a system that cannot succeed, and no amount of evaluation effort compensates for the grounding defect. The two disciplines meet in one artifact, the trace. The trace records the grounding evidence the agent received (which records, at what version and freshness, Chapter 8), the process the agent followed, and the outcome it produced, which is what lets a failure be attributed to bad evidence, bad reasoning, or a bad task definition rather than argued about.

The unit of evaluation

Evaluating a probabilistic component also needs a vocabulary the deterministic testing tradition does not supply. The terms below have largely stabilized across the industry (Anthropic, “Demystifying evals for AI agents,” 2026; see Chapter 21) and are used throughout the rest of this chapter.

Task. One test case: inputs, environment setup, and success criteria.
Trial. One stochastic attempt at a task. An agent remains stochastic under settings meant to reduce sampling variation, so a reliability claim about a task rests on repeated trials; a single trial is a smoke test or a diagnostic, not a reliability estimate.
Transcript. The record of what the model and its tools did during one trial: messages, tool calls, observations, intermediate results.
Outcome. The final, externally observable state the trial produced — the database rows, the files, the ticket status — as distinct from the agent’s account of it.
Grader. A procedure — deterministic code, a model call, or a human — that scores one aspect of a transcript or an outcome. A task usually has several.
Evaluation harness. The infrastructure that provisions tasks, isolates trials, captures transcripts and outcomes, runs graders, and aggregates results.
Evaluation suite. A versioned set of related tasks and their graders.

Two of the book’s established terms sit alongside these without being displaced. The trace (developed below) is broader than a transcript: it contains the transcript’s model-and-tool record plus the deterministic shell’s events — bounds, governance decisions, identity, memory access, cost, timing, approvals. An evaluation that reads only the transcript sees what the agent did; one that reads the trace also sees what the architecture did in response. And the agent harness (Chapter 19) is not the evaluation harness: the first runs the agent, the second runs the tests. The distinction matters when interpreting results, because an agent evaluation measures the model and the agent harness jointly — the same model under a different harness scores differently — so a comparison between systems is a comparison between pairs unless the experiment deliberately holds one fixed and varies the other.

The three layers of testing

An agentic system has three layers of testing concerns. Each layer needs its own techniques.

Layer 1: Deterministic substrate

The bounding layer, governance layer, memory gateway, tool adapters, and orchestration code are deterministic software. Standard practice applies: unit tests, contract tests, integration tests, fuzz tests where appropriate. This layer should have the same testing rigor as any production system; the agentic system’s reliability rests on it.

The architectural commitments from Chapters 5–6 make this layer testable: bounds, validators, and policy gates are deterministic and can be asserted directly. Tests should cover:

Bound enforcement (iteration, cost, time, action surface, data access scope, reversibility).
Schema validators (every tool, every output channel).
Policy gates (every rule, with positive and negative cases).
Approval-gate routing (correct actions route, others do not).
Risk-scoring (consistent, monotonic where claimed, no unexpected discontinuities).
Rollback paths (every reversible action class).

Treat these tests as the minimum; coverage gaps here cause incidents. A useful coverage metric is the governance event matrix: a list of every governance event class crossed with a list of every condition under which the event class fires, asserted to be covered by tests. Gaps in the matrix are gaps in the test suite.

Layer 2: Agent behavior within the envelope

The agent itself is the probabilistic component. Tests assert properties, boundedness, safety, policy compliance, output validity, rather than exact outputs. The standard techniques:

Bound-respect tests. Synthetic scenarios designed to provoke runaway behavior (ambiguous goals, contradictory feedback, tool failures); assert that the bounding layer aborts the run. These tests do not assert what the agent did inside the bound; only that the bound held.
Escalation tests. Trigger irreversible-action attempts; assert that the reversibility gate routes to approval rather than executing.
Refusal tests. Out-of-scope or out-of-policy requests; assert refusal or escalation, not silent compliance.
Property-based tests. Generate random valid inputs; assert that outputs satisfy structural properties (schema-valid, within tone bounds, no PII leakage).
Adversarial tests. Prompt-injection attempts, jailbreak attempts, tool-response-injection attempts; assert that governance defenses hold.
Cross-tenant tests. In multi-tenant systems, attempt cross-tenant reads and writes; assert refusal.

These tests are deterministic at the bound boundary even when the agent’s behavior inside the bound is not. That is the architectural payoff of Chapter 5 and Chapter 6 for testing.

A key insight: the agent’s behavior is probabilistic, but the envelope is not. The test suite asserts on the envelope. If the envelope holds, the agent’s variable behavior inside it is acceptable; if the envelope is breached, the agent’s behavior is unacceptable regardless of how reasonable it looks. This framing makes a previously-hard testing problem (testing a probabilistic component) into a tractable one (testing a deterministic envelope around a probabilistic component).

Layer 3: Quality of agent output

The hardest layer. Did the agent produce a good result? Good is domain-specific and probabilistic, but one rule orders everything else in this layer: grade the outcome, not the report of it. An agent that announces “the refund has been issued” has produced a claim in a transcript; whether a refund transaction exists in the payment system is the outcome, and the two diverge exactly when the system is failing in the way that matters most. Where a task has externally observable effects — database state, API state, files written, tests passing, a ticket transitioned, a reservation created — the primary grader queries the environment and verifies the change, deterministically. Asking a model whether the job was finished grades the agent’s ability to sound finished. The credible agent benchmarks take the same position, verifying properties of the final environment rather than the agent’s narration of its work (Anthropic, “Demystifying evals for AI agents,” 2026; Terminal-Bench; see Chapter 21).

An outcome check should usually admit multiple valid solution paths. Asserting that the agent called tool A, then B, then C produces brittle tests that fail on every valid approach the designer did not anticipate, and agents find such approaches regularly. Reserve exact-sequence assertions for the cases where the sequence itself is a policy or safety requirement — and then the assertion is an envelope property that belongs in Layer 2, not a quality judgment.

Outcomes cannot see everything, which is why process checks stay in the suite. The final state does not reveal a destructive intermediate step (the agent that dropped a table to update one row and restored it from backup reaches the same end state as the agent that updated the row), an unauthorized or unrequested action along the way, a policy violation en route, or the cost, latency, and redundant tool calls the trial spent getting there. These are properties of the transcript and the trace, assessed by graders — mostly deterministic, over typed events — that complement the outcome oracle. Interaction quality lives here too: whether the agent asked for clarification when the task was ambiguous, and how it communicated a refusal.

Where no environmental oracle exists — the output is a summary, an analysis, an answer — quality grading falls back on references and judgment: reference comparisons against curated known-good outputs, effective in narrow stable domains and brittle in open-ended ones; and pairwise preference, in which a judge compares two candidate outputs to rank a change without absolute scoring.

The model-based evaluator in those fallbacks has an industry name: LLM-as-a-judge. It is the most common way to test LLM quality at scale, and it is inherently compromised — the judge has the same probabilistic failure modes as the agent it grades (Zheng et al., 2023; see Chapter 21). A judge can be sycophantic, can be prompt-injected by the content it evaluates, and drifts with its own model version. Use it where nothing cheaper exists, treat its verdicts as the softest signal in the suite, and subject it to the calibration discipline developed later in this chapter (below). Deterministic structural compliance (Layers 1 and 2) is always more trustworthy than an LLM-as-a-judge score (Layer 3); when the two disagree, believe the deterministic check.

Quality testing is the layer where the practice is weakest in industry. The book’s position is that the team should commit to a small, stable set of quality signals tied to business outcomes and resist the urge to overfit to model benchmarks. Benchmarks measure benchmarks; the system’s quality is its production outcome.

A neglected dimension of quality testing is stability under input perturbation. The same task, phrased slightly differently, should produce similar-quality outputs. A test suite that includes perturbation — semantically-equivalent but syntactically-different inputs — catches the failure mode where the system is brittle to phrasing. Production-derived perturbations (real user phrasings of the same intent) are more valuable than synthetically-generated ones, because they reflect the distribution the system actually faces.

Calibrating the governance layer

The risk scorer and the policy thresholds it feeds are themselves components that can drift, and calibrating them is a recurring operational task rather than a one-time setup. Chapter 6 states the rule for a mature system: calibrate on incident data, so that actions that turned out to be problematic would have had risk scores above the escalation threshold. For a new system that rule is circular, because there is no incident data to calibrate against. The team that ships the scorer on day one with thresholds set by intuition ships a scorer that is, in effect, uncalibrated, and the first weeks of traffic will either flood the approval queue with false positives or let genuinely risky actions through unflagged. The greenfield case needs its own bootstrap procedure.

Bootstrap without incident data

The procedure begins from conservative defaults and tightens or loosens as evidence arrives, rather than waiting for an incident to supply the evidence. It has three steps.

First, set initial thresholds by action class, not as a single number. A refund, a code commit, and a configuration read are not the same kind of action, and a single risk threshold blurs the cliff between them. Class actions by reversibility and blast radius: read-only operations start below the elevated-logging threshold; reversible mutations start at the elevated threshold; irreversible or external-facing mutations start above the approval threshold. The defaults are deliberately conservative, because the cost of a false positive at startup (an approval that was not strictly needed) is a small delay, while the cost of a false negative is the incident the bootstrap exists to prevent.

Second, seed the calibration with red-team cases. Deliberately craft borderline actions, the kind the team expects the system to see in production: a refund just under a dollar threshold, a diff that touches a sensitive path but is small, a query that is plausibly analytical and plausibly exfiltration. Score each seeded case, and set the class’s threshold at the decile that catches the seeded risky cases while passing the seeded benign ones. The seeded set need not be large; a few dozen cases per action class is enough to find a threshold that is not obviously wrong, and it gives the team a concrete artifact (“the scorer routes these N cases correctly”) to review rather than an intuition to defend.

Third, run a tightening loop over the first N sessions. Sample real sessions in production, score every action, and have a human annotate a rotating fraction (the riskiest decile the scorer flagged, plus a random sample of the rest) as genuinely risky or benign. After each batch of annotations, adjust the thresholds: if the flagged set contains mostly benign actions, the threshold is too low; if the random sample contains risky actions the scorer let through, it is too high. The loop converges when the scorer’s flagged set is mostly genuinely risky and the unflagged set is mostly genuinely benign, which is the operational definition of “well-calibrated enough to operate.”

When to loosen

A scorer is well-calibrated enough to loosen when two conditions hold over a sustained window: the approval queue is dominated by actions a human reviewer consistently approves (the threshold is catching work it should not), and the random-sample annotation shows no risky actions passing unflagged. Loosening is then a measured change, replayed against the golden trace set (below) to confirm it does not let a known-incident shape through. Tightening is the same procedure in reverse, triggered by an incident, a near-miss, or a drift in the deny rate (below). Both are controlled changes to a versioned component, not prompt tweaks.

Testing the probabilistic governance components

A subtlety the bootstrap surfaces is that the risk scorer itself is often a model, and a model-based governance component is not covered by the Layer 1/2 testing that asserts on the deterministic envelope. The scorer’s routing is deterministic; the rule that acts on its score is fixed, but the quality of the score is a property of the scorer, and it drifts with the scorer’s own model version. The testing discipline for a model-based evaluator is therefore distinct from testing the envelope: track scorer calibration as a metric over time (the agreement between the scorer’s flagging and human annotation on the rotating sample), assert monotonicity where the scorer claims it (a larger refund scores higher than a smaller one), and test for drift by re-running the seeded red-team set against every scorer model upgrade before promotion. Where multiple model-based evaluators run in concert, the agreement between them is itself the signal, developed below under quality testing.

Trace discipline

The trace is the system of record. It is not logging in the operational sense; it is the artifact that makes every reasoning step, tool call, memory access, policy decision, and approval event observable, attributable, and replayable.

Trace discipline rests on four commitments.

1. Structured traces

Every event in the trace is a typed record with explicit fields: timestamp, agent identity, session identity, event class, parent event (where applicable), payload (input, output, metadata). Unstructured logs are not traces.

In 2026 there is no reason to invent a bespoke schema for this. The industry standard is OpenTelemetry (OTel; see Chapter 21), and its Semantic Conventions for GenAI define spans and attributes for model and agent telemetry, gen_ai.system, gen_ai.request.model, gen_ai.usage.input_tokens, and the like. (These GenAI conventions are still stabilizing as of 2026; treat the specific attribute names as illustrative and pin to a convention version.) Emit agentic traces as OTel spans, mapping the agentic event classes below onto span names and attributes. Building on OTel means the trace plugs into existing collectors, backends, and dashboards rather than a one-off pipeline, and it makes the agentic system observable with the same tooling as the rest of the stack.

Common event classes:

agent.thought, reasoning content (where the model emits it).
agent.action_proposed, the agent has proposed an action; payload includes the action.
bounds.check_passed / bounds.check_failed, the bounding layer’s decisions.
governance.validator, validator outcomes (with a result of pass or fail).
governance.policy_gate, policy decisions (with a decision of allow or deny).
governance.risk_score, the risk score computed for a proposed action.
governance.approval.{requested,granted,denied,timeout}, approval events.
tool.invocation, a tool call (input, output, latency, cost).
memory.{read,write}, memory access through the gateway.
skill.{loaded,activated}, skills layer events.
cost.tick, cost accumulation events.
error, exceptional conditions.

These names are illustrative; the principle is that every event is typed and the type is queryable.

A short trace excerpt makes the format concrete. The session below is a coding-assistant agent making a small change; the trace captures the bounding check, the governance pipeline, and the tool invocation:

{"ts": "2026-06-17T09:15:42.103Z", "session": "s_8f23", "event": "agent.action_proposed",
 "tool": "write_file", "args": {"path": "src/api/signup.ts", "diff_size_lines": 12}}
{"ts": "2026-06-17T09:15:42.105Z", "session": "s_8f23", "event": "bounds.check_passed",
 "tool": "write_file", "checks": ["iteration", "cost", "time", "action_surface",
                                   "data_access_scope", "reversibility"]}
{"ts": "2026-06-17T09:15:42.106Z", "session": "s_8f23", "event": "governance.validator",
 "tool": "write_file", "result": "pass"}
{"ts": "2026-06-17T09:15:42.109Z", "session": "s_8f23", "event": "governance.policy_gate",
 "gate": "no_secrets_in_diff", "decision": "allow"}
{"ts": "2026-06-17T09:15:42.110Z", "session": "s_8f23", "event": "governance.policy_gate",
 "gate": "protected_paths", "decision": "allow"}
{"ts": "2026-06-17T09:15:42.111Z", "session": "s_8f23", "event": "governance.risk_score",
 "action": "write_file", "score": 10}
{"ts": "2026-06-17T09:15:42.243Z", "session": "s_8f23", "event": "tool.invocation",
 "tool": "write_file", "latency_ms": 132, "cost_usd": 0.0008, "result": "ok"}
{"ts": "2026-06-17T09:15:42.244Z", "session": "s_8f23", "event": "cost.tick",
 "spent_usd": 0.34, "remaining_usd": 1.66}

Two architectural observations from the format:

Each event is independently meaningful. A query of the trace store filtered to event = "governance.policy_gate" AND decision = "deny" returns every blocked action across the system, sortable by gate, useful for both incident response and metric dashboarding.
The trace records what the agent proposed and what the architecture did in response. A failed bound check, a denied policy, an escalated risk score — each is a separate event. The trace is the record of the interaction between agent and architecture, not only the agent’s actions.

The schema for each event class is itself versioned. Trace consumers, the dashboards, the replay infrastructure, the alerting, depend on the schema, and trace schema changes go through the same review as code changes. Backward-incompatible trace schema changes are rare and explicit.

2. Per-session correlation

All events for a single agentic session share a session identifier. Events from related sessions (orchestrator delegating to a worker, agent invoking a sub-agent through handoff) are linked via parent-child relationships. The trace forms a tree (or, in multi-agent shapes, a DAG) per logical task.

This correlation makes incident response feasible. Without it, traces are an unsorted pile.

Correlation extends across system boundaries. When an agentic system interacts with an upstream or downstream service, correlation identifiers should propagate so that the full request lifecycle can be reconstructed. A customer-support agent’s session correlates to the CRM ticket; an operations agent’s session correlates to the incident response record. The trace is most valuable when it connects to the rest of the engineering record.

3. Full trace on flagged sessions

In high-volume systems, full tracing every session is impractical. The right architectural pattern is:

Sample at low rate for routine sessions. Aggregate metrics suffice.
Full trace on flagged sessions. Any session that triggers an alert (cost exceeds threshold, governance gate denied, error, approval declined, user complaint) is traced in full and retained for incident response.
Full trace on a small audit sample. A continuously rotating fraction of sessions is fully traced regardless, for drift detection.

Full tracing consumes storage and increases handling costs, but missing traces makes incident reconstruction far more expensive. Design the trace store around explicit cost-versus-retention tradeoffs.

Retention by event class is a useful refinement. Governance events (validator pass or fail, policy decisions, approvals) often have compliance retention requirements measured in years. Routine operational events (cost ticks, tool invocations) may have retention measured in weeks. A trace store that supports per-class retention reduces storage cost while preserving the data that matters for audit.

4. Replay capability

The trace must be replayable. Replay in this book means: given a trace, the agentic system’s deterministic substrate can be re-executed against it (or against a simulation of it) to:

Reconstruct what happened at every step.
Test hypotheses (“if the policy gate had been stricter, would this incident have been prevented?”).
Validate behavior against a new model version (“does the new model produce a trace within the envelope of the old one?”).
Train automated tests from production traces (“this incident becomes a regression test”).

Replay is not free. It requires that the trace captures enough information — inputs to model calls, tool responses, retrieval results — to re-run the deterministic surrounding code. Build replay as a first-class capability and exercise it routinely; do not discover at incident time that the trace is insufficient to reconstruct the failure.

Replay also requires mocking. A trace shows the agent calling DELETE /user/123; the replay must not actually hit production again. The trace store therefore doubles as an API mock server during replay: when the deterministic substrate re-issues a tool call, the mock returns the exact observation captured in the original trace rather than executing the call for real. This is what makes replay safe to run continuously and in CI: every external effect is served from the recorded trace, and the only thing executed live is the deterministic glue (and, for counterfactual replay, the single component being varied).

One subtlety follows from this, because adversarial and counterfactual replay (below) deliberately push the agent off the recorded path: the mock will eventually be asked for a call the original trace never captured. On such a miss it must never fall through to the live system. The mock returns a typed cache-miss observation, and the assertion moves one step earlier, to the bounds check or governance gate that precedes the unrecorded call, or the diverged call is routed to a sandbox that can answer it safely. Replay’s safety guarantee is precisely that a divergence reaches a cache miss or a sandbox, never production.

Replay loop

Replay also has security implications. A trace that captures sensitive content captures it for as long as the trace is retained. The trace store has access-control requirements at least as strong as the original data sources, with explicit purpose-of-access logging when traces are read by humans. This is part of the architectural design, not an afterthought.

Replay-driven test patterns

With trace discipline in place, several test patterns become possible.

Regression testing from traces

A historical trace plus a new system version: re-run the deterministic substrate against the trace; observe whether the agent’s behavior stays within the envelope. Within the envelope means: bounds still hold, governance still allows what it allowed, outputs still satisfy the same validators, end-task outcome (if known) is comparable.

This is the most valuable testing pattern enabled by the architecture. It catches drift between model versions, between policy versions, between memory states. It produces a continuous validation signal at the cost of trace storage and replay infrastructure.

A practical implementation: maintain a golden trace set, a curated collection of traces that represent the system’s behavioral envelope. The set includes routine successful sessions, near-failures the architecture caught, escalations, refusals, and a few representative incidents. New releases must reproduce the envelope-respecting behavior on every trace in the set. Add to the set when incidents reveal under-covered scenarios; retire traces when they no longer represent realistic input. A golden trace set records trials, though, not tasks: a trace documents one attempt, while a task defines the problem and its oracle, and several traces may belong to one task. The distinction, and the construction discipline that turns traces into tasks, is developed in Constructing evaluation tasks below.

Adversarial replay

Take a trace; modify the inputs (a different prompt, an injected tool response, a tampered retrieval result); re-run. Assert that the system’s defenses hold under the modified inputs. This is fuzz testing applied to agentic systems, with traces as the seed.

Adversarial replay is especially useful for security testing. Take a trace of a normal session; inject prompt-injection content into a retrieved document; re-run. Assert that the agent’s actions are still within the governance envelope. The test is reproducible (it is a trace replay), deterministic (the substrate is deterministic), and asserts on a real attack surface (the retrieval content).

Counterfactual replay

Take an incident trace; modify a single architectural parameter (a tighter bound, a stricter validator, a different policy); re-run. Assert whether the incident would have been prevented. This is the right way to make a case for policy or bound changes after an incident: data, not opinion.

Counterfactual replay also supports proactive architecture work. Before adopting a new pattern or tightening a bound, run the change against the trace archive; see what would have happened differently. If the change blocks five incidents and twenty legitimate operations, the team has concrete data about whether the change is worth making.

Trace-driven test generation

The trace archive is itself a source of tests. Routine processes can derive tests from traces:

Each governance deny in production becomes a unit test asserting that the deny remains in force.
Each approval-routed action becomes a test asserting that the routing remains correct.
Each refusal becomes a test asserting that the refusal remains.
Each successful end-to-end task becomes a regression test asserting that the task completes.

The test suite grows organically from production. The team does not have to imagine every test case at design time; the suite reflects the system’s actual operating envelope.

The loop from production to suite runs in five steps, and the adjudication step in its middle is what prevents misattributed failures from entering the suite:

Monitoring, a user report, or manual trace review identifies a candidate failure.
A domain expert adjudicates it: was the agent wrong, was the grounding data wrong (Chapter 8), was the task specification ambiguous, or was the grader wrong? Each attribution routes to a different fix, and only the first is an agent failure.
The adjudicated case becomes, or updates, a versioned evaluation task, constructed with the discipline of Constructing evaluation tasks below.
The task enters the capability suite or the regression suite according to its purpose (below).
Later releases replay the task in isolated environments.

Manual transcript review keeps its place in step 1 even after the automation matures, because automated graders detect only the properties they were designed to measure; the failures nobody anticipated surface first to a human reading transcripts.

The loop accumulates. Each adjudicated failure becomes a versioned task, so every later release is tested against every incident the system has already had, and the curated corpus of tasks and traces records how the system fails and what correct behavior looks like — knowledge the team retains even when the model belongs to a vendor. The trace infrastructure is what feeds this accumulation; the recurrences it prevents and the investigations it shortens are what justify building it.

Constructing evaluation tasks

The replay patterns grow the suite from what production already did; construction is the discipline that turns a captured trace, an adjudicated failure, or a requirement into a task the team can trust. An untrustworthy task is worse than a missing one, because it misallocates engineering effort: an ambiguous specification reads as agent unreliability, and a mis-wired grader reads as a capability gap.

The minimum record for a task is short enough to enforce:

Versioned instructions and fixtures. The task’s instructions and its environment setup, under version control like code.
Expected environmental changes or invariants. What must be true of the world afterward — the rows written, the state transitioned — or what must not have changed.
Expected outputs where exact values are meaningful. A computed refund amount is checkable to the cent; prose is not. Assert exact values only where the domain defines them.
Deterministic assertions wherever the check can be code.
A known-good solution where the domain permits one. A reference solution proves the task is solvable — for the exact fixture and environment it ran against — and exercises the graders. A known-good solution that fails signals a defect somewhere in the task package: the grader, the fixture, the environment, or the solution itself, to be located before any agent is blamed. For open-ended tasks, do not pretend one canonical trajectory exists; require a solvability witness — a worked resolution, a reference outcome — rather than a prescribed path.
A written rubric for judgment-based dimensions (developed in the next section).
Provenance. The production failure, requirement, policy, or expert decision the task came from.
Version history for instructions, fixtures, graders, and rubric, because a pass-rate trend is uninterpretable across unrecorded task changes.

Four review rules make ambiguity and grader defects visible before they contaminate results. First, two qualified reviewers should reach the same pass/fail judgment from the specification and rubric alone; persistent disagreement is evidence that the task or rubric is underspecified, and ambiguity in the specification becomes noise in the metric. Second, a near-zero pass rate from a capable agent is a reason to inspect the task’s solvability and the grader’s correctness before declaring a capability failure — broken tasks and mis-wired graders produce exactly that signature — though it is a diagnostic prompt, not proof the task is broken.

Third, where the capability has a decision boundary — search or answer from memory, escalate or resolve, act or abstain — the task set needs positive, negative, and abstention cases, or the suite optimizes the system toward one side of the boundary. Fourth, source tasks from real product failures, support cases, and manual release checks rather than synthetic coverage for its own sake: a modest set drawn from real failures is a better start than broad synthetic coverage, because each real failure is a documented way the system actually fails.

Internal suites need not reproduce the full benchmark review process, but the gates the public benchmarks apply are worth borrowing in lighter form. Terminal-Bench accepts a task only when its instructions describe every acceptable end state (specificity), its reference solution passes the tests (solvability), and the task cannot be passed by a shortcut that would not exist in a real deployment (integrity) — the last verified in part by an adversarial agent hunting for exploits (see Chapter 21). Such gates reject a large fraction of contributed tasks, which indicates how many defective tasks a naively assembled golden set contains.

Grader design and calibration

Graders come in three classes, and a mature suite uses all three.

Deterministic graders — environmental state checks, schema and exact-property assertions, test execution, static analysis, policy checks, resource caps — are fast, reproducible, and debuggable. They carry the outcome verification and process checks described under Layer 3, and they are preferred wherever the property can be expressed as code.

Model-based graders cover the open-ended dimensions deterministic code cannot evaluate: coherence, helpfulness, appropriateness of tone, quality of an explanation.

Human review writes the rubrics, calibrates the model-based graders, adjudicates disputed cases, and judges the genuinely subjective. Between the last two classes there is no fixed ranking. Humans are too slow and expensive to be the routine grader at scale, but scale is not the only criterion: a high-stakes release gate, an audit sample, or a dimension whose rubric is still immature is graded by humans even where a model grader could be built, because the calibration that would justify the model grader does not yet exist. The choice is set by risk, subjectivity, volume, and calibration maturity.

The model-based class is the one that needs design rules, because its failure mode is an uncalibrated score with no rubric or human labels behind it:

Decompose the judgment. A broad quality verdict hides more than it reveals. Name the rubric dimensions — task correctness, tool-use quality, error recovery, communication — and score each independently where a combined score would let strength in one dimension mask a serious failure in another.
Give the grader a way to abstain. A grader forced to choose between pass and fail on evidence it does not have will still answer, converting missing evidence into an unsupported verdict. An explicit “insufficient evidence” result makes the case visible instead.
Permit partial credit where it means something. An agent that correctly identified the problem and verified the customer before botching the refund is operationally different from one that failed at the first step. Score the difference — while preserving a separate pass threshold, so partial credit informs analysis without inflating the pass rate.
Calibrate against human labels, dimension by dimension. Run the grader over human-labeled examples, measure agreement per dimension, and read the disagreements rather than averaging them away; the disagreements are where the rubric, the grader, or the labels are wrong.
Calibrate the humans first. Human labels are not automatically ground truth. Establish inter-annotator agreement before treating the labels as the standard: a rubric two trained reviewers cannot apply consistently will not be applied consistently by a model either.
Recalibrate on change. A calibration holds for a particular judge model, rubric, task distribution, and product behavior. When any of those changes, the calibration is stale.

A panel of model judges with an agreement rule deserves a narrower role than it is often given. Agreement across a panel estimates the judges’ variance, not their correctness: judges built on similar models share biases and can agree confidently in the same wrong direction. A panel is useful after calibration, chiefly as a triage signal — where calibrated judges disagree, the case is borderline and belongs with a human — and never as a substitute for calibration against human labels.

Trial reliability

An agent remains stochastic under every setting intended to tame it; a low sampling temperature narrows the output distribution without collapsing it. A task is therefore measured over repeated trials, and the rule for aggregating those trials follows from the architecture around the agent. Two aggregations dominate (the second introduced by τ-bench; see Chapter 21):

pass@k — the task succeeded in at least one of k trials. Rises as k grows.
pass^k — the task succeeded in all k trials. Falls as k grows.

The same per-trial success rate tells opposite stories under the two rules. A task the agent completes 80% of the time succeeds at least once in three trials about 99% of the time, and in all three barely half the time (0.8³ ≈ 0.51). The arithmetic is an idealized illustration — it assumes independent trials with a stable success rate — and benchmark implementations report finite-sample estimators computed from observed successes and averaged across tasks, so a published pass^k is not reproduced by exponentiating an aggregate pass rate.

Each aggregation corresponds to a retry and supervision model. pass@k describes a system in which retries are permitted and a human or a deterministic selector chooses among results — a code-generation task whose output must pass the test suite can afford several attempts, because the selector discards the failures. pass^k describes a system in which every unsupervised execution must be dependable — a customer-facing action that commits without review has no selector to rescue the failed trials.

Report the per-trial success rate alongside the aggregate, with enough distributional information to reconstruct the operational risk, and never compare two systems without stating the number of trials, the aggregation rule, and the retry policy. A pass@10 figure is not comparable to a pass^3 figure, however similar the headline percentages look.

Trial isolation and evaluation integrity

Every trial runs in a fresh or deterministically reset environment. State that leaks between trials — leftover files, warm caches, credentials, database rows, prior sessions’ histories, exhausted resources — produces correlated failures that read as agent instability, and can just as easily inflate scores by handing a later trial the work products of an earlier one. The fixture and environment versions are recorded in the trace, so every result is attributable to the exact world it ran in — the same lineage discipline the ingestion pipeline applies to data (Chapter 8).

Graders are part of the deterministic evaluation substrate and need their own tests. A grader is code, with the usual defects plus one specific to its role: it can be satisfiable by something other than the intended accomplishment, and a system iterated against the suite will eventually find that shortcut. The checks are direct. Run the known-good solution and confirm the grader passes it; run known-bad and near-miss cases and confirm the grader fails them; and for high-value suites, run an adversarial attempt — an agent or a person instructed to pass the grader without completing the task — before trusting the results. This is the adversarial posture of Chapter 6 applied to the test infrastructure itself. The evaluation harness controls the results the team acts on, and it requires the same Layer 1 testing as the substrate it measures.

Quality signals that survive model changes

A common failure mode in agentic testing is to rely on quality signals that are sensitive to model version. The model upgrades; the test suite fails not because the system degraded but because the model phrases things differently. The team loses trust in the suite.

Quality signals that survive model changes:

Outcome-based metrics. Test pass rate, ticket-resolution rate, end-to-end conversion. Independent of how the agent phrases its work.
Structural compliance. Schema-validity, validator-pass rate, policy-pass rate. Architectural, not stylistic.
Trace-property assertions. “Every action passed through governance,” “no action exceeded the reversibility envelope without approval.” Independent of model.
Side-effect assertions. “No cross-tenant data appears in the output,” “no PII appears in the trace’s memory writes.” Pattern-matched, architectural.

Quality signals that do not survive model changes well:

Exact-output reference comparisons. Fragile to phrasing.
Embedding-similarity to reference. Drifts with embedding model upgrades.
Single-judge LLM evaluations. Sensitive to the judge model’s own behavior.

Use the reliable signals as the testing backbone; use the fragile ones cautiously, with the understanding that they fail often for non-substantive reasons.

For pairwise preference and reference-comparison testing, a panel of judges with an agreement rule — three judges, two-of-three to accept a verdict — reduces single-judge noise, and disagreement within the panel flags a borderline comparison for a human review queue at the same priority as an approval-bound action. The panel’s limits — agreement is not correctness — and the calibration that establishes whether the judges are right are covered in Grader design and calibration above.

Evaluation as continuous process

Evaluation is not a phase; it is continuous behavior of the production system. Three commitments:

In-production evaluation. A continuously running evaluator (an evaluator-optimizer pattern, Chapter 4) samples real sessions, scores outputs, and surfaces failures for review. The evaluator is itself bounded and observable.
Behavioral metrics over time. Per-week dashboards for bound-trigger rate, governance-deny rate, approval-request rate, cost per session, latency per session, refusal rate. Drift in any of these is a signal.
Periodic adversarial review. Scheduled adversarial testing, prompt injection, jailbreaks, tool-response injection, policy-evasion attempts, against the production system. Not as compliance theater; as actual probing.

These are operational practices, not testing tasks. They sit at the intersection of testing and observability and are owned by the team that operates the system.

The choice of evaluator in continuous evaluation matters. An evaluator that is itself a model call has its own failure modes; an evaluator that is a deterministic rule check is more reliable but covers less. The mature approach is a mix: deterministic checks on the structural properties (schema, policy, governance), model-based checks on the quality dimensions (relevance, coherence, helpfulness). Each is monitored separately; drift in either is investigated.

Production traffic carries no ground-truth labels, and that constraint splits the grounding measurements of Chapter 7 across two paths. Retrieval recall, and precision measured against labeled relevance judgments, is offline work: it needs a versioned query set annotating which evidence should have been retrieved, so it runs as an evaluation suite, never as live monitoring. What runs online is the reference-free remainder: structural compliance, policy and cost anomalies, claim support against the retrieved context, answer relevance, and the freshness and provenance completeness of what the agent consumed (Chapter 8).

Claim support is the notable member because it needs no labeled answer: the evidence the agent was given is in the trace, and the check asks whether the output stays within it. Reference-free means free of a reference answer, not free of judgment or of ground truth in the wider senses. The check is usually a model-based grader, subject to the calibration discipline above, and it measures support relative to the evidence supplied, which it cannot certify was itself complete, current, or authoritative — that is the grounding concern of Chapters 7 and 8.

The online signals and the offline suites’ ground-truth results are monitored separately (Chapter 18); conflating them on one dashboard hides drift in either.

Capability suites and regression suites

The suites the evaluation program runs divide by purpose, and the division carries lifecycle rules. A capability suite measures what the system cannot yet do reliably. Its tasks have headroom, its pass rate is expected to sit well below full compliance, and its job is to discriminate among candidates — a new model version, a new prompt, a reworked tool surface. A suite saturated on every dimension it measures has stopped discriminating capability and needs harder tasks; easy tasks keep their place only when they measure something else — consistency across trials, cost, latency, process safety — or anchor the suite against broad regressions. A regression suite holds the behaviors the system is already expected to preserve — every adjudicated incident, every governance property, every task the system once failed and now passes — and it is expected to stay close to full compliance; a dip blocks the release.

The rule connecting them is graduation. When a capability task saturates — current systems pass it consistently — it moves into the regression suite. It is not deleted for being easy: deleting a saturated task because current models find it trivial removes exactly the coverage that would catch the regression when a later change breaks it. The release integration — which changes gate on which suites, and how suite and grader versions are recorded with release results — belongs with the model-version canary discipline of Chapter 18.

Cost regression testing

Cost is a first-class quality dimension and the most likely to drift silently. The architectural commitment is:

Per-session cost recorded in the trace (Chapter 5; cost ledger).
Distribution of cost monitored, not just the average. The long tail is where incidents hide.
Cost regression alerts on shifts in the distribution that exceed historical variance.

A test that asserts “average cost remains under X” misses long-tail incidents. A test that asserts “the 99th percentile cost remains under Y” catches them. Operate on percentiles.

A specific cross-version failure deserves a named test: token inflation. A new model can be chattier than its predecessor, emitting 20% more tokens for the identical prompt and task, which is a silent cost regression even when quality and behavior are unchanged. Track token variance per task across model upgrades, not just dollar cost in aggregate, because a per-token price drop can mask a token-count rise (or the reverse) until the bill arrives. A model migration is not cost-neutral until the per-task token distribution has been measured on the new model.

Cost regression testing should also be tied to the same trace archive as quality regression testing. A new release’s expected cost can be projected by replaying the archive against the release; significant differences (whole percentiles shifting upward) are caught before release. Without this projection, cost regressions appear after deployment, often invisibly.

Drift detection

Drift is the agentic-system equivalent of slow leaks: behavior changes incrementally; no individual change is alarming; the cumulative effect is significant. The patterns:

Quality drift, the agent’s outputs degrade subtly over time. Caught by continuous evaluation against the golden trace set.
Cost drift, per-session cost rises without code changes. Caught by percentile monitoring.
Latency drift, responses get slower. Caught by latency percentile monitoring.
Governance drift, deny rates change, often because input distributions shift. Caught by governance-event dashboards.
Memory drift, the memory architecture starts surfacing different content than it used to. Caught by the read-path measurements of Chapter 7 — retrieval recall, precision, and ranking quality — on a stable, versioned query set.
Reasoning trajectory drift, progress scores fall, thrash rates rise, or reasoning-token ratios shift without a matching change in output quality. Caught by per-iteration metrics (below).

The architectural commitment is to make drift observable. Drift that is observed is investigable; drift that is hidden becomes incident.

Reasoning trajectory observability

Layer 3 testing asks whether the final output is good. Layers 1 and 2 ask whether the envelope held. Neither asks whether the agent made progress on the way there — whether each iteration moved the task forward or merely consumed budget retracing the same ground. That reasoning trajectory is observable from the trace alone, without grading the model’s internal chain-of-thought, and it is the earliest warning for several failure modes Chapter 11 names that output-quality metrics detect only after the damage is done.

Per-iteration progress metrics treat each harness turn as a step whose value can be scored from typed trace events, not from prose the model emitted about its progress. Useful signals include: new information introduced (a tool returned data not present in prior turns); state delta (a file changed, a record written, a test status flipped); plan stability (the declared next action aligns with the prior turn’s plan rather than restarting); and monotonic convergence toward a stated goal (remaining subtasks decrease). A session whose iterations show no state delta for several turns while cost and iteration counters climb is stalled even if each individual output looks plausible. The defense Chapter 11 names for infinite and thrashing loops — trace alerting on per-iteration progress — reduces to concrete thresholds: alert when the progress score stays below a floor for N consecutive turns, or when cost-per-progress-unit exceeds a fleet baseline.

Thrash signatures (Glossary) are recognizable shapes in the trace that precede bound aborts. Repeated near-identical tool calls with cosmetically different arguments (the same search query rephrased, the same file read several times without intervening writes) indicate the agent is re-deriving rather than building on observations. Oscillating plans — propose migration A, revert to B, return to A — show up as alternating tool sequences without cumulative effect. A high thought-to-action ratio (many model turns per tool call with no forward state change) often correlates with false termination: the model declares done without having executed the tools that would make it true (Chapter 11). These signatures are cheap to compute from the trace structurally; they do not require reading reasoning text.

Reasoning-token distribution adds a cost-shaped quality signal. Reasoning models expose, directly or via billing metadata, tokens spent on internal deliberation as distinct from output and tool traffic. Track the ratio of reasoning tokens to tool effects per task class: a refactor session that spends ten thousand reasoning tokens to produce a fifty-line diff may be acceptable once and alarming as a fleet median. Sudden shifts in that ratio after a model upgrade — more reasoning, same outcomes — are a form of behavioral drift distinct from output-quality drift, and they belong on the same dashboards Chapter 18 already use for cost and latency percentiles. Pair reasoning-token percentiles with progress metrics: rising reasoning cost with flat progress is thrashing before the iteration bound fires.

The architectural commitment is that trajectory observability stays outside the model. The harness records what happened; deterministic code scores progress and fires alerts; bounds still abort runaway sessions. This is Layer 3 observability — outcome and trajectory — not a fourth layer, and it complements rather than replaces golden-trace replay and LLM-as-a-judge evaluation.

The architectural payoff for testing

Each architectural commitment from earlier chapters pays a testing dividend:

Bounded autonomy (Chapter 5), bound-respect tests are tractable.
Governance (Chapter 6), validators, gates, and rollback are testable in isolation.
Memory (Chapter 7), scoping, lifecycle, and compaction are deterministically assertable.
Skills (Chapter 10), skill admission and activation become testable surfaces.
Trace discipline (this chapter), replay testing, counterfactual analysis, regression detection.

A team that has not made these commitments cannot test its agentic system meaningfully. The output of testing such a system is a vague sense that “it works most of the time.” That is not enough for production. The architectural commitments are what turn it into a testable property.

Building a test program

For a team starting on agentic-system testing, the recommended order of investment is:

Trace discipline. Without traces, nothing else is possible. Build structured trace capture, correlation, and storage first.
Bound and governance tests. Layer 1 testing. Standard practice; deterministic; the substrate everything else rests on.
Property-based and adversarial tests. Layer 2 testing. Begin with the highest-risk scenarios; expand as the system stabilizes.
Replay infrastructure. Once traces exist, build the capability to replay them against the substrate. This enables the most powerful test patterns.
An evaluation task corpus. Convert real failures and manual release checks into versioned tasks with outcome oracles (above), run in isolated environments over repeated trials. Add model-based graders only for the dimensions code cannot check, and calibrate them against human labels before trusting them.
Continuous evaluation. Wire up sampling, scoring, and dashboarding. Run continuously; split capability from regression suites as tasks saturate.
Drift monitoring. With the substrate above in place, monitoring becomes a question of which metrics to track and what thresholds to alert on.
Reasoning trajectory metrics. Progress scores, thrash detection, and reasoning-token ratios on production traces (above).

Skipping steps is tempting and unwise. A team that builds continuous evaluation without trace discipline cannot investigate the failures the evaluation finds. A team that builds replay without governance tests cannot trust the replay’s results. The order is sequential because each layer rests on the previous.

Summary

Testing an agentic system requires three layers: deterministic substrate testing (standard practice), agent-behavior-within-the-envelope testing (property-based assertions over bounds and governance), and quality testing, whose primary oracle is the outcome — the verified state of the environment, not the agent’s report of it. Of the three senses of “ground truth,” this chapter’s subject is the evaluation sense: versioned tasks with provenance, graders that are deterministic where possible and calibrated against human labels where not, and repeated trials in isolated environments, aggregated to match the production retry and supervision model. Trace discipline is the substrate that makes the higher layers possible: structured, correlated, replayable traces. Reasoning trajectory observability — progress per iteration, thrash signatures, reasoning-token ratios — connects envelope compliance to output quality. Evaluation is continuous, not periodic, and production failures return to the suite as versioned tasks through expert adjudication.

The architecture from Chapters 5–10 is what makes all of this practical. Chapter 13 opens the production-facing chapters with the interface through which humans supervise these systems, the first of the chapters that lead through enterprise integration and model routing to the system vignettes of Chapter 16 and the operational discipline of Chapter 18.