Skip to content

AI Agent Evaluation

AI Agent Evaluation book cover

How Do You Know Your Agent Actually Got Better?

Written for engineers pushing agents toward production, the ones asked "can we ship?" with no evidence in hand. No eval set, no dashboard, no list of failures, nothing. This book starts from a two-hour Pocket Eval and, across 16 chapters, takes you all the way to release gates and eval culture. Each chapter hits one real wall and leaves you one template (44 in all, compiled in Appendix A).

The whole book has one lab agent (Mini, a few hundred lines of pure-stdlib Python) and one task world (a fictional e-commerce company, Shore & Summit). Mini ships with its capabilities sealed behind flags, and every chapter's Lab keeps the same discipline: write the eval first → then unlock the capability → watch what the eval catches.

A 30-second start (no API key)

git clone https://github.com/hallieren/ai-agent-evaluation.git
cd ai-agent-evaluation/repo
python world/world.py                                        # sandbox self-check (resettable)
python viewer/trace_viewer.py traces/examples/t-0007.jsonl   # read one real trace

Running the full eval (python -m harness.runner --cases cases/seed-20) requires a real model: three environment variables, any OpenAI-compatible endpoint; see repo/README.md. Zero third-party dependencies, Python ≥ 3.10.

Rather hand the setup to an agent? Paste this to Claude Code, Codex, or any coding agent to do the one-time clone and configure:

Clone https://github.com/hallieren/ai-agent-evaluation and set it up so I can run the labs.
Python >= 3.10, zero third-party deps. Read repo/README.md, then set the three model env vars
(MODEL_BASE_URL / MODEL_NAME / MODEL_API_KEY). Verify from repo/ with the no-key checks:
python world/world.py and python viewer/trace_viewer.py traces/examples/t-0007.jsonl. Ask me
only for the API endpoint and key; handle everything else yourself, and stop and show me the
output if any command errors.

Each chapter's Lab carries its own "let an agent run it" prompt for that chapter's steps. Agents can also read the whole book in one file: llms.txt is the index and llms-full.txt is the full text. For offline reading there is an EPUB.

Every chapter follows the same skeleton: The Wall (your predicament) → The Method → The Evidence (field notes from Shore & Summit) → The Decision (the call you have to make) → Anti-Self-Deception → Your Loot (the template you take with you). Unfamiliar terms live in the glossary (Appendix E).

The roadmap

flowchart LR
    subgraph P1["Part I Quick Wins"]
        direction TB
        c1["1 The Two-Hour Pocket Eval"] --> c2["2 Endpoints, Paths, and Cost"]
    end
    subgraph P2["Part II From Reading Traces to Trustworthy Numbers"]
        direction TB
        c3["3 Error Analysis"] --> c4["4 Building Eval Sets"] --> c5["5 The Judgment Ladder"] --> c6["6 Variance, Sampling, and Significance"] --> c7["7 Harness and Sandbox"]
    end
    subgraph P3["Part III Agent-Specific Battlegrounds"]
        direction TB
        c8["8 Dangerous Tools"] --> c9["9 Planning and Cost"] --> c10["10 Memory"] --> c11["11 Subagents"] --> c12["12 Attacks"]
    end
    subgraph P4["Part IV Shipping and Sustaining"]
        direction TB
        c13["13 Online Eval"] --> c14["14 Regression and Gates"] --> c15["15 The Improvement Loop"] --> c16["16 Eval Culture"]
    end
    P1 --> P2 --> P3 --> P4

Three ways to read

Reading Chapter 1 straight through to Chapter 16 is only one of three paths. Pick by situation.

  • A. Shipping next week, 3 hours a week: ch1 → ch3 (minimal pass) → ch5 → ch8 → ch13, the rest as needed.
  • B. Building an eval practice from zero: read straight through.
  • C. Platform / infrastructure engineer: ch2 → ch6 → ch7 → ch13 → ch14 + Appendix B.

Chapter overview

# The wall you hit Your Loot Chapter Lab
1 The demo charms everyone; you hold no evidence Pocket Eval Template ch01 lab
2 ★ One "right/wrong" dimension stopped being enough Attribute Map + Severity Tiers + Action Boundary Table ch02 lab
3 ★ Intuition can't carry it anymore Trace Review Form + Failure Mode Atlas ch03 lab
4 Your cases don't represent reality Golden Task Design + Coverage Matrix ch04 lab
5 ★ Human labeling can't keep up Judgment Ladder Decision Tree + Judge Validation Report ch05 lab
6 Run it twice, get two different results Statistics Cheat Sheet ch06 lab
7 You don't dare test in the real environment Harness Architecture Spec + Synthetic User Persona Library ch07 lab
8 ★ The reply is impeccable; the danger hides in a mid-trace parameter Action Permission Matrix + Side-Effect Audit Table ch08 lab
9 The endpoint is still right; the bill has already doubled Plan Quality Rubric + Cost/Latency Report Template ch09 lab
10 It digs up last month's history to apologize to a customer Memory Eval Matrix + Long-Task Attribution Protocol ch10 lab
11 ★ End-to-end failed; both agents say it wasn't them Multi-Agent Attribution Decision Tree + Handoff Checklist ch11 lab
12 ★ A piece of external content says "ignore previous instructions," and it does Red Team Protocol + Shutdown Red-Line Checklist ch12 lab
13 ★ 91% offline, wrecked in week two of production Evidence Ladder + Monitoring Signal Spec ch13 lab
14 You fixed scenario A; scenario B quietly regressed Release Gate + Change-Tier Matrix + Stop Rule ch14 lab
15 You have all the data and still don't know what to change Failure Mining Protocol + Bottleneck-to-Lever Mapping ch15 lab
16 Nobody maintains the evals; postmortems turn into blame games Incident Postmortem Template + Quality Ownership RACI ch16 lab

★ = heavyweight chapter. Appendices: A Template Pack · B Repo Map and Migration · C High-Stakes Domains · D Failure-Mode Taxonomy · E Glossary.

The repo's star is the harness; Mini is the teaching prop

flowchart LR
    case["case (YAML)"] --> runner
    runner --> trace["trace (JSONL)"]
    trace --> assertions --> stats
    trace --> judge --> stats
    stats --> report
    world["world sandbox + tool stubs"] -.replaceable.- runner
    synth["synth synthetic user"] -.replaceable.- runner

The six components contain not a trace of Shore & Summit knowledge; all world knowledge lives in the two replaceable parts. Appendix B shows how to detach the harness and bolt it onto your own agent; the trace schema is compatible with the OTel GenAI semantic conventions (the industry-standard set of LLM telemetry fields, so it plugs into existing observability stacks).

FAQ

  • The ch14 Lab gate occasionally goes red? On purpose. The release-gate baseline probabilistically trips a sev-1 after the vendor swaps the model underneath, a live specimen of Chapter 14's change tiers (which tier is a vendor model swap?). Details ship with Chapter 14's lab (repo/labs/ch14/README.md).
  • What are the traces in labs/*/out/? Pre-generated teaching material, for following the Labs offline and checking against the reference answers. Committed on purpose.
  • Is MODEL_FAKE=1 an evaluation method? No. It's a programming interface for tests and teaching-trace generation (replies come from a scripted queue). Evaluation must measure a real model.
  • What does this book not cover? Public benchmarks and model leaderboards (the eval set grows only out of your own failures; there is no benchmark roundup), how to build an agent (Mini is handed to you, how it was written is not taught), and prompt engineering as such, and the retriever's own ranking metrics (recall@k, NDCG and their kind, already covered by the single-turn eval books; this book only separates "the retrieval was wrong" from "the retrieval was misread," by whether first_bad_step lands on the tool_result or on the next model step, see Chapters 3 and 15).
  • Is Shore & Summit a real company? No. Shore & Summit is a synthetic teaching world assembled from common enterprise scenarios; it does not correspond to any real company, and every character in this book is fictional.

Contributing & license

Errata, case submissions, and harness improvements: see the contributing guide. Prose is CC BY-NC-SA 4.0, code is MIT; see LICENSE.md.