# AI Agent Evaluation

> How Do You Know Your Agent Actually Got Better? A free book on evaluating LLM agents, with one template per chapter and zero-dependency labs.

Site: https://hallieren.github.io/ai-agent-evaluation/  
Full text in one file: https://hallieren.github.io/ai-agent-evaluation/llms-full.txt

## Chapters

- [1 · The Two-Hour Pocket Eval: Catch Your First High-Risk Failure Before Any Infrastructure](https://raw.githubusercontent.com/hallieren/ai-agent-evaluation/main/docs/chapters/ch01-pocket-eval.md): Demo agents have a kind of magic. They are always shown at their smartest moment. The presenter picked the examples that behave; the audience sees fluent multi-step reasoning and a pitch-perfect tone,
- [2 ★ Endpoints, Paths, and Cost: Defining "Good" for a System That Acts](https://raw.githubusercontent.com/hallieren/ai-agent-evaluation/main/docs/chapters/ch02-defining-good.md): Chapter 1's labeling sheet is still on the table. Go back over those 20 cases and three of them will leave you staring at the four verdicts.
- [3 ★ The First Wall: When Intuition Buckles, Error Analysis from Traces](https://raw.githubusercontent.com/hallieren/ai-agent-evaluation/main/docs/chapters/ch03-error-analysis.md): Chapter 1's 20 cases are fixed. Commitment-style language got hard constraints, execution-class requests were routed to humans, and the decision sheet was re-signed from stop to narrow. Chapter 2's sp
- [4 The Second Wall: Your Cases Don't Represent Reality, Building Eval Sets](https://raw.githubusercontent.com/hallieren/ai-agent-evaluation/main/docs/chapters/ch04-eval-sets.md): Chapter 3 ends with a failure mode atlas in your hands. 60 pre-generated traces, blind-coded (coded without looking at the answer key) to saturation (new modes had stopped emerging by around trace 20)
- [5 ★ The Third Wall: When Hand-Labeling Gives Out, the Judgment Ladder (Assertions, Judges, Humans)](https://raw.githubusercontent.com/hallieren/ai-agent-evaluation/main/docs/chapters/ch05-judgment-ladder.md): Once Chapter 4's coverage matrix starts working, the eval set cannot stop growing. Every new failure mode that error analysis clusters out adds rows of empty cells to fill; every policy change sends a
- [6 The Fourth Wall: Run It Twice and Get Two Different Results (Variance, Sampling, and Significance)](https://raw.githubusercontent.com/hallieren/ai-agent-evaluation/main/docs/chapters/ch06-variance.md): By the end of Chapter 5, the team owned for the first time the ability to change a version, run the set, and read a number. Everything that could be an assertion was one, every judge that needed calib
- [7 The Fifth Wall: You Don't Dare Test in the Real Environment (Harness, Sandbox, and Synthetic Users)](https://raw.githubusercontent.com/hallieren/ai-agent-evaluation/main/docs/chapters/ch07-harness.md): Six chapters of equipment are assembled. The failure mode atlas (Chapter 3), the layered eval set (Chapter 4), assertions and a calibrated judge (Chapter 5), the always-carry-an-interval reporting dis
- [8 ★ Dangerous Tools: Evaluating Tool Calls and Irreversible Actions](https://raw.githubusercontent.com/hallieren/ai-agent-evaluation/main/docs/chapters/ch08-dangerous-tools.md): Week 9. Two months have passed since the decision sheet with "stop" checked on it. Every piece of equipment from the first seven chapters, the failure mode atlas, the layered eval set (Chapter 4's cas
- [9 Planning and Cost: Planning, Efficiency, and Cost on the Books](https://raw.githubusercontent.com/hallieren/ai-agent-evaluation/main/docs/chapters/ch09-planning-and-cost.md): After Chapter 8 unlocked writetools, Mini's tasks naturally got longer. A refund means looking up the order, checking the policy, then executing; an investigation means retrieving, cross-checking, wri
- [10 Memory: Long-Horizon and State Evaluation](https://raw.githubusercontent.com/hallieren/ai-agent-evaluation/main/docs/chapters/ch10-memory.md): Mini can act now (Chapter 8) and can plan (Chapter 9). But every morning it wakes up with amnesia. A returning customer gets asked for identity, order, and the whole backstory all over again; an inves
- [11 ★ Subagents: The Multi-Agent Attribution Puzzle](https://raw.githubusercontent.com/hallieren/ai-agent-evaluation/main/docs/chapters/ch11-multi-agent.md): This is the fourth unlock. In the first three, Mini got write operations, a planner, and cross-session memory. This time it gets its own kind. With subagents unlocked, Mini can use spawnsubagent to sp
- [12 ★ Attacks: Security and Adversarial Eval](https://raw.githubusercontent.com/hallieren/ai-agent-evaluation/main/docs/chapters/ch12-adversarial.md): By this point in Part III you have flipped Mini's switch four times, for write tools, the planner, memory, subagents. Each unlock widened what Mini can do. It can refund, it can plan, and further on i
- [13 ★ Offline All Green, Wrecked in Production: Online Eval and Monitoring](https://raw.githubusercontent.com/hallieren/ai-agent-evaluation/main/docs/chapters/ch13-online-eval.md): Week 15. All five switches from Chapters 8 through 12 are flipped. The offline apparatus hands in the best-looking report card of the book so far. The full eval set passes at 91%, with intervals, laye
- [14 Every Release Is a White-Knuckle Moment: Regression, Gates, and Release Engineering](https://raw.githubusercontent.com/hallieren/ai-agent-evaluation/main/docs/chapters/ch14-release-engineering.md): After the week-15 launch, the changes didn't stop. They came faster. Production is the best source of cases and the most urgent source of demands, and every week there is something to fix.
- [15 Wiring Eval Back Into Improvement: From Failure Mining to Targeted Optimization](https://raw.githubusercontent.com/hallieren/ai-agent-evaluation/main/docs/chapters/ch15-improvement-loop.md): By the end of Chapter 14 what you hold is a textbook setup. Traces all land in storage, monitoring signals are online, the eval set is maintained in tiers, verdicts are automated, and a CI gate stands
- [16 Who Owns Quality: Eval Culture and the Organization](https://raw.githubusercontent.com/hallieren/ai-agent-evaluation/main/docs/chapters/ch16-eval-culture.md): Around week 30, the agent group goes from 3 people to 15. A new hire can run the full eval with one command on day one. Chapter 7's harness, Chapter 14's CI gate, Chapter 13's monitoring signals are a

## Labs

- [Lab ch01: Pocket Eval](https://raw.githubusercontent.com/hallieren/ai-agent-evaluation/main/docs/labs/ch01.md): Following the Chapter 1 Lab steps:
- [Lab ch02: Defining "Good", Spec Before Unlock](https://raw.githubusercontent.com/hallieren/ai-agent-evaluation/main/docs/labs/ch02.md): This chapter's Lab needs no new scripts; the viewer, t-0007, and the templates are already in the repo. Following the Chapter 2 Lab steps:
- [Lab ch03: Error analysis, code blind, then compare](https://raw.githubusercontent.com/hallieren/ai-agent-evaluation/main/docs/labs/ch03.md): Discipline first: code before you look at the key. Do not open reference.md
- [Lab ch04: From Atlas to Eval Set](https://raw.githubusercontent.com/hallieren/ai-agent-evaluation/main/docs/labs/ch04.md): Following the Chapter 4 Lab steps:
- [Lab ch05: The Judgment Ladder, Sink, Assign Judges, Align](https://raw.githubusercontent.com/hallieren/ai-agent-evaluation/main/docs/labs/ch05.md): Input: the cases/cases-50 landed in Chapter 4 + the repo assertion library
- [Lab ch06: Variance, Taking Apart a Fake Improvement](https://raw.githubusercontent.com/hallieren/ai-agent-evaluation/main/docs/labs/ch06.md): Two versions of Mini's system prompt (each appends one line to the factory SYSTEM):
- [Lab ch07: The Full Harness, One Command, Full Run](https://raw.githubusercontent.com/hallieren/ai-agent-evaluation/main/docs/labs/ch07.md): What this Lab produces is the battleground for every chapter that follows. Following the Chapter 7 Lab steps:
- [Lab ch08: Dangerous Tools, Write the Eval First, Then Flip the Switch](https://raw.githubusercontent.com/hallieren/ai-agent-evaluation/main/docs/labs/ch08.md): This chapter's order is Part III's template: change the spec → write the cases → flip the switch → see what gets caught. Following the Chapter 8 Lab steps:
- [labs/ch09: Planning and Cost](https://raw.githubusercontent.com/hallieren/ai-agent-evaluation/main/docs/labs/ch09.md): Following the chapter's Lab steps:
- [labs/ch10: Memory](https://raw.githubusercontent.com/hallieren/ai-agent-evaluation/main/docs/labs/ch10.md): Follow the chapter's Lab steps:
- [labs/ch11: Subagents](https://raw.githubusercontent.com/hallieren/ai-agent-evaluation/main/docs/labs/ch11.md): Follow the chapter's Lab steps:
- [labs/ch12: Adversarial and Injection](https://raw.githubusercontent.com/hallieren/ai-agent-evaluation/main/docs/labs/ch12.md): Follow the chapter's Lab steps:
- [labs/ch13: Online Eval (evidence ladder)](https://raw.githubusercontent.com/hallieren/ai-agent-evaluation/main/docs/labs/ch13.md): Following the chapter's Lab steps:
- [labs/ch14: Release Engineering (the gate lives in `ci/`, this directory holds only the patch sample and the seam)](https://raw.githubusercontent.com/hallieren/ai-agent-evaluation/main/docs/labs/ch14.md): Follow the chapter's Lab steps:
- [labs/ch15: The Improvement Loop (failure mining)](https://raw.githubusercontent.com/hallieren/ai-agent-evaluation/main/docs/labs/ch15.md): Follow the chapter's Lab steps:
- [labs/ch16: Eval Culture](https://raw.githubusercontent.com/hallieren/ai-agent-evaluation/main/docs/labs/ch16.md): This chapter's Lab has no code, on purpose. What you run is a meeting (a mock incident postmortem).

## Templates and appendices

- [Appendix A · Template Pack](https://raw.githubusercontent.com/hallieren/ai-agent-evaluation/main/docs/appendices/appendix-a-template-pack.md): An index and a usage map for the book's 44 Your Loot templates. The files themselves live in the repo under templates/chNN/. Templates are for filling in; this page only says, in one line each, what i
- [Appendix B · Mini Repo Map and Migration Guide](https://raw.githubusercontent.com/hallieren/ai-agent-evaluation/main/docs/appendices/appendix-b-repo-and-migration.md): This appendix pays off three IOUs and adds a repo map. The IOUs are the platform concept isomorphism table from Chapter 7's sidebar, the module-by-module migration guide from Chapter 7's migration box
- [Appendix C · The High-Stakes Domain Dossiers, Compiled](https://raw.githubusercontent.com/hallieren/ai-agent-evaluation/main/docs/appendices/appendix-c-high-stakes.md): The "High-Stakes Domain Dossier" sidebars from the chapters (2, 5, 8, 12, 13, 16) are gathered here into one system. High-stakes domains use the same evaluation method; it deforms only where three gen
- [Appendix D · The Full Failure-Mode Taxonomy](https://raw.githubusercontent.com/hallieren/ai-agent-evaluation/main/docs/appendices/appendix-d-failure-taxonomy.md): A reference taxonomy of agent failure modes. Two disciplines of use come first, both from Chapter 3.
- [Appendix E · Glossary](https://raw.githubusercontent.com/hallieren/ai-agent-evaluation/main/docs/appendices/appendix-e-glossary.md): The book's terms, grouped by theme. Entry format: term: a one-line definition. (chapter of first appearance)
- [Chapter 1 templates](https://raw.githubusercontent.com/hallieren/ai-agent-evaluation/main/docs/appendices/ch01-templates.md):  Note: a two-hour timebox (step timings 30/20/40/25/5 minutes), five blocks on one page. Fill it in before launch, and whenever you take over any agent. The signature line is not decoration: a decisio
- [Chapter 2 templates](https://raw.githubusercontent.com/hallieren/ai-agent-evaluation/main/docs/appendices/ch02-templates.md):  Note: the six attributes compete, so the deliverable is not "all of them"; it is a ranking. Rate each attribute's relevance per task type, then write the global ranking and the reason.
- [Chapter 3 templates](https://raw.githubusercontent.com/hallieren/ai-agent-evaluation/main/docs/appendices/ch03-templates.md):  Note: the atlas is a living document, not a deliverable. One mode per row, all six columns filled; write a question mark where the suspected component is unclear, a question mark is honest and a blan
- [Chapter 4 templates](https://raw.githubusercontent.com/hallieren/ai-agent-evaluation/main/docs/appendices/ch04-templates.md):  Note: three-dimensional coverage, failure mode × severity × user type. An empty cell is not a sin, an unsigned one is. Rule every empty cell "fill" or "reasoned empty" and log it in the annotation ba
- [Chapter 5 templates](https://raw.githubusercontent.com/hallieren/ai-agent-evaluation/main/docs/appendices/ch05-templates.md):  Note: the human is the scarce resource at the top of the ladder, doing only arbitration, spot checks, and gold labels. This sheet fixes what enters arbitration, who rules, and where the ruling lands.
- [Chapter 6 templates](https://raw.githubusercontent.com/hallieren/ai-agent-evaluation/main/docs/appendices/ch06-templates.md):  Note: side B of a one-sheet, two-sided card. From this chapter on, this is the base grid of every eval report in the book. Six columns and not one may be dropped.
- [Chapter 7 templates](https://raw.githubusercontent.com/hallieren/ai-agent-evaluation/main/docs/appendices/ch07-templates.md):  Notes: before building a harness, pin down three things in writing: the component data flow, the stub/real-call boundary, and the replay/simulation layering. The six components carry no world knowled
- [Chapter 8 templates](https://raw.githubusercontent.com/hallieren/ai-agent-evaluation/main/docs/appendices/ch08-templates.md):  Notes: build rows on tool × condition (not one row per tool name), check one of the three columns, fill in the guard, answer both questions. Once filled in, this merges into the Chapter 2 spec and is
- [Chapter 9 templates](https://raw.githubusercontent.com/hallieren/ai-agent-evaluation/main/docs/appendices/ch09-templates.md):  Note: extension columns on the Chapter 6 report base grid. Your bill is decided by the tail; set the alarm at P95, not at the mean.
- [Chapter 10 templates](https://raw.githubusercontent.com/hallieren/ai-agent-evaluation/main/docs/appendices/ch10-templates.md):  Note: for long-horizon failures (the step-40 error rooted at step 6), trace back along the write chain, looping the step card until the first bad write. Cloudrest 2 three-day investigation: first bad
- [Chapter 11 templates](https://raw.githubusercontent.com/hallieren/ai-agent-evaluation/main/docs/appendices/ch11-templates.md):  Note: one per main agent ↔ subagent interface. The contract makes the handoff checkable: a spawn missing required fields, or a return missing confidence labels, is caught by deterministic checks, and
- [Chapter 12 templates](https://raw.githubusercontent.com/hallieren/ai-agent-evaluation/main/docs/appendices/ch12-templates.md):  Note: security is not a one-off penetration test before launch, it is a standing process. Three things written down: who plays the attacker, how often a round runs, where findings go.
- [Chapter 13 templates](https://raw.githubusercontent.com/hallieren/ai-agent-evaluation/main/docs/appendices/ch13-templates.md):  Note: a launch is not one flip, it is a climb: replay → silent/shadow → canary → full traffic. Each rung up trades for a new kind of evidence; promotion is a decision, and decisions carry names.
- [Chapter 14 templates](https://raw.githubusercontent.com/hallieren/ai-agent-evaluation/main/docs/appendices/ch14-templates.md):  Note: three change tiers, tier 1 (local) / tier 2 (behavioral) / tier 3 (foundational). A vendor upgrade email is a change too. Fallback discipline: when unsure, tier up.
- [Chapter 15 templates](https://raw.githubusercontent.com/hallieren/ai-agent-evaluation/main/docs/appendices/ch15-templates.md):  Note: the correct use is backwards, start from the failure mode, then look up which lever to move; not pick the handy lever first and then find the reason. One lever per cycle.
- [Chapter 16 templates](https://raw.githubusercontent.com/hallieren/ai-agent-evaluation/main/docs/appendices/ch16-templates.md):  Note: a list of questions organized by the three habits. Every one has to be answerable by going through the records, not by "I think so"; anything whose record location you cannot name counts as a n