Chapter 15 templates

This page is generated from repo/templates/ch15/; the repo holds the latest version.

Bottleneck-to-Lever Mapping (Chapter 15)

Note: the correct use is backwards, start from the failure mode, then look up which lever to move; not pick the handy lever first and then find the reason. One lever per cycle.

Eight levers × failure classes

Lever	Failure class it treats	Cost and blast radius (regression width)
Edit the prompt (system instructions)	Behavior and phrasing: unauthorized commitment, answering flat when it should escalate, tone	Cheap; but globally coupled, any trajectory can be caught in it, full regression is mandatory
Edit the tool description	Wrong tool choice, wrong parameters, hallucinated tools	Cheap and narrow: only trajectories that use that tool are touched
Swap the model	Capability failures: several modes running high at once, nothing else moves them	Most expensive: the judge is fully recalibrated (ch5 expiry discipline), and ch14 change tiers put you on the largest suite
Add a confirmation gate (a row in the permission matrix)	Irreversible actions: duplicate refunds, unauthorized execution	Doesn't lower the error rate, lowers the harm; the price is human confirmation volume
Edit the handoff contract	Multi-agent context loss, reviewing off a summary	Medium; only the subagent path is touched
Fix the memory policy	Crosstalk, misremembering, cross-session contradictions	Medium; must be verified with multi-session replay
Fix the knowledge base or retrieved content	Grounding failures: the policy document itself is wrong or stale, the agent cites it correctly and answers entirely wrong	Cheap; touches every trajectory citing that content, and triggers gold-label relabeling (ch4 expiry policy)
Remove a component (a prompt instruction, a validation layer, a tool, a context reset)	Stale scaffold assumptions: patches written for the old model fighting each other on the new one, the cost tail	Cheap; blast radius same as the prompt, full regression is mandatory; verified by the ch14 non-inferiority criterion, and if nothing degrades the simpler version stays

Capability failures no lever reaches -> the gate: the harm is shut behind human confirmation, the bottleneck can wait, the harm cannot.

Fine-tuning is not in the table: this book does not teach how to build agents (boundary declaration in the Decisions & Principles internal design doc); if you really do fine-tune, handle it as a ch14 tier-3 change, judge fully recalibrated + the largest suite.

"Handy != on target" self-check

[ ] Was this lever chosen by evidence, or by habit? (The handiest lever for any team is always editing the prompt, and using it to fix a wrong tool choice is like fixing one broken door with the building-wide PA system)
[ ] Two suspected components and both plausible? -> Write both as falsifiable hypotheses, try the smaller blast radius first
[ ] Did this cycle move only one lever?

This cycle's selection record

Failure mode: _ Candidate levers: Chosen: _ Evidence:

Source file: templates/ch15/bottleneck-lever-mapping.md

Failure Mining Protocol (Chapter 15)

Note: the industrialized version of ch3's error analysis on production data. Mining wants the structure of failure, not a census of failure, so a pool with false positives and leaks is fine.

1. Failure-pool signal list (production "suspected failure" is defined by online signals)

[ ] escalation (handoff to a human)
[ ] negative user feedback and restatement
[ ] assertion hits (online red line)
[ ] judge escalation
[ ] the cost tail over budget
[ ] (your own) ____

2. Stratified sampling rules

[ ] Never sample in proportion to traffic (lookup-type tasks drown the batch)
[ ] Stratify by signal severity and task type
[ ] sev-1 signals all enter the pool, never sampled
[ ] Read the saturation curve separately per task type, stop at saturation
[ ] Keep a random slice of ____% outside the pool (no signal filter at all; the modes the signals cannot see surface only here)

Stratum (signal × task type)	In pool	Sampled

3. Coding and clustering steps (following ch3)

Read every trace by hand, code it with the Trace Review Form (verdict / first_bad_step / behavioral description / severity / suspected component)
Code blind, all four coding disciplines still apply (dirty data gets no exemption)
Cluster, name behaviorally

4. Atlas extension format (six-column row structure, following Chapter 3)

Name (behavioral verb phrase)	Definition and criterion	Representative trace IDs	Count	sev distribution	Suspected component

Once extended: a new sev-1 mode -> a red-line case; the suspected component -> look up the bottleneck-to-lever mapping, write the falsifiable hypothesis.

Source file: templates/ch15/failure-mining-protocol.md

Improvement Cycle Template (Chapter 15)

Note: one page per cycle. The rejection rule is written before the run; set the standard after the run and any result can be called "fixed".

Cycle #_ (dates _ to ____)

Target mode

Atlas row: _ (count _, sev ____)

Experimental hypothesis (falsifiable form)

"_ fails because of in the _ component; change it and the target mode's count should drop from to ____."

Lever moved (only one allowed)

☐ edit the prompt ☐ edit the tool description ☐ swap the model ☐ add a confirmation gate ☐ edit the handoff contract ☐ fix the memory policy ☐ fix the knowledge base ☐ remove a component
The change itself:

Pre-written rejection rule (signed before the run)

The target mode's count has to drop to: _ (paired, --repeat 5 passes, with intervals, how the interval is computed: _)
Full-regression gate line (by sev tier): ____
What result counts as failure: ____
Signature: _ Date: _

Two-part verification result (one part short doesn't count)

Did it get fixed? Target mode's count (paired ± interval, not the overall pass rate):
Did it break anything else? Full regression by sev tier:
Conclusion: ☐ keep it, in effect ☐ roll back and try the next hypothesis (a normal step in the cycle, not a setback)

Capability-set ledger

Graduated _ (passed --repeat 5 on two consecutive versions, moved into the regression set; sev-1 needs a deterministic assertion first) Newly added to the capability set _
Saturated? ☐ no ☐ yes, harder cases added this cycle: ____

Next cycle's candidates

Mode: _ Basis (failure-pool signal): _

Source file: templates/ch15/improvement-cycle-template.md