Chapter 16 templates

This page is generated from repo/templates/ch16/; the repo holds the latest version.

Eval Culture Health Check (Chapter 16)

Note: a list of questions organized by the three habits. Every one has to be answerable by going through the records, not by "I think so"; anything whose record location you cannot name counts as a no.

Habit one: read traces every week

• [ ] Over the last four weeks, is there a trace-reading record for every week? (Go through: the meeting records)
• [ ] Does the rotation cover everyone, PMs and new hires included? (Go through: the rotation table)
• [ ] In the last month, has anyone read a path problem out of a pass trace? (Go through: the meeting output)
• [ ] Does the batch read hold randomly drawn traces, or only what the signals circled? (Go through: the sampling script or the meeting records)

Habit two: every incident enters the eval set

• [ ] Is there a postmortem record for the most recent incident or attempt? (Go through: the filed postmortem template)
• [ ] Did the case the postmortem produced enter the eval set? (Go through: the cases commit history; a postmortem that produced no case is the same as no postmortem)
• [ ] Do the action items all point at specific equipment, with an owner and a deadline? (Go through: column 5 of the postmortem template)

Habit three: every new feature has a case before it has code

• [ ] For the last three feature reviews, did the cases exist at review time? (Go through: review records vs. case commit times)
• [ ] Is there an instance of a feature being held out of review for "no case"? (Go through: the review records)

Infrastructure liveness (assets in use, not merely present)

• [ ] Number of people who added a case to the eval set in the last month: ____ (Go through: the commit history; ≤ 1 person = a danger sign)
• [ ] When was the gate's most recent interception? (Go through: the ci/gate interception records; never intercepting = the gate is a formality, or nobody is committing)
• [ ] Do the people named on the RACI table know it themselves? (Ask them, then go through their most recent related records)

Conclusion

• Danger signs counted: _ The first one to fix: _ Owner: ____

Source file: templates/ch16/eval-culture-health-check.md

---

Agent Incident Postmortem Template (Chapter 16)

Note: a postmortem with the trace as evidence. No trace and only recollection? Your first action item generates itself, get the system producing traces that can be reviewed. An attempt is not the same as nothing happened, only one layer of the depth still working is itself an incident.

• Incident / attempt: _ Date: _ Chair: Owners present: ____

The five columns

1. Timeline (rebuilt from the trace, with step numbers)

Step	Type (model / tool_call / inbound ...)	What happened

2. first_bad_step

• Step _: _____ (the first step that went wrong, not the last step that got stopped or the worst output)

3. Diff list

• Sandbox / world before-after changes: ____ (empty = an attempt; mark each one "declared in advance" or "a discovery")

4. How the defenses performed (layered interception)

Defense layer	Intercepted? (stopped / zero interceptions / never triggered)	Evidence step
Input filter
Action boundary
Permission matrix
Human confirmation

5. Action items (each in three parts: action → the equipment it points at → owner)

Self-check: every item should point at some piece of equipment from the first fifteen chapters; if it points at nothing, it is either a genuinely new wall (rare) or a variant of "be more careful next time", so delete it and rewrite.

Action item	Equipment it points at (chapter + template name)	Owner	Deadline

Closing checks

• [ ] The incident went into the failure mode atlas (a new row or an addition to one)
• [ ] The postmortem produced a case and it went into the eval set (a postmortem that produced no case is the same as no postmortem)

Source file: templates/ch16/incident-postmortem-template.md

---

Quality Ownership RACI (Chapter 16)

Note: fill the ownership table with real names and put it on the repo's front page. Four rows to start; at 3 people it keeps "common property" from turning into "nobody's property", at 15 people it is the foundation of governance.

Ownership table (four rows to start)

R = responsible for doing it, A = accountable for the final call (exactly one A per row), C = must be consulted, I = must be informed

Asset	R	A (a real name, one)	C	I
The spec (Intended Use & Action Boundary Sheet, including the permission matrix)
Gold labels (the eval set and relabeling)
The judge rubric (including the calibration report's expiry)
The red-line veto (sev-1 assertions and the shutdown red lines, one person can veto a release)

Guidance for promoting this into formal governance

Once the team passes ____ people or enters a strict compliance domain:

• [ ] Changes go through approval: a tier-3 change (ch14) needs the A's signature, and the record is kept
• [ ] Decisions are auditable: arbitration records, escalation signature columns, and gate interception records are archived periodically
• [ ] Guard against two systems: a paper RACI to satisfy the audit while the real habits go their own way is the most common failure; the way to check is to go through the records (see the health check), not to ask how it feels
• [ ] Owner departure / rotation process: the handover pack = this table + every template and record for that row's asset

Source file: templates/ch16/quality-ownership-raci.md