Skip to content

Chapter 13 templates

This page is generated from repo/templates/ch13/; the repo holds the latest version.

Deployment Evidence Ladder (Chapter 13)

Note: a launch is not one flip, it is a climb: replay → silent/shadow → canary → full traffic. Each rung up trades for a new kind of evidence; promotion is a decision, and decisions carry names.

Four rungs × three questions

Rung Newly verifies Promotion signal Rollback signal
Replay Real input distribution; stub assumptions reconciled (ch7 register) Layered rates clear the gate (zero sev-1); every register row concluded; new failures harvested, fixed, rerun (offline, fix and rerun)
silent/shadow Live real-system read path; same-question comparison vs humans Disagreement postmortems acceptable; zero red-line hits on proposed actions Proposed action hits sev-1
Canary Real consequences and the world's reaction; write-path stub assumptions closed Zero red lines + signals no worse than baseline + inside SLO Any sev-1; signal breaks the band
Full traffic Nothing (only scale and the long tail) None Same as canary, plus drift alarms

Action type × mandatory rungs (the four-rung rule)

  • Autonomous actions whose rollback column is empty (or nominally reversible, in practice unrecoverable) → all four rungs, shadow not skippable
  • Purely read-only, or every write in the "needs confirmation" column → shadow may fold into the canary
Action type (from the permission matrix) Rollback column Mandatory rungs

Promotion signature line per rung

Promotion Date Evidence (report link / metrics) Signature
Replay → shadow
Shadow → canary
Canary → full traffic

Source file: templates/ch13/deployment-evidence-ladder.md

Monitoring Signal Spec (Chapter 13)

Note: online has no gold labels, so monitor only "what can be judged without a reference answer." A signal without a baseline is just a number, and a signal without a trigger action is just decoration.

Signal table (five columns)

Pre-filled with the four no-gold-label signal classes, in descending order of trustworthiness:

Signal Data source Baseline Band Trigger action
Deterministic red-line assertions run online (no_pii_disclosure scans outbound, amount_within_limit scans refund arguments, no_over_limit_commitment scans reply text) Production traces / outbound content Zero hits No band: a single instance trips sev-1 single instance = rollback
Escalation rate (spike = new inputs; a dip is more suspicious = bluffed answers) Human-handoff records
Customer repeat-contact rate (same customer, same matter, short window) Session records (human-support era = ready-made baseline)
Overturn-type signals (appeal reversals; human rejection rate on "needs confirmation" actions; shadow disagreement postmortems) Human ruling records
Prompt cache hit rate (the cost regression most easily shipped by accident, ch9) API usage Below the band = check the head of the prompt

Three-column cost basis (per ch11)

System cost = outer usage + the sum of every nested trace's usage.

Main agent Subagents Round trips

Three drift probes (answering "is the world still the world in the eval set")

Probe Watches Response (not rollback, harvesting)
Input distribution Task-type mix, approximate persona distribution, topic words
Tool error rate Real-system error codes (examines the "error recovery" dimension)
Escalation rate (on stage again) Slow climb = input drift's earliest symptom

Source file: templates/ch13/monitoring-signal-spec.md

Silent/Shadow Plan Template (Chapter 13)

Note: shadow = running online, producing no consequences. A shadow with no exit condition shadows forever; a shadow with no comparison is just a delayed launch.

1. Comparison baseline

  • Compared against: ☐ human handling ☐ old version ☐ current process
  • Same-question comparison method (how the entry-by-entry lineup works):

2. Interception point inventory (which layer stops each write)

Write tool Interception layer (where the action is caught) Where the proposed action lands (log / outbox) Red-line assertion scanned online?
refund refund_not_executed etc.
send_email no_pii_disclosure

3. Disagreement postmortem process and cadence

  • Cadence (daily / weekly): ________ Read the top ________ disagreements each time
  • Three-way call: ☐ Mini wrong ☐ human wrong (don't waste it, harvest) ☐ both right (different routes)
  • Record table:
Date Entry Three-way call Disposition (harvest / fix / archive)

4. Duration and exit conditions

  • Planned duration: ________
  • Exit (promotion) condition: disagreement rate stable below ________, and the "Mini wrong" share of postmortems ≤ ________; proposed actions hitting red-line assertions = 0
  • Exit (abandon) condition: ________
  • Honest-boundary memo: shadow cannot test the world's reaction to the agent (the customer's next line was spoken to a human); write-path stub assumptions wait for the canary to close.

Source file: templates/ch13/silent-shadow-plan-template.md