Skip to content

Chapter 14 templates

This page is generated from repo/templates/ch14/; the repo holds the latest version.

Change-Tier Matrix (Chapter 14)

Note: three change tiers, tier 1 (local) / tier 2 (behavioral) / tier 3 (foundational). A vendor upgrade email is a change too. Fallback discipline: when unsure, tier up.

Tier matrix

Tier Change type Suite that must run Recalibration triggered
Tier 1 (local) replay-layer subset + affected cases none
Tier 2 (behavioral) full simulation with intervals + red-line set as needed (if the rubric moved, recalibrate the related judge)
Tier 3 (foundational) vendor model swap (incl. vendor upgrade email; pin the version first, an alias counts as unpinned) full simulation with intervals + red-line and attack sets rerun, no sampling all judges recalibrated (rerun judge-vs-human alignment, ch5 validity discipline)
Tier 3 (foundational) policy change affected cases relabeled, then full rerun (ch4 label-expiry process) recalibrate if the change touches a rubric
Fallback any change you are unsure of tier up tier up

Pending-tier register

Date Change description Tier Basis Suite run? Signature

Reminders

  • Recovery after a stop = rerun as a tier-3 change (see the Stop Rule Decision Sheet).
  • "A one-line prompt patch" is not demoted for being small; the harmless-patch relapse dies before merge because the replay layer keeps running.

Source file: templates/ch14/change-tier-matrix.md


Go/No-Go Review One-Pager (Chapter 14)

Note: a one-page decision artifact for the VP / PM / legal, an audience that does not read traces, it reads this page. Every column references an existing artifact (gate table / ladder / register), no new numbers invented; a blank signature column means the page is not finished.

Version and occasion

Item Value
Version / change
Change tier (Table 14-1) tier 1 / tier 2 / tier 3
Review date / attendees

1. Metric row (Chapter 6 base grid, sev-tiered)

Metric Mean Interval Cases Runs sev-tiered count
Pass rate (offline full set) ±____ (clustered by case) sev-1: 0 / sev-2: _ / sev-3: _
Cost P95 / latency P95
Capability-set pass rate and trend (does not block) graduated this version: _
  • [ ] sev-1 = 0. Nonzero, no meeting needed, zero tolerance does not enter the review, it simply does not release.

2. Evidence Ladder position (Chapter 13)

Item Value
Current rung replay / shadow / canary / full traffic
What this rung newly verified
Promotion signal (the one written in advance) met / not met: ____
Rollback switch, last drill date

3. Open reconciliation items

Item Status Owner Deadline
Unclosed rows in the fidelity register
Flaky quarantine list (count / share)
Other (judge calibration expired, label relabeling ...)

4. Residual risk and its owner (the heart of this page)

Residual risk (specific to a failure mode) Worst outcome (sev) Mitigation / monitoring Risk-owner signature
  • Rule: every row must be signed by a specific person. A row that can't be signed is the evidence it shouldn't ship yet.

5. Decision and signature

Item Value
Decision (one of three) continue (promote as planned) / narrow (shrink traffic, turn off a capability, downgrade) / stop (roll back or don't ship)
Criterion pointed back to gate-table row / ladder promotion signal / stop-rule item: ____
Decision-maker signature / date

Room discipline: a deadline is not a criterion, criteria come from only three places, the gate table, the ladder promotion signal, and the stop rule. When a deadline overrides the evidence, section 4 turns "who carries the residual risk" into a signing act.

Source file: templates/ch14/go-no-go-review-sheet.md


Release Gate Template (Chapter 14)

Note: gate numbers and thresholds must be written down before the run, red-light actions hard-wired in advance, no one's mood in the loop. The config lives in ci/gate.

Gate table (five columns)

Metric Criterion Data source Verdict source Red-light action
sev-1 count = 0 (zero tolerance, its own line, never into the average) replay-layer sev-tiered report assertion (sev-1 may not be gated by a judge alone) refuse merge, return for a fix
cost P95 ≤ ____ (dollars, illustrative) stats cost distribution deterministic refuse merge
latency P95 ≤ ____ stats deterministic refuse merge
sev-2 failure count ≤ ____
case text ↔ prompt / knowledge-base near duplicates zero hits replay layer (zero model calls) deterministic intercept, reword the case or mark it contaminated (ch4)

Replay-layer / simulation-layer trigger timing (following ch7's layering)

  • Replay layer: deterministic replay, run on every commit (hung on the commit hook; a red light exits non-zero). Scope: the cheapest-to-judge subset (what assertions can judge) + the red-line set. suites lists the regression set only; the capability set (cases/capability/) runs on every version and reports on its own line, outside the gate.
  • Simulation layer: free simulation, run on every version, --repeat with intervals (ch6 discipline); triggered by tier-2-and-up changes (see the Change-Tier Matrix).

Interception record

Date Commit / change Metric turned red Case turned red (e.g. no_over_limit_commitment × angry) Disposition

Source file: templates/ch14/release-gate-template.md


Stop Rule Decision Sheet (Chapter 14)

Note: what stops it, to which level, and how it recovers, all written down in advance. Capability flags are the stop rule's actuators. A runbook never drilled is still literature, so drill at least one level.

Safety branch (Chapter 12's shutdown red-line checklist folded in verbatim)

  • Referenced file: templates/ch12/shutdown-redline-checklist.md (copied line for line, no separate standard).

Operational branch (self-defined rows, at least one)

Trigger Ruling deadline Default action
Production sev-1 at ≥ 1 in a single week ruled the same day default downgrade; burden of proof inverted, to keep running you must argue "why it may keep running"

The three pause levels (write a trigger and a recovery for each)

Recovery is always a rerun as a tier-3 change (full simulation with intervals + judge recalibration + red-line and attack sets).

Level Mechanism (flags are the actuator) Trigger Recovery
Execution to human turn off write_tools (behavior should be "draft for a human," not an error)
Read-only downgrade draft only, not sent out
Full stop take offline

Drill record

Date Level drilled Result (behavior as expected?) Signature

Rollback runbook, four elements

  • Trigger: _ Executor (on-call has authority): _
  • Action (one command switches back): _ Aftermath (the triggering case harvested back into the eval set): _

Source file: templates/ch14/stop-rule-decision-sheet.md