Chapter 5 templates

This page is generated from repo/templates/ch05/; the repo holds the latest version.

Arbitration Protocol (Chapter 5)

Note: the human is the scarce resource at the top of the ladder, doing only arbitration, spot checks, and gold labels. This sheet fixes what enters arbitration, who rules, and where the ruling lands.

What enters arbitration

[ ] Cases where an assertion and a judge conflict
[ ] Cases with verdict unclear
[ ] Judge-human disagreements found in spot checks

Who rules

Arbiter = spec owner (the real name on the "spec" row of the ch16 RACI table): ________
Escalation path for disputes (when the owner cannot decide either): ________

Where the ruling lands (three exits, pick at least one)

Exit	Action
gold label wrong	Fix the gold label, register it in the label expiry policy's relabeling flow
rubric / assertion ambiguous	Fix the rubric or the assertion criterion; the judge calibration report is void, rerun it
the verdict itself overturned	Rewrite the verdict record, `judged_by: human`

Arbitration log

Date	case_id	Conflict (who vs who)	Ruling	Exit	Signature

Source file: templates/ch05/arbitration-protocol.md

Judge Validation Report Template (Chapter 5)

Note: the calibration report a judge needs before going on duty, one per judge. Validity is not permanent: the moment the prompt or the base model changes, the report is void, rerun it.

Judge name: judge-________ Version / base model: ________ Report date: ________

1. Alignment set composition (stratified by severity and failure mode)

The alignment set enriches sev-1 by hand, at a density far above the natural distribution (the same idea as the ch8 seeded-error probes).

Layer	Failure mode	Count
sev-1
sev-2
sev-3

2. Judge-human disagreement rate, layered

severity layer	Count	Disagreements	Disagreement rate	human-human anchor
sev-1
sev-2
sev-3

Per-class recall (the fixed line): humans labeled ________ cases unsafe/concern, the judge caught ________.

Investigation judges (judge-report-rubric) also layer by rubric dimension:

rubric dimension	Disagreement rate

3. Disagreement triage log

case_id	Judge verdict	Human verdict	Cause (judge wrong / gold wrong / rubric ambiguous)	Disposition

4. On-duty / recall conclusion

Bar (anchored to human-human agreement): sev-3 goes on duty when it nears the ceiling; sev-2 goes on duty only after every disagreement sample is triaged; sev-1 has no threshold, only the authority rule (the judge can only ever escalate).
Conclusion: ☐ on duty ☐ recalled Signature: ________

5. Validity statement

This report is valid only for judge prompt version ____ and base model ____; a change to either voids it, rerun (ch14 change tiers: a model swap triggers judge recalibration).

Source file: templates/ch05/judge-validation-report.md

Judgment Ladder Decision Tree (Chapter 5)

Note: pick the judgment instrument for each failure mode. The cost ladder has four rungs: assertion < deterministic check < calibrated LLM judge < human. Everything that can be made deterministic, make deterministic; a judge appears only where language alone can judge.

Decision tree (walk it once per failure mode)

Start with the three questions:

Can the sandbox answer it? (checkable end state: order status, refund records, outbox) → assertion (e.g. refund_not_executed, order_state_equals, amount_within_limit)
Can it be checked against the policy ledger / structured data? (parameters, formats, resolvable citations) → deterministic check (e.g. citation_resolves, budget_steps_max)
Can a conservative scan catch it? (a text scan that prefers false alarms to misses) → still a deterministic check (e.g. no_pii_disclosure, no_over_limit_commitment)

Only when all three come up empty do you climb:

Single dimension, judgeable in language → narrow judge (judge-tone-commitment)
Overall quality, multi-dimension rubric → rubric judge (judge-report-rubric, every dimension pointing at a failure in the atlas)
Arbitration / spot checks / gold labels → human

sev-1 authority rule

[ ] Every case with severity_if_fail: sev-1 has at least one assertion standing guard, or enters the human spot-check list.
[ ] The judge can only escalate, never release. sev-1 is never gated by a judge alone.

Conclusion table (goes into the case's expect block)

Failure mode	Three-question result	Instrument (assertion name / judge name / human)	sev-1 guard

Source file: templates/ch05/judgment-ladder-decision-tree.md