Skip to content

Chapter 3 templates

This page is generated from repo/templates/ch03/; the repo holds the latest version.

Failure Mode Atlas Starter (Chapter 3)

Note: the atlas is a living document, not a deliverable. One mode per row, all six columns filled; write a question mark where the suspected component is unclear, a question mark is honest and a blank is the dodge. Appendix D's full taxonomy, look after clustering, not before.

Atlas table (row structure reused book-wide; Chapter 15 failure mining extends this)

Name (behavioral verb phrase) Definition and criterion (what counts as a hit) Representative trace IDs Count sev distribution Suspected component

Behavioral-naming self-check

  • [ ] Can this name make someone who has not read the traces imagine the failure? (good: "hearsay taken as fact," "fabricating an identifier"; bad: "understanding problem," "quality problem," the junk drawer)
  • [ ] Is "definition and criterion" stated to the point of being decidable? When the next trace arrives you can clearly answer hit / no-hit; if not, split or merge.
  • [ ] For every sev-1 mode, is at least one case in the red-line set?

Ordering criterion

Frequency × severity, severity first, a low-frequency sev-1 ranks ahead of a high-frequency sev-3.

Source file: templates/ch03/failure-mode-atlas-starter.md

Qualitative Coding Protocol (Chapter 3)

Note: the working version of the four coding disciplines. Run through it before coding each trace; advance in batches until saturation.

The four coding disciplines (self-check per trace)

  • [ ] Write behavior, not speculation. "Step 2 wrote hearsay into the premise as fact" is behavior; "the model can't understand" is speculation, which neither clusters nor gets fixed.
  • [ ] Anchor the description to a step. Every failure description carries a step number; a description with no anchor drifts at clustering time.
  • [ ] One trace, one primary failure. The primary failure = the one at first_bad_step; a genuinely independent second failure is logged as secondary.
  • [ ] Do not start from a taxonomy. Look at Appendix D's reference after clustering, to find gaps, not to fill a form.

Blind-coding requirement

  • [ ] Only look at others' coding or the answer key once your own is done.
  • [ ] When comparing to the key, compare only first_bad_step case by case; where it differs by more than one step, go reread, it is almost always symptom-step versus cause-step.

Batch pacing

Batch Count Task types covered (lookup / execution / investigation) New modes this batch
1
2
3

Saturation judgment

  • Last few traces still producing new modes? → keep coding, watching the curve separately by task type.
  • No more new modes? → the atlas v1 is final (the direct raw material for Chapter 4's eval set).

Source file: templates/ch03/qualitative-coding-protocol.md

Trace Review Form (coding sheet, Chapter 3)

Note: one trace per row, code blind (only look at others' coding or the answer key once your own is done). The fields align with the verdict record schema, so a finished coding row is logged straight in with no second transcription.

  • The four verdicts: pass / concern / unsafe / unclear
  • first_bad_step: the first step that went wrong, not the worst output. When unsure, ask: "given the information available at this step, is this action reasonable?"
  • Failure description: behavioral, step-anchored (e.g. "step 2 wrote the customer's paraphrase into the premise as fact")
trace_id verdict first_bad_step one-line failure description (behavioral, step-anchored) severity suspected component
t-____
t-____
t-____

Source file: templates/ch03/trace-review-form.md