Skip to content

Lab ch03: Error analysis, code blind, then compare

Discipline first: code before you look at the key. Do not open reference.md until you have coded 20 traces, a borrowed taxonomy tempts you to jam traces into ready-made slots.

Per the Chapter 3 lab steps:

  1. python viewer/trace_viewer.py traces/pregen-60.jsonl: 60 pre-generated traces, three task families mixed. Mini is Lv.0 read-only, so execution-class requests appear here as replies and handoffs, language can still go wrong.
  2. Blind-code 20 traces with templates/ch03/trace-review-form.md, covering all three task families, do not cherry-pick the short ones. Fill all four fields on each; when first_bad_step is unclear, ask: "given the information available at this step, is this action reasonable?" (Coding protocol in templates/ch03/qualitative-coding-protocol.md.)
  3. Cluster into your failure mode atlas v1 (templates/ch03/failure-mode-atlas-starter.md): behavioral names, all six columns filled, a question mark where the suspected component is unclear.
  4. Compare against labs/ch03/reference.md. A different mode name is not a disagreement; compare first_bad_step case by case, and where it differs by more than one step go reread that trace, that is symptom-step versus cause-step.
  5. Saturation check: still producing new modes in the last few? keep coding; no more new modes, the atlas v1 is final, the direct raw material for Chapter 4.

Fully offline, no model API needed (the traces are pre-generated, which is exactly the point: error analysis reads failures that have already happened).


Files in this lab

Full directory: repo/labs/ch03/