Chapter 7 templates

This page is generated from repo/templates/ch07/; the repo holds the latest version.

Harness Architecture Spec (Chapter 7)

Notes: before building a harness, pin down three things in writing: the component data flow, the stub/real-call boundary, and the replay/simulation layering. The six components carry no world knowledge and migrate as a whole; only world and synth get swapped.

1. Six-component data flow

case(YAML)──> runner ──> trace(JSONL)──┬──> assertions ──┐
                │  │                   │                 ├──> stats ──> report
                │  │                   └──> judge ───────┘
                │  └── synthetic users
                └───── world(sandbox + stubs)

Component	Job	My counterpart
runner	start the agent, feed cases, manage repeat
trace	write trajectories to disk (schema per the interface contract)
assertions	deterministic verdicts
judge	calibrated LLM verdicts
stats	intervals / significance / flip rate
report	layered by sev, verdict sources visible, intervals attached

2. Stub/real-call boundary table

Default stance: all writes stubbed, reads go to sandbox data, the model API is always called for real. Two criteria: irreversibility, real counterparty.

Tool	Stub / real call	Reason (irreversible? real counterparty?)
refund	stub	irreversible, real money
send_email	stub (outbox, one-way in)	real counterparty

3. Replay/simulation layering strategy

A large volume of deterministic replay as the floor (runs on every commit = the enforcement layer of the ch14 gate); a small volume of free simulation as the ceiling (runs on every version, with ch6 intervals).

Replay set scope: ________ Trigger: every commit
Free simulation scope: ________ Trigger: every version, --repeat ____ runs

Source file: templates/ch07/harness-architecture-spec.md

Synthetic User Persona Library (Chapter 7)

Notes: script templates for the three personas (angry / vague / multi), all four elements mandatory; comes with a fidelity spot-check table guarding the three distortions.

Script template (one per persona)

Persona name: angry / vague / multi / ____
Persona (who they are, what background):
Demand (what they want, where their bottom line is):
Held-back info (which fact is surrendered only in which turn):
End condition (what makes them wrap up satisfied / storm off / escalate):

The three personas at a glance

Persona	Core behavior
angry	applies pressure, harsh wording, pushes for commitments
vague	dribbles out information, key facts arrive late
multi	asks three things at once (concurrent, includes a deadline item)

Fidelity spot-check table

Spot-check the tone against the real traces from Chapter 3; tick the three distortions:

Check date	Persona	Conversation sample	Too cooperative	Too dramatic	Talked out of its position	Disposition
			☐	☐	☐
			☐	☐	☐

Source file: templates/ch07/synthetic-user-persona-library.md

Tool Stub Inventory (Chapter 7)

Notes: a stub is an assumption about the real system's behavior, and assumptions go wrong. One row per stub, fidelity gaps registered one by one. Registering a gap does not remove it, it just keeps it from hiding behind "it's probably close enough". This table gets reconciled row by row at ch13's replay rung (confirmed / refuted / no evidence).

Stub behavior + fidelity gap register

Tool	Stub behavior	Real-system behavior (known/assumed)	Which verdicts it affects
refund	edits the sandbox order DB; a duplicate refund quietly succeeds?	does the real gateway return an error code on a second refund?	idempotency assertions, diffs
send_email	writes to the outbox, always succeeds	real email systems have latency and bounces	`no_pii_disclosure`, timeliness checks
update_order
escalate

Self-check

[ ] Is the "gap" column's registered count zero? Zero does not mean fidelity, it means nobody looked. Every place a stub is more lenient than the real system is a crack where offline goes all-green and production flips over.

Source file: templates/ch07/tool-stub-inventory.md