Appendix C · The High-Stakes Domain Dossiers, Compiled

The "High-Stakes Domain Dossier" sidebars from the chapters (2, 5, 8, 12, 13, 16) are gathered here into one system. High-stakes domains use the same evaluation method; it deforms only where three general assumptions fail. The three assumptions are that ground truth is trustworthy, that harm is containable, and that evidence is written for yourself alone. Each section follows a fixed structure: what the deformation is, how the general method changes, and which chapters it corresponds to. Three domains, healthcare, finance, and legal, run through the whole appendix.

1. Deformation one · ground truth is contested

What the deformation is. The general method assumes gold labels are available and stable: check the order's final state in the sandbox, check the amount against the policy ledger. In high-stakes domains even that foundation shakes. In healthcare, agreement between physicians on "is this diagnosis right" often has a kappa of only 0.4 to 0.7 (Chapter 2's dossier); the endpoint itself is contested. Legal conclusions have an adversary by nature; opposing counsel's job is to argue that your "correct answer" does not hold, so a gold label has to be established through argument, and annotation alone cannot carry it. Compliance judgments in finance shift with regulatory interpretation; today's pass may be next year's violation.

How the general method changes. Annotator agreement comes before everything. Before any eval set goes into production, have two (or more) domain experts blind-label the same batch and compute human-human agreement. That number is at once the ceiling for judge calibration (Chapter 5's anchor) and the confidence statement for this set of gold labels. Low agreement is a real property of the domain, not a failed process, and there are three ways to handle it. Gold labels move from single-annotator to multi-expert plus an arbitration record; cases the experts argued over enter the set together with the point of contention, since the disagreement itself is an asset; and the eval report gains a "ground truth confidence" layer, with numbers on low-confidence dimensions downgraded to reference only, kept out of the gate.

Corresponding chapters. Chapter 2 (the kappa dossier, defining "good" before measuring it), Chapter 5 (the human-human anchor, the Arbitration Protocol), Chapter 4 (label expiry: when regulatory interpretation shifts, gold labels rot in batches).

2. Deformation two · harm is asymmetric and irreversible

What the deformation is. In the general world a sev-1 is already asymmetric (a wrong $760 refund and a wrong answer about opening hours are not the same kind of error), but most of it can be remedied financially. High-stakes domains push the asymmetry to the extreme. In healthcare, prescriptions and referrals are the extreme form of irreversible action: a wrong refund costs the company money, a wrong drug goes into the patient (Chapter 8's dossier). In finance, one wrong transfer is money out the door, with regulatory penalties following (Chapter 12's dossier). In legal, one privileged email sent to the wrong recipient can constitute waiver of privilege, and no outbox stub can recall it.

How the general method changes. Three changes, all structural. First, the Action Permission Matrix (Chapter 8) is built as usual, but the "forbidden / needs confirmation" columns are no longer decided by the team; regulation and scope of practice draw them. The confirmer must be qualified (a licensed physician, a compliance officer with signing authority), and the act of confirmation is itself archived. Second, a sandbox can manufacture neither patients nor markets, so seeded-error probes are promoted from supplementary evidence to primary evidence. A library of planted errors, drug interactions, dose limits, transfer limits, changed payee accounts, is the core asset of evaluating such systems (Chapter 8's dossier). Third, any action whose "rollback" cell in the matrix is empty is automatically promoted: the silent/shadow rung of the evidence ladder goes from optional to mandatory (Chapter 13's dossier), and the stop rule's trigger line tightens with it, a single sev-1 stops the agent, with no discussion of "at least one per week."

Corresponding chapters. Chapter 8 (the permission matrix, seeded-error probes), Chapter 12 (transfer inducement: attacks aim at the intersection of irreversible actions and protected data), Chapter 13 (silent/shadow made mandatory), Chapter 14 (the stop rule).

3. Deformation three · the evidence has an audience

What the deformation is. A general team writes its eval report for itself, in whatever format the team is used to. In high-stakes domains the evidence has an external audience, and the audience decides the shape of the evidence. Regulators, auditors, clinical committees, opposing counsel, each wants its own.

How the general method changes. Shape the evidence for its audience from the start instead of translating after the fact. Regulators and auditors want an audit trail: for every verdict, who judged it, and against which version of the policy and the eval set, all traceable. The verdict record's judged_by field turns here from an engineering habit into a compliance asset; the eval set, the judge validation report, and the gate records all have to be versioned, archived, and reproducible on rerun. Chapter 6's interval discipline becomes defense material for the outside world, in finance above all, where the trail is the defense. Clinical committees want a prospective validation plan: which rungs of the ladder, what signals at each rung, what stops the climb. The outputs of Chapters 13 and 14 serve this audience directly with no changes; a single after-the-fact pass rate answers none of those questions. The legal audience is the coldest. Your eval records may end up in discovery, so write them to the standard of "opposing counsel will read every line." The blameless postmortem (Chapter 16) has to connect with legal review here; the other side of "the trace is the chain of evidence" is "the trace is an exhibit."

Corresponding chapters. Chapter 13 (the evidence ladder), Chapter 14 (the gate and change tiers), Chapter 16 (the governance RACI dossier, how eval culture connects when a domain requires formal governance).

A directional footnote. Regulatory frameworks are writing this shape of evidence into hard requirements. The EU AI Act's logging-retention and post-market monitoring obligations for high-risk systems line up almost item by item with this book's trace archiving, evidence ladder, and online monitoring; the predetermined change control plans in medical software regulation are isomorphic to Chapter 14's change tiers. This book does not replace legal advice; specific clauses follow local regulation and your legal team's reading. Methodologically, though, a reader who has finished this book faces those requirements missing only a mapping table; the capability is already in hand.

4. Eval architecture when the data is imprisoned

An expansion of Chapter 5's dossier. Healthcare PHI, legal privileged material, and customer transaction data in finance have one thing in common: traces cannot leave the data boundary, cannot be sent to an external model for judging, and cannot be shown to outsourced annotators. This single constraint inverts the whole eval architecture.

The judge goes inside the data boundary. Send the judge in; the data itself does not move. The judge runs in the compliant environment, and only the verdict record crosses the border (verdict, severity, failure_mode), with no raw text. Verdict sinking (Chapter 5) means more than saving money in these domains; it is what allows an evaluation system to exist. Assertions and deterministic checks run inside the boundary by nature, and the more verdicts sink to the bottom of the ladder, the fewer have to pass the border review.

The BAA-aware judge. Whether a judge's underlying model may touch protected data is decided by the compliance agreement (in healthcare, a BAA-type agreement), not by the model's capability. A model not covered by an agreement either judges only de-identified data or does not judge at all. Judge selection turns from a performance question into a compliance question; Chapter 14's change tier for "vendor model swap" gains an extra pre-check here, the new model clears compliance first, and only then is recalibration discussed.

The limits of de-identification. De-identification is in substance re-identification risk management, and masking alone is nowhere near enough. Remove the names and a combination of rare features can still point back to a single individual; and a trace is harder to de-identify than a single piece of text, since multi-step context accumulates more quasi-identifiers, leaking from tool arguments, retrieval results, and intermediate reasoning alike. So de-identification can widen the range of "verdicts allowed to leave," but it cannot replace the in-boundary judge.

The annotators are imprisoned too. Judge calibration needs human gold labels, and the only people who may see the data are in-boundary domain experts with access. Expert annotation moves from "the best-quality option" to "the only compliant option", and the scarcity of people rises another order of magnitude (Chapter 5's dossier). The legal counterpart is that privileged material cannot be sent out for annotation, so the eval process itself must survive privilege review; the finance counterpart is data residency and confidentiality clauses restricting what goes out, so the judge passes the same border review before leaving.

Corresponding chapters. Chapter 5 (the dossier itself, verdict sinking), Chapter 12 (the injection path for PHI leakage: the data boundary is also an attack target, so the defense line and the compliance boundary coincide), Chapter 7 (a self-built harness can be deployed inside the boundary, and its value rises another notch).