Chapter 8 · Read the Results, Catch the Errors

Chapter companion

📋 Chapter 8 templates · 🗂 Template index · 💻 code/smol-army · 💻 code/persona-panel

This chapter's ladder. At this step AI downgrades on purpose. Batch splitting, recomputing, pattern scanning can be handed off. Those jobs have cheap ground truth, a recomputed number comes out right or wrong. The ruling on "does this count as evidence" is yours. The reason was written in the Chapter 3 snapshot, interpretation is sycophancy's home ground, you ask with excitement and it answers along with the excitement. Let it sit as a juror. The judge's seat is not for it.

Spine update. The Chapter 7 harness finished the full run. In this chapter you will see the most dopamine-rich line of numbers in the whole project, and how within twenty-four hours it went from +18 to −2.3.

This chapter delivers. The result interrogation checklist, the preregistered/post-hoc side-by-side report template, the surprise-result red flags.

8.1 The verdict column reads army_ahead

The evening of July 25, 2026, the full run closed out. 3,580 result lines, $5.58 in the ledger, 0 failures. How the harness was built and the five bugs the pilot caught, Chapter 7 covered. I opened results/report.md and went straight to the paired bootstrap table. Paired bootstrap, paired resampling, used to put an interval on the gap between two arms over the same set of problems.

The code row, the army 93.7%, frontier 96.0%, CI crossing zero. CI, confidence interval, the range the gap could fall in, and crossing zero means that range contains 0. Respectable and expected. The mmlu_pro row, the army 15 percentage points behind, CI [−21.3, −8.7]. The skeptics' script, also within the odds.

Then math. Army vote 0.856, frontier 0.673. A gap of +18.2 percentage points. The raw difference is 18.3, this book uses the bootstrap point estimate throughout. CI [+12.7, +24.2], and the verdict column, the column in the results table where the ruling is written, reads army_ahead.

What went through my head in those minutes, I can report faithfully. In Chapter 4 the private odds I gave myself were "narrow tasks have a shot," and now a narrow task had come in person to collect. This one line of numbers was enough for the cover story of the whole book, enough for the opening paragraph of the Chapter 9 memo. I wanted to send it to every colleague who had poured cold water on it, and I even started thinking about what to call this chapter.

I wanted to announce. That urge, by itself, is the first signal this chapter teaches.

Hitting the brakes was no virtue. This line of numbers was good beyond bounds. The same table held two more things that made no sense. First, the cost-matched self-consistency arm also beat frontier on math, by six points. If a 20B small model can overtake frontier on multi-sampling alone, the story should not be "team magic." Second, frontier scored only 0.673 on grade-school probability problems. A frontier model cannot do a third of grade-school math problems? Too good to be true and too bad to be true, on the same table.

The evening the results arrive is the most dangerous moment in the whole workflow.

8.2 The brake rule and the three interrogation questions

First see why this step is dangerous. Motivation is at its peak, weeks of work are waiting for this line of numbers to pay out, the degrees of freedom are still wide, how to split, how to tell it, which slice to stress are all undecided, and the error is silent, a wrong conclusion and a right one look identical in a report. The criteria locked in Chapter 6 govern "what counts as a win." They do not govern how fast you run out the door holding the word win.

The way AI rewrites this step has the same shape as the earlier steps, only the labor collapses in price. Splitting the results by problem, clustering the wrong examples, recomputing the statistics under a different assumption, these interrogation moves used to cost so much that you did them only when a reviewer, or the checker before delivery, forced you to. Dispatch them to AI now and the full set comes back in an hour. The guilt of not interrogating went up, not down. Not interrogating used to have an excuse, interrogation was too expensive. Now interrogation is cheap enough that the excuses run out. The ruling half is another matter. The three variables of Chapter 3 all press toward the human here. No machine can rule on whether a number is qualified to carry a conclusion, a wrong ruling raises no error, and retracting an announced conclusion is priced in reputation. So the ladder at this step falls rather than rises. Splitting and recomputing are handed off. The judge's seat is taken back.

The brake rule is one sentence, written into the process. Any result that makes you want to announce it at once goes through the interrogation before you announce it. You do not need to feel something is wrong to interrogate. Noticing that you want to announce is enough.

The interrogation is three questions.

Question one, can the scorer be trusted? Do not stop at the metric. Pull out the original text of the answers scored wrong and scored right and look. What do the wrong answers look like? Messy wrongness looks like a normal capability boundary. Tidy wrongness looks like the scorer or the gold answer itself being sick. All five bugs of Chapter 7 were this question's prey. You will see shortly that this question can dig deeper than the harness.

Question two, what does the data look like? Open the problem text itself. How many kinds of problem are there? Independent of each other, or batch variants from one mold? Your CI was computed under the assumption "samples are independent," and what that assumption is worth you only know after seeing the raw data.

Question three, where is the win concentrated? Spread the gap out by problem and by slice. Is the win spread evenly, or concentrated in a small handful of problems? Concentration is not a crime by itself, but it means your conclusion hangs on the quality of that handful, and they deserve a separate interrogation.

All three are jobs AI can be handed. How you dispatch decides the quality. Do not ask "is this +18 credible," that hands the judge's seat to sycophancy, the tendency to flatter in the direction of the question, which Chapter 11 will dissect. Dispatch mechanical work. "List the original text of every frontier wrong answer, the matching gold answer, and the numerical relation between the two, ordered by problem id." Let it sit as a juror and lay out the facts. "Does this count as evidence," you rule.

Before the interrogation begins, one more thing has to be set up, a reporting discipline. Report the preregistered numbers as is. Report the post-hoc breakdowns the interrogation produces side by side, labeled "post-hoc." At no time may a post-hoc number replace a preregistered one. This discipline must stand before the work starts, or there is no line between "interrogate" and "interrogate until I like it."

8.3 Spine update · the twenty-four hours from +18 to −2.3

Now I run the three questions in front of you. Every number below can be recomputed in the smol-army repo (scripts/audit_math.py, matching the 2026-07-25 post-hoc audit entry in docs/CHANGES.md). The audit itself is outside the preregistered scope and is labeled post-hoc by the discipline.

Question one, executed. I had AI pull out every frontier wrong answer on math, 49 in all. The first pattern surfaced on the spot. All 49 wrong answers fell in math-100 to math-149, one continuous stretch of the 150 problems. The second pattern sent a chill down my back. Every wrong answer was exactly 4 times the gold answer, with a percent sign attached. 49 times, no exception. A model that "cannot do the problem" does not look like this. It was doing a different problem, with extreme consistency.

Question two, executed. Open the problem text. math-100 to 149 are 50 variants of one GSM-Symbolic probability template. That benchmark generates different versions of the same problem from symbolic templates, built to detect fragile mathematical reasoning and data contamination (Mirzadeh et al., Apple, 2024, arXiv:2410.05229). The question reads "how much more likely...(as a percentage)". That English has two readings. The gold answer reads it as the absolute percentage-point difference (p₁ − p₂). Frontier answered the relative increase every time ((p₁ − p₂)/p₂). The base probability in this template is 1/4 in every variant, so the relative reading is always exactly 4 times the absolute one. All 49 "wrong answers" are 4 times with a percent sign, and that is the whole solution, the question itself is ambiguous, the math is not wrong. On these 50 problems frontier matched the gold answer's reading on only 1. It had firmly chosen the other, perfectly defensible, English reading.

Question three, executed. The win is entirely concentrated on the ambiguous template. On these 50 problems, army vote scored 0.61, self-consistency 0.23, frontier 0.02. The distribution of readings of one ambiguous English sentence across three model lineages happened to tip the majority toward the gold answer's side. Multi-lineage voting really does have an advantage, but the advantage is in guessing the reading. Reasoning gets no credit.

Remove the ambiguous template and recompute. Frontier 1.000, the army 0.977, self-consistency 0.983. The paired bootstrap gives the army −2.3 percentage points, CI [−4.0, −0.7], direction reversed. There is one more quiet but important comparison. On the clean subset the army's 0.977 against self-consistency's 0.983, teaming is about equal to multi-sampling. The steelman arm that Chapter 6 nearly left out earned its whole wage right here.

The theme sentence of Chapter 7's five bugs was "bugs zero out the strongest arm first." The sixth hid deeper, in the data's gold answers. The harness was innocent, the code all correct. This bug zeroed out nobody either. It handed me 18 points. A bug that zeroes an arm, you will run into sooner or later in an error message. A bug that hands you points, only the interrogation catches.

Let me say the ugly part here first. This round of post-hoc breakdown is itself a new batch of numbers, and the same discipline applies. The Chapter 10 red team will come back for it, and will find something.

8.4 150 problems that are not 150 problems

Question two had one more aggravating finding. Clicking through the problem text family by family, the 150 math problems are really only about 3 template families, each with 50 variants in sequence. That family count was clicked out problem by problem, not estimated backward from variance, and "about" is there only because the family boundaries were drawn by eye. The sampling script took the first 150 rows of the dataset without shuffling.

The damage to the statistics is structural. The 50 variants of one template are highly correlated. Template unambiguous, the whole family is right together. Template ambiguous, the whole family is wrong together. The preregistered bootstrap resampled them as 150 independent samples, so the CI it computed is overconfident. That respectable narrow interval [+12.7, +24.2] bought its narrowness with the assumption "problems are independent." The effective sample size does not reach 150. It is on the order of those roughly 3 template families that were clicked out.

This pit goes on my own account. Chapter 5 chose a "contamination-resistant variant set" to guard against memorized problems, but a variant set is by nature copies of a few templates. It blocked one contamination and introduced one correlation. Not shuffling at sampling time pushed that correlation to its maximum. The Chapter 6 preregistration locks in motive. It cannot lock in ignorance. Locking the criteria first guarantees I cannot pick data afterward. It does not guarantee the data I picked beforehand is healthy. For this class of error, the only detection mechanism is interrogation question two, seeing with your own eyes what the raw data looks like.

Now produce the deliverable by the reporting discipline. In the table, pp means percentage points.

Task family	Preregistered result (reported as is)	Post-hoc breakdown (labeled post-hoc)
math	Army vote +18.2pp [+12.7, +24.2], army_ahead	After removing the ambiguous template, the army −2.3pp [−4.0, −0.7], direction reversed; the army 0.977 ≈ self-consistency (SC) 0.983; this family has only ~3 template families, the preregistered CI is overconfident
mmlu_pro	The army −15pp [−21.3, −8.7], army_behind	No overturn. The SC arm −18pp, the fault lies in the small models' knowledge base, not in teaming
code	The army 94% vs frontier 96%, CI crossing zero; the army costs 1/5 of frontier	No overturn. The SC arm likewise level

Last, H goes through the interrogation. The preregistered falsification conditions were two. (a) The army trails by more than 5 percentage points on all three families. Not triggered, code's CI crosses zero. (b) The army catches up only at more than 2× the cost. Not triggered either, the army's bill is a fifth of frontier's on code, under half on mmlu, about equal to frontier on math, and no family bought a tie with money. So H was not falsified, but neither did it hold across the board.

The landing point is the question Chapter 5 planted, which tasks have a shot. Knowledge Q&A has none, teaming cannot rescue a weak knowledge base. On math, trust neither side, this exam has to be reissued first, with a template-shuffled problem set that has enough families. Code has a shot, and that is the sturdiest sentence in the whole case.

Read strictly by the preregistration, a gap has to be within ±2pp to count as a tie, and code has only 100 problems, so the interval cannot be squeezed that narrow. It can only count as "undecided."

As a decision-grade conclusion, directionally no frontier advantage is visible, and the bill is hard. How that sentence is written into a paper's limitations, and into the first line of the CTO memo, is Chapter 9's business.

The two readings side by side are the table below. They do not contradict each other. They answer two different questions.

Reading	code	mmlu_pro	math
Preregistered reading, ε=2pp, per family	Undecided, CI crosses zero	15 percentage points behind, a clean loss	The "overtake" died in the interrogation, the problem set has to be reissued
Decision-grade reading	Directionally no frontier advantage visible, at a fifth of the price	No shot	Trust neither side

8.5 Subplot mirror · FAIL gets interrogated too

The persona study's verdict is in. All three criteria failed outright, the preregistered falsification shape was reached, and "persona can replace real interviews" is dead. The full table of numbers has to wait for Chapter 12. Here it first fills in one lesson for this chapter, when "like a real person" counts as evidence, and why a falsified conclusion gets interrogated all the same. Readers who care only about the spine case can skip to the next section and take one sentence with them, the interrogation does not depend on direction.

The first thing takes one sentence. A single answer that "sounds like" a person never counts as evidence. Fluency is the model's default property, with zero correlation to distributional fidelity (the first item on the list when Chapter 11 does the subplot's case review, the fluency illusion). The only thing that counts as evidence is a match at the distribution level against criteria locked in beforehand. If the distribution distances for all six subgroups had cleared the thresholds, variance had not collapsed, and the cross-slices had all been green, at that moment "like a real person" would rise to evidence, and the scope of that rise would reach only as far as the question bank's domain. The persona case is a long way from that moment.

The second thing is the point of the mirror. For this book, FAIL is a desirable result. It confirms Chapter 11's average face prediction beautifully, that is, persona answers are the averaged-out result of a crowd of people smoothed together, more homogeneous than real people, and narratively you could not ask for better. By the symmetry discipline, the more desirable, the more it gets interrogated. Walk the three questions. The scorer, all three criteria are computable statistics, no LLM judge, program rerunnable. The data, 10,800 interviews, zero invalid answers. Where is the loss concentrated, and the answer is that it is not. All six subgroups fail, variance collapsed on eight in ten clean questions, and no slice can carry the blame alone. The more spread out the loss, the sturdier the falsification. Only after this pass does FAIL qualify for the Chapter 9 deliverables.

Only one ambiguity remains. In the contamination test, 10 of 15 questions were flagged, the answer distributions on the original question and the reworded one clearly disagree. Two readings. Persona memorized the original questions, or it is highly sensitive to wording (a close relative of sycophancy drift, and sycophancy drift is the sycophancy described earlier). The single-source design cannot tell these apart. Single-source means the real-interview arm was cut before the run started, leaving only the public question bank as a comparison, and the cost of cutting the real arm is booked in Chapter 12. The shape of the interpretation discipline here is do not force a single reading. Report carrying both readings, and check whether the conclusion stands under each. If it is memorization, what was memorized is the real people's question bank, and the effect can only push persona's distribution toward the real people's side, which makes FAIL only more conservative and cannot rescue persona. If it is wording sensitivity, that is itself another kind of distortion, and it supports "cannot replace" just the same. The falsification survives under both readings. Forcing one interpretation would be the real defect.

8.6 Lessons stolen from debugging

When a result surprises you, suspect your own code first. That is your muscle memory. Three lessons move over as is, and below I look only at where they break when moved onto interpretation.

Lesson one, an unexpected victory, suspect the scorer first, the world second. "'select' Isn't Broken" (The Pragmatic Programmer, first edition Tip 26, twentieth anniversary edition Tip #33). When you think the compiler is broken, it is almost always your own code that is broken. The interpretation version is identical. The data tells you "the frontier model cannot do grade-school probability," and there are two explanations on the table. The world turned over, or your gold answer is wrong. The second is boring, but it is cheap and common. The size of the surprise should be proportional to the strength of your suspicion. Most people make it proportional to the strength of their excitement. Where it breaks, in code, suspecting yourself has a stack trace to help you locate it. Here, suspecting yourself means pulling out the original text of the answers and looking with your eyes, so interrogation question one must be written into the process.

Lesson two, a green light proves the tests passed, not that the tests are right. Row 5 of the transfer map says verification infrastructure sets the radius of letting go. This chapter adds its dark side. When the infrastructure itself is wrong it raises no alarm, and the whole pipeline runs green all the way to the wrong answer. Chapter 7's five bugs were all the harness's fault, and tests and the ledger could catch them. The sixth bug was in the gold answers. Every line of the harness was right, every point was scored correctly, and what was wrong was the definition of "right." So the interrogation has to go all the way down to the problem text itself. Locking the criteria first locks in your degrees of freedom. It cannot lock out the error buried in the data.

Lesson three, excitement is the least trustworthy gauge. Row 3 of the transfer map, developers rated themselves 20% faster and measured 19% slower, self-perception decoupled from fact, calibration comes from measurement (Chapter 2 covered it, not repeated). The interpretation step's counterpart is the Chapter 11 law, the more desirable the result, the stronger the sense of mastery, the less checking. The brake rule's trigger is written as "wanting to announce" because the self-perception of that moment is the least trustworthy gauge reading in the whole workflow.

8.7 Swap in your project

Dig out your most recent "want to announce" result. Last week's experiment, that good-looking curve, the p-value that finally came out significant, any of them. Give it the interrogation it missed, budget two hours.

Write down the announcement sentence. The sentence you originally wanted to say out loud, written down without changing a word, and pinned beside you. When the interrogation ends it is either alive or dead. First give the interrogation an autopsy subject;
Question one. Pull out the original text of the key wrong and right examples that support the sentence (dispatch it to AI, with none of your expectations in the brief, template in the appendix). Look for a pattern. Messy wrongness or tidy wrongness?
Question two. Count your effective sample size, and do one thing based on the count. The counting is short. First write down the n in your report, then ask "how many groups among these n share one source," such as one template, one batch of cells, one crawl, one surveyed institution. Whatever shares a source counts as one independent unit.

Counted, act on it. If the number of effective units is below the reported n, write the number of effective units into the body text (not a footnote), in the format "n=150, 3 independent units". Keep the original interval as is, and add one sentence, "this interval was computed under the assumption that problems are independent, and at this number of independent units that assumption does not hold."

Do not quietly swap in a wider interval. Recomputing the interval is next round's job. This round's move is to turn the assumption from implicit into explicit. This step goes only this far. What comes after (a mixed-effects model? average within groups first?) depends on your field's conventions and your reviewers' standards, and this book cannot give a general prescription. "Write n and the number of independent units side by side" holds in every field, and it is enough to let whoever reads your conclusion reprice it; 4. Question three. Spread the effect out by slice and look at concentration. The small handful of samples the effect concentrates in gets its own quality pass; 5. Side-by-side report. Report the numbers under the original criteria as is, report the breakdowns the interrogation produced side by side, labeled "post-hoc"; 6. Closing self-test. How is the announcement sentence now? Alive as is, alive after narrowing, dead. All three are qualified outputs, and the second is the most common. Only one is unqualified, sent out without the interrogation.

This chapter's three tools, the result interrogation checklist, the preregistered/post-hoc side-by-side report template, the surprise-result red flags, are in the appendix in full fillable form.

Want an agent to run it with you? Paste this to your AI assistant or coding agent:

Help me put a result through the interrogation, the Swap in your project of Chapter 8. I will first write you the announcement sentence without changing a word, and you paste it as is at the top of every reply.
Question one, you follow the instructions in Template 1 of docs/appendices/ch08-templates.md to pull out the original text of the key wrong and right examples, with none of my expectations in the brief,
and whether the wrongness is messy or tidy is for me to say after I have read the originals. Question two, the n in my report and the groups sharing a source are mine to count, you only write it into the body text in the format "n=how many, how many independent units",
and you may not quietly swap the interval. Question three, you spread the effect out by slice, and where it concentrates is my ruling. The ruling field is signed by a human, you may not sign on my behalf.
Finally the side-by-side report, the numbers under the original criteria as is, the interrogation breakdowns labeled "post-hoc". If any command errors, stop and show me the output.

8.8 Sober reminders

Not interrogating before you announce is the number one failure mode of this step, and it dies quietly. The +18 flows on into the abstract, the weekly report, the next round's budget request, gaining value at every stop, until some stranger runs the audit for you. You have only two choices, be the first auditor yourself, or wait for someone else to be.
Replacing preregistered numbers with post-hoc ones is worse than not interrogating, because it wears the clothes of rigor. −2.3 cannot replace +18 as "the result of this experiment." It can only stand beside it, labeled post-hoc, and only the next round's experiment is entitled to retest it confirmatorily. The moment you replace, the interrogation degrades into another round of picking data, Chapter 6's degrees of freedom and motive colluding again, with a different moment to strike.
Interrogating only the undesirable results is the most hidden kind. The interrogation is triggered by surprise and by stakes, and direction is not in the trigger conditions. The honest account for this case, mmlu's −15 followed the literature's expectations, and I really did ask it fewer questions than I asked the +18. That asymmetry goes into the limitations. When you run your own interrogation, check once, did the favorable results and the unfavorable ones go through the same checklist?
Still exploring. Tools that let AI audit results end to end on its own are iterating. As of this writing, dispatched mechanical interrogation, pulling examples, clustering, recomputing, can be handed off. The nose for "what to be suspicious of" is not there yet, and which item on the interrogation checklist to run first is still the human's job. The book's online case library tracks it. Generalizing this interrogation into a systematic attack procedure against any conclusion, attacking in turn the scorer, the data composition, the independence assumption, the cost basis, is Chapter 10's business.

8.9 The unfair advantage you now hold

For any result that excites you enough to want to announce it at once, you hold an interrogation checklist that runs before the announcement, three questions, two hours. In this chapter it interrogated a +18pp "victory" until the direction flipped, and later it did not spare its own output either (Chapter 10). The people without this checklist are, right now, ordering champagne on their own +18pp.