Chapter 6 · Turn an Idea into a Falsifiable Test Plan

Chapter companion

📋 Chapter 6 templates · 🗂 Template index · 💻 code/persona-panel

This chapter's ladder. At this step AI is stable at assistant level. Drafting the plan and enumerating confounders can both be handed off. Collaborator level is still exploring. AI drafts a plan fast, but the most expensive flaws in a plan, unfair baseline, basis drift, a criteria backdoor, are exactly the kind it does not flag itself. Reviewing and signing off the criteria is the hard step that stays in your hands.

Spine update. Hypothesis H is already sharpened, and this chapter turns it into a plan signed off before the run. By the end you will see that the soul of the whole plan is one control arm I nearly left out.

This chapter delivers. The seven items of a test plan, the confounder checklist, plus a plan red-team prompt.

6.1 The eleventh minute of the review meeting

Friday afternoon review meeting, and you are on the third page. For the past two weeks you evaluated a new retrieval design, it beats the production baseline by a clear margin, pretty curves, a clean conclusion. At the eleventh minute the engineer sitting in the corner looks up. "The recall parameters on the baseline, those are the defaults from when it shipped, right? How long did you tune your new design?"

Two weeks. Your new design ate two weeks of your careful tuning, the baseline got not one minute. The room goes quiet. That is the verdict. How much of the gain comes from the design itself and how much from those two weeks of tuning, your data cannot answer. Two weeks of experiment, void.

Change the skin and the same scene plays out after submission in one line from a reviewer, "the baseline appears undertuned," and in an investment committee in one question, "how were the comparison companies picked?" The structure does not change. When the fairness of a test is not designed before the run, the hole always gets found after the results are in, by someone else, for you.

Rerun two weeks and that bill can still be counted. The bigger loss is that you have already seen the numbers. On the rerun you know which configuration produces the good result and which slice favors you, and from then on every "reasonable technical choice" you make carries a direction of preference. The first run, you are the experimenter. The rerun, you are an interested party.

Chapter 3 named this step already. In an evaluation whose criteria were added later, the conclusion always "happens to" support the plan that was finished first. This chapter takes that sentence apart at the mechanism and hands you a workflow that keeps it from happening to you.

6.2 Degrees of freedom colluding with motive

First be clear who the enemy is. Set fraud aside. It is rare, shameful, and relatively easy to catch. The enemy is a collusion between two things that are each innocent on their own.

The first is degrees of freedom. Not the parameter you compute with in a statistics class. Here it means how many decisions can be argued either way. Between an idea and a number, any test has dozens of decisions. Which slice of the task, which metric, how much tuning budget the baseline gets, whether outliers are dropped, when to stop running, which runs count. Each decision on its own has a defensible case both ways.

The second is motive. You want a certain result to hold. That is no disgrace. Someone with no preference would never take on this problem at all. But preference means one thing. Once dozens of "either way is reasonable" decisions are deferred until after you see the data, what pulls them needs no saying.

Statisticians call this the "garden of forking paths" (proposed by Gelman and Loken in a 2013 working paper, formal version in American Scientist 2014, titled "The Statistical Crisis in Science"). No single act of wrongdoing is needed. Follow the result at every fork and the destination is a beautiful false conclusion. Feynman said the same thing in his 1974 Caltech commencement address. "The first principle is that you must not fool yourself, and you are the easiest person to fool." ("Cargo Cult Science," in Engineering and Science, 1974)

This collusion is not a new disease of the AI era. Statistical software turned significance into one click and p-hacking got industrialized, the old plot of row 6 of the transfer map. After psychology got burned by the replication crisis (it erupted around 2011, when a large share of published results failed to reproduce once a different group reran them), the prescription written was preregistration. Sample size, test method, exclusion rules, decision thresholds, all locked in, filed, and signed before the data is seen. In 2013 OSF launched, a platform that hosts preregistrations, and Registered Reports arrived, a submission format that reviews the plan before it reviews the results. Those two are where the prescription landed (registration for clinical trials came earlier, and its spread through the social sciences really did follow the crisis). The pharmacology is simple. Lock the degrees of freedom before the data is seen and motive can find no fork.

What this chapter delivers is the field-neutral version of the preregistration idea. You do not have to be an academic researcher, and you do not have to use any platform. There is one core action. Before the run, write the plan, the basis, and the criteria, above all "what counts as losing," into a file with a timestamp on it. After the run, that file is your conclusion's alibi.

6.3 Drafting collapsed in price, sign-off did not

The way AI rewrites this step is isomorphic to the way it rewrote the literature review. Three things really changed.

Drafting a plan collapsed in price. Writing a plan is dull, running an experiment is tempting, and most people skip the first and go straight to the second. That is how the person in section 6.1 started. Now, from your hypothesis to a structurally complete first draft of a plan, minutes. The draft will have errors, but "a draft you can attack" and "nothing at all" are two different projects.

Enumerating confounders became AI's strong suit. A confounder is another cause that could explain the same result. Enumerating them is a classic breadth problem. It tests how many ways you have seen an experiment crash, not how deep you think. You dry up at five confounders, it lists twenty without breathing. Fifteen of them do not apply, and among the remaining five there are usually one or two you genuinely had not thought of.

Counter-plan generation went from a luxury to a commodity. It used to take a senior collaborator to think for you, "if I wanted to overturn this conclusion, how would I design the experiment?" Now that sentence is a prompt and the cost is near zero. Section 6.5 freezes it into a template.

One thing did not change, and it is the vital point of this step. Reviewing and signing off the criteria. The note in this chapter's ladder comes from the Chapter 3 snapshot. Unfair baseline, basis drift, a criteria backdoor, these three kinds of flaw AI does not flag itself. The reason is no mystery. They raise no error, they do not stand out, each one on its own looks like a reasonable technical choice, and they tend to grow along your preference. A model inclined to go along with the user will not pick a fight with your criteria on its own. Drafting, enumerating, playing the contrarian, hand them off. The final wording of every criterion you read word by word, then sign. What you sign is "if it's wrong, I answer for it," and no model today can carry that sentence.

6.4 Lessons stolen from test-first

You know test-first (TDD), write the test before the code, watch it go red before you make it green. The three lessons below look only at where it carries over to research and where it breaks.

Lesson one, locking the criteria first is the research version of test-first. A test written afterward grows into the shape of the code, and this one has bitten you. It verifies "what the code actually does," not "what the code was supposed to do." Criteria work the same way. Criteria set after the run grow into the shape of the result, and wherever the result lands is exactly where the "reasonable threshold" gets drawn. Writing the criteria first is the only way to make criteria independent of the result, and the virtue of rigor is secondary. Where it breaks. A test written afterward at least still catches regressions. Criteria written afterward are worth nothing at all. They only endorse the result.

Lesson two, watch it go red first. A test that has never been red does not count when it is green. The research counterpart is the falsification rehearsal. Once the criteria are written, invent a concrete set of numbers and check whether they would really rule against you. If you cannot think of any set of results that would trigger "H is falsified," what you wrote is decoration, not criteria. Hypothesis H from Chapter 5 came with its falsification condition (trailing by more than 5 percentage points on all three task families, or catching up only at more than 2× the cost) exactly for this moment. It can go red. Where it breaks. Watching a test go red takes one run. Watching criteria go red takes imagination, and nobody runs it for you, which is why this step is the easiest one to skip.

Lesson three, verification infrastructure sets the radius of letting go. Row 5 of the transfer map, a team with solid test coverage dares to let an agent into core modules. The research counterpart is in the plan in front of you. Chapter 3 said the ticket into collaborator level is an executable standard of verification, and no model, however strong, can buy it for you. Without locked-in criteria you have no standing to talk about spot checks. That ticket gets printed in this chapter. In the next step, execution (Chapter 7), what lets you turn AI loose on a whole eval pipeline without watching every line is that every number ends up hitting the criteria this plan locked in. Trust is no help here.

6.5 The seven items of a plan

Below is a practice you can copy as is. A test plan that holds up under examination has seven items, and each one missing leaves a class of accident waiting for you.

# Test plan (signed before the run; after sign-off, only appended change-log entries, no edits)

1 Question: what this test has to answer. One sentence.
2 Hypothesis H: falsifiable statement with its scope. (The Chapter 5 output, copied as is.)
3 Arm design:
   - Main arm (your design):
   - Baseline arm (control): lock in the tuning budget the baseline gets, equal to the main arm.
   - Steelman arm: write down the sentence the strongest opponent would use
     to call your result an artifact, then name the arm built to block it.
4 Basis: one primary basis (how it is computed, down to the formula); sensitivity basis listed separately.
   Locked-in commitment: if the two bases reach opposite conclusions, report it honestly, no picking.
5 Criteria and falsification condition: what counts as a win, as a loss, as undecided,
   written down to "what number triggers it." Statistical test, repeat count,
   random seed, stopping rule, all set beforehand.
6 Contamination and confounder checklist: run every line (see the appendix); for lines you cannot clear,
   write a mitigation, or write it honestly into the limitations.
7 Filing: stamp a timestamp (git commit / an email to yourself or the team /
   OSF, AsPredicted and other preregistration platforms, both free to use as of 2026).

One sentence each on the fourth and fifth items first. The sensitivity basis is a second way of computing, used to check whether the conclusion changes when the computation changes. The stopping rule says how many runs count as done and under what condition you may stop early.

Of the seven, item 3 needs the most unpacking, above all the steelman arm inside it, the control arm that makes a result interpretable. The core of arm design is growing the control arm on the right enemy. Merely "having a control" is not enough. The most common crippled design compares only against "doing nothing" or a straw-man baseline, which proves your design beats inaction and proves nothing about it beating the cheapest alternative explanation. When you design the arms, ask yourself one question. If the result comes out as I want, what would the person who least wants this conclusion to hold say? That sentence of his has to be pinned by an arm built for it. In section 6.6 you will watch me nearly crash on exactly this.

The confounder checklist in item 6, the in-text version carries only the five most painful lines, with the full version in the appendix. One, is the baseline fair. Two, are the budgets on both sides measured on the same basis. Three, is there a cheaper explanation that could eat your effect, and does it have an arm of its own. Four, could the evaluation data have been "seen" by the model or by your process. Five, are the metric, the slice, and the stopping rule unique and set beforehand.

Last, the plan red-team prompt. Once the plan is written, before you sign, hand it to AI for one round of attack.

This is my test plan: [paste the full plan]

Your job is to overturn it, not to improve it. Assume you are the reviewer
who least wants this conclusion to hold.
1 List every degree of freedom in the plan that "can still be moved after the results are in";
2 For each one, say which conclusion it would favor if adjusted after the fact;
3 Give one cheapest alternative explanation that, with the arm design unchanged,
  would produce the same result;
4 Point out which criterion's wording leaves a backdoor (words like "as appropriate," "a reasonable range");
5 If you were to add one arm built to make trouble for my conclusion, which arm would you add and why.

Raise only problems specific enough to act on, no general methodology advice.

There is one iron rule of use. What an AI red team produces is a draft list. The ruling is not in its hands. Go line by line, what you adopt turns into changes to the plan, what you reject gets one line of reason, and that "rejection record" becomes your ammunition at the defense later. Remember the note in this chapter's ladder. It can find wording backdoors, but the most expensive class of flaw it may not flag, and what answers for that is still the checklist and you. Red-teaming your conclusions is Chapter 10's business. What gets red-teamed here is a plan that has not run yet.

The complete fillable versions of this chapter's three templates are in the appendix.

6.6 Spine update · I nearly left out the steelman arm

Now watch the seven items run. At the end of Chapter 5 I held hypothesis H. Under cost alignment, the gap between an open-source small-model army and a single frontier model across three task families is ≤ 2 percentage points, which counts as a tie. The three families are math and logic reasoning, on a contamination-resistant variant set, knowledge Q&A, on an MMLU-Pro subset, and code, on a HumanEval+ or LiveCodeBench slice. Team topology takes three forms, voting, debate, division of labor.

My first draft of the arm design had two arms, a single call to the frontier model as the baseline, the small-model army as the main arm, topology as the independent variable. It looked complete, and for a while I thought item 3 could be ticked.

What stopped me was the third line of the checklist, is there a cheaper explanation that could eat your effect? I sat with that line for a while, came up with nothing, and nearly wrote "none." Then, following the process, I went back to the Chapter 4 controversy map, to the skeptics' column. On the Smit 2024 paper card was a distillation I had written by hand two weeks earlier. Under default settings, debate cannot reliably beat old single-model tricks like strong prompting plus self-consistency. The Chen 2024 card said performance is non-monotonic in the number of calls. Rereading the two cards side by side, I finally saw where the skeptics' core objection lands. The sentence they hold onto is "the gain from teaming is only the gain from spending more sampling budget," far more precise than "teaming is useless."

Rehearse my two-arm design against that sentence. Say the army beats frontier and I announce "teaming works." A reviewer on the skeptics' side takes it apart in one line. "You gave the army a budget of ten calls and gave the single model one. Give the same budget to one small model sampling ten times and taking the majority, is the gap still there?" My plan had no defense against that sentence at all. The army winning might only prove "spending works," and I would misreport it as "teaming works."

So a third arm enters, a single small model doing self-consistency at the same budget, the self-consistency those paper cards were talking about. One small model, sampling repeatedly at exactly the army's budget and taking the majority. If the army beats the third arm and the third arm beats the baseline, the team structure itself contributes. If the army only matches the third arm, the "team" in the army is decoration and the gain comes entirely from sample count. Without this arm, however pretty the result, "teaming works" cannot be told apart from "spending works." That is the steelman arm of this plan.

Looking back, the way this arm was nearly lost is representative. I knew Smit 2024 perfectly well. I filled that paper card in myself. Whoever designs a plan stands by default on the side of his own hypothesis, while the steelman arm grows on the enemy's argument, and you do not spontaneously design for the enemy. What pulled me back was one line on a checklist, nothing to do with inspiration. A process does not depend on the state you are in that day, which is exactly why you want a process. This chapter's ladder says the most expensive flaws in a plan "are exactly the kind AI does not flag itself." In fairness, I did not flag it either.

Once the third arm stood, the words "same budget" turned around and forced me to lock in the cost basis, or budget alignment itself becomes one more degree of freedom to fiddle with afterward. I set the primary basis as dollars per query, with two sets of books, one at API list price, one at amortized local deployment. The two sets of books match two real situations enterprise readers are in, and the conclusions may differ.

The sensitivity basis is listed separately, accounted in compute. The parameters actually active in one model run, times the tokens it produces, is the unit of this ledger. This ledger has nothing to do with list price or discounts. It is cost at the physical level. One commitment is locked into the plan. If the two bases reach opposite conclusions, report it honestly, no picking. The reason for the ranking goes into the plan too. Chapter 9 writes a one-page memo for the CTO, and that kind of reader thinks in dollars, so dollars are the primary basis. The compute basis answers "does the conclusion still hold when list prices change?"

The criteria item, locked in line by line, preregistration style. The concrete task sets and slices for the three families, the random seed, the repeat count per configuration, significance by paired bootstrap (paired resampling, used to put an interval around the gap), the data contamination check as a step of the process (whether the evaluation set might appear in the model's training data, run family by family), and H's falsification condition filed as is. Trailing by more than 5 percentage points on all three task families, or catching up only at more than 2× the cost, and H dies. The verdict for the gray band is locked in first too. A single family trailing by 2 to 5 percentage points is neither a tie nor a falsification, ruled "undecided," reported honestly, no picking a side.

For a few parameters still hanging I give a provisional basis, and the three sign-off items are listed here.

The gap threshold ε, the tie line signed last chapter, stays at 2 percentage points.
The small-model size tier is set at "the smallest practical tier among open-source models in service today," total parameters ≤21B.
The frontier control is provisionally one of the mainstream flagships at the time of writing.

There are two reasons for setting the size tier there. One, quantized it runs on a single consumer GPU, which keeps the promise that readers can reproduce it. Two, and this matters more, it keeps the word "small model" honest. Team up 100B-class "open-source large models" to tie frontier and a win does not answer the original question. The army's concrete members, together with the frontier model, are deferred and locked just before the runs start in Chapter 7, since list prices and availability change month to month and locking too early is false precision. The locking itself goes into the plan's change log. It does not happen quietly.

Once the plan is final, stamp a timestamp and file it. It will be the first file in the Chapter 7 eval harness repo, and the harness may only implement it, never revise it. Honesty layering as usual, label whatever needs labeling. At this moment I hold a design and no data, so H remains still exploring. The only thing this plan guarantees is that whatever Chapters 7 and 8 produce, you can reconcile it against today's file.

6.7 Subplot · what does the persona study use as ground truth

Move the same step onto the subplot and the difficulty jumps a level.

Chapter 5 ground the persona study into a testable question. Does the answer distribution of a persona model on a public survey question bank agree with the distribution of the corresponding real population subgroup to a preset threshold, with no systematic stereotype drift. Now it is the test design's turn, and the first question stops you cold. That "distribution of the real population" in the criteria, where does it come from?

The spine case has it easy at this step. Math problems have gold answers, code has test cases, ground truth is ready-made, and all I have to design is how to use it. The persona study has no such luxury. "How a real person would answer" is itself something you have to pay to find out. There are two candidate answers, using a large public survey dataset as ready-made comparison, plus small-sample real-person calibration. Public data means long-running cross-national social attitude surveys like the World Values Survey, with the public question bank of Pew, a polling organization, as backup. Each road has pits of its own. Public survey data, for instance, is very likely already in the model's training corpus, which is the textbook case of the "has the data been seen" line on the confounder checklist.

WVS licensing (skip on first read)

The seventh wave of WVS is free for non-commercial research. The price is registration, a citation obligation, and no redistribution of raw data.

When ground truth is not ready-made, half the work of verification design is designing the ground truth itself. The full version of this plan, its pits and its criteria, is revealed in Chapter 12.

6.8 Swap in your project

In Chapter 5 you ground out your own hypothesis. Now give it a plan, budget one evening.

Write down the strongest opponent's sentence. "If the result comes out as you want, how would he call it an artifact?" Then check your arm design. Does that sentence have an arm built to pin it? If not, add the arm before you go on;
Write the criteria down to "what number counts as losing." Then rehearse the falsification, invent a concrete set of numbers, and confirm it really triggers "I lost." If it cannot trigger, rewrite until it can go red;
Rank the bases. One primary basis, down to the formula; the sensitivity basis listed separately; write down "opposite conclusions, report it honestly";
Run the confounder checklist (full version in the appendix). Do not force through lines you cannot clear, write a mitigation, or write it honestly into the limitations;
Run one round of the plan red-team prompt, rule on every line, change what you adopt, write the reason for what you reject;
Stamp a timestamp and file it. git commit, an email to yourself, the team wiki, any form will do as long as it cannot be altered.

Finish with one test. Hand the plan to a colleague or an advisor, let them read only the plan, and have them guess "what result would make you admit you lost." If they cannot guess, the criteria are not locked in yet, go back to step 2.

Want an agent to run it with you? Paste this to your AI assistant or coding agent:

Help me run the Swap in your project of Chapter 6. Build the test plan of Template 1 in docs/appendices/ch06-templates.md as an empty form, and ask me through the seven sections one by one,
with hypothesis H copied as is from the Chapter 5 output. The strongest opponent's sentence is mine to write, and you check one thing only, whether the arm design has an arm built to pin it.
After I write the criteria down to "what number counts as losing," you invent a concrete set of numbers and run the falsification rehearsal, confirming it really triggers "I lost,"
and if it cannot trigger, send it back to me to rewrite. The confounder checklist is mine to run line by line, and you record the lines I cannot clear into the limitations. Then open a separate session, paste only the plan
and not my expectations, run the plan red-team prompt of Template 3, and the charges that come back are mine to rule on one by one. At the end, remind me to git commit for the timestamp.
If any command errors, stop and show me the output.

6.9 Sober reminders

Adjusting criteria after the fact is the number one failure mode at this step, and it always happens under a respectable name, such as "only once it was running did I find the original criteria unreasonable." Set the rule in advance. Criteria can change, but only by appending a change-log entry that states the change and the reason. Any conclusion under criteria changed after the results were seen is downgraded to exploratory, a lead and not a conclusion, and getting confirmatory status back means rerunning.
A missing control arm typically shows up as controlling against the wrong enemy. There usually is a control in the plan, it just compares against "doing nothing" or a straw man instead of against the cheapest alternative explanation. There is one test. Can every one of your main opponents' arguments point at some arm in the plan?
Reporting sensitivity analysis selectively is the basis version of "picking data afterward." Both bases went into the plan, so both go into the report. Presenting only the flattering one is the same act as deleting the ugly data points, just dressed in the clothes of sensitivity analysis.
Preregistration does not forbid exploration. Exploratory analysis outside the plan is free to do and is often the source of the next hypothesis, as long as it comes on stage labeled "exploratory" and does not wear the clothes of a confirmatory conclusion.
Still exploring. Tools that let AI draft a whole test plan end to end are iterating fast, and the output is already usable as a first draft. As of this writing there is no evidence that models reliably catch unfair baselines and criteria backdoors in their own drafts, so before criteria get signed as a contract, the step where a human reads them word by word cannot be skipped. The book's online case library tracks progress.

6.10 The unfair advantage you now hold

Before you run anything, you hold a plan with "what counts as losing" locked in. Three weeks later, however ugly the result, your conclusion is natively immune to the charge of "picking data afterward." Most people do not discover they need this file until the eleventh minute of the review meeting.