Chapter 7 · Execution

Chapter companion

📋 Chapter 7 templates · 🗂 Template index

This chapter's ladder. Execution is the step AI climbs highest on among the seven. Collaborator level is solid, and the door to autonomous researcher level opens a crack here, which counts as local autonomy. The reason is not mysterious. This step has cheap ground truth. Tests go red, the ledger adds up, a number can be recomputed, errors do not stay hidden long. The ladder carries one footnote. Building the pipeline and writing the tests can both be handed off. The one exception, "the harness itself is wrong," is the kind AI does not flag, and all five bugs in this chapter are that kind.

Spine update. The plan was signed off, and it was time to spend real money. I thought the pilot would be out in twenty minutes. Three hours later I had caught five bugs, and every one of them pinned the strongest arm to the floor at zero first.

This chapter delivers. The four pillars of a harness, a minimal checklist, and a pilot process that spends 3-5% of the total budget.

7.1 A twenty-minute plan, a three-hour reality

The day of the run my confidence had grounds. The Chapter 6 plan went into the repo ahead of any result. The harness, the homemade scaffolding the experiments run in, holds the data loading, the model calls, the scoring and the accounting, about four hundred lines, a model client, a ledger, tasks, topologies, a batch runner, plus two small pieces for reports and statistics, with 19 unit tests, all green. Under mock mode the whole chain had run end to end more than once. What was left, swap the fake model for the real API and spend a dollar or so on a 20-problem pilot, a small-scale trial run. I glanced at the clock and put twenty minutes on it.

The first real call, a 400 error. The plan says the frontier arm runs at temperature 0, that is, no random sampling, always take the most likely token. But GPT-5.6-terra, locked in just before the run, is a reasoning model, one that generates a stretch of internal thinking before it answers, and its API refuses a custom temperature and wants the token ceiling passed under a different parameter name. The code itself was not wrong. What was wrong was the code's assumptions about the world, and mock mode had faithfully mocked the whole world away.

Fixed, run again. This time it started, and halfway through the process died. qwen3.5-9b in the army is also a reasoning model, a 1024 token ceiling was not enough for it to finish thinking, the reply cut off midway came back with content null, my code used null as a string, and it crashed on the spot. The way it crashed was more embarrassing. My first batch runner was a naive serial loop. One problem throws, the whole run lies down, and the problems already finished have to be fished back by the resume mechanism. The mechanism was there, thankfully, but every crash cost me one more restart and one more stretch of watching it. The third time was OpenRouter, the middle layer that forwards requests to the various model providers, hiccupping. Status code 200, response body not JSON. Parsing blew up and the whole run lay down again.

By now I had to admit the crashes kept landing in the same place. The design was brittle. Why should one problem's failure put the whole experiment to death? The fix went into the skeleton of the batch runner. Tasks are flattened into small independent pieces, a single problem's failure goes onto a failed list only, and the run carries on. While I was there I opened concurrency to 32 lanes, since the tasks were flat anyway. That one change is the direct reason the full run later finished in a single pass. Serial, the full run would take about fifteen hours, and I would have to pray that not one bug showed up in those fifteen hours.

The pilot finally finished and the ledger stopped at $1.23. Then I looked at the numbers, and my blood pressure went higher than at any of the crashes. Frontier on math, 0. Code problems across every arm, all 0. Two new bugs. GSM-Symbolic, the contamination-resistant variant set the plan uses for the math family, has answers that often carry a % suffix. My scorer ran float parsing on them, a parse failure scored 0, and frontier happened to answer in the tidiest format of all, so it was wiped out. Scoring the code problems runs evalplus, an off-the-shelf test suite for code problems, whose tests import numpy, and this project's dedicated Python environment (venv) had no numpy, so the grader quietly recorded 0 for every problem in every arm.

Three hours. Five bugs. The part that stings most in review, not one bug came from "I can't write code," and not one of the 19 tests could stop them. My twenty minutes estimated the time for "the code is not wrong." It did not estimate the time for "the world does not follow my assumptions," and the real work of the execution step lives in the second kind of time. Readers who do not write code should not be shut out by this section. Swap the pipeline for your own data-gathering and accounting process, and all five bugs have counterparts there.

7.2 The step where it is easiest to look like you are working

Good news first. In the seven-step workflow, execution carries the thickest transfer dividend. Ten years of accumulated coding practice moves over almost as is, and AI happens to be most skilled at this step too. Most of my harness was written by AI, and going from an empty directory to the whole chain running under mock mode took hours. Three years ago this was one person's week or two.

The bad news lives next door. Execution is also the step where "looking like work" most easily passes for work. Logs scroll, the progress bar moves, the bill climbs, all of it the appearance of work, none of it proof that trustworthy numbers are coming out. The numpy bug is the perfect counterexample. With the dependency missing, the pipeline still ran diligently, still threw zero errors, and the code still looked elegant. Every number it produced was garbage.

The core proposition of this step follows. The credibility of execution comes from the structure of the harness, and the structure has to leave errors nowhere to hide. "Looks right" is an aesthetic judgment, and AI's code almost always looks right. "Cannot hide" is a structural judgment. Does a crash lose data, is there an account for the spending, can the criteria still be edited, can the scoring be replayed. Section 7.4 breaks this structure into four pillars.

The division line gets drawn here too. Writing the pipeline, writing the tests, writing the scaffolding, hand them off. But the kind it does not flag has to be seen clearly, and that kind is the harness itself being wrong. It does not know what terra's API contract looks like, does not know your venv is missing numpy, does not know this batch of answers carries a %. These bugs live on the seam between the code and the world, and the seam sits outside the view of whoever writes the code. Chapter 6 said the most expensive flaws in a plan are the kind AI does not flag. The execution step's counterpart, a green test suite only proves the code matches the world you defined, and the pilot is the only probe that reaches into the world's side.

7.3 Lessons stolen from pair programming

Pairing, tests, tracer bullets, all three lessons of this step are your daily routine, so I will only cover where they carry over to a research experiment and where they break.

Lesson one, pair on execution. The keyboard changes hands, AI writes, you review, and the bottleneck of Row 1 of the transfer map lands right here in the execution step. "Review" has to be layered, and you do not read every line along with it. Boilerplate (retries, writing to disk, command-line arguments) gets a glance and a pass. Lines that touch the criteria, the scoring function, the ledger's money formula, the resume key covered in the next section, get read word by word. The reason is cold. An error in these few places is directly an error in the number you sign off on two weeks later, and the program is the small matter. Seven tenths of the human hours I spent on the harness went into these three places, and in hindsight not one of them was wasted. Two of the five bugs, the % and the null, were caught right around this review line through scoring and parsing. Where it breaks. Pairing on production code reviews logic. Here you review the definition of right and wrong itself, and when the scoring function is wrong the tests stay green.

Lesson two, verification infrastructure sets the radius of letting go. Chapter 6's lesson three already established this, so here I only add the execution step's increment. Ground truth at this step is unusually cheap. Tests go red or green in seconds, every ledger line adds up, and scoring can be replayed offline. That is why collaborator level is real here and local autonomy is worth discussing. The same model downgrades on purpose at the interpretation step of Chapter 8, where there is no cheap ground truth. Section 7.1 already demonstrated the other side. The infrastructure covers only the world you defined, and all five bugs lived on seams the 19 tests could not reach. Whether the infrastructure itself can be trusted waits for the pilot to test.

Lesson three, fire the tracer bullet first. Walk the whole chain with mock first, then spend real money. This is the practice Chapter 2 stole from The Pragmatic Programmer, and at the execution step it lands as one switch, --mock. A fake model that always answers 42 with a fixed token count walks data loading, team topology, scoring, writing to disk and reporting through the whole chain offline. It cleared out a whole class of bugs for free, data formats, voting logic, the report pipeline, so that the first time I spent real money only the seams were left to blow up. Its boundary deserves an honest label too. Mock by definition cannot clear the bugs on the world's side, and all five are proof of what slipped through. So the tracer bullet is fired twice. Mock is the first shot, free, clearing structural bugs. The pilot is the second, a few percent of the total budget, clearing seam bugs. Skip the second shot and go straight to the full run, and the five bugs will ride along through your entire budget.

7.4 The four pillars of a harness

Four pillars, each one matching a class of "places errors hide," a few dozen lines of code all together, and the highest-value few dozen lines of the execution step.

Pillar one, a resume key. Every result written to disk carries a key that uniquely rebuilds it. Mine is a four-tuple (task family, problem, arm, seed). The run starts by scanning the result file for finished keys and striking them out of the task pool. The effect is that a crash mid-run drops from an accident to a lossless event. Problems already run are not rerun, money already spent is not spent again. Two things come with it. Tasks must be flattened into small independent pieces, because the mid-state of a long serial chain cannot be expressed as a key, and that was the root of my crashes in section 7.1. A single piece's failure goes onto the failed list and may not take down the run, and the resume retries it naturally. Two lines are enough to show it.

done = {(r["family"], r["item_id"], r["arm"], r["seed"]) for r in results}
jobs = [j for j in all_jobs if key(j) not in done]

Pillar two, an append-only ledger with a budget hard cap. Every API call writes one JSONL line, a text format with one record per line, recording the timestamp, the model, tokens in and out, and dollars. Append only, never edited. The moment cumulative spending passes the $35 hard cap, throw, and the whole run stops. This pillar later paid two dividends. First, during the pilot a batch of self-consistency result lines had been run with the wrong model, before the probe selection rule covered in the next section was executed. Cleaning up, the lines in the result file were deleted and rerun, and not one ledger line moved. The principle is one sentence. Results record the current understanding, and when the understanding is corrected they should be rewritten. The ledger records history, and history does not accept edits. Second, "how much did this experiment actually cost" went from a number recalled by impression to the sum of a column. Chapter 6's sign-off promise on cost basis is redeemed right in these JSONL lines.

Pillar three, mock mode. Lesson three in concrete form. Besides serving as the tracer bullet, it is the free regression test for every later change. Changed the topology logic? Mock reverifies the whole chain in three minutes. One calibration mark from experience, building mock cost a few dozen minutes and paid back an order of magnitude on the first afternoon.

Pillar four, the criteria enter the repo ahead of the results. prereg.md goes into the repo ahead of any result. The harness may only implement it, never revise it. Every change after the run starts may only be appended to the change log, with the date, what changed and why. The bug where terra refused temperature 0 is the textbook disposition under this discipline. The plan's assumption about the world was wrong, the plan itself does not change, the difference goes into the change log, the harness runs by reality, and when Chapter 8 reconciles, every entry is out in the open. Why not just fix the plan? "Just this once" has no stopping condition, and the change log does. The more changes there are, the uglier that file looks on its own.

The most beautiful execution of this discipline happened on the subplot, and it is worth a digression. Readers who care only about the spine case can skip to the fifth pillar. In the preregistration of the persona survey (its execution story opens up in Chapter 12), I had labeled Q120 of the WVS question bank a "competition" question from memory. While writing the data loader, following the step "a first-hand document you cite must be checked," I opened the official WVS-7 questionnaire. Q120 is the "risk of being held to account for taking a bribe" question, and the competition question is Q109. Same disposition. The locked preregistration does not change, the question number stays, the official wording replaces mine, and the difference goes into the change log. The host of this bug is worth recording. The code was not wrong, AI was not wrong, my own memory was wrong. It turned "checking" from a virtue into a step, and a step catches its own designer along with everyone else.

There is a fifth pillar. It was too cheap to list on its own, and it earned its name during the bug-catching day in the next section. Write the original text of the answers to disk, and let the score be a derived column. The result file stores the extracted original answer text, not only the score. So when the scorer's bug was fixed, one replay script recomputed every score at zero cost. The grade of this pillar gets tested once more in Chapter 10.

How readers who do not write code use these pillars

The pillars are about structure, not code. Doing interviews, running questionnaires, digging through archives, checking reports, all of them need the same pillars with different materials.

A resume key. Every time you finish one minimal unit (one interview, one document, one report), write it down and number it, and after an interruption pick up from the number, redoing nothing and missing nothing. Same precondition, break the work into small independent pieces.
An append-only ledger with a budget hard cap. Record the money and the hours spent on a sheet that only grows and never changes, lock in a ceiling before you start, and stop at the line, with no "just a little more."
Mock mode. Walk the whole process on one set of fake material, from gathering to scoring to aggregating, confirm every step connects, and only then touch the real material. After that, every time you change the process, walk it again on the fake material first.
Criteria leave a trace ahead of results. Once the criteria are written, send yourself or a colleague a timestamped email, or save them into a document with version history. Later changes append a note, they do not overwrite.
Raw records go to disk. Keep the recordings, the source excerpts, the screenshots, and derive the scoring and coding from them. When the scoring rule changes, recompute from them instead of collecting again.

7.5 Spine update · the five bugs go for the strongest arm

Now an autopsy report for those three hours. Five bugs, each one by where it landed.

#	Bug	The seam it hid in	Symptom	Who zeroes out first
1	The reasoning model's API contract	plan ↔ API	temperature 0 refused, a 400 error	The frontier arm, not one problem run
2	Truncated thinking returns null	token budget ↔ reasoning model	content is null, the harness crashes	The strongest reasoning member of the army
3	A 200 status code with a non-JSON response	provider ↔ client	the status says success, the body is garbage	Whoever it hits (the only one whose landing point is random)
4	The % suffix scoring miscarriage	scorer ↔ data	correct answers carrying % scored 0	Frontier's math all zero
5	The missing numpy dependency	grader ↔ runtime environment	code scoring silently all 0	The code set zeroed out across every arm

Read the landing column straight down and the pattern shows itself. Bugs zero out the strongest arm first. The mechanism is not mysterious. The strongest arm uses a reasoning model, whose contract is the most special and whose thinking burns the most tokens. Its answers are also the tidiest, carrying units and percent signs, feeding exactly into the parser's most fragile path. The grader it depends on is the heaviest too, since it has to actually run tests, so its dependency chain is the longest. The more capable it is, the more seams it has, and the higher the density of bugs. The weak arm is safer. A model that was going to answer wrong anyway cannot be wronged by much.

The corollary is ugly. If your harness has bugs and you did not catch them, the most likely direction of bias in your readings is systematically wronging the strongest contestant. Then you get a surprising "the weak beat the strong" result, exactly the kind you wanted most, and you write it into the report full of excitement. What saved me this time was absurdity. Frontier scoring 0 on math cannot be true. But bugs make no promise of absurdity. A bug that chews a whole arm down to zero screams on its own. A bug that only chews off 3 points says nothing, and 3 points is already enough to flip a verdict at ε=2pp, that is, two percentage points. So the alarm has to come from process. Go through the pilot numbers arm by arm, plus one dedicated question. Does the strongest arm's performance make sense?

The repair process verified the fifth pillar along the way. The fix for the two scoring bugs (the % and numpy) was to change the scorer and run the replay script, rescoring every historical answer for free. The 38 null answers from the truncation era were deleted and rerun.

The pilot has one more piece of preregistered business. The change log of the Chapter 6 plan locked in a rule, the self-consistency arm uses "the strongest army member on a 20-problem probe," a harder control against the objection that teaming is just multi-sampling. The probe readings came in. Of the three candidates gpt-oss-20b was highest at 0.867, ministral-14b 0.850, qwen3.5-9b 0.533. The rule executed, the SC arm locked to gpt-oss-20b, and the readings and the choice both went into the change log.

The probe brought back something else. qwen's 0.533 incidentally exposed its chronic illness. With too low a token ceiling it thinks past the timeout, and under a 4096 ceiling it gave no answer on 28 of the 60 probe problems. The ceiling went to 8192, and one rule was locked in on the spot, any further timeout gets reported honestly as a property of that member, with no more patching. Even "when do you stop tuning" has to be agreed in advance.

Then the full run. Five bugs fixed, the mock regression run once more, 32 lanes of concurrency started. Three numbers at the end, 3,580 result lines, $5.58 in the ledger (hard cap $35), 0 failures. Three hours of crashing bought a full run that passed in one go, and that trade comes out ahead however you count it. The full run's numbers also held a surprise that made my heart race, and this chapter's five bugs are the whole reason I later dared to face it in the right posture. Chapter 8 opens it up.

7.6 Swap in your project

In Chapter 6 you gave your hypothesis a plan. Before you start, give it a harness, and accept it with this minimal checklist, where every item matches a class of real accident from this chapter. Budget half a day.

Fire the mock tracer bullet first. Build a fake data source or fake model and walk the whole chain of "input → processing → scoring → writing to disk → aggregation" offline. Every break you hit is a break you would otherwise have paid real money to find;
Run a crash drill. Halfway through a real run (or a mock run), kill it by hand and restart. Check that results are neither duplicated nor missing, count the lines, reconcile the ledger. If you cannot, add the resume key before you start. The version for people who do not write code, close the laptop halfway through and see whether the records alone let you resume the next day;
Blow the fuse once. Set the budget hard cap to nearly zero, run, and watch it abort with your own eyes. A fuse that has never blown is not a fuse. Chapter 6's discipline of "watch it go red first" applies to infrastructure too. The version for people who do not write code, set the ceiling at a number you will hit within the hour and see whether you actually stop;
Check the timestamps. The commit of the criteria file must be earlier than the first line of results. If it is later, you already know which chapter to go back to;
Plant a poison pill. Deliberately drop in a task that must fail, and confirm it only goes onto the failed list and does not take down the run. The version for people who do not write code, mix in a piece of material you know is bad, say a fabricated citation, and see whether the process flags it on its own;
Run the pilot on 3-5% of the total budget. Look at the numbers arm by arm and configuration by configuration, spot-check the original text of the answers behind every 0 and every perfect score (the faithful extract, meaning the answer pulled out of the model's reply as is, without passing through the scorer), and finish with that sentence. Does the strongest configuration's performance make sense?

This chapter's checklist, design patterns and audit prompt are in the appendix in full fillable form.

Want an agent to run it with you? Paste the block below into Claude Code, Codex, or any coding agent:

In code/smol-army of the research-rewritten repo, help me run Chapter 7's Swap in your project. First run
uv run python -m smol_army.run --mock to fire a mock tracer bullet, and show me the whole chain's output verbatim. Then run the crash drill,
kill the mock run halfway and restart it, count the lines, reconcile the ledger, and tell me whether results are duplicated or missing. Set the budget hard cap
in config/run.toml to nearly zero and run once so I can watch the fuse blow with my own eyes, then change it back. Check the timestamps again, is the commit of docs/prereg.md earlier than
the first line of results in results/. All of that is practice on the book's harness. My own project's harness I accept myself, against
the checklist of Template 1 in docs/appendices/ch07-templates.md, with Template 3's "have AI audit the harness" prompt run in a separate session.
I sign the conclusion. If any command errors, stop and show me the output.

7.7 Sober reminders

Taking "it is running" for "it is producing" is this step's number one illusion. Logs, progress bars and the bill are all moving, and that does not make the numbers trustworthy. There is only one test, can every number point to the criteria, the ledger and the original answer text (the faithful extract).
The false safety of an all-green test suite comes second. Green only proves the code matches the world I defined, and all five bugs lived on seams outside that definition. The only test for a seam is a pilot that spends real money. Budget for it, do not save on it.
The temptation to edit while running is fiercest during execution. Every friction between the plan and reality argues for fixing the plan while you are in there. The rule again, the harness obeys reality and the document obeys discipline. Any change made after you have seen the results is downgraded to exploratory.
Running bare with no ledger. In an experiment with no ledger the cost numbers rest on later recall, and recall always leans toward making your method look cheap. Put that into a cost-matched comparison and it already counts as measurement failure.
Dating this. Every number in this chapter will expire, model versions, list prices and probe readings alike. The four pillars and the five classes of bug will not.
Still exploring. Frameworks that let an agent run experiments autonomously for long stretches and find and fix harness defects on its own are iterating fast. As of this writing, my observation is that they catch syntax bugs and logic bugs faster than people, and there is no evidence they catch seam bugs reliably. Contracts, dependencies, data quirks, exactly where all five bugs of this chapter are registered. The book's online case library tracks this front.

7.8 The unfair advantage you now hold

Before any experiment that costs money and days, you can walk the whole chain for free under mock mode, turn "crashing mid-run" into a lossless event and "blowing the budget" into an impossible one. Most people only find out on their own three-hour afternoon that these are everybody's floor, having treated them until then as a luxury for the cautious.