Experiment ledger index
The crime scene for every number in the book. Each experiment's preregistration entered its repository before any result did. The criteria existed before the results, which is the method Chapter 6 teaches, applied to the book itself.
| Experiment | Question | Preregistration | Raw results | Red team | Cost ledger | One-line conclusion |
|---|---|---|---|---|---|---|
| Naive demo (Start Here) | One 20B open-source model alone against the frontier model, the first 20 HumanEval+ problems | / | demo_naive.jsonl | The whole chapter is its red team | A few cents | 19/20 vs 20/20, the number is real, but seven reasons say do not trust it yet |
| smol-army main experiment | Small models from 9B to 20B in teams (voting, debate, division of labor), can they tie the frontier model on a cost-matched basis | prereg.md (entered before results; later changes in CHANGES.md) | report.md · results.csv · runs.jsonl | redteam-2026-07-25.md | ledger.jsonl ($35 hard cap) | Split by task family, code tie in doubt (CI crosses zero), math the army ahead, mmlu_pro the army behind. "Can it tie" has no one-word answer, Chapters 8 and 13 unpack it |
| persona-panel (subplot) | Does the answer distribution of an LLM persona panel look like the real subgroup it imitates (ground truth, the WVS-7 US sample) | prereg.md (entered before results) | report.md · answers.jsonl | See Chapters 11 and 12 | ledger.jsonl ($10 hard cap) | Preregistered verdict FAIL. The variance-collapse red line tripped and the subgroup cross-check failed. "Answers like a real person" does not hold at the distribution level, Chapter 12 reveals it |
The book's own verification ledger
The four counts quoted in Chapter 15, section 15.7, come from here. The fact-checking round during drafting registered 48 facts pending verification (Chapter 1, 7 items; Chapter 2, 17; Chapter 3, 4; Chapters 8 to 10, 5; Chapters 11 to 15, 15). 34 were verified, 1 core claim was overturned outright, and about 10 statements were corrected or narrowed. The item-level reports were retired from the book on 2026-09-05 and are kept in the author's archive.
Reproduction notes
- Both projects run the whole chain offline with
--mock, at zero cost. Output goes toresults_mock/and never touches the real results shipped with the repository. Unit tests are all offline. - Every API call lands in
ledger.jsonl, and a run aborts on its own when it hits the budget hard cap. Runs resume from where they stopped. - smol-army's
data/SHA256SUMSpins the task-set files. The original files from the evaluation were not kept, the hashes come from a 2026-09 re-fetch, and the 3,580 archived answers in the repository were rescored to verify that they match the original data, errata in CHANGES.md. persona-panel's WVS-7 raw data must be registered for and downloaded by you (the license does not allow redistribution, see data-license.md). - Got a different number? Tell me with the reproduction report template.