Chapter 8 templates
This page is generated from repo/templates/ch08/; the repo holds the latest version.
Action Permission Matrix (Chapter 8)
Notes: build rows on tool × condition (not one row per tool name), check one of the three columns, fill in the guard, answer both questions. Once filled in, this merges into the Chapter 2 spec and is part of the spec from then on; before unlocking any write tool, change this first.
Permission matrix
| Tool | Condition | Autonomous | Needs confirmation | Forbidden | Guard (assertion / diff / gate) |
|---|---|---|---|---|---|
| escalate | always | ✅ | none | ||
| refund | amount ≤ $500 and order not refunded | amount_within_limit, diff |
|||
| refund | order already refunded | ✅ | refund_not_executed (seeded probe) |
||
| update_order | after shipment (via Swiftlink interception) | ✅ | |||
| send_email | recipient not verified through the order binding | no_pii_disclosure |
|||
The two confirmation-and-rollback questions (answer per write tool)
| Tool | Who confirms? (no answer = no unlock) | How does it roll back? (nominally undoable ≠ actually recoverable) |
|---|---|---|
| refund | ||
| send_email | a sent email cannot be recalled → intercept up front | |
| update_order | ||
| escalate |
Differ semantics
Every sandbox before/after change is either declared as expected, or it is a finding.
Source file:
templates/ch08/action-permission-matrix.md
Side-Effect Audit Table (Chapter 8)
Notes: the three typical side effects of write tools × three detection methods. Once the verdict methods are in place, register every finding from every run in the last column. Every "change not declared as expected" on a diff list lands here.
Side effect × detection method
| Side effect | Definition | Assertion | Diff | Probe |
|---|---|---|---|---|
| partial write | a multi-step write breaks off halfway; the world rests in an intermediate state | before/after diff against the declared expectations | ||
| duplicate submission | the same action executed twice (idempotency missing, e.g. the second SH-88271 refund) | order_state_equals, refund_not_executed |
one extra row in the refund ledger | setup seeds "already refunded" |
| stale read | acting on expired state (e.g. refunding again without checking the refund ledger) | two-channel timing case (ticket + next-day email) |
Register of confirmed findings
| Date | case_id / trace_id | Side-effect type | Caught by which layer (assertion / diff / probe) | severity | Disposition |
|---|---|---|---|---|---|
Source file:
templates/ch08/side-effect-audit-table.md
Tool-Call Eval Checklist (Chapter 8)
Notes: walk your tool-call eval configuration through the five dimensions. You will find nearly every cell says "deterministic check." Tool calls have structured arguments; the judge does not get a turn.
| Dimension | Question | Recommended verdict method | Covered? (case / assertion name) |
|---|---|---|---|
| Selection | Was the right tool called? And no tool that shouldn't be? (fuzzy search_orders with an order ID in hand?) |
deterministic check (trace scan) | |
| Arguments | Are the arguments right? In bounds? (amount, order ID, recipient) | deterministic check (amount_within_limit etc.) |
|
| Ordering | Are the dependencies in order? (read before write; verify identity before sending out) | deterministic check (step-sequence scan) | |
| Error recovery | After a tool error: retry, reroute, or double down on the error? | deterministic check + seeded-error probe | |
| Hallucinated tools | Called a tool that doesn't exist / fabricated a tool result? | deterministic check (against the registry) |
Self-check
- [ ] At least one case per dimension, all five dimensions
- [ ] At least one seeded-error probe (mistake planted in setup; tests the defenses, not the capability, e.g. "this order is already refunded")
Source file:
templates/ch08/tool-call-eval-checklist.md