I direct and release every change. Coding agents build it inside a workflow I designed.
One AI coordinates the job; specialist agents research, plan, build and test it; models from two other vendors challenge the work before anything risky ships. The rules grew out of 112 documented runs, and every failure is kept.
Populations and datesShow less
The system this workflow ships trades only my own funds: no client assets, no investment service, no advice. Runs from 26 March to 1 September 2026, numbered 1 to 134 (22 numbers were skipped or abandoned). Complexity split 93 complex, 16 moderate, 1 simple, 2 unstated. Longest single session about 30 hours; longest calendar span 4 days (run 130). Measured 3 September 2026.
- Understand
- Routemy decision
- Plan
- Buildmy decision
- Review
- Decidemy decision
From a nine-line permissions file to an 11,009-line runbook
October 2025: the first AI-workflow file is a nine-line permissions list. Today the delivery command is 11,009 lines across 93 commits, and each addition is dated to the run that needed it.
The milestonesShow less
The growth is rules added after failures, not features. Gates carry the run that needed them: “added 2026-07-03, Run 108”, “operator-caught 2026-08-18”.
Three repositories, two servers, one map every agent must read first
The most important design choice is not the number of reviewers. It is the documentation system that tells each new AI where it is, what the surrounding system does, and which evidence controls the answer.
How the map worksShow less
The constitution
Architecture before code, and the real code wins when the documentation disagrees.
The navigation index
Points every component at the one section of the map that explains it.
The architecture map
How the system works today. No dates, no history, no debate.
The canonical contracts
Jobs and tables, on-chain decoders, operator-view separation: one source of truth each.
Source of truth
Every plan, review and archive ends at the real files.
- The constitution is a rules file every agent reads on every turn. The navigation index is generated from stable section markers, so an agent loads 200 lines instead of 20,000. The map is rewritten after every change as if the feature had always existed. Each contract must be updated in the same change as the code it governs, and a validator strips any line number or value that does not exist in the real files.
- An auditor starts at the navigation index, reads the one architecture section that matters, then the code. Three reads instead of a blind search across three repositories.
- After a run ships, the archive rewrites the affected architecture section and regenerates the navigation index. History goes into reports and archives; the map stays a map.
- A separate documentation audit has run 16 times since March 2026. It hunts undocumented code, obsolete statements, history creeping into the map, broken navigation and secrets, with a final adversarial pass over the documentation diff itself.
The rule every agent is bound by before it may touch code. Lightly shortened.
One typed sentence in. A gated, recorded release out.
Six stages a person can hold in their head; eleven phases and 62 steps a script executes, in the runbook's own order for a complex change. Open a phase, then a step, to see who acts, what they must read first, what evidence they leave, and what stops the run.
Phase 0Pre-flight
Scripts and the coordinator, before the request is even read3 steps
0.AAre the reference books intact?A broken skill link silently degrades every reviewer downstream
A script checks that every skill package, stored as a link, resolves. The runbook's reason: a stale brief is recoverable, but a quietly broken skill bundle invalidates the audit itself. Added after a 47-day silent degradation nobody noticed.
- Stops the run when
- A skill link is broken
0.BRefresh the official Solana playbookOnly for blockchain tasks, and deliberately non-blocking
The coordinator pulls the latest official Solana development skill from the Solana Foundation's public repository. Network failure or a partial download warns and continues.
- Evidence left behind
- A refresh trace file for the run
0Is there a half-finished run in the building?Leftover state from another task must never be resumed by mistake
The recovery files on disk are compared with the archive. Clean: proceed. Stale after an archive: pause and ask me. In flight: hard stop until I choose resume, archive or abandon. It exists because a research brief for task A was once used to audit the plan for task B.
- Stops the run when
- A previous run is still in flight
Phase 1Understand and route
The coordinating model · my decision at the end9 steps
1Read the requestOne sentence from me; a separate intake gate if it is a trading incident
The task text is parsed. If it is bound to a trading incident, an intake gate fires first: the incident register must be read, and nothing may be marked fixed before a signed closure certificate exists.
1.5Which system is this?Seven buckets decide the harness, the skills and whether money gates apply
The work is classified: analytics, trading engine, on-chain program, both Solana codebases, workflow hardening. This one label later decides which test harness fires, which expert skills are injected and whether the money-path gates apply. The current commit ID is pinned so a later gate can detect the code moving underneath the audit.
- Evidence left behind
- The audit brief: the single source of truth every later command reads
1.6Is there a screen involved?Dashboard scope and impact level
UI scope is detected and given an impact level, which later decides whether a design agent is spawned and whether the browser is photographed before and after.
2Architecture first, through the navigation indexRule Zero: “Use the navigation system, don't just grep”
The coordinator reads the navigation index, finds the matching entry, follows it into the 19,514-line architecture map, and then reads the actual source files it names. No production query is allowed yet.
- Must read first
- Navigation index → architecture map section → source files
2✓Checkpoint: prove you understood before you touch productionA tick-box gate, all boxes or no progress
An architecture-understanding block must be written, including what a healthy state looks like for this subsystem. “Do not proceed if any box is unchecked.” Navigation and server commands may not run in parallel.
- Evidence left behind
- The architecture-understanding block
- Stops the run when
- Any box unchecked
3Health-check the live systemOnly now, because now the coordinator knows what healthy looks like
Live processes, configuration flags, recent critical errors and a row count from the production database are read and saved as the system state for the run.
- Evidence left behind
- The system-state snapshot
3.0BPhotograph the browserA baseline to diff against after deployment
A headless browser captures a screenshot, the JavaScript error count and the list of API calls. The after-picture at the end of the run is compared with this.
- Evidence left behind
- Browser baseline: screenshot, error count, API inventory
4Present the route and waitSimple, moderate or complex, from risk rather than file count
The complexity assessment and recommended route are shown in plain and technical language, then the run stops. If the task auto-escalated for security reasons, such as keys, wallets or fund movement, the downgrade option is withheld: the complex route is mandatory.
- My decision
- Proceed, upgrade, downgrade or cancel. Every override is logged by name.
4.5Pick the external bug-hunter's depthLight, single, or manager mode where it spawns its own sub-hunters
The depth of the OpenAI Codex lane is set from complexity. Ten listed conditions escalate a moderate run to manager mode anyway, among them any authentication, wallet or money-movement logic.
Phase 2Research
A researcher agent gathers · a validator fact-checks · the coordinator validates8 steps
5.1Refresh the audit contextLive schema and recent errors, re-exported if older than a week
A script re-exports the production database schema and the recent error log so that planning happens against today's reality.
- Evidence left behind
- Fresh schema export and error digest
5.2Delegated research, validated upward“Subordinates research, boss validates”, never the other way round
A researcher agent maps affected files, callers, consumers, data sources and user-facing paths, reading vendor documentation from the vendors' own live documentation servers first. The coordinator validates every finding against local knowledge and records what was accepted and what was rejected. Mandatory for every moderate and complex task.
- Must read first
- Navigation, architecture map, code, live vendor docs
- Evidence left behind
- The research brief, to a fixed template
3.1Do the research files actually exist?An empty brief stops the run
A hard gate: if the research brief is missing or empty, stop.
- Stops the run when
- The research brief is empty
3.15Fact-check the researchEvery function, line, data source and column, against the real files
A validator agent re-reads every claim in the brief against the code. A failure re-runs the researcher. The reason, in the runbook: a brief once said a function had 18 signals; reality was 9 stale ones; nobody verified; three hours and forty-five minutes and a full revert were lost.
- Evidence left behind
- The validation report
- Stops the run when
- A claim in the brief is false
3.16Did we map the blast radius?For every change target: who calls it and who reads its output
The brief must contain a feature impact map with explicit full or partial coverage flags, so a change cannot fan out into consumers nobody listed.
- Stops the run when
- No impact map
SEPView separation gateFires from a computed flag, not from an agent noticing a keyword
A script decides whether the change touches the boundary between the operator's own view and any other account view. If it does, the brief must classify every changed path, and the canonical separation contract governs.
- Stops the run when
- The separation matrix is missing when the flag fires
3.2Verify the data says what we thinkA real sample from production for every data source touched
The coordinator queries production for an actual sample of every data source the task touches. Hard gate on mismatch. The incident: a dashboard showed three of five signals; reality was zero of thirteen; nobody had queried production before planning.
- Stops the run when
- The data does not match the assumption
3.2.5Do we need a data-flow trace?Four automated triggers
Complex work, a database migration, a new value written into a shared column, or a new background loop each require a data-flow trace at plan time. Written after run 37 shipped two catastrophic data-loss bugs that all four internal reviewers missed and only the external reviewer caught.
- Evidence left behind
- The trace-required flag in the audit brief
Phase 3Plan and validate
A planner agent writes · the coordinator and scripts check · revision loop5 steps
3.3The planWritten by a different model than the coordinator, with its own re-read of the architecture
A planner agent is told it is not executing the research brief blindly: it must independently re-read navigation, architecture and source. The plan includes tests, rollback, dependencies and a mandatory section naming the skills it used, with the file and lines actually opened.
- Must read first
- Navigation, architecture, code, the required skills
- Evidence left behind
- The plan, with its skills section
3.5The researcher validates the plannerSix checks, including a ground-truth cross-check of every table named
Completeness, pattern adherence, scope, integration, skills, and whether every database table the plan names was actually verified by the research. The reason: run 42's plan named five tables that did not exist and three same-lineage reviewers approved it.
- Evidence left behind
- Valid, minor fix in place, or a revision request
3.5.4Mechanical existence gateA thirty-second grep that catches every phantom name before the panel spawns
A script extracts every table, column, route, file path and environment variable in the plan's code blocks and checks each exists, with batched database queries and file tests. Deliberately not an AI. Complex runs only.
- Stops the run when
- Any phantom symbol
3.5.5Integration verificationDoes the plan match the code that already exists?
Existing patterns, SDK rules, cross-system contracts, database integration and explicit preservation of existing features are checked and scored.
- Evidence left behind
- A scored integration document
3.6Revision loopAt most two automatic iterations, then I handle it
Major issues go back to the planner with its memory intact. A reject verdict first sends the researcher back for missing ground truth. Audit-driven revision rounds later in the run are recorded by name; the deepest recorded is nine.
- My decision
- After two automatic iterations, the decision is mine
Phase 4Five-way plan audit
Three vendors in parallel · quorum three of four · then my approval5 steps
5Launch five reviewers in one messageNobody can read another's verdict, because none exists yet
A plan auditor and an architecture reviewer (Claude, maximum effort on complex runs), an OpenAI Codex lane and an xAI Grok lane through their wrappers, the coordinator's own verdict, and a cross-validator when dependencies or documentation servers change. Grok is explicitly non-blocking while it earns its place.
- Must read first
- The plan, the harness patterns, the injected skills
- Evidence left behind
- Five written verdicts
5.1Four mandatory preambles“If you have not read the actual source files, your review is invalid”
Every reviewer is told: you are the auditor, not a rubber stamp; architecture first; read the actual source files; and on trading work, a blocking SDK-compliance check with live verification of every external API call.
5.2QuorumThree of four legacy lanes must return valid output
No verdict until at least three of the four lanes have returned. A reviewer that omits the mandatory harness-patterns section is re-spawned once, then excluded from quorum; if quorum drops, the run blocks and asks me.
- Stops the run when
- Fewer than three valid lanes
5.3Extract the data-flow traceFail-closed: a missing or placeholder trace re-spawns the reviewer
When a trace is required, the architecture reviewer's trace is extracted between explicit markers and checked: non-empty, exactly five sections, no placeholder text.
- Evidence left behind
- The trace artifact the implementer must satisfy
- Stops the run when
- The trace is missing or malformed
5.4Check the checkersThe coordinator reads the code behind every finding
A mandatory to-do item: independently validate every plan-audit finding by reading the code and verifying the claim before acting on it.
- My decision
- Approve the plan for implementation, or send it back
Phase 5Implement
A separate agent writes the code · scripts prove every claim7 steps
P9Stamp the before-pictureOnce, and never re-stamped
Each repository's commit ID is recorded before implementation. It is never re-stamped on a resume or a fix round, so the diff the external reviewer later sees is cumulative.
- Evidence left behind
- Pre-implementation commit IDs
6A separate implementerNever the coordinator, on moderate and complex work
The coding agent's model and turn budget come from a tier matrix; blockchain work always gets the strongest tier. The implementer follows the plan exactly and adds nothing; if the code differs from the plan's assumptions it stops and the plan is revised rather than the scope silently expanded. It must return a skills-applied section naming the specific rule used per skill. If it runs out of turns it hands over to a continuation implementer, at most three deep.
- Must read first
- The approved plan, the research brief, the trace
- Evidence left behind
- The diff and the implementation report
G-29Implementer output freshnessSilent degradation converted into a loud one
If the implementer's report is missing or older than the plan, which happens when the coordinator applies fix-round edits itself, either a fresh implementer is spawned or a sentinel with per-skill file-and-line citations is written. Neither: the scoring pass must mark the skills checkpoints as failed, not as unverified.
- Stops the run when
- Stale or missing implementer output without a sentinel
6.1Post-implementation impact validation“Zero insertions or deletions means the implementer's claim is false”
The recorded diff is checked for every file the implementer claims to have changed. The tenant-separation flag is re-affirmed from the real diff. The last three runs' recurring bugs are grepped for.
- Stops the run when
- A claimed edit shows no change
6.2Skills gatesPresence is not application
A nine-check matrix maps task characteristics to skills that must appear. On blockchain work a stricter citation gate demands an actual rule number or pattern ID. This one is an advisory gate: after two retries it logs that it degraded and lets the run continue, so it is not counted among the gates that stop the run.
8.7Schema validation gateEvery column in the SQL diff, against the live database
The incident: a helper referenced a column that did not exist and 17 review rounds approved the code.
- Stops the run when
- A missing column
8.8Static linter gateAn unused variable the plan named is a wiring failure
The incident: a value was loaded from the database and never wired to the page; 17 review rounds approved; the user saw blank rows.
- Stops the run when
- An unused plan-named variable
Phase 6Five-way code audit
Same panel, now against the real code · the coordinator verifies the verifiers6 steps
7The panel, against the codeOn blockchain runs the cross-validator goes first and can stop the audit
A code auditor, the architecture reviewer, OpenAI Codex and xAI Grok in parallel; on traced runs a data-flow tracer joins, with fresh eyes: it does not read the plan-phase findings. Same quorum rule.
- Evidence left behind
- Five verdicts, the tracer's delta report
- Stops the run when
- Fewer than three valid lanes
7.1Functional testing requirementPick an input, trace it through the code, compare to a known-good reference
Reviewers must go beyond syntax: trace one concrete calculation step by step, test edge cases, and publish the results.
CP7The coordinator verifies the verifiersLine numbers exist, values match the database, then its own trace
Each reviewer's numeric trace is extracted; line numbers are checked to exist; values are checked against the live database; expected values must have a real source. Then the coordinator performs its own independent trace and cross-checks the reviewers against each other. Explicit anti-gaming measures; hard gates prevent a pass.
- Stops the run when
- A reviewer's trace does not hold up
7.2Official sources overrideA vendor's documentation beats a reviewer's claim about that vendor
Any external reviewer's claim about a provider's behaviour is checked against that provider's own documentation before it is accepted.
8.5Post-fix re-auditWhen a round had criticals and the fix touched a shared column
The fix itself is re-audited, by a different lane than the one that found the problem.
- Stops the run when
- Fires on the trigger; the fix must pass
6.6LSkill application ledgerOne row per rule, not per skill, each mapped to code
On blockchain runs the code and security reviewers must table every individual rule of the injected skills with a code location and a verdict. Any unmapped or violated row forbids approval. The incident: code shipped with an unchecked token freeze authority while the security report claimed to have applied the very skill section that forbids it.
- Stops the run when
- Any rule unmapped or violated
Phase 7Pre-deploy gates
Scripts execute what reviewers only read4 steps
8.6Live-execution gate“Do not proceed on any timeout. All other paths are forbidden.”
Write-and-time code is dry-run against the real database. New read helpers are run against a known canary entity. A zero-row mismatch or a missing canary on a complex run blocks.
- Stops the run when
- A dry run fails or the canary probe mismatches
7BRuntime test executionFund-moving code must have run, on a mainnet fork, in this run
A test runner must produce a transcript keyed to the run. The gate then re-extracts the cited test bodies from disk, cross-checks pass counts against the framework's own totals, and hashes the raw log against the archive. Missing transcript, count drift, empty proofs or zero entry-point coverage all block. The comment trail shows this gate re-hardened a dozen times against ways a forged transcript could slip through.
- Evidence left behind
- The signed test transcript
- Stops the run when
- No transcript, count drift, or zero coverage
7B.5Harness flip integrityMarking a pattern satisfied needs two mechanical citations
A non-trivial assertion and proof of reachability with no early exit. The incident: four patterns were flipped to satisfied by editing a count; one cited test asserted that true is true, fourteen others exited before reaching the code they claimed to test.
- Stops the run when
- A bare claim of coverage
10Security gateFirst: has the code moved since the audit began?
If the repository has moved since the commit pinned at step 1.5, the findings may be stale and the run blocks. Then a recommendation: security-relevant work spawns a security auditor that itself spawns three sub-auditors, blinded to its own findings to prevent anchoring. I always have the final say.
- Stops the run when
- The code moved underneath the audit
- My decision
- Spawn the security auditor, or accept the recommendation not to
Phase 8The external round
A different vendor's model, in a session I open myself6 steps
9ATriggerMandatory on traced work, money-moving work, and complex UI work
The origin: run 37's rollback bug would have wiped 22,102 production records and was caught only by the operator's external reviewer, while all four internal lanes approved. The money-path trigger became unconditional after 15 consecutive runs in which the external round caught what the internal panel missed.
- Stops the run when
- A required trace is missing: inferring “no file means not required” fails open
9A.1Freeze, assemble, redactNothing is reviewed dirty
The in-scope state is committed and its commit ID recorded. The brief builder refuses to render from dirty files. It pulls plan, brief, trace, changed files, implementer output and navigation ranges, and replaces every server address, private network address, database identifier, key path and dashboard credential with redaction tokens before writing, stamping a receipt of how many replacements it made.
- Evidence left behind
- The redacted brief and its receipt
9A.1-BRender the promptA universal template with two independence guards written in capitals
Known anti-patterns must be framed as rebuttable, never “do not re-flag”. Per-lane targets are a floor, never a ceiling, with explicit latitude to look outside the changed files, because “the orchestrator is a single model and will miss things.”
9A.1-CWho runs itDefault: I paste it into my own separate session
The default channel preserves full process independence: I paste the brief into a separate OpenAI Codex session and paste the verdict back. A fallback where the coordinator spawns the external model itself is labelled reduced independence and can only be launched by a script that refuses to run without my verbatim authorising words, logged before spawning.
- My decision
- Run the external session myself
9A.1.5The prompt gateTwenty-eight checks; fail-closed if the checker itself is missing
Any inline production access, unresolved template slot, wrong skill bundle for the system type, or empty navigation range blocks. If the checker script is missing, the run still blocks.
- Stops the run when
- Any of 28 checks, or the checker is absent
9A.4Receive and validate the verdictMechanical validation of every finding, and a downgrade guard
Freshness of the brief's commit, format conformance, artefact shape, degraded-round detection, then each finding validated mechanically. On money-moving work, downgrading an external critical or high finding requires a code receipt and, where traced, a runtime receipt proving the neutralising path executes. A second external auditor agreeing with the downgrade does not satisfy the gate, because that has happened and production disproved it within a day.
- Stops the run when
- A downgrade without receipts: the finding stays open
- My decision
- Accept, refute with receipts, or send back to plan revision. There is no timeout.
Phase 9Deploy and verify
Scripts stage exactly this run's files · machines test before I do4 steps
11DeployOnly from the default branch, only this run's files
A reconcile gate runs once per deployed repository and blocks if a foreign or missing file appears. Then pull and restart on the servers by script, a release build for the Rust engine or an on-chain deploy for the program, and ten minutes of monitoring.
- Evidence left behind
- Deploy receipt and monitor log
- Stops the run when
- A foreign or missing file in the staged set
- My decision
- Ship, or do not ship
9C-autoAutomated browser verificationMandatory-fire: the trigger once met and zero browser calls happened
The browser is driven for real; console errors and network calls are compared with the baseline from step 3.0B. Added after a run whose 3.7-megabyte transcript contained no browser tool call at all.
- Evidence left behind
- The after-picture, diffed against the baseline
- Stops the run when
- The precondition is met and the check did not fire
9C-dataData validation harnessThe same numbers across pages and filters must agree
For data-pipeline work, the harness cross-checks figures across pages and filter combinations. It evolves after every run.
- Stops the run when
- Cross-page disagreement
9CThe human checklist, with a twist“Did you test all these yourself?”
Before I see anything, every checklist item is tagged as browser-verifiable or human-only, and the coordinator runs every automatable item itself. I get only what genuinely needs a human. Then a hard wait: the run does not advance until I report back.
- My decision
- Confirm the result, or report defects and start another fix-and-review cycle
FinalDocument, archive, self-score
The coordinator rewrites the map · an archive agent freezes the evidence · the run grades itself5 steps
F1Session summaryEverything the run did, with real commit hashes
Artefacts, audit files, changed files, both five-way verdict sets, an executed-versus-read ledger, gate evidence, skills evidence, external-round evidence, post-deploy issues.
- Evidence left behind
- The session record
F1.5Rewrite the architecture mapBy the coordinator directly, never delegated
A fresh agent succeeded at this one time in four across earlier runs, so the coordinator does it itself: read the changed source files, not the plan; rewrite the affected sections as current state; sync any canonical contract whose domain was touched; regenerate the navigation index from section markers; run the drift test; spot-check two or three documented flows against the code.
- Must read first
- The changed source files
- Evidence left behind
- Updated architecture map and navigation index
- Stops the run when
- Any of seven quality checks fails
F1.99Was the summary actually written?A chokepoint added after five zero-byte session files
A hard gate that the session record exists and has content, plus a pre-archive guard for blockchain artefacts.
- Stops the run when
- A zero-byte record
F2ArchivePlan, diff, verdicts, the raw transcript, file fingerprints
An archive agent creates the dated archive, copies every artefact including the raw session transcript, and writes the report with a mandatory harness-enforcement stamp. If no artefact proves the gate fired, it must write “not found”, never infer it did.
- Evidence left behind
- The archive and its report
F3The run audits itselfCannot end without a review, an explicit opt-out, or a documented failure
The progress review scores the run and every reviewer lane, classifies each post-deploy bug as preventable or new, records gaps with an ID, a severity and a proposed rule, and checks that every red recommendation is shipped with evidence or retired with my words quoted. Four consecutive runs once skipped this step; the skip cost weeks of drift.
- Evidence left behind
- The scorecard and the gap entries
- Stops the run when
- No review artefact at all
About the numberingShow less
Where the runbook's numbering is out of sequence the order follows its executable to-do list. Steps fire on their triggers: the data-flow trace on data lineage or migrations, the Solana gates on the two Solana codebases, the runtime test gate on fund-moving code, the external round when the run is flagged for it. Simple runs skip planning, the panel and the archive, but still run the deploy gate, a browser regression check and a tracker entry.
Different vendors, different questions, and a record of who caught what
Lane count is not lane diversity. The panel mixes models from three vendors, two blind roles and deterministic scripts, and every lane is scored, so the ledger shows which ones earn their place.
Who caught whatShow less
| Lane | Role | Independence | In the ledger |
|---|---|---|---|
| Claude Opus | Plans, implements, and reviews plan, code and architecture; maximum effort on complex runs | Same lineage as the coordinator, different roles and contexts | The most frequent lane; 16 plan rounds once blessed a feature flag that did not exist |
| Claude Sonnet | Coordinates, validates research and findings against the real code and data, cross-validates citations | Fact-checker of everyone, including the coordinator | Run 122: executed a test probe and overturned the coordinator's own refutation |
| OpenAI Codex | Internal review lanes, a second data-flow trace, and the separately run external round | Different vendor; the external round runs outside the workflow on a frozen commit | Present in 111 of 112 runs; scored 9.33 of 10 across 51 scored runs; caught 15 critical findings in run 49 that Claude reviewers had approved |
| xAI Grok | Independent bug hunter on plan and code, added July 2026 | Third vendor; started non-blocking while it earned its place | 13 runs, 8.22 of 10 across 9 scored; run 128: contradicted a maximum-effort Claude critical, and the source proved Grok right |
| Data-flow tracer | Compares the promised data path with the code, at plan time and again in the code | Forbidden from reading the other reviewers' findings | 30 recorded passes, up to 35 of 35 assertions |
| Scripts | Existence gates, schema checks, linters, brief redaction, prompt gate, routing | Deterministic; no judgment, no fatigue | 146 hard-block instructions; a gate stays red until it has executed on representative data |
| Me | Scope, route, plan approval, release, verification sign-off, retirement of recommendations | The only party that can authorise a deploy or retire a red finding | Two of the six failures on this page were caught by me, not by any model |
- A retired lane is retired in writing. The Google lane's own epitaph in the ledger: “1 unique finding in 13 runs”.
- External catches are counted per run. The field “external caught what internal missed” reads yes 78 times and no 14 times across the recorded rows, with a recorded streak of 40 consecutive runs.
- More reviewers did not always mean more truth. The latest run records four deployments and eleven review rounds in which nobody opened the page; the fix was a live round trip through the real screen, not another reviewer.
78 of the 92 runs that record the field; 20 of the 112 runs do not record it, and the ledger does not break these catches down by severity. Scores are the workflow's own progress reviews, kept in the run ledger: an internal quality record, not an independent certification.
Official references first, and proof they were used
Twenty-two skill packages, mostly official vendor documentation, are injected into every agent that needs them. Citing a skill is worth nothing on its own; a seven-checkpoint chain follows each one from plan to code to review.
How use is provenShow less
The planner cites
The plan names the skills it used and the rules it applied. The plan auditor verifies each citation exists and says what the plan claims.
The implementer applies
The implementation report maps each skill rule to a file and line in the diff. The coordinator checks the rule applied to this task at all.
The auditor proves
The code auditor produces a compliance table, the harness patterns consulted are cited, and a review without them is rejected and re-run.
- False or shallow use is named, not graded: a citation to a rule that does not exist is a broken citation; a skill mentioned but not applied is name-dropped; a required skill not used is a missing rejection.
- Three testing harnesses hold 4,763 lines of known failure patterns, one per past incident. A new safety gate stays red until it has run on representative data; reading the code does not count as executing it.
- Community snippets are treated as untrusted. One shipped the wrong byte offsets for a trading venue; official sources are now a hard gate, with pinned, digest-verified reference files.
The skill packages as they sit on disk. Several are official vendor documentation packaged for the models.
How a run survives the model losing its memory, and lasts for days
The durable unit is not the conversation but the run ID plus its evidence on disk. Five commands hand that state to each other, which is why a run can span days and several memory resets.
How a run survives memory lossShow less
The run
The state machine described above: assess, research, plan, review, implement, review again, deploy, verify.
The save file
Before the model's memory is compacted, a timestamped handover records run identity, plan revision, phase, pending reviewers, the live to-do list and every owed gate.
Hypothesis, then proof
A resumed model must restore the tracker, verify it against the live repository, re-arm every owed gate and re-read the architecture. The handover is a hypothesis; verification commands are the proof.
The frozen record
After deploy and testing: report, evidence package, file fingerprints, the rewritten architecture section, the regenerated index. Disposable artefacts are staged, never blindly deleted.
The retrospective
Scores the run and each reviewer lane. When no single transcript exists, it reconstructs the review from every handover, verdict and report, and sweeps all handovers for durable findings.
The periodic check
Sixteen times so far: undocumented code, obsolete statements, history in the map, broken navigation, secrets, then an adversarial pass over the documentation diff.
- Run 49 ran about 30 hours across two memory resets. Run 51, the workflow auditing itself, ran about 28 hours over two days across two resets and two handovers. Run 134 took three days and about 26 review passes.
- A run once dropped six findings that its first two handovers had deliberately recorded, because the review swept only the last handover. I caught it. The rule since: every handover in the chain is swept, and each carries machine-readable durable findings.
- 678 handover files sit in the archives, one for every time a run paused, compacted or changed sessions.
Six times something broke, and what exists because of it
Six real failures from the run ledger, and the permanent control each one produced. Two of the six were caught by me, not by any model.
3 moreShow less
From the run ledger, verbatim: “Every single one was found by widening the check, never by re-reading the same layer harder.”
Every run is graded, and so is every reviewer
After every run a separate command scores the run against a five-tier rubric and grades each reviewer lane, so the ledger shows which reviewers earn their place.
- Claude Opusplans, implements, reviews architecture and code; maximum effort on complex runs
- Claude Sonnetcoordinates, validates plans and findings against the real code and data
- OpenAI Codexinternal review lanes and the separately run review; 9.33 of 10 across 51 scored runs
- xAI Grokindependent bug hunter, added July 2026; 8.22 of 10 across 9 scored runs
- Data-flow tracerchecks the promised data path at plan time and again in the code; 30 passes recorded, up to 35 of 35 assertions
How the rubric worksShow less
The rubric has five tiers: 17 coordinator gates, 16 reviewer checks, 8 security and verification checks, 9 Solana gates, 6 harness-health rows. Each reviewer lane gets a plain-English grade, star to silent, and exactly one is named star performer. The best signal of a reviewer's value is its unique catches: findings nobody else found. A “no” is a gap by definition; every post-deploy bug is classified as preventable, partly preventable or new and mapped to the gate that should have caught it.
Not every gap becomes a rule. The ones that return, or are red, must.
A gap is anything that should have fired and did not; each gets a stable ID, a severity and a proposed fix. A triage rule then forces a decision, not necessarily a fix.
The triage rule, and four gaps that became rulesShow less
- Red “must fix” recommendations must be dispositioned before the run can end: shipped, with file evidence, or retired, with my own words quoted verbatim. The coordinator may not paraphrase or infer a retirement. Yellow and green stay advisory.
- The rule exists because of a measurement: of six red recommendations in July 2026, none had reached a workflow file, while all six harness notes from the same reviews had. The only difference was the word “must” versus “may”.
- When an external reviewer catches what every internal lane missed, the harness is updated immediately or the waiver cites the rule that already covers it. In the ledger's words: the tracker is the diary, not the checklist.
- For test patterns there is a universality table: a mechanism seen in two or more runs, or affecting two of the three codebases, or with a greppable signature, becomes a pattern; a one-off becomes a note. Severity escalates on repetition, a claimed fix is grepped for before it counts, and a rule is retired only after three clean runs.
Gap IDs are never reused or renumbered. A retired recommendation is a legitimate outcome; an absent decision is not.
A claim without evidence stops the run
Thirty-four catalogued gates. Each one turns a sentence an AI might say into a check a script performs, and blocks the run when the check fails.
Three more, and the gates in the fileShow less
From the runbook, lightly shortened. Each gate carries the run that needed it.
Each gate is traceable to the run that needed it.
Two subscriptions and a thirty-dollar plan run all of it
One Claude Max subscription, one ChatGPT plan for Codex, one thirty-dollar Grok plan. Review cost is routed by risk rather than capped, and anything that can lose money always runs at full depth.
How the cost is controlledShow less
- Claude runs on a single Max subscription: the coordinator, every specialist agent and every Claude review lane count against that one quota. OpenAI Codex runs on a ChatGPT plan; xAI Grok on a thirty-dollar monthly plan. Those are my bills, not figures from the repository.
- Review cost is routed, not capped. A lighter fork of the whole workflow saves 40 to 55 percent of Claude tokens on moderate work by dropping duplicate lanes, and it never drops the external round. Since July 2026 a deterministic policy picks the model and effort per lane by complexity. In the measured A/B, deep reviews used 84.8 percent fewer tokens and standard runs 57.5 percent fewer, with zero missed high or critical findings across three samples.
- Money-moving work never routes cheaper: it always runs the full high-effort fleet. And no monetary budget is enforced anywhere in the workflow; the only budgets in the routing code are wall-clock timeouts. Endurance comes from handovers and archives, not from spending more.
Subscription costs are the operator's statement. Token savings are from the routing A/B recorded in the repository: three samples, which is enough to enable a route, not to certify it; one missed high or critical finding rolls a route back. A control run before routing consumed about 13.4 million tokens.
Measured, with the populations printed
Six numbers from the run ledger, the archives and git, each with the population it was counted on.
Where the numbers come fromShow less
Every figure is from the run ledger, the archives or git, and the ledger is written by the same system it evaluates. It is a high-resolution internal record, not an independent certification. Multiple AI reviewers are not an independent security audit; where money or security is at stake, the boundary is a human specialist.
The archive and the system are the evidence.
In an interview I can walk through one archived change and one live product flow: the evidence, the release decision, the result, and the limits of this approach.
One archived change, end to end
A real run from the archive: the request, the plan, the review decisions and fix rounds, the separate review's verdict, my release decision, and what production showed afterwards.
One live product flow
The system this loop built, running: a signal arriving, the engine executing, the record it leaves, and the numbers behind case study 1.
the boundaryWhere I would not use this: a security-critical change under time pressure, or anything a regulator would need to sign. Those get a human specialist and a slower path, not a review panel.
In a team I would own: scope and acceptance criteria, the review gates, release decisions, and the retrospective loop. Engineers own the code; the workflow makes their review load smaller, not larger.
I own the product intent, the system boundaries and constraints, the acceptance criteria, the release decision and production verification. AI agents provide much of the implementation-level technical depth: they research, propose designs, write the code and review one another independently. I resolve their disagreements through evidence and decide what ships. I do not claim the unaided technical depth of a senior backend or Rust engineer.