Case study 2 of 2 · how I ship

I direct and release every change. Coding agents build it inside a workflow I designed.

One AI coordinates the job; specialist agents research, plan, build and test it; models from two other vendors challenge the work before anything risky ships. The rules grew out of 112 documented runs, and every failure is kept.

A note on authorship

I am not a backend, Rust or smart-contract engineer and do not represent myself as one. I define what the system must do, its boundaries and acceptance criteria; I direct coding agents through research, design, implementation and review; I validate the resulting behaviour and own the release decision. The implementation-level expertise described below is substantially agent-mediated.

112documented runs
3AI vendors on every review panel
4 dayslongest run, start to ship
Populations and datesShow less

The system this workflow ships trades only my own funds: no client assets, no investment service, no advice. Runs from 26 March to 1 September 2026, numbered 1 to 134 (22 numbers were skipped or abandoned). Complexity split 93 complex, 16 moderate, 1 simple, 2 unstated. Longest single session about 30 hours; longest calendar span 4 days (run 130). Measured 3 September 2026.

UnderstandRouteMY DECISIONPlanBuildMY DECISIONReviewDecideMY DECISION
  1. Understand
  2. Routemy decision
  3. Plan
  4. Buildmy decision
  5. Review
  6. Decidemy decision
How it grew

From a nine-line permissions file to an 11,009-line runbook

October 2025: the first AI-workflow file is a nine-line permissions list. Today the delivery command is 11,009 lines across 93 commits, and each addition is dated to the run that needed it.

0k4k8k12kborn · 284 linessub-agentsharness · reviewsskills · self-auditthird vendor · routingrun 134 · 11,009Dec 25Mar 26Jun 26Sep 26
Lines in the workflow file, from git · growth is rules added after failures, not features
The milestonesShow less
Oct 2025
A nine-line permissions file. The whole AI workflow, on day one.
Nov 2025
The delivery command, 284 lines. A second vendor's model reviews every plan from the first version.
Dec 2025
The first gate scripts: file fingerprints and stale-archive detection. The file quadruples in five weeks.
Feb 2026
Six specialist agents split the work: researcher, planner, implementer, two auditors, a validator.
Apr 2026
The first testing harness and the progress review that scores every run.
May 2026
The hardening quarter: +4,015 lines in one month, skills, handovers, and the workflow auditing itself for 28 hours.
Jul 2026
A third vendor's model joins the panel, with routing that cuts review cost without touching money-critical work.
Sep 2026
Run 134. Twenty-six agent roles, 22 skills, three harnesses, 34 gates.

The growth is rules added after failures, not features. Gates carry the run that needed them: “added 2026-07-03, Run 108”, “operator-caught 2026-08-18”.

Keeping AI in scope

Three repositories, two servers, one map every agent must read first

The most important design choice is not the number of reviewers. It is the documentation system that tells each new AI where it is, what the surrounding system does, and which evidence controls the answer.

27,493lines of architecture documentation
16documentation audits since March
3canonical contracts, same-change updates
How the map worksShow less
Always loaded

The constitution

Architecture before code, and the real code wins when the documentation disagrees.

2,110 lines

The navigation index

Points every component at the one section of the map that explains it.

19,514 lines

The architecture map

How the system works today. No dates, no history, no debate.

3 contracts

The canonical contracts

Jobs and tables, on-chain decoders, operator-view separation: one source of truth each.

The code

Source of truth

Every plan, review and archive ends at the real files.

  • The constitution is a rules file every agent reads on every turn. The navigation index is generated from stable section markers, so an agent loads 200 lines instead of 20,000. The map is rewritten after every change as if the feature had always existed. Each contract must be updated in the same change as the code it governs, and a validator strips any line number or value that does not exist in the real files.
  • An auditor starts at the navigation index, reads the one architecture section that matters, then the code. Three reads instead of a blind search across three repositories.
  • After a run ships, the archive rewrites the affected architecture section and regenerates the navigation index. History goes into reports and archives; the map stays a map.
  • A separate documentation audit has run 16 times since March 2026. It hunts undocumented code, obsolete statements, history creeping into the map, broken navigation and secrets, with a final adversarial pass over the documentation diff itself.
CLAUDE.md · Rule Zero, verbatim
Your FIRST action on ANY code task MUST be reading the architecture. No exceptions. No "quick grep first."
1. Read the navigation index and find the section for the component
2. Read the matching section of the architecture map
3. Read the actual source files and understand the existing flow
4. Then analyse or make changes
The actual source code is always the decisive factor. If the docs say one thing and the code does another, the code is correct.

The rule every agent is bound by before it may touch code. Lightly shortened.

The run, step by step

One typed sentence in. A gated, recorded release out.

Six stages a person can hold in their head; eleven phases and 62 steps a script executes, in the runbook's own order for a complex change. Open a phase, then a step, to see who acts, what they must read first, what evidence they leave, and what stops the run.

Phase 0

Pre-flight

Scripts and the coordinator, before the request is even read3 steps
0.AAre the reference books intact?A broken skill link silently degrades every reviewer downstream

A script checks that every skill package, stored as a link, resolves. The runbook's reason: a stale brief is recoverable, but a quietly broken skill bundle invalidates the audit itself. Added after a 47-day silent degradation nobody noticed.

script
Stops the run when
A skill link is broken
0.BRefresh the official Solana playbookOnly for blockchain tasks, and deliberately non-blocking

The coordinator pulls the latest official Solana development skill from the Solana Foundation's public repository. Network failure or a partial download warns and continues.

Claude Sonnet
Evidence left behind
A refresh trace file for the run
0Is there a half-finished run in the building?Leftover state from another task must never be resumed by mistake

The recovery files on disk are compared with the archive. Clean: proceed. Stale after an archive: pause and ask me. In flight: hard stop until I choose resume, archive or abandon. It exists because a research brief for task A was once used to audit the plan for task B.

scriptClaude Sonnet
Stops the run when
A previous run is still in flight
Phase 1

Understand and route

The coordinating model · my decision at the end9 steps
1Read the requestOne sentence from me; a separate intake gate if it is a trading incident

The task text is parsed. If it is bound to a trading incident, an intake gate fires first: the incident register must be read, and nothing may be marked fixed before a signed closure certificate exists.

Claude Sonnet
1.5Which system is this?Seven buckets decide the harness, the skills and whether money gates apply

The work is classified: analytics, trading engine, on-chain program, both Solana codebases, workflow hardening. This one label later decides which test harness fires, which expert skills are injected and whether the money-path gates apply. The current commit ID is pinned so a later gate can detect the code moving underneath the audit.

Claude Sonnet
Evidence left behind
The audit brief: the single source of truth every later command reads
1.6Is there a screen involved?Dashboard scope and impact level

UI scope is detected and given an impact level, which later decides whether a design agent is spawned and whether the browser is photographed before and after.

Claude Sonnet
2Architecture first, through the navigation indexRule Zero: “Use the navigation system, don't just grep”

The coordinator reads the navigation index, finds the matching entry, follows it into the 19,514-line architecture map, and then reads the actual source files it names. No production query is allowed yet.

Claude Sonnet
Must read first
Navigation index → architecture map section → source files
2✓Checkpoint: prove you understood before you touch productionA tick-box gate, all boxes or no progress

An architecture-understanding block must be written, including what a healthy state looks like for this subsystem. “Do not proceed if any box is unchecked.” Navigation and server commands may not run in parallel.

Claude Sonnet
Evidence left behind
The architecture-understanding block
Stops the run when
Any box unchecked
3Health-check the live systemOnly now, because now the coordinator knows what healthy looks like

Live processes, configuration flags, recent critical errors and a row count from the production database are read and saved as the system state for the run.

Claude Sonnet
Evidence left behind
The system-state snapshot
3.0BPhotograph the browserA baseline to diff against after deployment

A headless browser captures a screenshot, the JavaScript error count and the list of API calls. The after-picture at the end of the run is compared with this.

script
Evidence left behind
Browser baseline: screenshot, error count, API inventory
4Present the route and waitSimple, moderate or complex, from risk rather than file count

The complexity assessment and recommended route are shown in plain and technical language, then the run stops. If the task auto-escalated for security reasons, such as keys, wallets or fund movement, the downgrade option is withheld: the complex route is mandatory.

Me
My decision
Proceed, upgrade, downgrade or cancel. Every override is logged by name.
4.5Pick the external bug-hunter's depthLight, single, or manager mode where it spawns its own sub-hunters

The depth of the OpenAI Codex lane is set from complexity. Ten listed conditions escalate a moderate run to manager mode anyway, among them any authentication, wallet or money-movement logic.

Claude Sonnet
Phase 2

Research

A researcher agent gathers · a validator fact-checks · the coordinator validates8 steps
5.1Refresh the audit contextLive schema and recent errors, re-exported if older than a week

A script re-exports the production database schema and the recent error log so that planning happens against today's reality.

script
Evidence left behind
Fresh schema export and error digest
5.2Delegated research, validated upward“Subordinates research, boss validates”, never the other way round

A researcher agent maps affected files, callers, consumers, data sources and user-facing paths, reading vendor documentation from the vendors' own live documentation servers first. The coordinator validates every finding against local knowledge and records what was accepted and what was rejected. Mandatory for every moderate and complex task.

Claude OpusClaude Sonnet
Must read first
Navigation, architecture map, code, live vendor docs
Evidence left behind
The research brief, to a fixed template
3.1Do the research files actually exist?An empty brief stops the run

A hard gate: if the research brief is missing or empty, stop.

script
Stops the run when
The research brief is empty
3.15Fact-check the researchEvery function, line, data source and column, against the real files

A validator agent re-reads every claim in the brief against the code. A failure re-runs the researcher. The reason, in the runbook: a brief once said a function had 18 signals; reality was 9 stale ones; nobody verified; three hours and forty-five minutes and a full revert were lost.

Claude Sonnet
Evidence left behind
The validation report
Stops the run when
A claim in the brief is false
3.16Did we map the blast radius?For every change target: who calls it and who reads its output

The brief must contain a feature impact map with explicit full or partial coverage flags, so a change cannot fan out into consumers nobody listed.

Claude Sonnet
Stops the run when
No impact map
SEPView separation gateFires from a computed flag, not from an agent noticing a keyword

A script decides whether the change touches the boundary between the operator's own view and any other account view. If it does, the brief must classify every changed path, and the canonical separation contract governs.

script
Stops the run when
The separation matrix is missing when the flag fires
3.2Verify the data says what we thinkA real sample from production for every data source touched

The coordinator queries production for an actual sample of every data source the task touches. Hard gate on mismatch. The incident: a dashboard showed three of five signals; reality was zero of thirteen; nobody had queried production before planning.

Claude Sonnet
Stops the run when
The data does not match the assumption
3.2.5Do we need a data-flow trace?Four automated triggers

Complex work, a database migration, a new value written into a shared column, or a new background loop each require a data-flow trace at plan time. Written after run 37 shipped two catastrophic data-loss bugs that all four internal reviewers missed and only the external reviewer caught.

script
Evidence left behind
The trace-required flag in the audit brief
Phase 3

Plan and validate

A planner agent writes · the coordinator and scripts check · revision loop5 steps
3.3The planWritten by a different model than the coordinator, with its own re-read of the architecture

A planner agent is told it is not executing the research brief blindly: it must independently re-read navigation, architecture and source. The plan includes tests, rollback, dependencies and a mandatory section naming the skills it used, with the file and lines actually opened.

Claude Opus
Must read first
Navigation, architecture, code, the required skills
Evidence left behind
The plan, with its skills section
3.5The researcher validates the plannerSix checks, including a ground-truth cross-check of every table named

Completeness, pattern adherence, scope, integration, skills, and whether every database table the plan names was actually verified by the research. The reason: run 42's plan named five tables that did not exist and three same-lineage reviewers approved it.

Claude Sonnet
Evidence left behind
Valid, minor fix in place, or a revision request
3.5.4Mechanical existence gateA thirty-second grep that catches every phantom name before the panel spawns

A script extracts every table, column, route, file path and environment variable in the plan's code blocks and checks each exists, with batched database queries and file tests. Deliberately not an AI. Complex runs only.

script
Stops the run when
Any phantom symbol
3.5.5Integration verificationDoes the plan match the code that already exists?

Existing patterns, SDK rules, cross-system contracts, database integration and explicit preservation of existing features are checked and scored.

Claude Sonnet
Evidence left behind
A scored integration document
3.6Revision loopAt most two automatic iterations, then I handle it

Major issues go back to the planner with its memory intact. A reject verdict first sends the researcher back for missing ground truth. Audit-driven revision rounds later in the run are recorded by name; the deepest recorded is nine.

Claude OpusMe
My decision
After two automatic iterations, the decision is mine
Phase 4

Five-way plan audit

Three vendors in parallel · quorum three of four · then my approval5 steps
5Launch five reviewers in one messageNobody can read another's verdict, because none exists yet

A plan auditor and an architecture reviewer (Claude, maximum effort on complex runs), an OpenAI Codex lane and an xAI Grok lane through their wrappers, the coordinator's own verdict, and a cross-validator when dependencies or documentation servers change. Grok is explicitly non-blocking while it earns its place.

Claude OpusOpenAI CodexxAI GrokClaude Sonnet
Must read first
The plan, the harness patterns, the injected skills
Evidence left behind
Five written verdicts
5.1Four mandatory preambles“If you have not read the actual source files, your review is invalid”

Every reviewer is told: you are the auditor, not a rubber stamp; architecture first; read the actual source files; and on trading work, a blocking SDK-compliance check with live verification of every external API call.

Claude Sonnet
5.2QuorumThree of four legacy lanes must return valid output

No verdict until at least three of the four lanes have returned. A reviewer that omits the mandatory harness-patterns section is re-spawned once, then excluded from quorum; if quorum drops, the run blocks and asks me.

Claude Sonnet
Stops the run when
Fewer than three valid lanes
5.3Extract the data-flow traceFail-closed: a missing or placeholder trace re-spawns the reviewer

When a trace is required, the architecture reviewer's trace is extracted between explicit markers and checked: non-empty, exactly five sections, no placeholder text.

script
Evidence left behind
The trace artifact the implementer must satisfy
Stops the run when
The trace is missing or malformed
5.4Check the checkersThe coordinator reads the code behind every finding

A mandatory to-do item: independently validate every plan-audit finding by reading the code and verifying the claim before acting on it.

Claude Sonnet
My decision
Approve the plan for implementation, or send it back
Phase 5

Implement

A separate agent writes the code · scripts prove every claim7 steps
P9Stamp the before-pictureOnce, and never re-stamped

Each repository's commit ID is recorded before implementation. It is never re-stamped on a resume or a fix round, so the diff the external reviewer later sees is cumulative.

script
Evidence left behind
Pre-implementation commit IDs
6A separate implementerNever the coordinator, on moderate and complex work

The coding agent's model and turn budget come from a tier matrix; blockchain work always gets the strongest tier. The implementer follows the plan exactly and adds nothing; if the code differs from the plan's assumptions it stops and the plan is revised rather than the scope silently expanded. It must return a skills-applied section naming the specific rule used per skill. If it runs out of turns it hands over to a continuation implementer, at most three deep.

Claude Opus
Must read first
The approved plan, the research brief, the trace
Evidence left behind
The diff and the implementation report
G-29Implementer output freshnessSilent degradation converted into a loud one

If the implementer's report is missing or older than the plan, which happens when the coordinator applies fix-round edits itself, either a fresh implementer is spawned or a sentinel with per-skill file-and-line citations is written. Neither: the scoring pass must mark the skills checkpoints as failed, not as unverified.

scriptClaude Sonnet
Stops the run when
Stale or missing implementer output without a sentinel
6.1Post-implementation impact validation“Zero insertions or deletions means the implementer's claim is false”

The recorded diff is checked for every file the implementer claims to have changed. The tenant-separation flag is re-affirmed from the real diff. The last three runs' recurring bugs are grepped for.

script
Stops the run when
A claimed edit shows no change
6.2Skills gatesPresence is not application

A nine-check matrix maps task characteristics to skills that must appear. On blockchain work a stricter citation gate demands an actual rule number or pattern ID. This one is an advisory gate: after two retries it logs that it degraded and lets the run continue, so it is not counted among the gates that stop the run.

Claude Sonnetscript
8.7Schema validation gateEvery column in the SQL diff, against the live database

The incident: a helper referenced a column that did not exist and 17 review rounds approved the code.

script
Stops the run when
A missing column
8.8Static linter gateAn unused variable the plan named is a wiring failure

The incident: a value was loaded from the database and never wired to the page; 17 review rounds approved; the user saw blank rows.

script
Stops the run when
An unused plan-named variable
Phase 6

Five-way code audit

Same panel, now against the real code · the coordinator verifies the verifiers6 steps
7The panel, against the codeOn blockchain runs the cross-validator goes first and can stop the audit

A code auditor, the architecture reviewer, OpenAI Codex and xAI Grok in parallel; on traced runs a data-flow tracer joins, with fresh eyes: it does not read the plan-phase findings. Same quorum rule.

Claude OpusOpenAI CodexxAI GrokClaude Sonnet
Evidence left behind
Five verdicts, the tracer's delta report
Stops the run when
Fewer than three valid lanes
7.1Functional testing requirementPick an input, trace it through the code, compare to a known-good reference

Reviewers must go beyond syntax: trace one concrete calculation step by step, test edge cases, and publish the results.

Claude OpusOpenAI CodexxAI Grok
CP7The coordinator verifies the verifiersLine numbers exist, values match the database, then its own trace

Each reviewer's numeric trace is extracted; line numbers are checked to exist; values are checked against the live database; expected values must have a real source. Then the coordinator performs its own independent trace and cross-checks the reviewers against each other. Explicit anti-gaming measures; hard gates prevent a pass.

Claude Sonnet
Stops the run when
A reviewer's trace does not hold up
7.2Official sources overrideA vendor's documentation beats a reviewer's claim about that vendor

Any external reviewer's claim about a provider's behaviour is checked against that provider's own documentation before it is accepted.

Claude Sonnet
8.5Post-fix re-auditWhen a round had criticals and the fix touched a shared column

The fix itself is re-audited, by a different lane than the one that found the problem.

Claude Opus
Stops the run when
Fires on the trigger; the fix must pass
6.6LSkill application ledgerOne row per rule, not per skill, each mapped to code

On blockchain runs the code and security reviewers must table every individual rule of the injected skills with a code location and a verdict. Any unmapped or violated row forbids approval. The incident: code shipped with an unchecked token freeze authority while the security report claimed to have applied the very skill section that forbids it.

Claude Opusscript
Stops the run when
Any rule unmapped or violated
Phase 7

Pre-deploy gates

Scripts execute what reviewers only read4 steps
8.6Live-execution gate“Do not proceed on any timeout. All other paths are forbidden.”

Write-and-time code is dry-run against the real database. New read helpers are run against a known canary entity. A zero-row mismatch or a missing canary on a complex run blocks.

script
Stops the run when
A dry run fails or the canary probe mismatches
7BRuntime test executionFund-moving code must have run, on a mainnet fork, in this run

A test runner must produce a transcript keyed to the run. The gate then re-extracts the cited test bodies from disk, cross-checks pass counts against the framework's own totals, and hashes the raw log against the archive. Missing transcript, count drift, empty proofs or zero entry-point coverage all block. The comment trail shows this gate re-hardened a dozen times against ways a forged transcript could slip through.

script
Evidence left behind
The signed test transcript
Stops the run when
No transcript, count drift, or zero coverage
7B.5Harness flip integrityMarking a pattern satisfied needs two mechanical citations

A non-trivial assertion and proof of reachability with no early exit. The incident: four patterns were flipped to satisfied by editing a count; one cited test asserted that true is true, fourteen others exited before reaching the code they claimed to test.

script
Stops the run when
A bare claim of coverage
10Security gateFirst: has the code moved since the audit began?

If the repository has moved since the commit pinned at step 1.5, the findings may be stale and the run blocks. Then a recommendation: security-relevant work spawns a security auditor that itself spawns three sub-auditors, blinded to its own findings to prevent anchoring. I always have the final say.

scriptClaude OpusMe
Stops the run when
The code moved underneath the audit
My decision
Spawn the security auditor, or accept the recommendation not to
Phase 8

The external round

A different vendor's model, in a session I open myself6 steps
9ATriggerMandatory on traced work, money-moving work, and complex UI work

The origin: run 37's rollback bug would have wiped 22,102 production records and was caught only by the operator's external reviewer, while all four internal lanes approved. The money-path trigger became unconditional after 15 consecutive runs in which the external round caught what the internal panel missed.

script
Stops the run when
A required trace is missing: inferring “no file means not required” fails open
9A.1Freeze, assemble, redactNothing is reviewed dirty

The in-scope state is committed and its commit ID recorded. The brief builder refuses to render from dirty files. It pulls plan, brief, trace, changed files, implementer output and navigation ranges, and replaces every server address, private network address, database identifier, key path and dashboard credential with redaction tokens before writing, stamping a receipt of how many replacements it made.

script
Evidence left behind
The redacted brief and its receipt
9A.1-BRender the promptA universal template with two independence guards written in capitals

Known anti-patterns must be framed as rebuttable, never “do not re-flag”. Per-lane targets are a floor, never a ceiling, with explicit latitude to look outside the changed files, because “the orchestrator is a single model and will miss things.”

script
9A.1-CWho runs itDefault: I paste it into my own separate session

The default channel preserves full process independence: I paste the brief into a separate OpenAI Codex session and paste the verdict back. A fallback where the coordinator spawns the external model itself is labelled reduced independence and can only be launched by a script that refuses to run without my verbatim authorising words, logged before spawning.

MeOpenAI Codex
My decision
Run the external session myself
9A.1.5The prompt gateTwenty-eight checks; fail-closed if the checker itself is missing

Any inline production access, unresolved template slot, wrong skill bundle for the system type, or empty navigation range blocks. If the checker script is missing, the run still blocks.

script
Stops the run when
Any of 28 checks, or the checker is absent
9A.4Receive and validate the verdictMechanical validation of every finding, and a downgrade guard

Freshness of the brief's commit, format conformance, artefact shape, degraded-round detection, then each finding validated mechanically. On money-moving work, downgrading an external critical or high finding requires a code receipt and, where traced, a runtime receipt proving the neutralising path executes. A second external auditor agreeing with the downgrade does not satisfy the gate, because that has happened and production disproved it within a day.

Claude SonnetMe
Stops the run when
A downgrade without receipts: the finding stays open
My decision
Accept, refute with receipts, or send back to plan revision. There is no timeout.
Phase 9

Deploy and verify

Scripts stage exactly this run's files · machines test before I do4 steps
11DeployOnly from the default branch, only this run's files

A reconcile gate runs once per deployed repository and blocks if a foreign or missing file appears. Then pull and restart on the servers by script, a release build for the Rust engine or an on-chain deploy for the program, and ten minutes of monitoring.

scriptMe
Evidence left behind
Deploy receipt and monitor log
Stops the run when
A foreign or missing file in the staged set
My decision
Ship, or do not ship
9C-autoAutomated browser verificationMandatory-fire: the trigger once met and zero browser calls happened

The browser is driven for real; console errors and network calls are compared with the baseline from step 3.0B. Added after a run whose 3.7-megabyte transcript contained no browser tool call at all.

script
Evidence left behind
The after-picture, diffed against the baseline
Stops the run when
The precondition is met and the check did not fire
9C-dataData validation harnessThe same numbers across pages and filters must agree

For data-pipeline work, the harness cross-checks figures across pages and filter combinations. It evolves after every run.

script
Stops the run when
Cross-page disagreement
9CThe human checklist, with a twist“Did you test all these yourself?”

Before I see anything, every checklist item is tagged as browser-verifiable or human-only, and the coordinator runs every automatable item itself. I get only what genuinely needs a human. Then a hard wait: the run does not advance until I report back.

Claude SonnetMe
My decision
Confirm the result, or report defects and start another fix-and-review cycle
Final

Document, archive, self-score

The coordinator rewrites the map · an archive agent freezes the evidence · the run grades itself5 steps
F1Session summaryEverything the run did, with real commit hashes

Artefacts, audit files, changed files, both five-way verdict sets, an executed-versus-read ledger, gate evidence, skills evidence, external-round evidence, post-deploy issues.

Claude Sonnet
Evidence left behind
The session record
F1.5Rewrite the architecture mapBy the coordinator directly, never delegated

A fresh agent succeeded at this one time in four across earlier runs, so the coordinator does it itself: read the changed source files, not the plan; rewrite the affected sections as current state; sync any canonical contract whose domain was touched; regenerate the navigation index from section markers; run the drift test; spot-check two or three documented flows against the code.

Claude Sonnet
Must read first
The changed source files
Evidence left behind
Updated architecture map and navigation index
Stops the run when
Any of seven quality checks fails
F1.99Was the summary actually written?A chokepoint added after five zero-byte session files

A hard gate that the session record exists and has content, plus a pre-archive guard for blockchain artefacts.

script
Stops the run when
A zero-byte record
F2ArchivePlan, diff, verdicts, the raw transcript, file fingerprints

An archive agent creates the dated archive, copies every artefact including the raw session transcript, and writes the report with a mandatory harness-enforcement stamp. If no artefact proves the gate fired, it must write “not found”, never infer it did.

Claude Opus
Evidence left behind
The archive and its report
F3The run audits itselfCannot end without a review, an explicit opt-out, or a documented failure

The progress review scores the run and every reviewer lane, classifies each post-deploy bug as preventable or new, records gaps with an ID, a severity and a proposed rule, and checks that every red recommendation is shipped with evidence or retired with my words quoted. Four consecutive runs once skipped this step; the skip cost weeks of drift.

Claude Sonnetscript
Evidence left behind
The scorecard and the gap entries
Stops the run when
No review artefact at all
34catalogued gates
146hard-block instructions in the runbook
8checkpoints that are mine
About the numberingShow less

Where the runbook's numbering is out of sequence the order follows its executable to-do list. Steps fire on their triggers: the data-flow trace on data lineage or migrations, the Solana gates on the two Solana codebases, the runtime test gate on fund-moving code, the external round when the run is flagged for it. Simple runs skip planning, the panel and the archive, but still run the deploy gate, a browser regression check and a tracker entry.

Who reviews

Different vendors, different questions, and a record of who caught what

Lane count is not lane diversity. The panel mixes models from three vendors, two blind roles and deterministic scripts, and every lane is scored, so the ledger shows which ones earn their place.

Claude Opusplans, implements, reviews
Claude Sonnetcoordinates, fact-checks
OpenAI Codexbug hunter · external round
xAI Grokindependent bug hunter
Tracerdata flow, blind to other findings
Scriptsdeterministic gates
Mescope, release, retirement
3model vendors on the panel
2blind roles
78 / 92runs where the outside reviewer caught what the panel missed: why that round is never skipped
Who caught whatShow less
LaneRoleIndependenceIn the ledger
Claude OpusPlans, implements, and reviews plan, code and architecture; maximum effort on complex runsSame lineage as the coordinator, different roles and contextsThe most frequent lane; 16 plan rounds once blessed a feature flag that did not exist
Claude SonnetCoordinates, validates research and findings against the real code and data, cross-validates citationsFact-checker of everyone, including the coordinatorRun 122: executed a test probe and overturned the coordinator's own refutation
OpenAI CodexInternal review lanes, a second data-flow trace, and the separately run external roundDifferent vendor; the external round runs outside the workflow on a frozen commitPresent in 111 of 112 runs; scored 9.33 of 10 across 51 scored runs; caught 15 critical findings in run 49 that Claude reviewers had approved
xAI GrokIndependent bug hunter on plan and code, added July 2026Third vendor; started non-blocking while it earned its place13 runs, 8.22 of 10 across 9 scored; run 128: contradicted a maximum-effort Claude critical, and the source proved Grok right
Data-flow tracerCompares the promised data path with the code, at plan time and again in the codeForbidden from reading the other reviewers' findings30 recorded passes, up to 35 of 35 assertions
ScriptsExistence gates, schema checks, linters, brief redaction, prompt gate, routingDeterministic; no judgment, no fatigue146 hard-block instructions; a gate stays red until it has executed on representative data
MeScope, route, plan approval, release, verification sign-off, retirement of recommendationsThe only party that can authorise a deploy or retire a red findingTwo of the six failures on this page were caught by me, not by any model
  • A retired lane is retired in writing. The Google lane's own epitaph in the ledger: “1 unique finding in 13 runs”.
  • External catches are counted per run. The field “external caught what internal missed” reads yes 78 times and no 14 times across the recorded rows, with a recorded streak of 40 consecutive runs.
  • More reviewers did not always mean more truth. The latest run records four deployments and eleven review rounds in which nobody opened the page; the fix was a live round trip through the real screen, not another reviewer.

78 of the 92 runs that record the field; 20 of the 112 runs do not record it, and the ledger does not break these catches down by severity. Scores are the workflow's own progress reviews, kept in the run ledger: an internal quality record, not an independent certification.

Skills and harnesses

Official references first, and proof they were used

Twenty-two skill packages, mostly official vendor documentation, are injected into every agent that needs them. Citing a skill is worth nothing on its own; a seven-checkpoint chain follows each one from plan to code to review.

22skill packages
7checkpoints per skill, per run
4,763lines of harness patterns
How use is provenShow less
Checkpoint 1–2

The planner cites

The plan names the skills it used and the rules it applied. The plan auditor verifies each citation exists and says what the plan claims.

Checkpoint 3–4

The implementer applies

The implementation report maps each skill rule to a file and line in the diff. The coordinator checks the rule applied to this task at all.

Checkpoint 5–7

The auditor proves

The code auditor produces a compliance table, the harness patterns consulted are cited, and a review without them is rejected and re-run.

  • False or shallow use is named, not graded: a citation to a rule that does not exist is a broken citation; a skill mentioned but not applied is name-dropped; a required skill not used is a missing rejection.
  • Three testing harnesses hold 4,763 lines of known failure patterns, one per past incident. A new safety gate stays red until it has run on representative data; reading the code does not count as executing it.
  • Community snippets are treated as untrusted. One shipped the wrong byte offsets for a trading venue; official sources are now a hard gate, with pinned, digest-verified reference files.
.claude/skills/ · directory listing
anchor-test-harness birdeye data-accuracy-check
design-system-audit frontend-design helius
helius-phantom integrating-jupiter meteora
orca owasp-security postgres-patterns
pumpfun raydium rust-skills
safe-solana-builder solana-dev squads
ui-ux-pro-max vulnhunter web-design-guidelines
21 of 22 shown; one name is withheld because it names a product area not covered here

The skill packages as they sit on disk. Several are official vendor documentation packaged for the models.

One machinery, five commands

How a run survives the model losing its memory, and lasts for days

The durable unit is not the conversation but the run ID plus its evidence on disk. Five commands hand that state to each other, which is why a run can span days and several memory resets.

30 hlongest single session, run 49
4 dayslongest calendar span, run 130
678archived handover files
How a run survives memory lossShow less
audit-task

The run

The state machine described above: assess, research, plan, review, implement, review again, deploy, verify.

audit-task-handover

The save file

Before the model's memory is compacted, a timestamped handover records run identity, plan revision, phase, pending reviewers, the live to-do list and every owed gate.

resume

Hypothesis, then proof

A resumed model must restore the tracker, verify it against the live repository, re-arm every owed gate and re-read the architecture. The handover is a hypothesis; verification commands are the proof.

audit-archive

The frozen record

After deploy and testing: report, evidence package, file fingerprints, the rewritten architecture section, the regenerated index. Disposable artefacts are staged, never blindly deleted.

audit-task-progress

The retrospective

Scores the run and each reviewer lane. When no single transcript exists, it reconstructs the review from every handover, verdict and report, and sweeps all handovers for durable findings.

documentation-audit

The periodic check

Sixteen times so far: undocumented code, obsolete statements, history in the map, broken navigation, secrets, then an adversarial pass over the documentation diff.

  • Run 49 ran about 30 hours across two memory resets. Run 51, the workflow auditing itself, ran about 28 hours over two days across two resets and two handovers. Run 134 took three days and about 26 review passes.
  • A run once dropped six findings that its first two handovers had deliberately recorded, because the review swept only the last handover. I caught it. The rule since: every handover in the chain is swept, and each carries machine-readable durable findings.
  • 678 handover files sit in the archives, one for every time a run paused, compacted or changed sessions.
Failures → controls

Six times something broke, and what exists because of it

Six real failures from the run ledger, and the permanent control each one produced. Two of the six were caught by me, not by any model.

What broke
The AI reported “done”. The file was unchanged.
What exists now
Every claimed edit is proven by the recorded changes before any reviewer sees it.
Run 31 · Apr 2026
What broke
An agent declared a test tool “not installed” after one shell lookup. It was installed.
What exists now
A standing rule: never assert a tool is missing from one lookup; the check looks in every valid install location.
Operator-caughtJun 2026
What broke
A handover said the tests passed. The test record showed three failures, on transaction-critical code.
What exists now
Pass or fail is derived by a script from the test record, never from prose.
Run 108 · Jul 2026
3 moreShow less
What broke
Twelve external rounds said yes. The deploy locked the whole dashboard out for an hour: a refusal shipped before its data existed.
What exists now
A deploy-sequencing gate: populate before you enforce. Plans carry a release condition per feature.
Run 127 · Aug 2026
What broke
The safety test was green because its inputs were chosen where old and new code agree.
What exists now
Widened to 25 cases it went red: 28 production cases would have been handled wrongly. That corpus is now a gate.
Run 132 · Aug 2026
What broke
Eleven review rounds read the code; nobody opened the page. The feature would have refused every request.
What exists now
One live round trip through the real screen before reviewers are dispatched. Caught before release by the first test that did.
Run 134 · Sep 2026

From the run ledger, verbatim: “Every single one was found by widening the check, never by re-reading the same layer harder.”

Scorecards

Every run is graded, and so is every reviewer

After every run a separate command scores the run against a five-tier rubric and grades each reviewer lane, so the ledger shows which reviewers earn their place.

One real scorecard · run 133 · 19 Aug 2026 · from the written progress review
Coordinator gates11 / 12
one recorded as not run, never as passed
Reviewer checks10 / 10
Security, UI, verification8 / 8
Harness patterns cited14 / 14
plus 3 new patterns added
Skill-use chain7 / 7
citations at every step
OpenAI Codex lane10 / 10
no degraded rounds
xAI Grok lane8 / 10
unique catch the other two missed; text output, not structured
Who does what · five lanes
  • Claude Opusplans, implements, reviews architecture and code; maximum effort on complex runs
  • Claude Sonnetcoordinates, validates plans and findings against the real code and data
  • OpenAI Codexinternal review lanes and the separately run review; 9.33 of 10 across 51 scored runs
  • xAI Grokindependent bug hunter, added July 2026; 8.22 of 10 across 9 scored runs
  • Data-flow tracerchecks the promised data path at plan time and again in the code; 30 passes recorded, up to 35 of 35 assertions
121written progress reviews
5reviewer lanes graded per run
30data-flow passes · up to 35/35
How the rubric worksShow less

The rubric has five tiers: 17 coordinator gates, 16 reviewer checks, 8 security and verification checks, 9 Solana gates, 6 harness-health rows. Each reviewer lane gets a plain-English grade, star to silent, and exactly one is named star performer. The best signal of a reviewer's value is its unique catches: findings nobody else found. A “no” is a gap by definition; every post-deploy bug is classified as preventable, partly preventable or new and mapped to the gate that should have caught it.

Gaps

Not every gap becomes a rule. The ones that return, or are red, must.

A gap is anything that should have fired and did not; each gets a stable ID, a severity and a proposed fix. A triage rule then forces a decision, not necessarily a fix.

235
gap IDs recorded since March 2026
87
cited inside a step, gate, script or test
36
open today
The triage rule, and four gaps that became rulesShow less
  • Red “must fix” recommendations must be dispositioned before the run can end: shipped, with file evidence, or retired, with my own words quoted verbatim. The coordinator may not paraphrase or infer a retirement. Yellow and green stay advisory.
  • The rule exists because of a measurement: of six red recommendations in July 2026, none had reached a workflow file, while all six harness notes from the same reviews had. The only difference was the word “must” versus “may”.
  • When an external reviewer catches what every internal lane missed, the harness is updated immediately or the waiver cites the rule that already covers it. In the ledger's words: the tracker is the diary, not the checklist.
  • For test patterns there is a universality table: a mechanism seen in two or more runs, or affecting two of the three codebases, or with a greppable signature, becomes a pattern; a one-off becomes a note. Severity escalates on repetition, a claimed fix is grepped for before it counts, and a rule is retired only after three clean runs.
AUDIT_TASK_RUN_TRACKER.md · four gaps that became rules, shortened
G-29 runs 55–56 Skills accountability silently degraded when the coordinator implemented inline → a freshness gate: implementer output must be newer than the plan, or a re-spawn, or per-skill file-and-line citations SHIPPED 2026-05-17
G-71 run 70 The longest run had three handovers and no single transcript, so the review could not run → a transcript-substitute mode rebuilt from handovers, verdicts and reports SHIPPED same day
G-122f run 2 of a hardening series The review swept only the last handover and dropped six recorded findings; I caught it → every handover in the chain is swept and carries machine-readable durable findings SHIPPED 2026-07-27 · exercised in 5 runs
G-115a run 115 A money-moving run was handed a vanilla external brief; the external model then repeated the identical shortcut → the handoff now cross-checks the run's own flags SHIPPED
G-60 run 67 Pre-existing navigation drift hard-blocked an innocent run → proposed: block only drift this task introduced DESIGN PENDING

Gap IDs are never reused or renumbered. A retired recommendation is a legitimate outcome; an absent decision is not.

Where it stops

A claim without evidence stops the run

Thirty-four catalogued gates. Each one turns a sentence an AI might say into a check a script performs, and blocks the run when the check fails.

The claim
“I edited the file.”
What the workflow does
The recorded changes show nothing changed. The claim is false. Back to implementation.
The claim
“The plan uses this piece of data.”
What the workflow does
That data does not exist in the database. Blocked before a line is written.
The claim
Transaction-critical code, no record that the tests ran.
What the workflow does
No deploy. The test record must exist and match this run.
34catalogued gates
146hard-block instructions in the runbook
62steps a script or a model executes
Three more, and the gates in the fileShow less
The claim
The external brief contains a server address.
What the workflow does
Hard block. The brief never leaves the machine.
The claim
Fewer than three of the four core reviewers finished.
What the workflow does
No verdict, no deployment. A thinner panel is not a panel.
The claim
The external verdict is not back yet.
What the workflow does
Wait. There is no timeout. The point of the gate is the independent opinion.
.claude/commands/audit-task.md · four of the 34 gates, shortened
Step 8.6 Live-execution gate, stage A: detect write + datetime trigger files; dry-run first; HARD BLOCK on dry-run failure (Run 46)
Step 8.7 Schema validation gate: every <alias>.<column> in the SQL diff is checked against the live schema; HARD BLOCK on missing columns (Run 47)
Step 8.8 Static linter gate: lint every changed Python file; HARD BLOCK on unused plan-named variables (Run 47)
Step 9A.1.5 Prompt gate: fires before every external auditor handoff; HARD BLOCKs on a dirty brief
… "Zero insertions or deletions means the implementer's claim is false."

From the runbook, lightly shortened. Each gate carries the run that needed it.

Each gate is traceable to the run that needed it.

Economics and endurance

Two subscriptions and a thirty-dollar plan run all of it

One Claude Max subscription, one ChatGPT plan for Codex, one thirty-dollar Grok plan. Review cost is routed by risk rather than capped, and anything that can lose money always runs at full depth.

Before routing: every lane at maximum100%Standard runs after routing42.5%Deep reviews after routing15.2%Money-moving work, always100%
Token use per review as a share of the pre-routing control run, which consumed about 13.4 million tokens. From the routing A/B recorded in the repository.
−84.8%tokens on deep reviews after routing
−57.5%on standard runs
0missed high or critical findings in the three A/B samples
How the cost is controlledShow less
  • Claude runs on a single Max subscription: the coordinator, every specialist agent and every Claude review lane count against that one quota. OpenAI Codex runs on a ChatGPT plan; xAI Grok on a thirty-dollar monthly plan. Those are my bills, not figures from the repository.
  • Review cost is routed, not capped. A lighter fork of the whole workflow saves 40 to 55 percent of Claude tokens on moderate work by dropping duplicate lanes, and it never drops the external round. Since July 2026 a deterministic policy picks the model and effort per lane by complexity. In the measured A/B, deep reviews used 84.8 percent fewer tokens and standard runs 57.5 percent fewer, with zero missed high or critical findings across three samples.
  • Money-moving work never routes cheaper: it always runs the full high-effort fleet. And no monetary budget is enforced anywhere in the workflow; the only budgets in the routing code are wall-clock timeouts. Endurance comes from handovers and archives, not from spending more.

Subscription costs are the operator's statement. Token savings are from the routing A/B recorded in the repository: three samples, which is enough to enable a route, not to certify it; one missed high or critical finding rolls a route back. A control run before routing consumed about 13.4 million tokens.

Measured · 3 September 2026

Measured, with the populations printed

Six numbers from the run ledger, the archives and git, each with the population it was counted on.

112
documented runs, numbered 1 to 134
26 March to 1 September 2026, about one run every 1.4 days. 93 complex, 16 moderate, 1 simple.
235
gap IDs recorded · 87 written into a step, gate, script or test
36 open today. A gap leaves the open list only after three confirming runs.
121
written progress reviews
Each scores gates fired, reviewer checks, skills used, harness patterns cited.
201
archives since December 2025
188 with a written report. Plan, code changes, review decisions and file fingerprints frozen per run.
78 / 92
recorded rows where the external review caught what internal reviews missed
Peak streak recorded: 40 consecutive runs.
62 / 112
runs claiming zero post-deploy bugs
The others are documented too, including one severity-one incident and one shipped-with-reverts.
Where the numbers come fromShow less

Every figure is from the run ledger, the archives or git, and the ledger is written by the same system it evaluates. It is a high-resolution internal record, not an independent certification. Multiple AI reviewers are not an independent security audit; where money or security is at stake, the boundary is a human specialist.

See it running

The archive and the system are the evidence.

In an interview I can walk through one archived change and one live product flow: the evidence, the release decision, the result, and the limits of this approach.

01

One archived change, end to end

A real run from the archive: the request, the plan, the review decisions and fix rounds, the separate review's verdict, my release decision, and what production showed afterwards.

02

One live product flow

The system this loop built, running: a signal arriving, the engine executing, the record it leaves, and the numbers behind case study 1.

the boundaryWhere I would not use this: a security-critical change under time pressure, or anything a regulator would need to sign. Those get a human specialist and a slower path, not a review panel.

In a team I would own: scope and acceptance criteria, the review gates, release decisions, and the retrospective loop. Engineers own the code; the workflow makes their review load smaller, not larger.

Lukasz Rodzen
product owner · release owner · operator
Poland · remote

I own the product intent, the system boundaries and constraints, the acceptance criteria, the release decision and production verification. AI agents provide much of the implementation-level technical depth: they research, propose designs, write the code and review one another independently. I resolve their disagreements through evidence and decide what ships. I do not claim the unaided technical depth of a senior backend or Rust engineer.