FireTrident: a colony, not a queue

Scaling benchmark, 2026-10-12. Live runs with real Artifacts commits (the serial trunk writer on the production deployment; group commit and the scale runs on preview deployments) and a discrete-event simulator calibrated on them, comparing FireTrident with the common design: exclusive file locks plus one serialized merge queue. Every chart and tile says whether its points are MEASURED or SIMULATED, and every count of failed calls names its run.

Throughput on the live workloadMEASUREDSIMULATED

Synthetic agents on the production deployment with the serial trunk writer, 4 runs at 10 to 1,000 concurrent agents. Each change adds one file in one of 64 directories, and every landing is a real trunk commit pushed with compare-and-swap. The lines are the simulator replaying the same workload with costs fitted to these runs: it lands within about 10% of every measured point.

Trunk group commit, built and MEASURED (2026-10-12-g1, on a preview deployment, before the scale fixes). The trunk writer now takes every lane batch that queued while its previous cycle ran: it re-validates each proposal against trunk and against the batches ahead of it, builds one commit per batch chained on the one before (so lineage and why stay per landing), pushes trunk once and all lineage notes once. Same driver, same workload: 10 agents 216/min (serial writer 92/min), 200 agents 690/min (serial writer 126/min), 1,000 agents 515/min (serial writer 138/min), stopped early by the driver's error guard.

Scale: 1,000 to 5,000 agents on one colonyMEASURED

These are synthetic agents: no model calls, no agent-side git and no work time, with the review layer turned off on the colony (POST /api/colonies/:id/decide/policy {"enabled":false}), so the runs measure coordination and landing only. Same driver and workload as above: each change adds one file in one of 64 directories, and every landing is a real Artifacts commit pushed with compare-and-swap. 120 s per run with a 10 s ramp; outcomes reach the agents over the live WebSocket with a 20 s fallback poll; group commit on; one colony (g6-scale5k) on a preview deployment with its own Durable Objects and a pre-forked workspace pool. Raw runs: ../2026-10-12-scale/.

What changed since the group-commit run: takes share one read of each region's claims, at most 128 takes score at once and waiting agents are paced, outcomes are pushed to agents instead of polled, stopped agents release their claims at once, leasing a workspace is pure SQL with one token kept per slot, and lineage notes use git's two-character fanout (docs/ARCHITECTURE.md, Scale).

Failed calls, per run. 1,000 agents: 0 of 31,308 calls failed. 2,000 agents: 0 of 45,034 calls failed. 5,000 agents: 78 of 50,368 calls failed (0.15%), all in the first 33 s. The error kinds the driver recorded (it keeps the first eight distinct messages) are connection failures on the bench client (ENOTFOUND, ECONNRESET, UND_ERR_CONNECT_TIMEOUT), while one bench machine held up to 3,597 open sockets around the end of the ramp, and the service answering 500 Durable Object reset because its code was updated. (a Durable Object restarting on newly deployed code).

Lineage. After the runs, stock git fetched trunk and refs/notes/firetrident from colony g6-scale5k's Artifacts repository: 5,894 of 5,894 landing commits carry a lineage note (0 without), 60 sampled git notes show name their landing (0 mismatches), and the notes tree is 256 two-character fanout directories with 0 flat notes.

Next limit. The next limit is TrunkDO, the single trunk writer. Each commit cycle took 3.4–3.9 s at the median and carried 195–196 proposals, against its cap of 200 (CYCLE_MAX_PROPOSALS), so throughput is flat past 1,000 agents: 2,714–2,924 landings/min from 1,000 to 5,000 agents. About 6 s of queueing for the writer (5.7–6.0 s p50) dominates time to land (12.5–14.8 s p50). Inside a cycle, reading proposal blobs out of workspace forks takes longest (1.8–2.2 s p50), then the trunk push (1.0–1.2 s).

Earlier runs on the same colony, not in the table: 3,485/min at 1,000 agents and 3,342/min at 2,000 agents with the build before the take gate; the trunk cycle grew from 2.9 s to 3.9 s p50 over the 5 completed runs as the bench's trunk tree grew (each intent adds a file). 2 attempts at 5,000 agents stopped early: build before the take gate (10:56Z): ColonyDO isolate exceeded its memory limit about 20 s in (thousands of takes in flight, each holding up to 200 scored candidates) (error rate 84% over 5 s at 20 s); and take gate build, started 1.5 min after its deploy: a burst of dropped connections (cause not recorded yet) and 'reset because its code was updated' tripped the guard at 15 s (error rate 35% over 5 s at 15 s). summary.json records all of them; their raw files were not kept.

Throughput from 10 to 100,000 agents on hono's historySIMULATED

Write sets drawn from 2285 real changes to honojs/hono, grown along co-change edges. Read sets are the written files' direct imports. The repo is a monorepo of 100 copies (regions at depth 3), and agents work a log-normal median of 60 s per change. Mean of 3 seeds; hover a point for the seed range.

The seizure detector: one landing, 30 stale returnsMEASURED

Live, colony bench-w2-cascade. 30 agents in src/core read src/core/config.ts and kept working. A real commit to that file landed, and all 30 in-flight proposals came back stale, each attributed to that one landing. σ jumped to 30, so the lane's cadence stretched from 1 s to 8 s. The reworked proposals then landed in inhibited batches until σ fell below 1. σ points are the sigma events on /api/colonies/:id/live; the line is GET /field polled every second.

Tables

Method and assumptions

• Live (MEASURED). Synthetic REST agents take, propose, and poll their outcome on the in-platform schedule (1.5 s, ×1.6, capped at 15 s). Each proposal writes a new file whose blob already exists in every workspace fork. That makes the trunk writer do the real work (read the blob from the agent's fork, build the tree, push trunk with compare-and-swap, write the lineage note), while the agents need no git of their own. The review prong (reflex and LLM triage) was turned off on the two bench colonies, so the numbers isolate coordination and landing. Latencies are laptop-side and include about 100 ms of internet round trip.
• Calibration. In-platform take 250 ms and propose 30 ms; lane overhead 100 ms. Trunk writer per commit: 620 ms + 72 ms per proposal (fitted on the swarm runs and a single-region batch experiment). The baseline lands a change in 692 ms (the same writer with one proposal) plus 350 ms to rebase a stale change (fetch and replay, from the Artifacts spike). Checks cost 0 ms in the headline runs, the most favourable setting for the baseline.
• FireTrident model. Mirrors the deployed code: take scores the 200 oldest open intents with core chooseIntent, using repulsion from claims and admissionOpen per region. Each proposal joins the lane of its primary region, where core validateProposal and selectBatch run. The tick interval is laneIntervalMs × inhibitionFactor(σ), and σ is computed with core branchingRatio over recorded causes, where returns are never parents. One trunk writer per colony serves the lanes' commits one at a time and re-validates each batch. A returned intent reopens and is redone in full.
• Baseline model. Exclusive locks on write sets, offered from the same 200 oldest tasks, with the same 3 s retry hint. One FIFO merge queue: a stale change is rebased and its checks re-run, and a change whose read or written files changed since its base fails (the same rule as read-set validation). It goes back to its agent, which keeps its locks, reworks half of the change, and resubmits.
• Scale runs (MEASURED). The same synthetic agents with no work time, the review layer off, 120 s per run, outcomes pushed over the live WebSocket (20 s fallback poll), and every run's agents stopped at its end so their claims never hold the next run's admission. Throughput is counted from 25 s to 120 s, after the ramp.
• Not modelled. The coordinator's own request capacity (one ColonyDO per colony) is assumed to scale in the simulator. Live, it held in the serial trunk writer run (1,000 agents: 0 of 29,307 calls failed, take p99 3,617 ms); in the group-commit run, which pushed 3.7× as many landings through it, it saturated at 1,000 agents (816 of 9,927 calls failed (8.22%); the driver stopped the run: error rate 38% over 5 s at 60 s); after the scale fixes: 1,000 agents, 0 of 31,308 calls failed; 2,000 agents, 0 of 45,034 calls failed; 5,000 agents, 78 of 50,368 calls failed (0.15%), all in the first 33 s. Fork creation is not modelled (the pool is assumed warm).

What changed in RegionDO for the demo

• σ treats returns as leaves. A returned proposal changed nothing on trunk, so it cannot make anything stale. Counting every event in the window as a parent pinned a burst of N returns from one landing at N/(N+1) < 1 and hid every cascade. σ is now stale returns per landed proposal, with core branchingRatio's window semantics (60 s window, 5 s settle).
• σ is re-evaluated after the lane empties. While σ > 1, or while a recent return may still raise it, an empty lane keeps a 2 s alarm. Previously σ was sent only after ticks that had proposals, so neither the spike (which arrives after the settle time) nor the recovery reached the UI. An enqueue into an empty lane replaces that alarm, so the inhibited cadence still holds.
• Inhibition is published. Every alarm sends sigma, and while the factor is above 1 (and once when it lifts) it also sends metric events inhibition:<region> and laneDelayMs:<region>. Deployed 2026-10-12 06:14 UTC; DO tests pass, including a new case where 4 returns from 1 landing give σ = 4.

Findings and open issues

• Trunk group commit lifts the one-commit-at-a-time ceiling (MEASURED). At 1,000 agents the colony landed 515 changes/min, 3.7× the serial writer's 138/min, with 0 returns across the group-commit runs. Failed calls in the group-commit runs: 10 agents, 0 of 732 calls failed; 200 agents, 22 of 7,429 calls failed (0.30%); 1,000 agents, 816 of 9,927 calls failed (8.22%). The 1,000-agent run stopped early (error rate 38% over 5 s at 60 s): the one ColonyDO that serves every take, poll and propose saturated, so that run's rate is measured over a window it only partly worked in. The writer ran 24 cycles/min, each landing 11.8 lane batches (16.6 proposals) as chained commits behind one trunk push and one notes push. A cycle took 890 ms at the median: pull 132 ms, build 43 ms, push 517 ms. Each lane batch is still its own commit with its own lineage note, so why stays per landing.
• The scale fixes remove the coordinator limit (MEASURED; synthetic agents, no model calls, review off). 1,000 agents 2,924/min, 2,000 agents 2,714/min, 5,000 agents 2,778/min, against 515/min for the group-commit run at 1,000 agents before them. Failed calls: 1,000 agents, 0 of 31,308 calls failed; 2,000 agents, 0 of 45,034 calls failed; 5,000 agents, 78 of 50,368 calls failed (0.15%), all in the first 33 s.
• Next limit: the trunk writer. The next limit is TrunkDO, the single trunk writer. Each commit cycle took 3.4–3.9 s at the median and carried 195–196 proposals, against its cap of 200 (CYCLE_MAX_PROPOSALS), so throughput is flat past 1,000 agents: 2,714–2,924 landings/min from 1,000 to 5,000 agents. About 6 s of queueing for the writer (5.7–6.0 s p50) dominates time to land (12.5–14.8 s p50). Inside a cycle, reading proposal blobs out of workspace forks takes longest (1.8–2.2 s p50), then the trunk push (1.0–1.2 s).
• Fixed: landed proposals reported as returned. A lane's TrunkDO.land call could fail on the lane's side after the trunk writer had landed the batch (the logs show Durable Object reset because its code was updated during deploys). The retry then reported those proposals as already landed in lnd_… (13 times at 1,000 agents), and a lane re-validating them saw them stale against their own new blob (on the Hono run one landed proposal was returned and its intent taken and abandoned 19 times). Outcomes are now idempotent per proposal id: the trunk writer reports earlier landings in prior, and a lane asks TrunkDO.landingsOf before returning a stale-looking proposal, so a landed proposal is always settled as landed.
• Admission ignores the shared trunk writer. The floor of 3 agents per region admits far more agents than one writer serves when a colony has many regions. With the 200-oldest candidate window and repulsion, contended intents clog the window and are eventually taken anyway. That is why rework is about 42% vs 22% for locks on hono's history.
• σ was per region (fixed since). A stale read caused by a landing in another region got no cause, so cross-region cascades stayed invisible to the governor. A stale return with no local cause now records the other region's landing as its parent (RegionDO.landedFor).
• The review prong triages every proposal (deployed 06:32 UTC). It returned 41 of 87 synthetic proposals and added seconds each, so it was turned off on the two bench colonies with POST /api/colonies/:id/decide/policy {"enabled":false}. The 10-1,000 agent runs predate it. Two other deploys happened during the runs (06:24 and 06:32 UTC); the 200-agent run spans the first.
• Account use. 286 workspace forks (bench-w2-load 249, bench-w2-cascade 37), 8 seeds, 2 trunk imports. Both colonies are kept for re-runs and video capture; DELETE /api/colonies/:id removes them. The group-commit runs used one more colony on a preview deployment (g1-bench: 253 workspace forks, 4 seeds, 1 trunk import). The scale runs used one colony on another preview (g6-scale5k, with a workspace pool pre-forked to a target of 6,000).

Reproduce

# live runs talk to a deployment: FT_URL and FT_TOKEN (its admin token) come from the environment or ~/.config/firetrident/env
node packages/bench/src/workload.ts --repo /tmp/ft-w2/hono --tree <checkout of hono HEAD> --out packages/bench/data/hono-workload.json
FT_URL=<your deployment> FT_TOKEN=<admin token> node packages/bench/src/live.ts --colony bench-w2-load --agents 200 --duration 90 --out results/live-n200.json
FT_URL=<your deployment> FT_TOKEN=<admin token> node packages/bench/src/scenarios/batchcost.ts --colony bench-w2-load --sizes 2,10,25,50 --out results/batchcost.json
FT_URL=<your deployment> FT_TOKEN=<admin token> node packages/bench/src/scenarios/cascade.ts --colony bench-w2-cascade --readers 30 --out results/cascade.json
node packages/bench/src/sim/sweep.ts --mode live --agents 10,50,200,1000 --seeds 1,2,3 --out results/sim-live.json
node packages/bench/src/sim/sweep.ts --mode history --agents 10,100,1000,10000,100000 --seeds 1,2,3 --duration 1200 --warmup 300 --out results/sim-history.json
node packages/bench/src/sim/sweep.ts --mode history --agents 10,100,1000,10000,100000 --test-ms 30000 --out results/sim-history-test30s.json
# group commit: same driver, intents topped up so the supply never runs out
FT_URL=<your deployment> FT_TOKEN=<admin token> node packages/bench/src/live.ts --colony g1-bench --agents 200 --duration 90 --intents 3000 --out results/2026-10-12-g1/live-n200.json
# scale: synthetic agents on a colony with review off (POST /api/colonies/<colony>/decide/policy {"enabled":false}) and a pre-forked pool
FT_URL=<your deployment> FT_TOKEN=<admin token> node packages/bench/src/live.ts --colony <colony> --agents 5000 --duration 120 --out results/2026-10-12-scale/live-n5000.json
node packages/bench/src/report.ts packages/bench/results/2026-10-12 packages/bench/results/2026-10-12-g1 packages/bench/results/2026-10-12-scale

Data: live-n*.json, sim-live.json, sim-history.json, sim-history-test30s.json, cascade.json, batchcost.json, results.json next to this page; ../2026-10-12-g1/ (group commit); ../2026-10-12-scale/ (scale runs, notes-check.json, summary.json). The production runs behind the README's results (Hono, a new project from a spec, an external agent over MCP, the vaccination meter) are in ../2026-10-12-prod/. Code: packages/bench. Generated 2026-10-12T20:57:41.536Z.