Verify-Loop CI for Agent-Written Code

Once agents author most code and PR review speeds up, CI is the constraint. Cite Anthropic’s public numbers only: 25x CI jobs in 6 months, Claude ~80% of shipped code, 8x ship rate vs 2021–2025, tests 10x. Do not invent Institute throughput or revenue figures. Smaller, more granular agent PRs and overnight/weekend agent activity raise the CI floor; bursts still follow human approval.

Do not run every test on every change. Anthropic’s service is a deterministic test-impact / selection layer: a listener records results from every CI run; a selector reads history and package relevance to decide which tests run on which opened PRs. Agents need a specific valid set to self-verify; humans can ignore irrelevant failures, agents cannot. Vendor test-selection products exist in this category — build or buy, same split.

A single-process listener/selector cannot shard: one writer holding per-test history blocks horizontal scale. Anthropic’s path: patches (bigger machine, then per-package shards, then daily restarts) bought less time each round. The durable design offloads history to a store: any listener worker appends a job-result to a journal and stays stateless; a small consumer rolls the journal into per-test history; the selector looks up history. Do not keep the running selector state inside the process.

Listener lag is the failure mode: stale selector data runs the wrong tests (already-flaky or already-failing suites) and misses new/fixed tests. Anthropic’s operator metric: the same number of CI jobs coming in should equal the same going out. Instrument queued unprocessed job-result events. Use those signals as the agent’s eyes so it can hill-climb incrementally. A lagging listener is a verify-loop bug, not a reason to skip CI.

Anthropic’s advice, from the same public post: whether you build or buy, assume the architecture will be at a 25x load within two quarters. Account for 10–20x perceived scale in v0 if budget allows. Quick patches buy a fraction of the time they did a year ago; rewriting the service also takes a fraction of the time now that authoring is not the bottleneck. The insight is not “buy a bigger box” — it is plan the exponential before the singleton pages.

Dario Amodei’s pacing essay (https://darioamodei.com/post/we-must-pace-the-frontier) commits Anthropic to embedded third-party evaluators (example: METR) with employee-like access to verify safety practices and report incidents. Use that here only as optional instrumentation context: expose jobs-in = jobs-out, selector lag, and selection coverage so an embedded reviewer can see the verify loop. This playbook does not restate the essay’s coordination steps. METR Inspect Hawk remains the catalog eval harness — separate from test selection.

The agent saying it is done is not evidence; the system of record is. Microsoft and Hugging Face released ThinkingBox-Bench through OpenEnv (3 Oct 2026): 507 stateful business workflows (retail, auto insurance, travel, neobank, consulting) run 20 times each from a clean backend and graded on terminal backend state and side effects. In a 121,680-trial ablation across 12 models, 79,853 attempts failed executable checks; 67.24% of those failures still ended cleanly, called a state-changing tool, and reported no tool error. Among failures: wrong field values 77.61%, unintended extra effects 43.30%, missing required effects 25.36% (overlapping). Best observed 20/20 was 241 of 507 tasks (47.53%, Claude Opus 5.5 and Opus 5). Cost per dependable task ranged $6.80–$24.36 (estimates, not bills). About 79.9% of failures were classed as tool handling (failed recovery from tool errors, preconditions, empty lookups), not reasoning. Operator recipe: gate merges, payouts, and closures on executable end-state checks; report every-of-k (not best-of-k) for any agent touching customer records; fix retries and tool-error recovery before swapping models. Session traces are not an audit trail — see /playbooks/harness-permission-outside-agent/.