Case study · red-build study
Which of Paperclip's red builds an agent could have caught before the push
Why Paperclip
Paperclip is a very busy repository with pull requests from coding agents. Its last 300 runs of the
PR workflow cover only about 14 hours, from 20:37 UTC on November 10 to 10:27 UTC on November 11, 2026.
It already selects its own tests: a "Select Vitest files" job and an incremental CI plan. So DiffCI wouldn't save
much by selecting fewer tests in Paperclip's CI. The question here is a different one: how many red builds would
never have been pushed if the agent had run the right checks in its own loop first?
Method
For each red run, every failing job was sorted into one of two groups. Pre-push jobs are checks an agent can run locally: repository-rule checks, Typecheck + Release Registry, General tests, Sandbox provider tests, Runner support suite, Verify Paperclip Runner, Build, and Verify serialized server suites. CI-only jobs are the E2E shards, Canary Dry Run, Docker context integrity, and the lockfile and migration-ordering steps of the policy job.
Paperclip's master has no test workflow on push and there is no merge queue, so there is no base run
to compare against. Instead, each failing test was identified from its check-run annotation (file, test name and
first message line). A failure counts as the PR's own when no other PR in the window failed the same way. Failures
shared across PRs, such as one report-assets test on 10 PRs and one heartbeat-plugin test on 9, are a broken base
or an always-failing test the author couldn't fix. Failures that are only timeouts count as load or flakiness. A run
is agent-written when its head commit has an agent co-author line or its branch has an agent prefix such as
codex/.
Results: 74 red runs
| Red runs | Runner-minutes | |
|---|---|---|
| All red PR runs (60 PRs) | 74 | 25,563 |
| Preventable: every failure was the PR's own, in a pre-push job | 12 | 4,048 |
| — of which agent-written | 11 | 3,702 |
| Partly: an own failure, plus E2E, shared or unknown ones | 14 | 4,848 |
| — of which agent-written | 6 | 2,082 |
| Not the PR's own: shared, timeout or no usable annotation | 36 | 12,504 |
| CI-only: E2E shards only | 11 | 3,832 |
| Infrastructure | 1 | 331 |
Inside the agent's loop, 11 of the 74 red runs (15%) could have been stopped before the push. All 11 were agent-written. Together they cost 3,702 runner-minutes, and each was a wait of about half an hour: a red run took a median of 32 minutes from start to finish. In 6 more agent-written runs the agent would have fixed its own failure, but the run would still have gone red from an E2E or shared failure.
Most red runs were not the author's fault. 36 failed on shared, timed-out or unannotated tests and 11 only in E2E shards. A pre-push check can't help with those, and this page doesn't count them.
The 12 preventable runs
The failures are the kind DiffCI's selection and checks are built for: Vitest unit and integration tests, plus
tsc. Each run links to its public GitHub Actions page.
| Run | Agent-written | Failing job | First failure |
|---|---|---|---|
| 42922381980 | Yes | Verify serialized server suites | A vi.mock of ../services/task-retry.js without the DECLINED_TASK_RETRY_REASONS export |
| 42940191943 | Yes | General tests | Assertion failure in chat-channels.integration.test.ts |
| 42951846997 | Yes | General tests | TypeError: timer.unref is not a function in event-loop-stall-monitor.ts |
| 42962300744 | Yes | General tests (3 jobs) | Failures in chat-channels and voice-session-store integration tests |
| 42963271556 | Yes | Typecheck + Release Registry | '"codex_app_server"' is not assignable to '"mock" | "remote" | "runner"' in packages/paperclip-runner |
| 42964700326 | Yes | General tests | Assertion failure in native-failed-workspace-recovery.test.ts |
| 42968455542 | Yes | General tests | paperclip_runner_binary_missing in native-codex-runner.ts |
| 42973958854 | No marker | Verify Paperclip Runner | A failed insert in the native-restart test fixture |
| 42976147787 | Yes | General tests | Sandbox creation has already transferred ownership |
| 42976154239 | Yes | General tests | Assertion failure in plugin-fresh-workspaces.integration.test.ts |
| 42989667294 | Yes | General tests | Assertion failure in attention-retention-work.test.ts |
| 42991185935 | Yes | Verify Paperclip Runner | A failed insert in the native-restart test fixture, then Maximum call stack size exceeded |
The vi.mock case is worth noting: a mock that doesn't return an export the code now needs is a
mock-edge case, which DiffCI's graph tracks. The type error is the kind of failure a test-selection tool can't see
at all. It is caught by the typecheck, as in the own-repository
catch replay and the LobeHub replay.
What this shows and what it doesn't
It shows that in a busy, agent-heavy repository, almost every red build that a pre-push check
could have stopped was agent-written: 11 of 12. The failures were ordinary Vitest and tsc failures,
not E2E or infrastructure.
It doesn't show that DiffCI would have caught each of them. DiffCI's actual plan wasn't computed for these commits. It would miss one only if its plan left the failing file out, and checking that needs a replay of these heads, which Paperclip hasn't been asked for. Other limits:
- Fourteen hours of one repository. Repeat over a longer window before quoting a rate.
- "Own" means no other PR failed the same way. A one-off flake on a single PR would still count as own. No run in the window passed on a rerun without a new commit, so there is no direct evidence of flakes either way.
- The agent count is a lower bound. Markers were read on the head commit only, and fork PRs often carry none. 49 of the 74 red runs came from forks.
- Some data is missing. Two run attempts returned HTTP 502 from GitHub and were skipped.
Method, scripts and the per-run data (analysis.json, own-failures.json,
agent-authorship.json and the check-run annotations) are in DiffCI's research repository
(research/ci-replay/paperclip/red-builds/).