Case study · replay

Would DiffCI have caught agent-introduced CI breakage before the push?

Published November 11, 2026 · Replays run October 30 to November 2, 2026 · By Aditya Kale, primary maintainer

Both repositories are ours: DentalPresence, the product DiffCI was first built for, and DiffCI.com, this site's repository. Most of their commits are written by coding agents. That makes this the most complete data we have and the least independent. Read it as what DiffCI does on agent-built code, not as evidence about anyone else's repository.

32 of 38introduced breakages caught before push

~70%of red commits only inherited a broken main

1 of 57clean commits wrongly failed

185 of 186seeded bugs caught on narrowed runs

The question

When an agent pushed a commit that broke CI, would running diffci verify before the push have caught it, for the same reason CI later failed? And what does DiffCI cost on the commits that pass?

Method

Every replay ran in its own Cloudflare Container. For each commit, DiffCI ran diffci verify --checks --baseline-on-failure against the previous commit CI had run, as a pre-push hook would. Then the repository's real gate ran on the same commit. If a gate stage failed, it ran again at the base. A failure counts as introduced only if that stage passes at the base. Otherwise it was inherited from a commit that was already broken. "Caught" means DiffCI failed the commit on the same check, or on a failing test the gate also reported.

The corpus is all 73 DentalPresence commits behind its failed Cloudflare builds (August 14 to October 28) plus 40 passing builds, and all 62 DiffCI.com commits with a failed CI run (August 21 to October 29) plus 30 passing pushes. 204 of 205 replays completed.

Most red builds were inherited, not introduced

	DentalPresence	DiffCI.com
Commits whose CI failed	73	60
Failure reproduced in the replay	73	50
Inherited: the base already failed the same stage	51	35
Introduced by the commit itself	23	15

About 70% of failing commits only inherited a breakage. In agent-era CI, much of the waste is a red main branch that hides new failures, not too many tests.

DiffCI caught 32 of the 38 introduced breakages

Most catches came from checks, not tests. In the first run, 19 of 30 catches were typecheck or repository-rule checks (check:social-utm, validate:cloud-deployments and others). A test-selection tool can't catch any of these.
Narrowed runs still caught breakage. Where DiffCI selected tests rather than falling back, it ran a median of 15 of 362 test files in DentalPresence and 10 of 46 in DiffCI.com.
The first run caught 30 of 38. Two more came from a fix to verify, which used to block when the repository's own tests wrote files into the working tree. That brought it to 32. An unreleased fix that reruns an already-failing check at the base reaches 33 of 38.
The one outright miss was a test that scans every file under a directory, which DiffCI didn't model. It was fixed in CLI 0.3.25.
On clean commits, DiffCI wrongly failed 1 of 57 in the release-gate scorecard, on a test that was already failing at the base.

Replaying the build history with those catches, DentalPresence's 104 red builds would have dropped to 25, and DiffCI.com's 44 to 25. That assumes the commits after each fix would have been the same.

Seeded bugs: is narrowing safe?

Real breakages don't tell you what a narrowed run skipped. So we planted 393 one-line bugs (flipped comparisons, swapped && and ||, off-by-one bounds) across 100 DentalPresence commits where DiffCI ran only part of the suite. Three were in files the commit changed and one in a file a changed file imports. The full suite caught 186 of them, and DiffCI's selected tests caught 185 (95% interval 97.0–99.9%). It ran a median of 8% of the tests in about a fifth of the time. The miss was a path-escaping bug, since fixed in Core but not yet re-measured.

No test caught the other seeds, about half, the full suite included, so they say nothing about selection.

What this shows and what it doesn't

It shows that before the push, DiffCI catches most of the breakage agents introduce, and half of it through checks rather than tests. It also shows that a narrowed run kept nearly every bug the full suite would have caught.

It doesn't show anything independent. There are two repositories with one owner, and there is no human-written baseline. The replay ran DiffCI after the fact. It shows what DiffCI would have reported, not that agents would have run it or fixed the commit. In practice, agents called it in only a small share of test runs. The seeded bugs are seeded, not real.

The first outside repository with the same kind of replay is LobeHub. The August–September studies across 20 public repositories are in the DiffCI Open Evidence Study 2026.