Case study · replay
Replaying LobeHub against its own affected-test planner
Why LobeHub
LobeHub's CI runs the full app and server Vitest projects, about 3,650 test files, on
every pull request. A core maintainer opened an affected-test planner,
lobehub#20145, in September 2026. It uses
vitest list --changed <base> plus rules that trigger a full run. Its description gives the
reason: on GitHub's free plan, with 20 concurrent jobs shared across the organisation, test CI is about 64% of
runner minutes and queues reach 1–4 hours at peak. At the time of the replay the planner was still unmerged.
That makes LobeHub a fair test. The baseline isn't "run everything" but a planner the maintainers wrote for their own repository.
Selection: 40 commits
The corpus is the last 40 first-parent canary commits before
9a594a6, each against its parent. Each replay
cloned LobeHub, ran pnpm install --ignore-scripts, then diffci observe --no-send, then
the planner's vitest list calls. No tests were executed in this part.
| DiffCI 0.3.22 | DiffCI 0.3.24 (38 commits) | DiffCI 0.3.31 | LobeHub planner | |
|---|---|---|---|---|
| Mean share of test files run | 87.7% | 32.9% | 32.7% | 32.5% |
| Full runs | 35 | 3 | 2 | 1 |
| Planner-picked tests DiffCI lacks | n/a | 0 | 0 | – |
| Median analysis time | 87 s | 73 s | 66 s | 92 s |
The first replay was bad, and that is the useful part. On CLI 0.3.22, 35 of 40 commits fell back to
the full suite. One broken import type in a single test file made the graph UNSAFE for nearly every
change. Selection also followed type-only imports, and computed locale imports had no edge. Fixing those in
Core brought DiffCI to the planner's level by 0.3.24 without missing any test the planner picked.
On 0.3.31, when both narrow, DiffCI selects about six more test files than the planner and never fewer. Its two
full runs are a workspace package.json dependency edit and a root package.json change.
Catches: would DiffCI have stopped the broken pull requests?
The second study replayed 72 pull-request heads, 47 that failed CI and 25 that passed, against their merge-base with
canary. Each ran diffci verify --checks --baseline-on-failure. The full test, lint and type
commands then ran at the head and at the base to label each failure as introduced by the PR or inherited.
71 of 72 completed (CLI 0.3.31). 18 reproduce a failure the PR introduced.
| Introduced failure | Commits | Caught | How |
|---|---|---|---|
Lint (lint:ts) | 11 | 11 | The lint check. Three of these PRs also broke tests, and lint stopped them first |
| Type error | 1 | 1 | The type check, on a narrowed run |
| Test only, select mode | 4 | 3 | One narrowed selection picked the failing test, two ran the full suite, one analysis timed out |
| Test only, verify mode | 2 | 0 | Both ran the full suite and hit the sandbox's 27-minute step limit first |
No pre-existing failure was reported as introduced. Narrowed verify runs took a median of 8.9 minutes. Most of the catches are lint and type checks, which a test-selection tool can't see at all. That matches what the own-repository catch replay found.
What this shows and what it doesn't
It shows that on a large outside Vitest monorepo, DiffCI reaches the selection of a planner the maintainers built by hand, without missing a test that planner picks. Before the push, it catches most introduced failures, mainly through checks.
It doesn't show savings in LobeHub's real CI, because DiffCI never ran there. It doesn't show that agents would run it or act on it. The selection numbers predate later Core fixes and are reported as measured on 0.3.31. The catch sample is small: 18 introduced failures, and the two verify-mode test catches were lost to a sandbox time limit rather than to selection.
Method, scripts and the per-commit tables are in DiffCI's research repository (research/ci-replay/lobehub/).
The cross-repository record of the August–September studies is the
DiffCI Open Evidence Study 2026.