DiffCI

Open-source verification for AI coding agents

Tells AI coding agents what their change breaks.

Agents test the file they edited and push. DiffCI follows the change through the dependency graph, finds the tests and checks it affects (typecheck and repo rules as well as tests), runs them before the push and leaves a receipt of what ran and what was skipped. When evidence is missing, it falls back to the full suite.

It has no permission to skip, cancel, or modify a CI run. Local checks may execute test commands, but required CI remains unchanged.

Agent verificationShortest verified route

only work reached by the change · proof retained

10 real merges across two repositories, full and selected paths both executed and timed
12 of 13 executions where the runner honored the selection; the 13th is withheld, not counted
0 production CI runs ever altered

There is no headline percentage on this page. The honest number is repository-specific: the same selection quality produced 44% and 90% job-level reduction on two different real repositories. A single number would be an average of two things that aren't alike.

See it run

A short walkthrough of DiffCI following a change to its affected tests.

DiffCI demo
Plays from YouTube (privacy-enhanced mode) once you press play. Watch on YouTube.

The problem

A twelve-line change to one package re-runs four thousand tests in ninety other packages.

Everyone knows this is happening. Almost nobody knows what it costs them, because measuring it means either trusting a tool to start skipping things, or designing an experiment nobody has time for.

DiffCI is the measurement, delivered before the risk.

Start with the implementation guides, learn how to run affected tests conservatively, review language support, or calculate GitHub Actions cost from measured jobs.

Install DiffCI

DiffCI has local, agent, Action and hosted-observer entry points. The local check command may run inferred test commands; none of the entry points can skip, cancel or reorder your required CI jobs.

AI coding agents

Give agents a default validation command

Seed AGENTS.md, CLAUDE.md, Cursor rules, Copilot instructions, and diffci.config.json. Agents then have one safe command to run before calling a change PR-ready.

npx "@diffci.com/diffci@latest" init
npx "@diffci.com/diffci@latest" check

DiffCI changes no CI behavior. check may run inferred local test commands to validate the change. All agent installs →

MCP server

Connect DiffCI as a native agent tool

Remote MCP clients can request a safe local validation plan without sending repository content. The local server also exposes diffci_check, diffci_init, and diffci_verify_workflow.

https://diffci.com/mcp

The HTTPS endpoint provides stateless, read-only validation guidance. Use npx -p "@diffci.com/diffci@latest" diffci-mcp when tools must access a local checkout. MCP setup and tools →

npm CLI

Run it locally or inside CI

The npm package is @diffci.com/diffci. Use npx for a one-off observation, or install it as a dev dependency if you want a pinned local tool.

npx "@diffci.com/diffci@latest" check
npx "@diffci.com/diffci@latest" observe
npx "@diffci.com/diffci@latest" verify-workflow

npm install --save-dev @diffci.com/diffci

observe writes a JSON report outside the checkout by default. Nothing is sent to DiffCI unless an endpoint and token are explicitly configured.

GitHub Action runner

Add one non-blocking observer job

Put DiffCI in its own job so a DiffCI failure cannot become your workflow's conclusion. For a reproducible installation, the example pins the Core Action to a reviewed commit SHA. That Action uses the published @diffci.com/diffci@0.3.57 CLI.

jobs:
  diffci:
    runs-on: ubuntu-24.04
    continue-on-error: true
    permissions:
      contents: read
    steps:
      - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # immutable checkout pin
        with:
          fetch-depth: 0
      - uses: DiffCI/core@08deacbc326304533f8ff5915b516df34f1522b1 # action-v0.3.57

The Action writes a report artifact, prints a summary, and cannot skip tests, cancel jobs, comment on pull requests, or change required checks.

GitHub App

See what each pull request affects, without editing a workflow

The App is the lowest-friction path: on every pull request of a public repository it posts one advisory "DiffCI change impact" check listing the affected tests and why. The check never blocks a merge. The App also observes pushes and completed workflow runs from outside your CI.

How it works

  1. Observe

    A GitHub App sees pull requests, pushes and completed workflow runs. It reads metadata, contents, Actions and pull requests; its one write permission is checks, for its advisory check.

  2. Analyze

    DiffCI builds a real TypeScript compiler-backed dependency graph, computes what is reachable from the changed files, and produces a confidence-scored plan: which tests a safe system would have run. Changes it cannot reason about — lockfiles, workflow files, root config — force a mandatory fallback to "run everything". That is the default, not the exception path.

  3. Reconcile

    When your CI finishes, DiffCI reads the real outcome and timings and asks the only question that matters in hindsight: would that plan have missed a failure your CI actually caught?

  4. Report

    Runs observed, compute consumed, compute avoidable, missed failures — and, explicitly, what could not be measured.

The report

Illustrative layout — placeholder values, not data from any repository
Last 7 days · your-org/your-repo
CI runs observed———measured
Runner-hours consumed———measured
Runner-hours potentially avoidable———estimated
Observed missed failures———measured
Selected-run execution———not measured

Three labels, and they mean exactly what they say. Measured is a number a clock produced. Estimated is an inference from a cost model whose weaknesses we will tell you about. Unknown is where a lesser tool would have guessed.

DiffCI will not turn an estimated figure into an invoice. Ever. If it eventually charges for anything, it charges against savings that were executed and timed.

Why there is no single number

Two real repositories. Comparable selection quality — one or two test files selected out of hundreds, in both. Wildly different savings, for a reason that has nothing to do with how good the selection was.

Install & setup — paid either way Tests — the part selection shrinks

cal.com

full
657.7s
selected
356.9s −44%

deepseek-harness

full
760.2s
selected
58.7s −90%
Measured wall-clock, same horizontal scale across both repositories. cal.com PR #29940 and deepseek-harness PR #2760, each run in full and with DiffCI's selection, in DiffCI's sandbox. cal.com spends 336.8s installing before a single test runs; deepseek-harness spends 38.6s. That fixed cost is the entire difference between a 44% job and a 90% one.

The evidence behind it

Nothing here comes from a demo. Every figure comes from real repositories: real test suites executed with the full run and the selected run both timed, or replays of real commits and CI failures. See every study.

Replay · our repositories

Agent breakage, caught before push

32 of 38introduced breakages caught
~70%of red commits inherited a broken main
185 of 186seeded bugs caught on narrowed runs

Every red CI commit in two agent-built repositories, replayed. Half the catches came from typecheck and repository-rule checks that test selection can't see.

Read the catch replay →

Replay · lobehub/lobehub

Level with the maintainers' own planner

32.7% vs 32.5%share of tests run, DiffCI vs planner
0planner-picked tests missed
15 of 18introduced failures caught

40 commits against a hand-built Vitest affected-test planner, plus 72 pull requests checked for whether DiffCI would have stopped their CI failures.

Read the LobeHub replay →

Red-build study · paperclipai/paperclip

The preventable red builds were agent-written

11 of 12preventable red runs agent-written
15%of red runs stoppable in the agent's loop
~3,700runner-minutes in those runs

74 red pull-request runs in 14 hours, classified job by job: which failures were the PR's own and could have been caught by a check before the push.

Read the Paperclip study →

Case study

cal.com

86–91.6%test-stage reduction, net
44.2%complete-job reduction, net
8 of 8selections honored exactly
3 of 3measurable regressions caught

Five merges chosen by a rule written down before anyone looked at what they would cost, plus one measured six ways. We publish both reduction figures and lead with the smaller one.

Read the cal.com case study →

Case study

deepseek-ai/deepseek-harness

83.5–94.3%test-stage reduction, net
79.5–89.5%complete-job reduction, net
3 of 3measurable regressions caught
1 of 5results withheld, not counted

The repository that was allowed to be messy: 16–18 tests fail on every run before anyone changes anything. A naive safety check reads that as "the regression was caught" every single time. One merge's result is withheld entirely because the runner did not honor the selection.

Read the deepseek-harness case study →

Case study

Our own CI

$0.0048–$0.0127per CI job (median $0.0053)
1,960tests, currently green
0GitHub Actions minutes billed

Our suite runs on ephemeral Cloudflare containers, 2–6 minutes a run. We built our own runner fleet because we needed the timing data to be ours — and found a bug that had been hiding fifteen test files from our own green CI.

Read how →

Open study · CC BY 4.0

Every number, in one place

2,000real commit deltas, 20 repositories
29.8%of deltas where selection was even possible
4.2%aggregate reduction vs a path rule
0 of 32false greens, measurable mutations

Two weeks of pre-registered experiments, published as a study anyone can reuse with attribution: the page, a CSV of every figure with its evidence level and source, and a PDF. It leads with the number that argues against the product.

Read the open study →

Compare all case studies and their source evidence →

Isn't this just Nx affected, Turborepo, or a path rule?

Those tools answer a different question. They tell you what to run. DiffCI's output is the part they don't ship: evidence, after the fact, that a selection would have been safe on your real CI, and what it would have been worth — reconciled against what your CI actually did.

We are not claiming DiffCI is cheaper than any of them. Whether DiffCI's own analysis cost nets out ahead of a simple path rule depends on the repository, and we have not measured it on enough external repositories to say. Where it can be computed, the pilot report includes that comparison for your repository and reports the sign either way.

Compare dependency-graph selection, path filters and task-graph tools →

Check it yourself

You shouldn't have to take our word for any of this. Each item below is something you can check without asking us, and none of it depends on a logo or a program membership.

Releases

Signed, built in public CI

Every @diffci.com/diffci release is published from the public Core repository's GitHub Actions workflow with npm provenance (SLSA v1). The provenance ties each package to the commit and workflow run that built it.

npm audit signatures

See the provenance on npm →

Supply chain

OpenSSF Scorecard and CodeQL

Core runs the OpenSSF Scorecard and GitHub CodeQL on every change to main. We link to the live score instead of quoting it here, so it can't go stale or be rounded up.

See the live Scorecard →

Source

MIT, all of the engine

The analyzer, CLI, MCP server and CI integrations are MIT-licensed in DiffCI/core. Contributions are signed off under the DCO; there is no CLA to sign over your rights. The code that decides what runs is the code you can read.

Read the release history →

Evidence

Benchmarks you can rerun

The figures above come from public repositories and published replays, with the commits, scripts and raw results attached. We don't publish adoption numbers yet because there isn't outside adoption to count. When there is, the count will come from the same usage data our data handling page describes.

Back to the evidence →

Works with

DiffCI is independent. It is not affiliated with or endorsed by GitHub, GitLab or any agent vendor.

What we won't tell you

This section is here on purpose, and it is not going to be moved to the footer.

  • Nobody's CI has ever been made faster by DiffCI. Nothing has ever been skipped in a production pipeline. The savings above were measured by executing both paths in our own sandbox.
  • There are no customers. cal.com, deepseek-harness and the unjs projects are public repositories analyzed from public data. None of them use DiffCI. None of them have endorsed it.
  • Our failure prediction is not good yet. Replayed against 24 of our own historical CI failures, the preflight checks would have caught 16 of 23 evaluable ones — 0.696. Every one it caught was a typecheck or configuration failure. Every unit-test failure in that dataset was a miss.
  • There is no external pilot yet. At the time of writing, no repository outside our own two has installed the App. An earlier internal shadow cohort of public repositories was small, and one repository in it turned out to be unanalysable and was excluded rather than quietly retried. None of that is traction, and we are not going to describe it as such.
  • Small selections make our estimator optimistic. A run selecting zero of forty tests still pays the suite's startup cost, and the estimator does not model that. It is being calibrated per repository from real measurement, not patched with a guessed constant.

The pilot

What you giveRead-only access to one repository, for seven days.
What you getA report of what your CI ran, what was avoidable, and whether any change DiffCI would have de-prioritized turned out to break something.
What it costsNothing. The pilot is free, and so is DiffCI Core. DiffCI Cloud is free for open source: an organization whose connected repositories are all public on GitHub gets the Team plan at no cost. Sign in to start →
What we storeSee below, and data handling → for the full version.
Rather try it yourself?Run DiffCI on your own checkout in a few minutes, with nothing installed in your CI, and share the result. Try it on your repository →

Where your code goes

"Read-only" answers what DiffCI can change. It does not answer what DiffCI can see, so here is that answer, permission by permission.

ContentsFor each commit it analyzes, DiffCI makes a shallow clone of your repository at that commit, inside an ephemeral container on infrastructure operated solely for DiffCI. This is what the contents permission is for: there is no way to build a real dependency graph without reading the source.
What happens to the cloneThe graph is computed and the container is destroyed. The clone does not outlive the analysis. No copy of your source tree is kept anywhere, and no file contents are written to any store.
What is keptCommit SHAs, the paths of changed files, the tests DiffCI would have selected, the reason it fell back to a full run when it did, and the reconciled outcome and timings of the CI run that actually happened. Plus evidence archives for those analyses, so a result can be audited rather than taken on trust. Never file contents. Never credentials: DiffCI is not given any.
For how long90 days at most, counted from the analysis that produced each record. Uninstalling the App deletes that repository's records sooner, automatically.
Actions, checksRead the results and timings of workflow runs that have already finished. This is the same information the Actions tab shows you, used to check DiffCI's prediction against what your CI actually did.
Pull requestsNot requested. DiffCI does not read pull request titles, descriptions, diffs, or review comments. The App asked for this scope in an earlier version without using it; it was removed. If pull-request-level analysis is built, the App will ask for the permission then, and GitHub will prompt you to approve it.
Who else sees itNobody. Nothing is sold, shared with a third party, or used to train anything, and nothing naming your repository is published without your written agreement to that specific text.

If your repository is private and the clone is the part you are not comfortable with, that is a reasonable position, and the right answer is not to install. If you do install and change your mind, uninstall, and the records go with it.

Install the App

DiffCI can only analyze a repository with a tsconfig.json at its root, active GitHub Actions, and recent commits. If yours doesn't qualify, its report stays empty rather than being built on nothing — but it does not yet tell you why, so an unexplained empty report after a few pushes most likely means one of those three conditions is missing.