Test impact analysis for GitHub Actions

How affected-test selection works

A conservative analyzer starts with the files changed between a base and head commit. It resolves repository structure and import relationships, follows the dependency graph to tests, and then maps those tests into commands or GitHub Actions jobs.

The result must explain why each test was selected. Directly changed tests, tests that import changed source, package-level dependencies, and an explicit always-run policy are different evidence classes. An empty selection without that provenance is not evidence that no tests exist.

1. Read the real Git delta

   Use the actual base and head commits rather than a hand-entered file list.

2. Build the dependency and CI model

   Resolve supported source edges, test files, workflow jobs, conditions, matrices, and reusable commands.

3. Select or fall back

   Produce affected tests only when the reachable graph and command plan are sufficiently complete.

4. Retain the proof

   Record the selected tests, fallback reasons, commands, commit identity, and report hash.

When a full run is the correct answer

Some changes can alter every test or change how tests execute. A conservative system should run the full command for lockfile changes, workflow changes, root configuration, unsupported import patterns, missing base commits, or incomplete command synthesis.

GitHub Actions adds another layer. A test command may depend on install, generation, services, artifacts, reusable workflows, job conditions, or matrix values. Selecting a test without its prerequisites does not reproduce the job that CI would have run.

• Keep an always-run policy. Security, configuration, validation, and workflow guardrails may need to run for every change.
• Treat unsupported evidence as unknown. Unknown is not an empty affected set.
• Do not let an observer become authoritative. Evaluation should happen alongside required CI until the evidence is sufficient for a separate adoption decision.

Measure the job, not only the test list

Reducing 90% of test files does not necessarily reduce 90% of a GitHub Actions job. Checkout, dependency installation, build generation, service startup, and framework boot time still run. Parallel jobs can also make total compute fall while wall-clock latency barely changes—or the reverse.

A useful pilot executes the full and selected commands against the same checkout, confirms that both passed, includes analysis overhead, and labels whether the number describes a test stage or the whole job-equivalent workload.

DiffCI's controlled cal.com replay measured 86–91.6% reduction in the test stage but 44.2% in a job-equivalent install, pretest, and test workload. That is one sandbox result, not a production claim or a forecast for another repository.

Read the cal.com timings and method →

Evaluate DiffCI without changing required CI

From a checkout with Node.js 22.5 or later and Git:

npx "@diffci.com/diffci@latest" check

check analyzes the current commit and runs inferred full and selected commands when safe. Test commands may write generated files. For analysis without test execution:

npx "@diffci.com/diffci@latest" observe --no-send

To add the separate, non-blocking observer Action, start with init --workflow and verify the generated workflow before committing it. The Action does not skip or cancel other jobs.

What this approach does not prove

A green selected run is not proof that all omitted tests were irrelevant. Static analysis has support boundaries, tests can depend on runtime behavior that imports do not express, and historical replay is not production adoption. The useful output is therefore a selection plus its evidence, limitations, and fallback policy—not a claim that full CI is obsolete.

Continue with the affected-tests implementation guide, the comparison with path filters and task-graph tools, or the open benchmark.