Technical guide

Test impact analysis for GitHub Actions

Published and last updated September 25, 2026 · By Aditya Kale, primary maintainer

Test impact analysis asks which tests and CI jobs a code change can actually affect. The difficult part is not producing a short list. It is knowing when that list is complete, when to fall back, and whether running it saves anything after setup cost.

Run a local check See the evidence

Test impact analysis must preserve the workflow prerequisites around a selected test command, not only calculate a list of files.

How affected-test selection works

A conservative analyzer starts with the files changed between a base and head commit. It resolves repository structure and import relationships, follows the dependency graph to tests, and then maps those tests into commands or GitHub Actions jobs.

The result must explain why each test was selected. Directly changed tests, tests that import changed source, package-level dependencies, and an explicit always-run policy are different evidence classes. An empty selection without that provenance is not evidence that no tests exist.

Read the real Git delta
Use the actual base and head commits rather than a hand-entered file list.
Build the dependency and CI model
Resolve supported source edges, test files, workflow jobs, conditions, matrices, and reusable commands.
Select or fall back
Produce affected tests only when the reachable graph and command plan are sufficiently complete.
Retain the proof
Record the selected tests, fallback reasons, commands, commit identity, and report hash.

Add an isolated observer job

During evaluation, keep the observer outside the dependency graph of required jobs. It should have read-only contents permission, full comparison history, a time bound, and continue-on-error so its failure cannot change the workflow conclusion.

name: DiffCI observation
on: [pull_request]

permissions:
  contents: read

jobs:
  diffci:
    runs-on: ubuntu-24.04
    timeout-minutes: 20
    continue-on-error: true
    steps:
      - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # immutable checkout pin
        with:
          fetch-depth: 0
      - uses: DiffCI/core@8a96f7a9aa1eb827b2c58846188778da8eacfaa3 # action-v0.3.38

No required job should list diffci in needs, and the observer itself should not be configured as a required check. Verify generated or edited workflow text before committing it:

npx "@diffci.com/diffci@latest" verify-workflow

When a full run is the correct answer

Some changes can alter every test or change how tests execute. A conservative system should run the full command for lockfile changes, workflow changes, root configuration, unsupported import patterns, missing base commits, or incomplete command synthesis.

GitHub Actions adds another layer. A test command may depend on install, generation, services, artifacts, reusable workflows, job conditions, or matrix values. Selecting a test without its prerequisites does not reproduce the job that CI would have run.

Keep an always-run policy. Security, configuration, validation, and workflow guardrails may need to run for every change.
Treat unsupported evidence as unknown. Unknown is not an empty affected set.
Do not let an observer become authoritative. Evaluation should happen alongside required CI until the evidence is sufficient for a separate adoption decision.

Measure the job, not only the test list

Reducing 90% of test files does not necessarily reduce 90% of a GitHub Actions job. Checkout, dependency installation, build generation, service startup, and framework boot time still run. Parallel jobs can also make total compute fall while wall-clock latency barely changes—or the reverse.

A useful pilot executes the full and selected commands against the same checkout, confirms that both passed, includes analysis overhead, and labels whether the number describes a test stage or the whole job-equivalent workload.

DiffCI's controlled cal.com replay measured 86–91.6% reduction in the test stage but 44.2% in a job-equivalent install, pretest, and test workload. That is one sandbox result, not a production claim or a forecast for another repository.

Read the cal.com timings and method →

Evaluate DiffCI without changing required CI

From a checkout with Node.js 22.5 or later and Git:

npx "@diffci.com/diffci@latest" check

check analyzes the current commit and runs inferred full and selected commands when safe. Test commands may write generated files. For analysis without test execution:

npx "@diffci.com/diffci@latest" observe --no-send

To add the separate, non-blocking observer Action, start with init --workflow and verify the generated workflow before committing it. The Action does not skip or cancel other jobs.

Common GitHub Actions failure modes

Symptom	Why it happens	Safe response
Base commit not found	`actions/checkout` used shallow history.	Use `fetch-depth: 0` in the observer job or retain the refusal and run normal CI.
Selection omits setup	The test command depends on generation, services, artifacts, or another job.	Model the prerequisite or keep the selection advisory.
Matrix result differs	OS, runtime, or matrix variables change discovery or execution.	Evaluate each relevant environment; do not copy one matrix cell's selection to another.
Observer blocks a merge	It became required, lacks `continue-on-error`, or sits in another job's `needs` chain.	Restore isolation before collecting more observations.

Review the language and framework support matrix before interpreting an ecosystem-specific fallback.

What this approach does not prove

A green selected run is not proof that all omitted tests were irrelevant. Static analysis has support boundaries, tests can depend on runtime behavior that imports do not express, and historical replay is not production adoption. The useful output is therefore a selection plus its evidence, limitations, and fallback policy—not a claim that full CI is obsolete.

Continue with the affected-tests implementation guide, the comparison with path filters and task-graph tools, or the open benchmark.