Optimize GitHub Actions from measured job cost.

Separate four different outcomes

1. Test-stage runtime

   The time inside the test command after prerequisites are ready.

2. Job-equivalent runtime

   Analysis, installation, generation, pretest steps, and test execution.

3. Workflow latency

   The critical path users wait for, which can differ from total compute when jobs run in parallel.

4. Billed compute

   Runner minutes or self-hosted resource cost across every executed job.

State which number is being reported. A 90% test-stage reduction can become a much smaller job-level reduction when installation dominates the job.

Start with cheaper optimizations

Before selecting tests, remove duplicate installs, restore correct dependency caches, cancel superseded workflow runs where appropriate, right-size matrices, and move unrelated work off the critical path. These changes can reduce cost without deciding that any test is irrelevant.

Then compare the repository's existing path rules or task graph with dependency-based selection. A more precise analyzer has its own runtime cost and can lose to a cheap path filter on small suites.

Calculate the billable baseline

Start with the provider's bill, not a test count. For a repository and billing period, group completed jobs by runner type and operating system, then calculate:

monthly runner cost = Σ(job duration in billable minutes × applicable runner rate)

Keep public-repository allowances, private-repository included minutes, hosted-runner rates, storage, and self-hosted charges separate. GitHub can change prices and billing rules, so treat copied rates as dated inputs and verify them against the official GitHub Actions billing documentation.

For optimization decisions, also retain the number of workflow runs, jobs per run, fallback rate, queue time, and the critical-path duration users actually wait for. Lower aggregate compute and shorter feedback time are related but not interchangeable outcomes.

Run a paired pilot

A credible pilot keeps real CI unchanged and measures both arms against the same commit:

npx "@diffci.com/diffci@latest" check

DiffCI reports measured savings only when the full and selected commands both pass and checkout identity remains valid. One run is preliminary evidence; repeat comparisons and account for cache order and warm-state effects.

• Record fallbacks as zero selection savings. Do not drop them from the denominator.
• Withhold unhonored selections. If the runner executes more tests than proposed, the proposed selection was not measured.
• Do not extrapolate one repository. Suite shape, setup cost, and change distribution determine the result.

What the current evidence says

In a controlled cal.com replay, the same selected tests produced 86–91.6% test-stage reduction but 44.2% job-equivalent reduction once install and pretest work were included. In deepseek-harness, measured job-level reduction was 79.5–89.5% on counted runs, while one run was withheld because execution did not honor the proposed selection.

Those are sandbox measurements on public repositories. They are not production savings, endorsements, or a prediction for a new repository.

cal.com case study · deepseek-harness case study · open evidence study

Decide with repository-specific thresholds

Adoption makes sense only if the measured net benefit exceeds operational complexity and the safety evidence meets the repository's risk tolerance. Define that threshold before inspecting favorable results. If the selected path is not cheaper, keep the full run and retain the analysis only if its explanations are useful.