TRUE Coverage: How We Achieved 90% Faster CI by Measuring What Tests Actually Do

Anthony Dawson — Wed, 22 Jul 2026 15:41:29 +0000

TL;DR: Use per-test coverage data to build a reverse map (file → tests that touch it). Git diff + map lookup = run only relevant tests. 43min → 4min CI time.

The Problem Everyone Has

Your test suite runs for 45 minutes. You change one file. Should you:

Run all 2,000 tests? (Safe but slow)
Run tests with matching filenames? (Fast but broken)
Manually tag tests? (Gets stale immediately)

We tried all three. They all failed.

The Insight: Stop Guessing, Start Measuring

What if we just measured what each test actually executes?

Coverage tools have done this for years:

{
  "src/MyComponent.js": {
    "statements": { "0": 5, "1": 12, "2": 0 }
  }
}

Key insight: Coverage tools report per test run, not per test file.

All we needed: save coverage after each individual test.

The Architecture (5 Steps)

Step 1: Instrument the Code

// Build config (webpack/vite/rollup/etc)
export default {
  plugins: [
    process.env.COLLECT_COVERAGE &&
      coveragePlugin({
        include: 'src/**/*',
        exclude: ['**/*.test.*']
      })
  ]
}

Works with: Istanbul (JS), coverage.py (Python), SimpleCov (Ruby), JaCoCo (Java)

Step 2: Collect Coverage Per Test

// Test framework afterEach hook
afterEach(async () => {
  const coverage = getCoverageData();
  saveCoverage(`coverage-${testName}.json`, coverage);
});

Step 3: Build the Reverse Map

const map = {};

for (const coverageFile of allCoverageFiles()) {
  const testName = extractTestName(coverageFile);
  const coverage = JSON.parse(read(coverageFile));

  map[testName] = [];

  for (const [sourceFile, data] of Object.entries(coverage)) {
    if (wasExecuted(data)) {
      map[testName].push(sourceFile);
    }
  }
}

Example output:
{
  "tests/user-profile.test.js": [
    "src/components/Profile.jsx",
    "src/components/Card.jsx",
    "src/utils/formatters.js"
  ]
}

The magic: The test told us what it executed. No parsing, no guessing.

Step 4: Detect Tests

CHANGED=$(git diff origin/main...HEAD --name-only)
TESTS=$(lookupTestsInMap "$CHANGED")
npm test $TESTS

Step 5: Auto-Update

npm run build-coverage-map
git add coverage-map.json
git commit -m "Update coverage map [skip ci]"
git push

The Results

┌───────────┬────────┬───────┐
│  Metric   │ Before │ After │
├───────────┼────────┼───────┤
│ Tests run │ 47     │ 4     │
├───────────┼────────┼───────┤
│ Time      │ 18 min │ 2 min │
├───────────┼────────┼───────┤
│ Accuracy  │ 15%    │ 95%+  │
├───────────┼────────┼───────┤
│ CI cost   │ $12    │ $1.20 │
└───────────┴────────┴───────┘

Real PR example:

Changed: 1 component
CI time: 43 minutes → 4 minutes

Language Examples

JavaScript (Istanbul + Jest/Playwright)
afterEach(async () => {
  const coverage = global.__coverage__;
  saveCoverage(coverage);
});

Python (coverage.py + pytest)
@pytest.fixture(autouse=True)
def save_coverage(request):
    yield
    save_coverage_data()

Ruby (SimpleCov + RSpec)
RSpec.configure do |config|
  config.after(:each) { save_coverage }
end

Java (JaCoCo + JUnit)
@AfterEach
void saveCoverage() {
    save(getCoverage());
}

The Tradeoffs

What Works:

✅ Zero maintenance (auto-updates)
✅ Handles refactors naturally
✅ Incremental updates

What Required Tuning:

⚠️ Build tool noise (bundlers include common code)
⚠️ Shared utilities (some files in 50+ tests)
⚠️ Git history churn (auto-commits)

Solutions: Keyword filtering, threshold checks, minimum change limits

The Key Insight

Traditional approaches model relationships:

Import graphs → static dependencies
Pattern matching → naming conventions
Manual tags → human intent

We don't model. We measure.

Try It Yourself

~500 lines in any language:

Instrument (100 lines)
Hook (50 lines)
Parse (200 lines)
Lookup (100 lines)
CI (50 lines)

Complexity is in filtering noise, not the pattern.

Conclusion

Stop guessing. Start measuring.

Coverage tools have measured execution for decades. We just use them per test instead of per run.

If your CI is slow, this will fix it. Not by being clever, but by being honest.

Questions? Comment below or reach out on Twitter/LinkedIn

DEV Community: Anthony Dawson