DEV Community

Cover image for TRUE Coverage: How We Achieved 90% Faster CI by Measuring What Tests Actually Do
Anthony Dawson
Anthony Dawson

Posted on

TRUE Coverage: How We Achieved 90% Faster CI by Measuring What Tests Actually Do

#testing #ci #devops #architecture

TL;DR: Use per-test coverage data to build a reverse map (file → tests that touch it). Git diff + map lookup = run only relevant tests. 43min → 4min CI time.

The Problem Everyone Has

Your test suite runs for 45 minutes. You change one file. Should you:

  • Run all 2,000 tests? (Safe but slow)
  • Run tests with matching filenames? (Fast but broken)
  • Manually tag tests? (Gets stale immediately)

We tried all three. They all failed.

The Insight: Stop Guessing, Start Measuring

What if we just measured what each test actually executes?

Coverage tools have done this for years:

{
  "src/MyComponent.js": {
    "statements": { "0": 5, "1": 12, "2": 0 }
  }
}
Enter fullscreen mode Exit fullscreen mode

Key insight: Coverage tools report per test run, not per test file.

All we needed: save coverage after each individual test.

The Architecture (5 Steps)

Step 1: Instrument the Code 

// Build config (webpack/vite/rollup/etc)
export default {
  plugins: [
    process.env.COLLECT_COVERAGE &&
      coveragePlugin({
        include: 'src/**/*',
        exclude: ['**/*.test.*']
      })
  ]
}
Enter fullscreen mode Exit fullscreen mode

Works with: Istanbul (JS), coverage.py (Python), SimpleCov (Ruby), JaCoCo (Java)

Step 2: Collect Coverage Per Test 

// Test framework afterEach hook
afterEach(async () => {
  const coverage = getCoverageData();
  saveCoverage(`coverage-${testName}.json`, coverage);
});
Enter fullscreen mode Exit fullscreen mode

Step 3: Build the Reverse Map 

const map = {};

for (const coverageFile of allCoverageFiles()) {
  const testName = extractTestName(coverageFile);
  const coverage = JSON.parse(read(coverageFile));

  map[testName] = [];

  for (const [sourceFile, data] of Object.entries(coverage)) {
    if (wasExecuted(data)) {
      map[testName].push(sourceFile);
    }
  }
}

Example output:
{
  "tests/user-profile.test.js": [
    "src/components/Profile.jsx",
    "src/components/Card.jsx",
    "src/utils/formatters.js"
  ]
}
Enter fullscreen mode Exit fullscreen mode

The magic: The test told us what it executed. No parsing, no guessing.

Step 4: Detect Tests 

CHANGED=$(git diff origin/main...HEAD --name-only)
TESTS=$(lookupTestsInMap "$CHANGED")
npm test $TESTS
Enter fullscreen mode Exit fullscreen mode

Step 5: Auto-Update 

npm run build-coverage-map
git add coverage-map.json
git commit -m "Update coverage map [skip ci]"
git push
Enter fullscreen mode Exit fullscreen mode

The Results 

┌───────────┬────────┬───────┐
│  Metric   │ Before │ After │
├───────────┼────────┼───────┤
│ Tests run │ 47     │ 4     │
├───────────┼────────┼───────┤
│ Time      │ 18 min │ 2 min │
├───────────┼────────┼───────┤
│ Accuracy  │ 15%    │ 95%+  │
├───────────┼────────┼───────┤
│ CI cost   │ $12    │ $1.20 │
└───────────┴────────┴───────┘
Enter fullscreen mode Exit fullscreen mode

Real PR example:

  • Changed: 1 component
  • CI time: 43 minutes → 4 minutes

Language Examples 

JavaScript (Istanbul + Jest/Playwright)
afterEach(async () => {
  const coverage = global.__coverage__;
  saveCoverage(coverage);
});

Python (coverage.py + pytest)
@pytest.fixture(autouse=True)
def save_coverage(request):
    yield
    save_coverage_data()

Ruby (SimpleCov + RSpec)
RSpec.configure do |config|
  config.after(:each) { save_coverage }
end

Java (JaCoCo + JUnit)
@AfterEach
void saveCoverage() {
    save(getCoverage());
}
Enter fullscreen mode Exit fullscreen mode

The Tradeoffs

What Works:

  • ✅ Zero maintenance (auto-updates)
  • ✅ Handles refactors naturally
  • ✅ Incremental updates

What Required Tuning:

  • ⚠️ Build tool noise (bundlers include common code)
  • ⚠️ Shared utilities (some files in 50+ tests)
  • ⚠️ Git history churn (auto-commits)

Solutions: Keyword filtering, threshold checks, minimum change limits

The Key Insight

Traditional approaches model relationships:

  • Import graphs → static dependencies
  • Pattern matching → naming conventions
  • Manual tags → human intent

We don't model. We measure.

Try It Yourself

~500 lines in any language:

  1. Instrument (100 lines)
  2. Hook (50 lines)
  3. Parse (200 lines)
  4. Lookup (100 lines)
  5. CI (50 lines)

Complexity is in filtering noise, not the pattern.

Conclusion

Stop guessing. Start measuring.

Coverage tools have measured execution for decades. We just use them per test instead of per run.

If your CI is slow, this will fix it. Not by being clever, but by being honest.

Questions? Comment below or reach out on Twitter/LinkedIn

Top comments (0)