DEV Community

TECHDEV
TECHDEV

Posted on

Measured on real public CI logs: what two small tools actually did, and where they fell short

#typescript #testing #devops #opensource

Most tool READMEs show numbers from their own fixtures. We wanted numbers from logs nobody on our side wrote, so we ran two public repos, context-diet 0.5.0 and ci-failure-detective 0.1.0, cloned from GitHub and built and run exactly as their READMEs describe, on two real public CI logs from other projects. Everything below is what the runs actually printed, including the misses. Lines and bytes are exact. Token counts are the tools' own chars/4 estimates, labeled as estimates throughout, never as measured tokens. We did not run a real tokenizer, so we cannot say whether chars/4 overcounts or undercounts these logs.

The two logs

Log A: prettier/prettier, workflow "Dev", run 40391802031 (PR #20363, "Update babel to v8.0.8", 2026-10-26), job "[Full Test] Node.js 26 on Linux". Downloaded log: 2,589 lines, 251,942 bytes, 62,967 estimated tokens (chars/4). Ground truth from the log itself: 1 failed suite (tests/format/js/errors/export-default-from/format.test.js), 4 failed snapshot tests, 1,577 passed suites, 103,099 passed tests, 5 skipped, 103,108 total. The failures are snapshot mismatches: babel v8 changed parser error wording ("Unexpected token" versus "Unexpected token, expected from"), so stored snapshots no longer match. That is a real test failure caused by an upstream message change, not a bug in prettier's own formatting code.

Log B: jestjs/jest, workflow "Node CI", run 40415519415 (branch renovate/eslint-plugin-unicorn-79.x, 2026-10-26), job "Windows with shard 1_4, Node v25.x". Ground truth: 2 failed tests in e2e/tests/customReporters.test.ts ("Custom Reporters Integration" - "prints reporter errors" and "supports reporter written in ESM"), 2 failed / 11 skipped / 1,394 passed / 1,407 total in the failing invocation. The failure cause visible in the log is a Windows temp-directory cleanup error (EPERM, permission denied, on fs.rmSync). A later invocation in the same job log passed everything (1,396 passed, 11 skipped, 1,407 total). In other words: a Windows-only environment flake, not a code regression.

Method note: read this before the numbers

GitHub's downloadable logs prefix every line with a timestamp. ci-failure-detective's test parsers match markers at line starts, so on raw downloaded logs they extract nothing; we verified this on both logs (zero tests extracted from each raw log, framework reported as unknown). To reproduce the input the tool was built for (piped test output, no timestamps), we stripped only the timestamp prefixes and ANSI color codes from Log B before running the parser, and verified the strip changed nothing else: same line count, every line identical after the strip. Log A was run both raw and stripped; the parser gap there is different (see below) and survives the strip. context-diet was run on Log A exactly as downloaded.

context-diet on Log A

We replicated the procedure from the tool's own README example: pack with a 400-token budget and the query "test failure". We also ran dedupe alone and extract alone, because a single pack number hides which stage did the work.

Run Lines Bytes Est. tokens
Input (Log A) 2,589 251,942 62,967
dedupe alone, raw log 2,589 251,942 62,967
pack, budget 400, query "test failure" 13 1,580 395
pack, budget 4000, same query 163 15,771 3,941
extract alone, same query 1,694 149,870 (not reported)

Three findings, in order of importance.

First, dedupe removed nothing from the raw log: 0 duplicate lines, 0 duplicate blocks. Every line carries a unique timestamp, so no two lines are identical. The tool does not normalize timestamps today, so on raw GitHub logs its dedupe stage is idle. As a secondary check only, we ran dedupe on the same log with timestamps and color codes stripped: it then removed 266 lines (115 duplicate lines plus 26 blocks covering 151 lines), taking 2,589 lines to 2,323 (10.3% of lines; bytes 174,192 to 158,340 on the stripped input). Near-duplicate collapsing found nothing here, which is expected: this log contains no repeated traces differing only in memory addresses. The timestamp effect is therefore measurable on this log: 0% versus 10.3% of lines.

Second, the headline pack saving is mostly budget truncation, not duplicate removal. At budget 400, the manifest reads: strip-ansi removed 2,698 bytes and 0 lines; dedupe removed 0 lines; extract kept 10 chunks (2,589 to 1,694 lines); summarize dropped 1,663 lines as over-budget plus 18 noise lines. So the 99.5% line saving (2,589 to 13) is the budget doing the cutting, and every output line is still verbatim from the input (verbatim: true). That is a property worth keeping, but it is not evidence that the log was 99.5% duplication.

Third, at budget 400 the actual failure did not survive. The 13-line output contains two node warnings and eleven PASS lines, but no FAIL line and no mention of the failing suite. The mechanism is in the summarize code: it picks lines by priority (error-matching lines first) then document order until the budget is met, and its error pattern also matches PASS lines under errors/ directories. In the extracted text, 50 such lines precede the FAIL line, and the 400-token budget was spent before reaching it. At budget 4000 the failure does survive (163 lines, FAIL present), at a 93.7% line saving that is still mostly budget truncation (summarize dropped 1,531 lines). The honest summary: on this log, pack is a lossy budget summary, and the budget you pick decides whether the failure you care about is still in the output. Protect the lines that must survive, or check the output before trusting it.

ci-failure-detective on Log A and Log B

On Log A, the tool found the first error correctly: line 449 of the downloaded log, FAIL tests/format/js/errors/export-default-from/format.test.js, with the snapshot diff as context. That matches ground truth. But it extracted zero failing tests and reported the framework as unknown, raw and stripped alike. The reason is visible in the parser code: the jest parser needs bullet failure blocks or checkmark/cross result lines, and prettier's custom format-test runner emits neither. Its failing checks appear as "snippet: #0 > format" blocks, which no parser pattern matches. Verdict: unknown, with no history supplied. The tool pointed at the right suite and stopped there; it could not name the four failed checks.

On Log B (after the timestamp and color strip described above), the tool found the first error correctly: FAIL e2e/tests/customReporters.test.ts (9.769 s), and reported the framework as jest. It named both failing tests correctly ("Custom Reporters Integration" - "prints reporter errors" and "supports reporter written in ESM"), matching ground truth. But it attributed both to the wrong file (packages/jest-environment-node/src/tests/globals_cleanup_3.test.ts instead of customReporters.test.ts). The reason is structural: the bullet failure blocks sit in the summary section at the end of the log, after all file lines, so the parser's "current file" is whatever file line came last, not the failing suite's file. Verdicts for both tests: unknown, confidence 0, reason "No history supplied". That is the honest answer here, but it is not the flaky verdict the ground truth would support.

Why verdicts cannot go further on logs like these: the tool's record command stores failing tests plus per-test pass records, but pass records only exist when the parser sees individual passes, which needs verbose test output. Jest's own CI log contains zero checkmark/cross lines, so no pass records exist at all. Reading the scoring code, without pass records the same-commit verdict (0.95), the flip-rate verdict (0.70), and the regression verdict (0.80) cannot fire; the reachable verdicts are unknown or "real" at 0.55 for a test that failed before and never passed in recorded history. The in-log retry heuristic also needs retry markers, and this log contains none. Separately, the file misattribution above means history keys (file plus test name) would not match pass records even if verbose logs existed. We did not manufacture history from the same run to force a verdict; that would be circular.

Where rivals beat us

Stated plainly, because these are the gaps the measurements exposed:

  • GitHub's own failure annotations beat this tool on position: they show the failing suite directly on the run page, with no parsing step and no format caveats.
  • A plain text search for FAIL beats it on Log A: the first error was one grep away, while per-test verdicts needed markers the log does not have.
  • Real tokenizers beat chars/4 on measurement: our token numbers are estimates by construction, and we did not verify them against a real tokenizer.
  • The projects' own READMEs already list the honest comparisons: ci-failure-detective names hosted flaky platforms (dashboards, alerting, auto-quarantine) that it does not aim to replace, and context-diet keeps every output line verbatim instead of rewriting, which is a smaller claim than tools that summarize freely. We keep both framings: local, deterministic, no log leaves the machine, and narrower than the platforms above.

What we did not measure

Two logs are not a benchmark. We did not test other frameworks' logs, other budgets beyond 400 and 4000, or history-based verdicts, because the logs we used cannot support them (see above). We did not measure tokens with a real tokenizer. Nothing here claims these tools beat any rival on detection quality; the claim is narrower: on two real public logs, first errors were found correctly twice, per-test verdicts were absent twice for stated, checkable reasons, and pack savings track the budget you choose more than the duplication in the log.

Sources

Top comments (0)