We benchmarked a security detector against 500k lines of Go. It found a real bug.

StopThatSlop — Mon, 24 Aug 2026 19:05:48 +0000

We're building deslop — rule packs that stop coding agents from
re-introducing bugs that were already fixed. One half of that is
detection: if a rule claims agents keep writing a certain kind of bug,
there should be a detector that catches it, and that detector should be
measured before it's allowed to fail anyone's build.

So we benchmarked one of our security detectors against roughly 503,000
lines of Go across eight well-known open-source projects.

It produced exactly one finding. The finding was real.

The detector

The rule is simple to state: secrets and tokens must be compared in
constant time. A plain string equality check (==) against a secret leaks
information through timing — on shared infrastructure, an attacker who can
measure how long a comparison takes can, in principle, recover the secret
piece by piece. The standard fix is crypto/subtle.ConstantTimeCompare.

Writing a detector that fires on every == next to a variable named
token is easy and useless. Real code compares all kinds of things to all
kinds of things, and most of them are fine. Our detector only fires when
an operand has a traceable credential source — a value that came from
r.Header.Get(...), os.Getenv(...), or an assignment chained from one
of those. It skips test files and vendored code. It resolves named helper
functions one level deep, because CheckOrigin: isValidOrigin and
CheckOrigin: func(...) { return true } are the same bug wearing
different clothes.

The benchmark

Eight repositories, pinned commits, ~503k lines of Go (counted as
non-blank lines, tests and vendor excluded):

Repo	What it is	LOC	Findings
caddy	web server / reverse proxy	104,660	0
lazygit	TUI git client	141,754	0
restic	backup / storage infra	88,525	0
goreleaser	release automation	92,357	1
fzf	CLI fuzzy finder	33,306	0
task	task runner	23,162	0
chi	HTTP router	12,082	0
viper	configuration	7,194	0

Seven clean repos matter as much as the hit. A detector that cries wolf on
if key == "border-native" (fzf compares lexer tokens, not credentials)
or if setting.Key == "vcs.modified" (task dispatches on config keys)
is a detector nobody will leave enabled. Our first version produced
exactly those false positives — twenty-two of them. We hardened the
operand analysis (require the credential source, not the name) and
re-ran. One finding.

The finding

goreleaser, internal/client/gitlab.go, in checkUseJobToken:

ciToken := os.Getenv("CI_JOB_TOKEN")
if ciToken == "" {
    return false
}
// ...
if ctx.Config.GitLabURLs.UseJobToken {
    return token == ciToken
}

token is the user-configured GitLab API token. ciToken is the CI job
token from the environment. The comparison decides which API client to
use — and it's done with plain string equality.

To be clear about severity: this is hardening, not a zero-day.
Exploiting a timing side channel over a network on a comparison like this
is largely theoretical, and ConstantTimeCompare still leaks length
(though CI_JOB_TOKEN lengths are fixed per environment). We said exactly
that in the PR. The maintainers' response treated it as a reasonable
correctness improvement — which is the right way to think about it.

The fix

import "crypto/subtle"
// ...
return subtle.ConstantTimeCompare([]byte(token), []byte(ciToken)) == 1

Two hunks, one line changed. The PR is here:
goreleaser#6813.

What we take from this

Zero findings on 430k lines is the feature. The hard part of static analysis isn't finding things — it's not finding things that aren't there. Every clean repo in that table is what makes the single finding worth a maintainer's time.
Source tracking beats name matching. token == "giteatoken" in goreleaser (a placeholder comparison) looks identical to a credential check if you match on identifiers. Trace where the value came from instead.
This is the bar for rules that gate CI. If a rule can't run clean across half a million lines of real code, it has no business failing anyone's PR.

deslop is open source: github.com/Amaresh/deslop.
The detector that found this, and the benchmark that measured it, are part
of the repo.