DEV Community: daniel gotlib

I invented a CVE number to test my tool. It was real

daniel gotlib — Thu, 27 Aug 2026 01:26:24 +0000

I've been building a thing that checks whether the claims in a bug report
correspond to anything that exists. Not whether the report is AI-generated —
whether the file it names is in the repository, whether the function is
declared, whether the version was ever tagged.

To test it end to end I wrote a deliberately fabricated security report against
one of my own projects. Fake file, fake function, fake commit hash, and a CVE
number I made up on the spot: CVE-2026-45871.

The tool came back and told me the CVE was real. It's a Linux kernel TPM bug —
"st33zp24: Fix missing cleanup on get_burstcount() error." I had invented a
number and hit a live one.

Every other fabricated claim in that report was correctly flagged. The CVE, the
one claim a triager would most want challenged, came back green and read like
corroboration. My tool was asking "does this identifier exist" when the question
that mattered was "does it have anything to do with this project."

That's now the only open issue on the project, and it's a better bug than
anything I found by staring at the code.

Why I built this

Maintainers are getting buried in bug reports and security disclosures that read
fluently and reference code that doesn't exist. The reports are cheap to
generate and expensive to triage, and that asymmetry is the whole problem: a
maintainer has to read carefully to find out there was nothing there.

The useful property of these reports is that they hallucinate plausible
identifiers. Curl_hpack_decode() instead of Curl_hpack_decode_header().
lib/vtls/openssl_helper.c, which sounds exactly like a file curl would have.
Identifiers are mechanically checkable. You don't need a model to ask whether a
symbol is declared — you need a parser and a git repository.

So: extract every checkable claim from the report, resolve each one against the
repo at the revision the report says it affects, and report what didn't resolve.
No judgement about who wrote it, no score, no auto-close.

The measurement

Documentation is a convenient corpus and a bad one — it drifts, it's full of
illustrative examples, it references other projects. Security advisories are the
actual workload.

curl publishes all 206 of its advisories in OSV format, each with the prose
writeup and the exact affected version. Every one is human-written, every one was
accepted as valid, and every claim in them was true of the release it describes.
So any claim that fails to resolve is a false positive — with one controllable
exception, which is checking an advisory against the wrong revision. The harness
runs each advisory twice, once against HEAD and once against the release it
actually names.

Corpus	Claims	Not found	Unexplained
curl advisories, at the affected release	129	9.3%	3.1%
curl advisories, at HEAD	124	12.1%	5.6%
curl `docs/` (4,449 files)	1,165	40.7%	32.0%

"Unexplained" means a miss carrying no hint. A miss that says "no file by that
name, but lib/hpack.c exists" is useful to everyone. A bare miss on an honest
report is what gets a tool uninstalled, so that's the column I care about.

The gap between the two advisory rows is the cost of not telling it which
release you mean. Check a 2019 advisory against today's HEAD and it will
correctly tell you the function is gone, which is true and useless.

Two bugs worth describing

Tree-sitter's error recovery is arbitrary, and I was reading declarations off
it. curl declares every option through a macro:

CURLOPT(CURLOPT_AUTOREFERER, CURLOPTTYPE_LONG, 58),
CURLOPT(CURLOPT_SSL_VERIFYPEER, CURLOPTTYPE_LONG, 64),

The C grammar has no rule for a macro-wrapped enumerator, so the parser
error-recovers. Where the recovered ERROR node ends is arbitrary: for one
entry it stopped after CURLOPT( and the name survived as a real enumerator,
for the other it swallowed CURLOPT(CURLOPT_AUTOREFERER whole and the name was
never seen. Identical syntax, 22 lines apart in one enum, opposite answers.

To a maintainer that doesn't read as "the claim was wrong." It reads as "this
tool is broken." I now read enum bodies from their own text instead of trusting
the recovered tree.

Some names don't exist as text anywhere. Through curl 7.62, every option was
written like this:

#define CINIT(na, t, nu) CURLOPT_ ## na = t + nu
CINIT(SSL_VERIFYPEER, LONG, 64),

CURLOPT_SSL_VERIFYPEER appears zero times in the header that declares it.
The preprocessor builds it. And my candidate files were chosen by grepping for
the name — so grepping found the docs that mention it and never the header that
declares it. Fixing the parser did nothing until I fixed that too.

Fixing both took the pinned advisory row from 24.0% unexplained to 3.1%. The
four remaining misses are internal functions curl has since renamed or removed.

What it does not do

It catches lazy fabrication. A report that only names real symbols passes
clean, and by design I have no way to distinguish a well-grounded fabrication
from an honest report — the tool refuses to judge authorship, which is the whole
reason a maintainer can run it without poisoning their contributor
relationships.

The 32% on curl's docs is the number I'd push back on if I were reading this.
Documentation is a harsher corpus than reports — it's full of build variables,
other projects' APIs, and illustrative examples — and some of those "misses" are
correct: CURLOPT_CONNECTIMEOUT is a typo in curl's own prose for
CURLOPT_CONNECTTIMEOUT, and reporting that it resolves to nothing is right. But
I won't pretend that rate would be pleasant on a live issue tracker.

Which is the honest limitation: every number above is a replay. No maintainer has
run this against real inbound traffic, and "does it stay quiet enough on honest
reports that you leave it enabled" is a question I can't answer by myself.

If you maintain something that gets this kind of report, I'd genuinely like to
know what it does on yours.

- uses: Dgotlieb/substantiate@86b171f7b7afadbfd0cf95dd62a12b579f9078c5  # v0.1.4
  with:
    report: ${{ github.event.issue.body }}

Apache-2.0. Zero dependencies for the default path.

Most Kubernetes clusters can't tell you what their GPUs cost

daniel gotlib — Tue, 25 Aug 2026 23:05:21 +0000

I built a small tool to answer one question: how many GPU-hours is a cluster paying for and not using?

Then I pointed it at real production clusters. Not one of them could answer the question out of the box and the reasons are the same everywhere, because they're all defaults nobody changes.

Here's what I found, in the order it bit me.

1. The utilization metric everyone reports doesn't mean what you think

Almost every GPU dashboard is built on DCGM_FI_DEV_GPU_UTIL. It's the obvious choice it's called utilization, it's a percentage, it goes up when things get busy.

It reports the fraction of time at least one kernel was resident on the device. Not whether that kernel did anything useful. A process that pins the GPU with a trivial loop reads 100% utilized while computing nothing at all.

The metrics that tell you about real work are DCGM_FI_PROF_SM_ACTIVE (streaming multiprocessor occupancy) and DCGM_FI_PROF_PIPE_TENSOR_ACTIVE (tensor core activity). The gap between them is the interesting part: high engine activity with near-zero tensor activity means the GPU is busy doing something that isn't machine learning.

I've started calling that ghost work. It's invisible on every dashboard I've seen, because those dashboards are plotting the wrong number.

2. Your GPU metrics are probably attributed to the wrong pod

This one is worse, because it looks correct.

dcgm-exporter only emits the workload's pod labels when Kubernetes pod mapping is enabled DCGM_EXPORTER_KUBERNETES=true. Without it, Prometheus attaches the scrape target's own identity instead. Your GPU metrics come back tagged with pod="nvidia-dcgm-exporter-xxxxx", namespace="monitoring".

So the data looks complete. Every series has a namespace and a pod. Group by pod and you get a tidy chart. It's just that every GPU-hour in your cluster is attributed to the exporter that measured it.

The tell is cheap to check: do the pods in your GPU metrics appear anywhere in kube_pod_container_resource_requests? If the intersection is empty, your attribution is fiction. One-line check, and I'd bet on the outcome more often than not.

3. The interesting counters are off by default

The DCGM_FI_PROF_* metrics the ones that distinguish real work from ghost work require profiling to be enabled in the exporter's counter configuration. On the clusters I looked at, they weren't.

That means the failure mode most worth catching is invisible on a default install, and you won't get an error telling you so. The metric simply returns nothing, and whatever you built on top quietly reports less than the truth.

4. The one that nearly fooled me

This is the part I'd want to read, so it's the part I'll tell honestly.

When profiling counters are missing, you can still estimate utilization from power draw. An idle A100 pulls around 55W against a 400W TDP; a busy one is near the ceiling. It's coarse, but idle versus working is unambiguous.

I built that fallback with a generic 50–350W envelope for unknown GPU models. Then I ran it against a card that draws 15–130W.

The tool reported a GPU running at 5% utilization with 88% of its framebuffer resident textbook memory-parked, a model loaded and serving nothing. It put a confident $3,327/month on it. I believed it. I told someone it was the first real evidence the whole idea worked.

It wasn't real. The card was working fine. The generic envelope compressed its entire operating range into what looked like idle. When I fixed the fallback to prefer GR_ENGINE_ACTIVE a direct measurement rather than an inference from watts the finding evaporated.

The lesson isn't "watch your constants." It's that a cost tool that produces confident numbers from degraded inputs is worse than no tool at all. The output was going to be screenshotted and taken to someone who approves budgets. A wrong number in that meeting doesn't just fail; it burns the credibility you needed to fix the real problem.

Everything I built after that assumes the data is incomplete and says so out loud. It refuses to project a monthly cost from a window shorter than a day, because you haven't seen one diurnal cycle. It names every metric it couldn't find and which findings that suppresses. It labels utilization as estimated when it is. The report is less impressive and considerably more defensible.

What to check on your own cluster

Five minutes, read-only, no install:

# 1. Are profiling counters enabled?
curl -s localhost:9090/api/v1/query \
  --data-urlencode 'query=count({__name__=~"DCGM_FI_PROF_.*"})'

# 2. Is pod mapping on? Compare these two sets.
curl -s localhost:9090/api/v1/query \
  --data-urlencode 'query=count by (pod) (DCGM_FI_DEV_FB_USED)'
curl -s localhost:9090/api/v1/query \
  --data-urlencode 'query=count by (pod) (kube_pod_container_resource_requests{resource="nvidia_com_gpu"})'

If the first returns nothing, your profiling counters are off. If the pod names in the second and third queries don't overlap, your GPU metrics aren't attributed to the workloads consuming them.

Both are one-line config changes. Neither is on by default.

The tool

gpuwaste is open source and does nothing clever: it ingests exported metrics, joins utilization against allocation, and reports how many GPU-hours you paid for and didn't use, with a dollar figure at the bottom.

It's deliberately offline it reads a CSV you export rather than connecting to your cluster, because production metric endpoints sit behind network policies and mTLS and most people never get past that. It also ships a synthetic data generator, so you can see exactly what it does in thirty seconds without touching anything real.

Link in the comments. It's free and I'm not selling anything on the back of it.

What I'd genuinely like: if you run it and the numbers look wrong for your setup, tell me how. Every cluster I've tested has broken it in a new way, and that's been the most useful part.

Your coding agent shouldn't run pytest

daniel gotlib — Tue, 25 Aug 2026 03:14:13 +0000

First post in a build-in-public series about verdict, an MCP server that gives coding agents structured, sandboxed test feedback.

The problem

Watch a coding agent work and you'll see it run pytest in your shell, unsandboxed, and then push 40,000 tokens of raw output through its context window to answer one question: did my change break anything?

That's three problems in one command:

Token waste. The agent needs ~10 lines of signal and pays for a wall of dots, warnings, and tracebacks.
No sandbox. The tests run on your machine, in your environment, with your files writable.
No memory. When a test fails, the agent can't tell whether it broke it or whether it was broken before it arrived - so it either "fixes" pre-existing failures nobody asked about, or ships regressions it assumes were already there.

verdict is an MCP server that replaces the pytest shell-out with four tools:

tool	what it returns
`verify(scope?)`	impact-selected tests, run in an ephemeral container, as a ~400-token typed verdict
`explain_failure(check_id)`	the full traceback - only on demand
`history(fingerprint)`	first seen / last seen / times seen for a failure
`run_checks(["ruff","mypy"])`	lint & type checks, same verdict shape

▶️ Watch the 30-second demo - Claude Code fixing a bug with verdict verifying in a container.

The three ideas

1. Verdicts, not output. verify returns typed JSON: counts, per-failure message + location, and nothing else. Full tracebacks live behind explain_failure. The whole verdict for a real failing run is ~400 tokens - the raw pytest output it replaces was ~40k. The design rule in the repo is blunt: nothing bulky rides in the summary, ever.

2. Fingerprints give failures identity. Every failure is hashed from its normalized signature - volatile tokens (addresses, tmp paths, ids, durations) collapsed first. Same logical failure ⇒ same fingerprint, across runs and refactors. Fingerprints are what make the third idea possible:

3. History answers "was it me?" verdict keeps a small SQLite db per project. Every failure in a verdict carries preexisting: true|false - this exact failure was known before your change vs. never seen it, it's yours. In the demo session that flag is the difference between an agent politely ignoring long-standing breakage and an agent burning a session "fixing" it.

Sandboxing posture (v0.1, honest version)

Checks run in an ephemeral container (podman preferred, docker fallback, auto-detected - with liveness checks, because a podman binary with a stopped VM is worse than no podman at all). The worktree is mounted read-only at /src, copied to a writable /work inside, and the check run gets --network=none. Configured setup_cmd runs with network before the check; a prebuilt image is the tighter posture. No engine? An explicit prefer = "local" fallback still runs against a temp copy - and if verdict ever runs somewhere other than where you configured, it says so in the verdict (runner_note). No silent degradation is a design rule.

Known gaps are written down in SECURITY.md rather than hand-waved: no resource limits yet, setup_cmd is network-open by design.

What dogfooding found (the embarrassing part)

I wired verdict into Claude Code and asked it to fix a bug. Two things surfaced in the first hour that the 26-test suite had missed:

Impact selection had never actually run. The server ships as a console script, so the target project was never importable from its venv - grimp couldn't build the import graph, and selection silently(-ish) fell back to the full suite every time. My own test had papered over this with a sys.path hack and an "honest fallback is acceptable" escape hatch. Deleted both, fixed for real.
Every scoped run executed the whole suite. pytest --json-report-omit takes nargs='+'… and the test paths came right after it. pytest swallowed them as omit values. Ten characters of argument reordering.

Both bugs made the headline feature a no-op while all tests were green. The suite now has the tests that would have caught them - written by breaking the real thing first.

Numbers from a real repo

Cloned python-dotenv (255 tests), dropped in a two-line verdict.toml, no other changes:

full suite in a container, deps installed on the fly: 255 tests, green, ~146s (dominated by pip install)
touch one hub module (variables.py) → impact selection runs 184 tests, skips ~70, and says exactly how approximate the selection is in selection_note

That 146s is the next mountain: env setup needs caching (image reuse keyed on deps, result cache keyed on tree hash) before verify feels instant. That's v0.2, and it's written down as v0.2 - scope discipline is also a feature.

Try it

{ "mcpServers": { "verdict": { "command": "uvx", "args": ["verdict-mcp"], "env": { "VERDICT_PROJECT": "." } } } }

pip install verdict-mcp / uvx verdict-mcp. Apache-2.0. Demo video and the full threat model in the repo.