I work in security as a consultant, and I can't read code. That's not modesty. Hand me a Python diff and I cannot tell you whether it's correct.
Last week I published a CLI tool called triage-lens. Claude Code wrote every line of it. I wrote none, and I didn't read any of it either. Even so, this isn't a toy that may or may not work: it shipped with 145 passing tests, and I checked its judgments against real-world data before merging.
The trigger was mundane. My Claude Max subscription had tokens going unused every month, and I was curious how a tool for my own field, vulnerability management, would actually come together. This post is a record of that experiment. There's no code walkthrough here. Instead, I'll describe how someone who can't read code verified what the AI produced.
(Note: this is a translation of my Japanese article on Zenn. The tool's reports are currently Japanese-only; English output is planned for Phase 2.)
The tool: sorting scanner output into "fix this first" order
If you've ever run a vulnerability scanner like Trivy, you know the output: a wall of high-CVSS findings. Hundreds of 9.8s. "Fix everything now" isn't a plan, and in practice you can't move until you decide the order.
Here's the thing about severity scores: most high-CVSS vulnerabilities never get exploited. Two public data sources help separate the dangerous few from the noise. EPSS estimates the probability a vulnerability will be exploited, and CISA's KEV catalog lists vulnerabilities with confirmed exploitation in the wild. Layer those over CVSS and the pile shrinks to a handful you should look at first.
That's all triage-lens does. Feed it Trivy's JSON output, and it queries the public EPSS and KEV APIs for each CVE, then produces a single Markdown report sorted into four tiers, from P0 (known exploited) down to P3. Python, MIT licensed, and since every API is free, it costs nothing to run.
Three mechanisms I set up instead of reading code
The first decision was how to get quality assurance without code review. My worry going in was specific: once it's done I can tell whether it works, but midway through, I have no way to judge whether the approach is sound or the implementation is any good. Something had to fill that gap.
I set up three things.
First, a rules file in the repository. In a file called CLAUDE.md I wrote hard constraints: no direct pushes to main, no code without tests, no implementing anything outside the current phase's requirements. Requirements lived in a separate file, and Claude Code had to present a plan and get my approval before writing anything. Since plans and reports are the only artifacts I can actually read, I also made the reporting format a rule: plain language for non-engineers, with verification commands I can copy and paste.
Second, acceptance checks on behavior, standing in for code review. A normal sample should produce a report. A malformed JSON file should fail with a clear error. With the network cut, the tool should not spin out; it should state plainly that data couldn't be fetched and continue with what it has. I had Claude Code run all three scenarios and show me the full output. I can't judge code, but I can judge behavior.
Third, and I think this is the heart of the experiment: checking the report's claims against outside data. If the tool says a CVE is in the KEV catalog, I download CISA's actual catalog, bypassing the tool entirely, and look. When I did this, all three P0 findings (Log4Shell, Heartbleed, Spring4Shell) were really in the catalog, and all nine findings ranked P1 or below were really absent. A perfect match. The point is to get your reasons for trusting the tool from somewhere outside the tool.
At this stage the first implementation had 111 tests passing and green CI across three Python versions.
I still didn't trust it, so I brought in a second AI
Honestly, the implementation part was almost anticlimactic. Hand over requirements, approve the plan, and a tested PR shows up. Every number green.
But my past experience with AI had taught me that every additional review pass turns up new flaws. Green tests and correct implementation are different claims. The same AI wrote both the code and the tests, so they can share the same blind spots. I figured a different model would bring a different set of eyes, so I exported the full PR diff and handed it to another AI (the codex CLI) with one instruction: review this adversarially and list problems in order of severity.
It came back with nine findings. And here the same rule applied in reverse: don't take the reviewer's word for it either. I had Claude Code attempt to reproduce each finding to confirm it was real. Six reproduced and needed fixing. The rest were plausible in theory but out of scope for this phase, so I deferred them.
The worst defect: displaying "unknown" as "low"
Of the six, one stood out to me as the most serious.
When the EPSS score couldn't be fetched, the report's explanation column read "exploitation probability is low." The correct word is "unknown." The data was simply missing, but a reader would see "we checked, and it's low." That error runs in the dangerous direction: it makes risky things look safe, which is the one kind of lie a triage tool must never tell.
A second finding was related. When the same CVE appeared in two places, once in the OS packages and once in an application dependency, deduplication silently dropped one of them. That's a missed fix waiting to happen.
Neither of these was catchable through my behavior checks alone, because the happy path worked beautifully. Security judgments are not a place for rounding off. If you couldn't fetch the data, say you couldn't fetch the data. Build so there's no room for interpretation. Obvious principles, but whether an AI-written implementation honors them only shows up when you poke at the failure paths.
Some of the nine findings I couldn't evaluate on my own knowledge. For those, I had the AI break the issue down until I could explain it to someone else in my own words, and only then made the call. Saying "fix it" without understanding it felt like crossing the one line I'd held throughout.
After the six fixes, the suite stood at 145 tests, including 22 new regression tests, all passing. I merged.
Not being able to read code hasn't been the limit. Yet.
My conclusion so far: I haven't hit a wall that's attributable to not reading code.
The tool's small size surely helps. Still, I can verify correct behavior with my own hands, and I can check the tool's judgments against external ground truth. For reviewing the implementation itself, splitting the work across two different AIs is at least better than trusting one completely. It doesn't match a human expert's review, but "can't read code" turned out not to mean "can't assure anything."
What mattered more was everything outside the code: writing the rules down first, cutting requirements into phases, fixing the acceptance criteria in advance. Which is to say, the same things I do in my day job.
Phase 2 will add CycloneDX (SBOM) input and English reports. The repository is public if you want to take a look:
Top comments (0)