DEV Community: secleeman

The Day I Became the One Being pip Installed: My Pre-Release Checks Caught 3 Leaks

secleeman — Mon, 31 Aug 2026 03:15:19 +0000

(Translation of my Japanese article on Zenn.)

This is part 4 of a series where I keep delegating implementation to AI without being able to read the code, building a vulnerability triage CLI called triage-lens. This installment is about distribution rather than the tool's internals: the tool had been sitting on GitHub, and I published it to PyPI so a single pip install triage-lens brings it in.

A confession first. Shipping took more nerve than any of the feature work did. And three "leaks" actually turned up right before release.

From the installing side to the installed side

I can't read code, but I have typed pip install before. Years ago I dabbled in Python out of curiosity, and the one thing that stuck was the experience of a useful tool arriving in one line. Now that I'm the one publishing, the other side of that one line finally became concrete. Someone builds a thing, shapes it into a package, and puts it on the public shelf called PyPI. That's why it installs in one line anywhere in the world. My turn to put something on the shelf.

I delegated the release work to AI too: package metadata, the release workflow, and one thing I insisted on. Instead of an API token, authentication to PyPI uses Trusted Publishing (OIDC). Nothing like a long-lived password gets stored anywhere; you declare "trust publishes from this workflow in this GitHub repository" and that's it. A secret you never hold is a secret that can't leak.

The pre-release check caught three real ones

In this project, nothing goes out to a public repository without passing a mechanical check. Procedures and tests, not eyeballs, verify that no personal or development-only information is mixed in. For three releases it came up empty. That's what insurance looks like.

On the fourth run it caught something real. Three somethings.

First, test code had slipped into the distribution. The packaging tool's default behavior had a path where the whole development test suite gets bundled along. I was about to scatter files nobody needs into users' environments.

The second one scared me. The name of my private development repository was about to appear in a published artifact. I run this whole activity anonymously, and keeping the development side separate from the public face is a design I care about. That boundary nearly got crossed. The funny part is how it got in: the test that checks whether the dev repository name has leaked had that very name hardcoded in its own source, so the check itself was the source of the leak. Building a check doesn't make you safe. The check is something to distrust too.

The third was mundane: relative links in the README turn into 404s on the PyPI page. Links that work on GitHub break when displayed somewhere else.

All three were fixed before release and never went out. The point I want to make is not "that was close." It's the opposite. A fixed procedure calmly caught things eyeballs would have missed. The value of turning verification into a mechanism shows up on the fourth run, after three swings at nothing.

Traps waited after the publish button too

It didn't end with a clean release. Two lessons came after publishing.

One: the description shown on the PyPI page is baked into the artifact at the moment you publish. Fix the README on GitHub afterward and the published version's page never changes. I had scheduled the README update for "after the release succeeds," so the page right after launch showed stale instructions. I ended up shipping v0.4.1, identical in content down to the last character, just to line the page up. That lesson became one more line on the pre-release checklist.

Two: the post-release smoke check failed with "that version does not exist." Not the tool's fault, not a config mistake. PyPI takes time to propagate; the dashboard showed the release while pip couldn't see it for a few dozen seconds. One more line on the checklist: confirm the new version is visible to pip before running the check.

Distribution turned out to be a pile of small traps like these. None fatal. All of them unknown to me until I tried.

It doesn't feel real yet

A night after publishing, honestly: it doesn't feel real. Anyone in the world can now install it in one line. But "can install" is not "is installing."

Whether this tool is useful will show up from here on as download counts, quantitative and merciless. I've already built weekly automation that collects article reactions and package usage. The zeros might just keep lining up. Recording that too is what this series is for, so when the numbers come in, good or bad, I'll write them up.

The repository is public. pip install triage-lens gets you in.

https://github.com/secleeman/triage-lens

I Built a Vulnerability Triage Tool Without Knowing What EPSS or KEV Meant

secleeman — Sat, 29 Aug 2026 04:46:13 +0000

In the last two posts I described how, unable to read code, I delegated implementation to Claude Code, built a vulnerability triage CLI called triage-lens, and then let it run Phase 2 overnight while I slept. Both posts ended with the same promise: next time, the substance. How EPSS and CISA KEV fit together.

I sat down to write that third post and stopped.

I cannot explain what EPSS is. Or KEV. Honestly, even CVSS is shaky. My own tool runs on this data, passes 373 tests, and has shipped as v0.2.0. And I could not have defined the three scores it is built on.

I could fake it. An AI could draft a plausible explainer in an hour. But the spine of this series has been admitting what I don't understand and then showing how quality gets assured anyway. So this post delivers the promised explanation while starting from the awkward fact that the person who built the tool understood it least.

(Translation of my Japanese article on Zenn.)

What I did have: the memory of the floor

One thing in my defense. I didn't know the score definitions, but I knew exactly what a score-only list does to a team.

Years ago I worked as a PMO on automotive software development. Nothing to do with security; my job was keeping delivery on track. The engineering side's honest position was simple. We both know fixing everything is impossible, so give us an order.

Forwarding a list of findings from right to left isn't management. It exhausts the team and fixes nothing. Deciding the order is the manager's actual job, and we had no basis for it. Half my motivation for building triage-lens was that memory. The theory came later. The hunger for an ordering tool came first.

The three words, as the AI explained them to me

So, belatedly, I had the AI break it down until I could restate it myself. The weather analogy is the one that stuck.

CVSS is severity. A 0-to-10 score, a shared global ruler. It measures how much it hurts if you fall. It says nothing about whether you will fall. This is the misunderstanding at the center of everything: a list of several hundred CVSS 9.8 findings is a list of painful cliff heights, not a list of places about to collapse.

EPSS is the forecast. FIRST.org computes, with machine learning, the probability that a vulnerability will actually be exploited in the next 30 days, refreshed daily. It's the chance of rain. Plenty of high-CVSS vulnerabilities carry a tiny EPSS, and only a small fraction ever get exploited at all.

CISA KEV is the record. A catalog maintained by the US security agency CISA of vulnerabilities with confirmed exploitation in the wild. Not a forecast. A log of where it already rained.

And the layering works like this. Sort by pain alone and everything looks top priority. Add probability and add fact, and a short list of things to look at first rises out of the pile. In triage-lens, the top rank P0 means listed in KEV: it already rained there, no debate. P1 means both likely and painful, EPSS at or above 0.1 with CVSS at or above 7.0. Everything else falls in behind.

Written out, it is simple arithmetic. I understood it after my tool was already public.

How I trusted a thing I didn't understand

This is the part I actually wanted to write. How does someone with no theory trust a tool that runs on theory?

Looking back, every check I ran lived outside the theory. Take the CVEs ranked P0, download CISA's actual KEV catalog without going through the tool, and compare. The three P0 findings (Log4Shell, Heartbleed, Spring4Shell) were really listed. The nine ranked P1 or lower were really absent. For EPSS, hit the public API in a browser and confirm the numbers match.

I still don't know how EPSS is calculated. But whether the tool's claims agree with the world's facts can be verified with zero theory. Understanding and verification are different skills, and verification is the one you need first.

Understanding caught up later, and honestly it landed better for the delay. After fighting defects in the failure paths, like the one that displayed a missing exploit probability as "low," the difference between CVSS and EPSS finally meant something. If I had read the textbook first, it would have gone in one ear and out the other.

The order: verify from day one, understand as you go

My conclusion. When you use something you don't understand, the thing to secure first is not understanding. It is verification. As long as you can check the tool's claims against outside facts, it stays trustworthy while your understanding catches up. Understanding without verification, on the other hand, is indistinguishable from the plausible explainer an AI can write in an hour.

The PMO I used to be wanted exactly this. Explain the theory later; give me a defensible order now. I think triage-lens became the tool I wished someone had handed me back then.

The repository is public. The reason column on P0 rows says "listed in KEV, actively exploited" in plain words. That line is for people who can't read the scores yet. People like me, three weeks ago.

https://github.com/secleeman/triage-lens

I Wrote a "Contract" So AI Could Ship Code While I Slept

secleeman — Sat, 29 Aug 2026 02:31:50 +0000

Last time I wrote about building triage-lens, a vulnerability triage CLI, by delegating all implementation to Claude Code despite not being able to read code. This is the sequel. For Phase 2, I had the AI do everything unattended, overnight, while I was asleep.

When I woke up, the test suite had grown from 145 to 330, an adversarial review by a second AI was already done along with the fixes, and one pull request sat waiting, unmerged. Also, at some point during the night, a PR had deadlocked and been quietly rescued without me.

Here is the record of that night.

(Note: this is a translation of my Japanese article on Zenn. The tool's reports support Japanese and English as of v0.2.0.)

Before bed: replacing approval with a contract

Until now, our process had one central rule: present a plan, get my approval, then implement. That gate is where quality comes from. But I can't approve anything while asleep, and dropping the gate entirely felt reckless.

What I believed going in was that risk control lives in the instructions and the design, so I checked the prompt carefully before handing it over. In place of the approval gate, I wrote four things into it.

A frozen scope. Tonight's allowed work: CycloneDX input support, English reports, README updates. Anything else was explicitly forbidden.
Hard lines. No direct pushes to main. No merging. No syncing to the public repository.
A rule for doubt. If a requirement was ambiguous, err on the side of not implementing, and leave the question in an issue. If blocked by an error, push the work-in-progress and describe the situation.
A morning report format. What it can now do (three lines, for a non-engineer), verification commands I can copy and paste, test results, and whether anything needs my judgment.

In short: instead of asking me mid-flight, leave a trail of your decisions. The approval didn't disappear. It became something I could grant after the fact, over coffee.

The morning report: the fixes scared me more than the numbers

The report was waiting when I woke up. 330 tests green, CI green on three Python versions, PR unmerged. Exactly as instructed.

The interesting part came next. After finishing the implementation, Claude Code had followed the instructions and handed the entire codebase to a different AI (the codex CLI) for an adversarial review. Seven findings came back. Rather than accepting them at face value, it reproduced each one, fixed the four that were in scope, and filed the three debatable ones as issues for me to rule on later.

Two of the fixed defects were the kind a security tool cannot afford.

One: the tool had been reading "unaffected version" records as "fixed versions," which in some conditions produced output that recommended downgrading to an older release. Two: the input format check was loose enough that Trivy output could be misclassified, yielding a clean report with zero findings. Both errors run in the same direction, making dangerous things look safe, and both belong to the same family as the defect from my previous article, where a missing exploit probability was displayed as "low."

Three builds, three defects of this exact family. At this point I stopped treating it as coincidence and started treating it as the default suspicion to bring to any AI-written implementation.

The deadlock: handling what I couldn't understand

Buried in the report was this passage. Deleting a branch had auto-closed an unmerged PR, which led to a trap: a closed PR can't change its base branch, and a PR whose base branch is gone can't be reopened. So the AI temporarily restored the branch, reopened the PR, repointed its base, and deleted the branch again.

Honest admission: reading this first thing in the morning, I did not understand what had happened. People fluent in Git operations probably find this routine. I had no way to judge it.

What let me keep reading was the shape of the report: what happened, why, how it was recovered, and what the current state is, ending with claims I could verify (same PR number, still open, based on main, mergeable). I couldn't evaluate the content, but I could evaluate whether the form was complete. When someone who can't read code manages an AI's work, that form is what you lean on. Later I had the AI break the incident down until I could explain it in my own words, and only then felt settled about it.

Round two: making the tests prove they catch the bug

In my morning ruling, I picked one filed issue to fix before anything went public. When the same vulnerability was detected in two places, deduplication kept whichever came first, so under the wrong conditions the higher CVSS score could silently vanish. That understates risk, which put it in the must-fix family.

What impressed me about the fix PR was how the tests were validated. Passing tests after a fix proves little, so I had it deliberately revert the implementation to the old first-wins behavior and confirm that 17 tests fail. Do the tests actually catch the defect, or were they merely written to pass? You don't need to read code to demand that check.

One more decision I've grown fond of. We added a rule that when two records disagree on the fixed version, the newer one wins. But version comparison was limited to plain digits-and-dots notation like 1.2.10. Trying to be clever about Debian's epoch-prefixed strings and similar formats risks inventing a false ordering, so anything uncomparable falls back, safely, to "unknown." Declining to handle everything intelligently is also a security decision.

The suite ended at 373 tests, and v0.2.0 is now public.

What can run unattended, and where a human must stand

My conclusion after one night: where you draw the line depends on what you refuse to compromise.

For me, that was "never make danger look safe." It's the one lie a triage tool must not tell, so that property gets verified with a human involved. Everything protecting it, the second-AI review, the reproduction tests, the report format, went into the contract, and once it was there, the implementation itself was free to proceed while I slept.

If this were a writing project instead, the thing a human would guard is whether the piece stays worth reading. Deciding what to hand to the machine and where to keep your own eyes: that design work is the actual job of the person doing the delegating.

The repository is public. Next time I'll write about the tool's substance: what goes wrong when you rank vulnerabilities by CVSS alone, and how EPSS and CISA KEV change the picture.

https://github.com/secleeman/triage-lens

I Can't Read Code. I Had Claude Code Build a Vulnerability Triage CLI Anyway.

secleeman — Fri, 28 Aug 2026 13:43:44 +0000

I work in security as a consultant, and I can't read code. That's not modesty. Hand me a Python diff and I cannot tell you whether it's correct.

Last week I published a CLI tool called triage-lens. Claude Code wrote every line of it. I wrote none, and I didn't read any of it either. Even so, this isn't a toy that may or may not work: it shipped with 145 passing tests, and I checked its judgments against real-world data before merging.

The trigger was mundane. My Claude Max subscription had tokens going unused every month, and I was curious how a tool for my own field, vulnerability management, would actually come together. This post is a record of that experiment. There's no code walkthrough here. Instead, I'll describe how someone who can't read code verified what the AI produced.

(Note: this is a translation of my Japanese article on Zenn. The tool's reports are currently Japanese-only; English output is planned for Phase 2.)

The tool: sorting scanner output into "fix this first" order

If you've ever run a vulnerability scanner like Trivy, you know the output: a wall of high-CVSS findings. Hundreds of 9.8s. "Fix everything now" isn't a plan, and in practice you can't move until you decide the order.

Here's the thing about severity scores: most high-CVSS vulnerabilities never get exploited. Two public data sources help separate the dangerous few from the noise. EPSS estimates the probability a vulnerability will be exploited, and CISA's KEV catalog lists vulnerabilities with confirmed exploitation in the wild. Layer those over CVSS and the pile shrinks to a handful you should look at first.

That's all triage-lens does. Feed it Trivy's JSON output, and it queries the public EPSS and KEV APIs for each CVE, then produces a single Markdown report sorted into four tiers, from P0 (known exploited) down to P3. Python, MIT licensed, and since every API is free, it costs nothing to run.

Three mechanisms I set up instead of reading code

The first decision was how to get quality assurance without code review. My worry going in was specific: once it's done I can tell whether it works, but midway through, I have no way to judge whether the approach is sound or the implementation is any good. Something had to fill that gap.

I set up three things.

First, a rules file in the repository. In a file called CLAUDE.md I wrote hard constraints: no direct pushes to main, no code without tests, no implementing anything outside the current phase's requirements. Requirements lived in a separate file, and Claude Code had to present a plan and get my approval before writing anything. Since plans and reports are the only artifacts I can actually read, I also made the reporting format a rule: plain language for non-engineers, with verification commands I can copy and paste.

Second, acceptance checks on behavior, standing in for code review. A normal sample should produce a report. A malformed JSON file should fail with a clear error. With the network cut, the tool should not spin out; it should state plainly that data couldn't be fetched and continue with what it has. I had Claude Code run all three scenarios and show me the full output. I can't judge code, but I can judge behavior.

Third, and I think this is the heart of the experiment: checking the report's claims against outside data. If the tool says a CVE is in the KEV catalog, I download CISA's actual catalog, bypassing the tool entirely, and look. When I did this, all three P0 findings (Log4Shell, Heartbleed, Spring4Shell) were really in the catalog, and all nine findings ranked P1 or below were really absent. A perfect match. The point is to get your reasons for trusting the tool from somewhere outside the tool.

At this stage the first implementation had 111 tests passing and green CI across three Python versions.

I still didn't trust it, so I brought in a second AI

Honestly, the implementation part was almost anticlimactic. Hand over requirements, approve the plan, and a tested PR shows up. Every number green.

But my past experience with AI had taught me that every additional review pass turns up new flaws. Green tests and correct implementation are different claims. The same AI wrote both the code and the tests, so they can share the same blind spots. I figured a different model would bring a different set of eyes, so I exported the full PR diff and handed it to another AI (the codex CLI) with one instruction: review this adversarially and list problems in order of severity.

It came back with nine findings. And here the same rule applied in reverse: don't take the reviewer's word for it either. I had Claude Code attempt to reproduce each finding to confirm it was real. Six reproduced and needed fixing. The rest were plausible in theory but out of scope for this phase, so I deferred them.

The worst defect: displaying "unknown" as "low"

Of the six, one stood out to me as the most serious.

When the EPSS score couldn't be fetched, the report's explanation column read "exploitation probability is low." The correct word is "unknown." The data was simply missing, but a reader would see "we checked, and it's low." That error runs in the dangerous direction: it makes risky things look safe, which is the one kind of lie a triage tool must never tell.

A second finding was related. When the same CVE appeared in two places, once in the OS packages and once in an application dependency, deduplication silently dropped one of them. That's a missed fix waiting to happen.

Neither of these was catchable through my behavior checks alone, because the happy path worked beautifully. Security judgments are not a place for rounding off. If you couldn't fetch the data, say you couldn't fetch the data. Build so there's no room for interpretation. Obvious principles, but whether an AI-written implementation honors them only shows up when you poke at the failure paths.

Some of the nine findings I couldn't evaluate on my own knowledge. For those, I had the AI break the issue down until I could explain it to someone else in my own words, and only then made the call. Saying "fix it" without understanding it felt like crossing the one line I'd held throughout.

After the six fixes, the suite stood at 145 tests, including 22 new regression tests, all passing. I merged.

Not being able to read code hasn't been the limit. Yet.

My conclusion so far: I haven't hit a wall that's attributable to not reading code.

The tool's small size surely helps. Still, I can verify correct behavior with my own hands, and I can check the tool's judgments against external ground truth. For reviewing the implementation itself, splitting the work across two different AIs is at least better than trusting one completely. It doesn't match a human expert's review, but "can't read code" turned out not to mean "can't assure anything."

What mattered more was everything outside the code: writing the rules down first, cutting requirements into phases, fixing the acceptance criteria in advance. Which is to say, the same things I do in my day job.

Phase 2 will add CycloneDX (SBOM) input and English reports. The repository is public if you want to take a look:

https://github.com/secleeman/triage-lens