What happens when you have 52 tools
I run my own panel with 52 security tools spread across 7 categories, and it is not a toy.
It scans ports, resolves DNS, checks headers, enumerates subdomains, hunts for secrets, reviews vulnerabilities, and then I have another one that audits Windows itself.
The problem showed up the day I chained the tools into a pipeline and ran the whole thing.
That chain produced 25 tools in 35 seconds and 24 findings on the table.
That is where the real problem started, and it is not finding things. It is that someone has to look at all of them, sort them, and decide which ones matter before the day ends.
The temptation was to add another tool, make it super complex and intimidating, and that was the wrong answer.
What was missing was a judgment layer on top of the list, something that would read the 24 findings and say which are real, which are noise, and which it does not trust enough.
For that I integrated Jev, which is not an LLM but a new tool, the model TypeSafe published on September 15 (Source: docs.typesafe.ai).
Jev does not generate text, it evaluates a state against typed questions and returns an answer with an associated probability, and that is exactly what I needed. When I saw it announced I said this is perfect (and cheap).
I did not touch a single line of the scanners, because the AI layer hooks into the findings list the panel was already aggregating.
The numbers
The nuclear pipeline launched 25 tools in 35.15 seconds and produced 24 findings.
Jev evaluated all 24 in a single pass and split 12 true positives from 12 noise.
A tiny cost for working with 24 findings.
That kills the argument that AI is not affordable for a small or large SOC. The cost of the judgment layer is statistical noise next to the time this beast saves, and by the way this has existed for a long time but someone had the good sense to commercialize it.
Does the system doubt?
What convinced me most was not a hit, it was a dry 0.32.
An alternate HTTP port came out with that confidence, and the panel did not flag it as a threat, it flagged it as review.
The missing security headers came out at 0.86 and 0.89, those are true without discussion.
Tasks that simply completed, like whois or DNS, came out as noise at 0.92.
The difference between 0.32 and 0.89 is the difference between an alarm you chase for half an hour and one you document and close.
Then the panel reorders the whole queue by severity multiplied by the AI verdict, which is what actually changes your morning.
The case that made me laugh
The second layer came from the other side.
My PowerShell audit project walks a Windows machine through 10 modules and sends its findings to the same panel.
One of those audits flagged two high-severity threats.
The first was an unsigned driver loaded from my user's temp folder.
The second, a process running in memory whose binary was no longer on disk.
Read cold, those two are a rootkit and an injection, and anyone who has done triage would put them at the top of the queue.
The driver is called GPU-Z-v8.sys (I thought of Dragon Ball) and it belongs to the GPU monitoring tool that leaves its file in the temp folder, and the file is from July 28 with the program not even installed on the machine anymore.
Jev gave it 0.76 confidence and a 24% probability of false positive.
The other came out at 0.79 with 21%.
That percentage is the piece no text model gives me with that precision, because it did not tell me it was a rootkit, it does not talk: it calculates, and what it told me was that it was quite likely and that there was a 1 in 5 chance it was something legitimate.
Investigating both took me less than a minute, and that is the work I no longer do by hand.
And on top of that the report writes itself
The third piece does not judge, it writes.
A local 27B model on my own machine served with llama.cpp takes the verdicts and writes the executive summary in Spanish plus an explanation per finding with what it is, why it matters, and what I would do right now.
None of that leaves my machine, not a finding, not a machine name, not a path.
The report that in a normal audit took me 1 hour of writing comes out written, in my language, and with prioritized actions.
OK, so what then?
I will say it before anyone asks, this is one sample and 24 findings with 12 false positives can be a good Tuesday or it can be luck.
I still need to measure calibration in Spanish, which is another language and not the same thing.
I still need to count false negatives, the ones the judgment layer sends to noise that were actually real, because without that data I cannot say the layer is reliable, only that it is being useful to me.
And I still need to take it to the log viewer, where the natural state is not findings but bursts of events.
This is what there is, a 52-tool panel that finishes in 35 seconds and a calibrated judgment layer that costs 4 euro cents.
A local model that writes the report and 2 false alarms this setup resolved in 1 minute.
Think about it, gentlemen.
Originally published at https://sammideblas.com/notas/ai-verdict-layer-on-52-security-tools
Top comments (0)