ClinicLog, my practice-management product for therapists, runs in two regions, Israel and the US. Its server logs are where personal data hides: user names, IP addresses, the text of an error. Its privacy policy promises that error logs are stripped of personal data before any outside service sees them. Plain threshold alerts either flood a small team or miss the slow problems: a disk that fills a little every day, an error count that doubles overnight.
An AI model is good at reading slow signals. The obvious build hands it the logs and lets it say whether all is well. That breaks two things: the logs leave our servers, and a model gets to decide that nothing is wrong.
Figure: Example. The names and IP addresses are invented. The rules follow the health gate's code; the model's PASS is supposed, to show what the rule in code does with it. See the animated figure in the original note.
The calls I made
The model gets counts, not log lines. Before anything goes to the model, the lines are dropped. It gets numbers instead, such as how many logins by users nobody expected, how many server errors, how many malware hits, along with what the checks in code found. The lines stay in a private file on our servers, and an alert email points there instead of quoting them. I turned down sending the lines with the names masked: a mask has to recognise every name it will ever meet, and a count has nothing to mask. The model loses little: to see a disk filling or errors doubling, it needs the numbers and the last seven runs, not the words.
Code has the last word on anything critical. Plain checks in code run first and give each finding a severity. The model reads the counts and those findings and answers PASS or ALERT. Then one rule in code: if any finding is critical, the answer is ALERT, whatever the model said, and if the model had said PASS, the email says it was overridden. If the model is down, the checks decide alone. The obvious alternative is to leave it to the instructions the model gets, which say the same thing. It fails because an instruction is a request, and a rule in code runs every time. Below critical, the model does decide, and it may play a warning down. Code guarantees the floor; the model filters the noise above it.
Email only on an alert. A run that passes sends no email, only one routine message to a chat channel, and slow trends wait for a weekly report. The obvious alternative is an all-clear email after every run. It fails because four a day teaches you to stop opening them, and the one that matters goes unread with the rest.
What happened
The gate runs at 01:00, 07:00, 13:00 and 19:00 UTC. The last run before I wrote this, 1 November 2026 at 07:00, was all clear: no email. [1] Five tests plant a patient name or an IP address in the input and check it never reaches the model. [2]
One false alarm taught me the most. On 22 July the gate emailed a critical: the certificate had expired. It hadn’t. A check had failed, the code stored the failure as minus one days, and the model read minus one as expired. No rule in code had raised it, so there was nothing to overrule. Worse, the certificate check in code had a hole: it would have missed a certificate that really had expired. Both were fixed that day. [3]
Under the hood
About forty shell commands per host, over ssh for the remote region, collect disk, memory, containers, backups (local and offsite), certificate dates, failed logins, intrusion-detection events, database errors. Plain threshold checks turn those into findings, each with a severity. The model receives the metrics, the findings and the previous seven runs, and returns a fixed shape: PASS or ALERT, a summary, and findings with evidence. [4] The history matters most: thresholds can’t see “this has grown at each of the last seven runs”, a model reading a trend can.
The barrier is an if statement, right after the model’s answer is parsed:
# SAFETY BARRIER: override verdict if deterministic checks found CRITICAL
has_critical = any(
f["severity"] == "CRITICAL" for f in deterministic_findings
)
if has_critical and verdict.get("verdict") != "ALERT":
verdict["verdict"] = "ALERT"
verdict["summary"] = (
f"[OVERRIDDEN] AI said PASS but deterministic checks found "
f"CRITICAL issues. {verdict.get('summary', '')}"
)
Log lines carry bound SQL parameters, exception messages, user names in auth logs. Any field ending in _samples is dropped, and a finding’s “first matching line” is cut before it is sent. That is a naming convention, not a detector: a new field that carries text has to follow it.
Figure: Invented names and documentation IP ranges. See the animated figure in the original note.
The model learns that there were three SSH logins by users nobody expected. Which users, and the lines themselves, stay in a private history file on the server that runs the gate. One of the five privacy tests checks that raw findings from earlier runs never reach the model either.
The false alarm of 22 July, in full: one of the forty ssh connections the gate opened back to back was reset during the handshake, the certificate check came back empty, and the code stored “check failed” as minus one days. The threshold only looked at 0 to 7 days, so it never fired. Replay it in the diagram above, under 2026-07-22.
Two fixes went in that day. The gate’s metric commands now share one multiplexed ssh connection. And a certificate check that fails is reported as unknown, a warning, never as minus one, while a negative number is now its own critical. In October the same rule reached the login and malware counts: a failed count of logins isn’t “no logins”, it’s “I couldn’t look”. I am still applying it check by check.
What transfers
- Let code set the floor; let the model read the trend. Thresholds for what must never be missed, a model for what moves slowly.
- Send counts, keep text. Most of the judgment survives without a single raw line.
- A failed measurement is not a measurement. “Unavailable” has to be a value the whole pipeline understands.
Sources
- Runs at 01:00, 07:00, 13:00 and 19:00 UTC; last run 1 Nov 2026, 07:00: all clear, no email sent
- 14 tests; 5 plant a name or IP and check it never reaches the model
- A false alarm on 22 Jul 2026 and its fix, documented in the code
- The gate script: 1,721 lines, 51 commits since 22 Mar 2026
Originally published at yussieik.com, where the figures are animated.
Top comments (0)