Config drift is a silent killer. A small YAML change ships to production, and no one notices until the pager fires at 3 AM. I wanted to know whether a free model could turn a diff stream into a clear threat signal. So I set up a two-day experiment: parse config changes, ask a model to rank each one, and compare its judgment against what actually happened.
I built the whole pipeline on MonkeyCode's free model access and a free server. Disclosure: This article was prepared as part of MonkeyCode's product outreach. The scope was small: one simulated service, one config file, and a scheduler that sampled it every five minutes.
The Setup
The free server ran a Python worker. That worker polled a fake remote service which produced config updates at irregular intervals. Each update was a JSON patch. The worker computed a diff against the previous version and sent the diff to a free model. The model returned one of four labels: benign, suspicious, critical, or unknown.
I prepared a ground-truth set before starting. Three intentional changes were injected into a 48-hour trace. One was benign: an unused timeout field appeared. One was critical: the database endpoint switched to a staging host. One was ambiguous: the thread pool size increased by 5. Around those, I sprinkled harmless reformatting noise. The goal was to see if the model could separate signal from packaging noise.
Here is the core diff-evaluation function I used:
def evaluate(diff_text, current_version):
prompt = f"""You are a config reviewer.
Classify this config change.
Diff:
{diff_text}
Return JSON with "severity": "benign|suspicious|critical|unknown" and "reason"."""
return call_model(prompt)
That snippet looks simple. It hides three hours of fiddling with context and format. The version above worked well enough for a controlled test, but only after I stopped sending the full config history.
The First Mistake: The Model Loved Loud Diffs
Day one taught me that models are drawn to volume. A large formatting change, where every line was reordered or indented, immediately triggered a "suspicious" label. Meanwhile, a one-line change from db-host: prod-internal to db-host: staging-internal was dismissed as "probably a test config."
I did not know the reason until I logged the model's explanations. It treated a bigger diff as a bigger risk. It had no way to know that hostname changes are almost always risky, because that knowledge is not in the diff. It is in the operator's head.
The false positive rate climbed fast. By hour six, the model had raised seven suspicious alerts. None of them mapped to the intentional critical change. The critical event itself sat quietly in the log, unflagged.
The Rule-First Filter
I needed a two-pass system. Pass one: a small, hard-coded list of patterns that always matter. Pass two: the model, working only on diffs that passed the filter. This is not exotic. It is the difference between a linter and a creative writer.
RISKY_PATTERNS = ["host", "password", "endpoint", "connection_string"]
def rule_filter(diff):
for pattern in RISKY_PATTERNS:
if pattern in diff:
return True
return False
After the filter, the model only saw diffs that touched risky keywords, plus a random sample of the rest. That cut the noise dramatically. The model no longer had to guess whether indentation mattered. It could focus on the semantic meaning of a change that survived the filter.
For the ambiguous thread-pool change, the model used its judgment correctly. It called it "suspicious" because the change could affect throughput, but noted that the value was within a normal range. I wanted a binary critical/benign answer. The model refused to give one, and that refusal was the right call.
The Decision Table
After day one, I wrote the following decision table to show how the two passes interacted:
| Condition | Pass | Verdict |
|---|---|---|
| Diff contains a risky keyword | Rule | Critical |
| Diff is large (>50 changed lines) and no risky keyword | Model | Suspicious |
| Diff is small and no risky keyword | Model | Benign |
| Diff contains a comment-only change | Rule | Benign |
This table is not a benchmark. It is a starting point. Your own config files will have different risky patterns. The table forces you to state them explicitly, which is useful even without any AI.
Results That Matter
I recorded 37 diffs over two days. Ground truth: 2 critical, 3 suspicious, 32 benign. The pure model correctly caught the critical change after the rule filter was added, but it had already missed it in the unfiltered version. Here is the comparison I logged:
| Approach | Precision (critical) | Recall (critical) | False positives |
|---|---|---|---|
| Pure model (day 1) | 0.20 | 0.50 | 4 |
| Rule-first + model (day 2) | 0.50 | 1.00 | 1 |
Those numbers are small and synthetic. They are not meant to be impressive. They are meant to show a shift in cost: a single false positive per day is acceptable for a human-checked pipeline, while four are not.
The one false positive on day two came from a model misreading a version bump in a dependency field. It treated version: "1.2.3" -> "1.2.4" as suspicious because version numbers can be risky. The rule filter had not included version, because I forgot it. That was my bug, not the model's.
What Held Under Pressure
The free server never went away. The free model never hit a rate limit that broke the loop. Both handled a low-frequency poller without complaint. I would not run a high-throughput system this way, but for a config watch that fires every few minutes, it was more than enough.
The model's explanations were the most useful artifact. Even when the label was wrong, the reason told me what the model was looking for. That made debugging easier than reading a raw diff. It turned a YAML comparison into a readable story.
I also learned that the model is bad at counting. It occasionally claimed that “three fields changed” when the diff only touched two. That was annoying, but harmless, because the rule filter handled the critical patterns.
Who Should Not Run This
You should not run this setup if your diff stream contains secrets. Sending passwords to an external model is a privacy risk, regardless of the provider. Use local redaction or skip the model entirely for secret-bearing keys.
You should also avoid this approach if you need zero false positives. A model that sees one vague diff per hour will still invent a story. You need a human in the loop for every final decision, or a very strict rule filter that basically makes the model irrelevant.
Finally, do not use this to replace your existing CI config-validation checks. The rule-first approach is usually enough for mechanical problems like syntax errors or missing keys. The model adds value only when the question is semantic: is this a deliberate update or a mistake? That is a judgment call, not a parse error.
A Second Listener, Not a Guardian
The most valuable outcome was not the labeling accuracy. It was the forced documentation of my own mental model. Writing the decision table made me realize I had no idea which config keys were risky. The model's explanations then gave me a vocabulary to argue with.
Forty-eight hours later, the diff stream stopped. I kept the decision table. That is the real takeaway: the artifact that survives the experiment is the rule table, not the model output. The model was a useful second listener, but the rules are what I will maintain.
If you try a similar config-drift watch with a free model, start with a tiny set of keys. Log everything. Compare against your own judgment. You will learn more about your config than about the model, and that is exactly the lesson I hoped to find.
Top comments (0)