DEV Community

Cover image for Letting an LLM write alerting rules, but never letting it flip the switch
Sarit Chauhan
Sarit Chauhan

Posted on AI-assisted

Letting an LLM write alerting rules, but never letting it flip the switch

This is the platform behind taabi Nexus, which Taabi Mobility launched this week. The launch video has three real calls it made into trucks. This is the engineering side.

Last Wednesday a truck driver heard his dashcam intercom tell him, in Hindi, that his seat belt was off. He said yes, he could hear. It asked if he was wearing it now. He pulled the belt across while answering. Nobody had dialled that call. A fleet manager had typed a sentence into a text box a few days earlier, and that sentence did.

I've spent the last few weeks building the thing in between. Dashcams raise alerts: drowsiness, phone use, seat belt, hard braking, about forty kinds. A manager writes what should happen, in English. The platform messages, waits, re-checks, calls the truck, and writes down the follow-ups. Here is what I'd tell someone starting the same build.

The model writes a document. Code runs it.

My first instinct was to let the model read the sentence and act. That's a chatbot with side effects, and nobody is going to let one ring their drivers. So the LLM's only job became translating the sentence into a rule document with six keys, and a plain engine executes that:

{
"trigger": { "event_types": ["fatigueWarn"] },
"where": { "field": "speed_kmph", "op": "gt", "value": 60 },
"aggregate": { "window": { "type": "sliding", "size": "PT10M" }, "group_by": ["vehicle_no"],
"function": { "name": "count" }, "having": { "op": "gte", "value": 2 } },
"actions": [ { "action": { "type": "notify", "channel": "telegram", "to": { "role": "manager" } } },
{ "action": { "type": "call", "to": { "selector": "driver" } }, "wait": "PT10M",
"recheck": { "event_types": ["fatigueWarn"], "same": ["vehicle_no"] } } ],
"throttle": { "cooldown": "PT1H", "dedup_key": "{vehicle_no}" }
}

That is "drowsy above 60 km/h twice in ten minutes, tell the manager; still drowsy ten minutes later, call the truck". The schema is a Pydantic v2 model, exported as JSON Schema and as TypeScript types, so the engine, the simulator, the UI's plain-words summary and the evals all point at the same thing. Freezing it early was the best decision in the project, and the one I almost skipped because it felt like premature ceremony.

Where the flow stops

The authoring agent is a LangGraph graph with two interrupts. If the sentence is missing a threshold or a channel, the graph stops and asks. It doesn't guess. Once a draft validates (schema plus about thirty semantic checks, written in code, with the errors fed back to the model for up to three repairs), it is replayed over the tenant's last seven days and comes back with a number: "would have fired 27 times, 58 held back by the throttle". That number is what turns "sounds right" into "yes". It has also caught a rule that would have fired four thousand times.

Then the second interrupt: a person approves, and approval only saves a draft. Activating is a separate button in a separate service behind a role check. The agent has no tool that can press it. The same shape carries into the calls: a call step is placed by the notifier, never by a model, uncertain calls wait in a queue for a manager, and every call ends with follow-ups a person closes. I stopped calling this "human in the loop" in meetings and started saying "this is where the state machine stops", which is what it actually is.

Two bugs I'd rather not have found in production

"At most one call per vehicle per hour" came out of the composer as a cap per rule, which in my schema means one call per hour for the entire fleet. The schema was fine. The problem was that the summary the manager approves said "at most one call per hour" and let both readings through. Now it says exactly what the schema means, because the summary is the thing people actually read.

The other one was latency. A yes-or-no decision inside a live phone call went through the full agent subprocess and took 18 to 30 seconds, which is most of the call. One direct Messages API call with a tight schema took 1.2 seconds. Heavy machinery where you need tools and memory; not inside a call.

What real trucks taught me

Tests and a simulated intercom get you to the door. On the other side: engine noise under every word the driver says, Hindi speech recognition that is fine on the agent's own lines and shaky on the driver's, and no way for the agent to see that the belt actually went on. So it asks yes-or-no questions, treats an unclear answer as "not confirmed", and re-checks the alert stream after the call instead of trusting the transcript. Budget more time for the microphone than for the model. I did not, and I would now.

The boring parts that let me sleep

Every table carries tenant_id and Postgres row-level security is forced, so a forgotten WHERE returns nothing rather than everything. The tenant comes from the JWT, never from a request body or the model, and the MCP server the agent reads through takes a per-tenant token where the token is the tenant; there is no argument for the model to fill with someone else's id.

Every alert gets a trace_id at ingest that rides in a Kafka header and is bound into every log line downstream; with OpenTelemetry on, it's the trace id too. Someone asks what happened to an alert, I paste one id into Grafana and see it accepted, persisted, matched and delivered across four services. Under load, 96,000 alerts went through with ingest p99 at 94 ms and nothing in the dead-letter queue, and when I killed the persist consumer on purpose an 86,704-message backlog drained in under five minutes with nothing lost. I wrote those numbers into the service contracts so they would be argued with, not remembered.

Stack

Python 3.12 and FastAPI in a uv workspace, Pydantic v2 for the contracts, Kafka through aiokafka (a compacted topic for rule updates, so an engine restart rebuilds its rules from Kafka alone), Postgres 16 with RLS and Alembic, Redis for dedup and cooldowns. LangGraph for the agents, Claude via the Agent SDK, a custom MCP server of curated read-only tools, Langfuse for traces, prompts and the eval datasets, with pytest golden suites failing the build. OpenTelemetry to Tempo, Prometheus rules with promtool tests, Grafana provisioned from code. React 18, Vite and TanStack on the front, Playwright end to end. Docker Compose for now; the rules compile to an IR so the engine can move to Flink without the rules changing.

If you've shipped natural-language rules or agents with a person in the path, I'd genuinely like to know where you put the switch. Mine is two buttons and a queue.

Much more to come next.

Top comments (3)

Collapse
 
anp2network profile image
ANP2 Network •

Your role check guards the button, and by your own stack description it sits beside the write path rather than in front of it. The engine rebuilds its ruleset from the compacted topic alone after a restart, so producer access to that topic is the actual authority over what runs. A backfill or a replay tool with producer credentials lands a rule in the engine without the button ever being pressed. Cheap close: have the engine reject a rule record whose provenance field does not name the activation service. Then the topic enforces what the button currently only implies.

The replay number has a similar gap. "Would have fired 27 times, 58 held back" is measured over the tenant's last seven days at drafting time. Activation is a separate act with no bound on how much later it lands. A rule drafted after a quiet week and activated after a route change carries the quiet week's base rate into a different traffic mix. The 27 that converted "sounds right" into "yes" is never re-asserted against what the rule will really see. Re-run the replay at activation, and refuse if the count leaves a stated band.

On the per-fleet cap bug, the instance is closed and the class is not. The summary is generated from the schema, so prose and schema are two renderings of one interpretation, and the next ambiguous key produces two agreeing renderings of the wrong reading. Replay output grouped by key is the one artifact that cannot hedge. "27 firings across 3 vehicles, at most 1 per vehicle per hour" makes a fleet cap arithmetically wrong on the screen instead of textually plausible.

If that topic has exactly one producer and activation follows approval quickly in practice, the first two are bookkeeping.

What else can produce to that compacted rule topic besides the activation service?

Collapse
 
sarit_chauhan profile image
Sarit Chauhan •

Fair on all three, and thanks for reading it that closely.

Some comments may only be visible to logged-in visitors. Sign in to view all comments.