DEV Community: Abinivesh M

Your On-Call Agent Needs Hindsight, Not a Bigger Model

Abinivesh M — Thu, 05 Nov 2026 15:34:56 +0000

Description: An on-call SRE agent is only as good as the incidents it can recall. Structured retention, evidence-backed ranking, and a contradiction guard built on Hindsight.
It is 02:14. Checkout p99 has gone from 400 ms to 3.9 seconds, the payment provider is retrying, and an alert is burning through the error budget.

The engineer who fixed this exact failure six weeks ago is asleep. The migration that solved it was applied by hand and never written down. So the person on call searches, guesses, and restarts the pods.

The restart works for forty minutes.

I kept coming back to that gap. The model is not the bottleneck. A good model has read every runbook and every public postmortem. What it does not have is your history: this failure, on this service, with these signals, fixed in six minutes by a specific person, and whether the fix held.

I built Déjà Vu to test that claim seriously — an on-call SRE agent whose entire value comes from remembering incidents, and whose memory layer is Hindsight.

What the system actually does

Déjà Vu watches an alert queue. When one fires, it builds a query from the service, title, body and signal fingerprint, recalls candidates from a Hindsight bank called dejavu-oncall, re-ranks them, and answers with evidence. One bank per team: your incidents only, never another team's.

The answer is not a paragraph. It is a hypothesis plus receipts:

INC-1006 — missing composite index on orders(created_at, status) — 6 minutes — Priya Raman — resolved
INC-1001 — same cause class — resolved
INC-1002 — same class — resolved

Same alert, memory off: "generic triage, restart the affected pods", 31% confidence, no incident, no owner, no fix time. Memory on: the index migration, 97% confidence, the receipts, a named engineer. The second answer is not smarter. It is informed.

Making memory the actual product

The naive version of this is a vector store full of postmortems with a prompt that says "use the context below." It demos well and fails quietly, for three reasons.

It retrieves paragraphs, not incidents — so nothing is auditable. It treats similarity as identity — so a look-alike incident with a different cause gets recommended with full confidence. And it never learns from what happened after the fix.

So the memory is structured, not chunked. One document per incident, with metadata that survives retrieval:

resp = client().retain(
    bank_id="dejavu-oncall",
    content=text,                       # symptom, signals, cause, fix, time, outcome
    document_id=incident_id,
    context=f"{inc.get('service')} {inc.get('severity')} incident on {ts[:10]}",
    metadata={
        "incident_id": incident_id,
        "service": inc.get("service"),
        "root_cause": inc.get("root_cause"),
        "fix": inc.get("fix"),
        "outcome": inc.get("outcome"),
        "fix_minutes": str(inc.get("fix_minutes")),
        "engineer": inc.get("engineer"),
        "severity": inc.get("severity"),
    },
    tags=[inc["service"], inc["severity"], inc["outcome"]],
)

That metadata is the difference between a receipt and a rumour. When the agent says "six minutes, Priya Raman, resolved," those values came from memory, not the model's imagination.

Recall is deliberate: the alert's fingerprint becomes the query, Hindsight returns candidates by meaning, and the application re-ranks them — retrieval similarity and operational similarity are not the same thing:

similarity = 0.50 * signal_similarity + 0.45 * text_similarity + 0.05 * coverage
score = 0.82 * similarity
score += 0.10 * min(1.0, support / 3)      # how many incidents back this cause
score += 0.08 * success_rate               # did the fix actually hold
score += 0.02 * recency                    # recent precedent wins ties
if stale:
    score -= 0.06                          # flag it, do not trust it

Fifty percent of the decision is the signal fingerprint — the numbers, not the prose. p99_ms 3900 against 3800. pool_wait_ms 1240 against 1180. cert_age_days 2 against 61. That last one is why a certificate incident gets ruled out even when the text looks identical.

The part that matters: the contradiction guard

Two incidents can share every word and need opposite fixes. Incident INC-1009 has the same service, the same pool timeout, the same graph shape, and 53% similarity to the live alert. But its cause was a pool size capped during a config rollout, and its fix was a config change plus a redeploy — not an index.

Before committing, Déjà Vu checks the runner-up failure class. If a different cause is close, it surfaces the look-alike explicitly: the incident ID, the similarity, the one signal that separates them, and the different fix it required.

This is the failure mode of every naive retrieval system. A text search returns the nearest paragraph. The agent has to tell you what it ruled out, and why.

Measurement, not a claim

Every memory product says it improves. Almost none measure it, because honest measurement is uncomfortable.

The repo includes a replay that walks the incident corpus in chronological order. At each incident, the agent may only use memories that existed before that timestamp. No peeking. No LLM in the grading loop.

Across 22 graded incidents: 50% top-1, 64% top-3. First six graded alerts: 33%. Last six: 83%.

That pair is the thesis. The agent is measurably better at the end of the quarter than at the start, because it lived through the quarter. The Learning screen plots every graded incident, including the ones it got wrong. A memory system that only shows its wins is a marketing surface.

The loop has to close

Resolving an alert is not the end of the interaction. It is the write. When an engineer resolves an incident, Déjà Vu stores a postmortem, retains it with source: live, and marks the alert closed. Fire the same fingerprint again and the first receipt is the incident that was just created, answered in milliseconds.

That closed loop is also where a second agent earns its place. In the Band room, DejaVu-Recall posts the hypothesis and receipts, then DejaVu-Critic answers the recall — not the alert — with four deterministic checks: at least two receipts, no stale precedent, precedent fixes that held, confidence above threshold. It can block the fix from ever reaching a human.

I wrote the critic as code, not a model call: a reviewer should check evidence, not generate more of it.

What I got wrong

Three things, in order of embarrassment.

First, I scoped recall to the same service. That hid the lesson: a missing index on payments-api is the same failure as one on checkout-api. Cross-service transfer drove most of the late-corpus accuracy.

Second, I graded at string level before switching to failure classes. "Missing index on orders" and "missing index on settlement_date" are different strings and the same lesson. String-level grading made the agent look worse than it was.

Third, I trusted retrieval scores for too long. The signal fingerprint is weighted above text on purpose now. The text is how you find candidates; the numbers are how you decide.

Where it still breaks

First-time failures cannot be recalled. If the team has never seen it, memory has nothing, and the Learning screen shows exactly those incidents as misses.

Per-service histories hold eight to ten incidents, so cross-service transfer helps but is not evidence of production-scale performance. The corpus is synthetic — real incident data is confidential — and the weights were tuned on it. A production deployment needs re-tuning against real signals and a plan for the long tail of incidents that happen once.

And memory can be wrong. Precedent is not proof. That is why every answer carries confidence, receipts, and a visible trace of which path served it. When Hindsight is unreachable, the app serves from a BM25 index over the same SQLite mirror and the header badge changes from HINDSIGHT to BM25 FALLBACK. A demo should not die because a service did not start, but it should never pretend which path answered.

What I took away

The next improvement in on-call tooling is not a bigger context window. It is a memory that is specific to your systems, weighted by what actually worked, honest about what it does not know, and measured against its own past.

Hindsight made the memory layer the easy part: a bank, a retain with real metadata, a recall that returns meaning. The hard part was deciding what counts as the same failure — and being willing to show the moment the agent gets it wrong.

At 2 a.m., the question is not "what does a good SRE do?" It is "what did we do last time, did it work, and who can I ask?"

If you are building agents with memory, start with Hindsight's open-source memory engine and read the Hindsight documentation. It is also worth getting clear on what agent memory actually means before you design a schema you will regret.

Running agent: dejavu-tau.vercel.app. Source, including the replay harness: github.com/abinivesh-m/dejavu.

"RoadSOS AI: An AI-Powered Intelligent Road Safety and Emergency Response System"

Abinivesh M — Sat, 15 Aug 2026 18:38:02 +0000

RoadSOS AI: An AI-Powered Intelligent Road Safety and Emergency Response System

Technical project documentation — SmartAIthon 2026

This write-up is the deeper technical companion to our SmartAIthon presentation deck. Where the deck says "we detect drowsiness and phone usage," this document explains how — models, pipeline, data, performance, and the parts that are still in progress.

1. Overview

RoadSOS AI is a laptop-based prototype that uses computer vision to monitor a driver in real time, catch dangerous behaviour before it turns into a crash, and — when an accident does happen — kick off an emergency response workflow.

Most road-safety tools focus on a single moment: either they warn you before something goes wrong, or they react after a crash. RoadSOS tries to cover both ends in one pipeline: Predict → Prevent → Protect.

One-line honest description: "RoadSOS AI is currently a laptop-based AI road safety prototype that uses computer vision to detect driver distraction and drowsiness, calculates real-time driving risk, and demonstrates an emergency SOS workflow for accident situations."

2. Problem Statement

Road accidents are commonly linked to:

Driver drowsiness and fatigue
Mobile phone usage while driving
General driver distraction
Delayed emergency response after a crash has already happened

Most existing tools address only one of these stages in isolation — a drowsiness alarm, or a crash-detection black box, or a manual SOS button. There's a gap for a system that monitors, warns, assesses risk holistically, and supports the emergency response step.

3. System Architecture

Everything currently runs on a single laptop with a connected webcam — there is no external hardware in the loop yet (see §16, Limitations).

4. Mobile Phone Usage Detection

Approach: YOLOv8 (nano variant, for real-time inference on CPU) run on each captured frame to determine whether the driver is holding or using a phone.

What happens on detection:

A visual warning is shown on the dashboard
A voice alert is triggered ("Mobile usage detected. Focus on the road.")
The event is logged
The driver's risk score increases

5. Drowsiness Detection

Approach: MediaPipe Face Mesh (468-point facial landmark model) tracks the driver's eyes, and the system computes the Eye Aspect Ratio (EAR) to determine whether the eyes have been closed for an abnormal duration.

MediaPipe solution used: Face Mesh / Face Landmarker
EAR formula: standard six-point eye landmark ratio — vertical eye distances over horizontal eye distance
EAR threshold used to flag drowsiness: 0.25
Consecutive-frame requirement: 20 consecutive frames below threshold (roughly 0.8 seconds at ~25 FPS), to avoid false positives from a single blink
Inference speed / FPS: ~18–22 FPS on a standard laptop webcam feed

What happens on detection:

A warning is displayed on the dashboard
A voice alert is triggered ("Drowsiness detected. Please take a break.")
The event is logged
The driver's risk score increases

6. Real-Time Risk Engine

Rather than treating each unsafe event independently, RoadSOS combines them into a single real-time risk score. The intuition: a driver using a phone while also showing signs of drowsiness is more dangerous than either signal alone.

Current inputs to the risk score:

Phone-usage alert status (on/off, from YOLO)
Drowsiness alert status (on/off, from EAR/MediaPipe)
Simulated vehicle speed (keyboard-controlled in this prototype)

Scoring logic: A weighted-sum, rule-based model. Each active signal contributes a fixed weight to a 0–100 risk score:

Phone usage detected: +40
Drowsiness detected: +40
Speed factor (scaled from simulated speed, capped): up to +20

Risk bands: 0–30 = Low, 31–60 = Medium, 61–100 = High. Crossing into Medium/High triggers stronger dashboard highlighting and more frequent voice alerts.

The resulting score is displayed live on the dashboard and drives downstream alerting.

7. Speed Simulation

Since the current prototype runs entirely on a laptop with no vehicle interface, speed is simulated via keyboard input:

W — increases simulated speed
S — decreases simulated speed

This value feeds into the Risk Engine exactly as a real speed sensor reading would, which makes it straightforward to swap in a real speed source later (e.g. OBD-II, GPS-derived speed, or an accelerometer) without re-architecting the risk logic.

8. Accident Simulation and Emergency (SOS) Workflow

The full emergency response chain is implemented and demonstrable end-to-end, with the accident trigger itself simulated for the laptop demo:

X — simulates an accident event
On trigger, the system starts an SOS countdown
C — cancels the emergency during the countdown window
If not cancelled in time, the SOS workflow fires

On SOS trigger, the system:

Updates the dashboard to reflect accident status
Logs the accident event
Invokes the SOS workflow
Attempts to use the GPS/emergency modules
Announces (via voice) that emergency contacts have been notified

Architecture modules already in code:

GPSManager
SOSManager
AccidentManager

Intended full workflow: Accident detected → SOS countdown → cancellation opportunity → GPS location obtained → SOS sent to emergency contacts.

Honesty note for judges: in the current laptop demo, the accident event itself is simulated via keypress, not derived from real crash-sensor data (e.g. an IMU/accelerometer threshold). The GPS/SOS software architecture exists and runs, but real-world GPS/GSM hardware integration is future work (§16, §15).

9. Dashboard

A real-time OpenCV-based dashboard acts as the central monitoring interface, showing:

Live camera feed
FPS
Simulated speed
Driver status
Phone-use detection status
Drowsiness info + EAR value
Accident confidence
Risk score
Emergency countdown
SOS-sent status
Real-time safety messages

10. Voice Alert System

Alerts are delivered asynchronously so the driver isn't required to keep looking at the screen. Example alerts:

"Mobile usage detected. Focus on the road."
"Drowsiness detected. Please take a break."
"Accident detected. Sending emergency SOS."
"Emergency contacts have been notified."

TTS engine used: pyttsx3 (offline text-to-speech — chosen for low latency and no internet dependency, important for a safety-critical alert path).

11. Event Logging & Analytics

The system logs safety-relevant events for later analysis:

Phone detections
Drowsiness detections
Accident events
Overspeed events

Storage format: CSV logs (timestamp, event type, confidence score, risk score at time of event).

This logging layer is the foundation for planned features like trip history, driver behaviour reports, safety analytics, and (eventually) fleet-level monitoring.

12. Technology Stack

Layer	Technology
Object detection	YOLOv8 (nano)
Facial landmarks / drowsiness	MediaPipe Face Mesh
Computer vision / dashboard	OpenCV
Language	Python
Voice alerts	pyttsx3
System modules	Risk Engine, Accident Manager, SOS Manager, GPS Manager, Audio Manager, Analytics Manager, Event Logger
Interface	Custom OpenCV dashboard
Hardware (current)	Laptop + USB webcam
Hardware (planned)	Raspberry Pi 5, ESP32, MPU6050, GPS module, GSM/SMS module

13. Current Implementation Status

✅ Built and working:

Live camera processing
YOLO-based phone detection
MediaPipe-based drowsiness detection (EAR monitoring)
Real-time risk scoring
Voice alerts
Interactive OpenCV dashboard
Simulated speed input
Simulated accident trigger with SOS countdown and cancellation
Accident confidence display
Event logging and analytics counters
GPS/SOS software architecture (modules exist and run in simulation)

❌ Not yet completed:

Fully integrated, automatic hardware-based accident detection (no IMU/accelerometer trigger yet)
Real vehicle speed input (currently keyboard-simulated)
Fully validated GPS location integration in the field
Reliable GSM/SMS hardware integration
Real-world crash-data testing/validation
Mobile application
Fleet dashboard
Personalized AI risk model
Automatic near-miss detection

14. Cost Analysis

Current prototype (software-only, laptop-based):

YOLO, MediaPipe, OpenCV — open-source, ₹0 licensing cost
Runs on an existing laptop + standard USB webcam, no special hardware

Estimated future embedded hardware (per unit):

Component	Estimated Cost (₹)
ESP32 microcontroller	300–500
MPU6050 accelerometer/gyroscope	150–250
GPS module (e.g. NEO-6M)	500–700
GSM/SMS module (e.g. SIM800L)	600–900
Sensor subtotal	~1,500–2,500
Raspberry Pi 5 (8GB)	8,500–9,500
Active cooler + Pi Camera v3	2,800–3,500
MicroSD (32GB A2) + power adapter	1,200–1,600
Pi 5 subtotal	~11,000–14,500
Estimated total per unit	~13,000–17,500

These are component-price estimates for a proposed embedded version, not figures from a shipped product — further cost optimisation and real-world benchmarking are needed before any commercial deployment claim.

15. Roadmap

Phase 1 — Hardware Integration

ESP32 + MPU6050 for automatic crash detection
Real GPS location tracking
GSM/SMS-based emergency alerts
Real vehicle speed input (replacing keyboard simulation)

Phase 2 — Intelligence

Near-miss detection in addition to accident detection
Personalized, AI-driven driver risk modelling over time

Phase 3 — Scale

Companion mobile application for drivers and emergency contacts
Fleet management dashboard for organisations
Real-world testing and validation with crash data

16. Limitations (In Our Own Words)

We'd rather be upfront about this than have it discovered during Q&A:

The accident trigger in the current demo is a keypress, not a sensor reading. It demonstrates the workflow, not automatic crash detection.
Vehicle speed is keyboard-simulated, not read from a real vehicle or GPS.
GPS/GSM modules exist in code (GPSManager, class structure ready) but are not yet wired to working hardware and tested in the field.
No real-world crash dataset has been used to validate accident-confidence logic.
Detection accuracy numbers (mAP, FPS, false-positive rate) are not yet benchmarked and published — we're reporting the architecture honestly rather than citing numbers we haven't measured.

17. Conclusion

RoadSOS AI demonstrates how computer vision and a simple risk-scoring approach can move road safety earlier in the timeline — from "respond after a crash" to "warn before it happens" — while still keeping a working emergency-response path for when prevention isn't enough. The current MVP proves out real-time driver monitoring, phone and drowsiness detection, risk assessment, preventive alerts, and an SOS workflow, all as a laptop-based prototype. The next milestone is moving that logic onto embedded hardware and validating it against real-world data.

RoadSOS AI — Predict. Prevent. Protect.

Written for SmartAIthon 2026, Round 2 submission. This document is intended to accompany, not duplicate, the presentation deck — it goes into implementation depth for judges evaluating technical merit.