DEV Community: Vedas Kudalkar

Cutting LLM Calls in Incident Triage: Sliding Windows, Log Reduction, and a pgvector Cache

Vedas Kudalkar — Sun, 01 Nov 2026 17:03:27 +0000

A Kafka-to-WebSocket pipeline where the model is the last resort, not the first step.

The problem

The obvious way to add an LLM to log monitoring is to send logs to a model and ask what is wrong. That breaks quickly. Most log lines are not errors. During an outage, the same error repeats thousands of times with only timestamps, UUIDs and counters changing. A raw stack trace with framework frames is mostly noise. And the same database timeout will show up on orders today and inventory tomorrow, so you pay for the same diagnosis again.

I built telemetry-incident-dashboard around one rule: every step before the model must be cheaper than the model, and must be able to stop the pipeline.

It is a working prototype. I'll say clearly later what has and hasn't been exercised.

Architecture

flowchart TD
    A[Apps / log shipper] --> K[Kafka topic app-logs]
    K --> C[Spring Boot consumer<br/>virtual threads]
    C -->|every log| W[STOMP /topic/logs]
    C -->|ERROR only| D[Per-service sliding window<br/>10+ errors in 60s]
    D -->|below threshold| X[Stop: no tokens]
    D -->|burst| R[LogPayloadReducer]
    R --> E{Exact fingerprint<br/>in cache?}
    E -->|yes| H[CACHE_EXACT]
    E -->|no| V{pgvector cosine<br/>>= 0.90?}
    V -->|yes| S[CACHE_SEMANTIC]
    V -->|no| L[LLM call via LangChain4j]
    L --> P[(PostgreSQL + pgvector<br/>diagnosis_cache)]
    H --> I[(incidents hypertable)]
    S --> I
    L --> I
    I --> A2[STOMP /topic/alerts]

Stack: Java 21, Spring Boot 3.3, Apache Kafka (KRaft), PostgreSQL with TimescaleDB and pgvector, LangChain4j with OpenAI (gpt-4o-mini and text-embedding-3-small by default), and a React + Vite dashboard.

Step 1: detect bursts without a model

Each service gets its own ErrorWindow. It holds the errors from the last 60 seconds, fires when the count reaches the threshold (default 10), and then stays quiet for a cooldown (default 120 seconds).

synchronized List<LogEvent> record(LogEvent event, long nowMs, long windowMs, int threshold,
                                   long cooldownMs, int maxBuffered) {
    errors.addLast(new Stamped(nowMs, event));
    while (!errors.isEmpty() && nowMs - errors.peekFirst().arrivalMs() > windowMs) {
        errors.pollFirst();
    }
    while (errors.size() > maxBuffered) {
        errors.pollFirst();
    }
    if (errors.size() < threshold || nowMs - lastAlertMs < cooldownMs) {
        return List.of();
    }
    lastAlertMs = nowMs;
    return errors.stream().map(Stamped::event).toList();
}

Info, debug and warning logs never get here. They are counted and streamed to the dashboard, capped at 200 non-error events per second. A malformed Kafka record is counted and dropped. The listener never rethrows, so one bad record can't stall a partition.

Visual instruction: Capture this exact code section in VS Code using the OneDark-Pro theme. Use a readable font size and crop unnecessary editor UI.

The cooldown matters. Without it, a long outage would trigger analysis on every incoming error once the window is full.

Step 2: shrink the burst

LogPayloadReducer turns a burst into a short context string:

strips ANSI codes, ISO timestamps, UUIDs and hex addresses
drops framework frames (java.base/, jdk.internal., Tomcat, Spring AOP and CGLIB, JUnit)
keeps the top 12 stack lines and notes how many were cut
groups identical errors by fingerprint and prints them as xN
caps the whole thing at 2,000 characters and 5 distinct errors

The fingerprint is a SHA-256 of the text after volatile parts are normalised. Numbers become <n>, so Timeout after 3001ms and Timeout after 9999ms hash the same:

private static String normalize(String s) {
    return NUMBER.matcher(scrub(s)).replaceAll("<n>");
}

For the 15-error connection-pool burst in sample-data/02-db-timeout-burst.ndjson, the README reports the model input going from 8,415 characters (about 2,100 tokens) to 506 characters (about 127 tokens). That is one synthetic burst and the token count is a four-characters-per-token estimate, so treat 94% as an illustration, not a benchmark.

The reduced output looks like this:

service=orders errors=15 distinct=1
[1] x15 ERROR HikariPool-1 - Connection is not available, request timed out after 30001ms
java.sql.SQLTransientConnectionException: HikariPool-1 - Connection is not available, request timed out after 30001ms
    at com.zaxxer.hikari.pool.HikariPool.createTimeoutException(HikariPool.java:696)
    at com.zaxxer.hikari.pool.HikariPool.getConnection(HikariPool.java:197)
    at com.shop.orders.Repository.save(Repository.java:48)
    at com.shop.orders.Service.handle(Service.java:71)

Step 3: a two-level cache

LogAnalysisEngine.resolve goes cheapest first.

Exact match. One UPDATE ... RETURNING both bumps the hit counter and fetches the stored diagnosis:

List<String> exact = jdbc.queryForList(
        "UPDATE diagnosis_cache SET hits = hits + 1 WHERE fingerprint = ? RETURNING diagnosis",
        String.class, fingerprint);

Semantic match. If there's no exact hit, embed the context and ask pgvector for the nearest row. A cosine similarity of 0.90 or more counts as a hit. This is what lets a database timeout on inventory reuse the diagnosis made for orders:

SELECT id, diagnosis, 1 - (embedding <=> ?::vector) AS similarity
FROM diagnosis_cache
WHERE embedding IS NOT NULL
ORDER BY embedding <=> ?::vector
LIMIT 1

The table uses an HNSW index with vector_cosine_ops. The schema is 1536 dimensions to match text-embedding-3-small. If you change the embedding model, you change docker/init.sql and EMBEDDING_DIMENSIONS together. The engine checks the returned vector length and skips the vector cache on a mismatch instead of failing.

LLM call. Only on a miss. The prompt asks for under 150 words in three headings (root cause with confidence, impact, fix) and says not to invent details. The answer and its embedding are written back, so the next occurrence is free.

Visual instruction: Capture this exact code section in VS Code using the OneDark-Pro theme. Use a readable font size and crop unnecessary editor UI.

Engineering decisions

Single-flight for identical bursts. Two services can hit the same failure at once. diagnoseOnce keeps a ConcurrentMap<String, CompletableFuture<Diagnosis>> keyed by fingerprint. The first caller does the work; the rest join() its future. That avoids paying for duplicate LLM calls in the gap before the first answer lands in the cache.

Never throw from the pipeline. If the cache query, the embedding call, the database or the LLM fails, the incident is still recorded with source FALLBACK and the raw condensed telemetry attached. A flaky dependency degrades the diagnosis, not the stream.

Virtual threads. Each Kafka consumer loop and each analysis task runs on a Java 21 virtual thread. Analysis is blocking I/O (HTTP to a model, JDBC), which is the case virtual threads fit. The analysis is also submitted off the consumer thread so a slow model call can't hold up log ingestion.

Timescale for history. Cached diagnoses live in diagnosis_cache. Every incident, including cache hits, goes into an incidents hypertable with its source, similarity and tokens used, so you can see how often the model is actually needed.

Testing

There are two unit test classes, five tests:

ErrorWindowTest: fires at the threshold, respects cooldown, and expires old errors from the window.
LogPayloadReducerTest: truncates stacks to 12 lines plus a marker, collapses 200 repeated errors to x200 under the 2,000-character budget, and gives two events with different numbers and UUIDs the same fingerprint.

Nine NDJSON scenarios in sample-data/ cover normal traffic, bursts (DB timeout, NPE, OOM, upstream 502, JWT failures), a similar failure on a second service, malformed lines, and a below-threshold case.

Limitations

I'd rather list these than have you find them:

The successful LLM path (real diagnosis written to the cache and then served as a hit) has not been run against a funded OpenAI account. The fallback path, reducer, window logic and the dashboard demo have been run.
The dashboard's live mode hasn't been tested against the backend.
Kafka offsets auto-commit, so a crash mid-analysis can lose that burst.
No auth on the WebSocket endpoint. No cache eviction. Single Kafka broker.
No throughput or latency benchmarks.

Try it

cp .env.example .env            # optionally set OPENAI_API_KEY
docker compose up -d            # Kafka (KRaft) + TimescaleDB with pgvector
set -a; source .env; set +a
mvn spring-boot:run             # http://localhost:8080, STOMP at /ws

cd dashboard && npm install && npm run dev   # http://localhost:5173

scripts/send-sample-logs.sh sample-data/02-db-timeout-burst.ndjson

Without an API key the app still starts, and bursts are stored as FALLBACK. There is also a browser-only demo that simulates the same rules with hand-written diagnoses and is labeled as a simulation. Click "Play the story" to see an LLM call, a below-threshold burst, a similar-error hit and an exact hit.

What's next

Not implemented yet, but the obvious gaps: manual offset commits after analysis, cache eviction, authentication on the WebSocket, and a real run of the LLM path with measured cache hit rates.

Code is on GitHub: axololtos/telemetry-incident-dashboard

If it's useful, a star helps. Issues and PRs are welcome (there's a CONTRIBUTING guide). Follow me here on DEV for more write-ups like this.

Ofrenda: a keeper for stories that runs entirely on your laptop

Vedas Kudalkar — Sat, 31 Oct 2026 06:56:16 +0000

This is a submission for the Hacktoberfest Open-Source AI Challenge Week 4: Ghost in the Machine

What I built

Ofrenda is a command-line tool. You give it a story about someone, and it keeps it. Later you ask a question and it answers from what you told it, and from nothing else.

$ python ofrenda.py remember "Abuela" "She burned the first tortilla on purpose, for the house. She sang boleros while washing dishes, always the same one, Sabor a Mi. On Sundays she made caldo de res and nobody was allowed to start eating before she sat down."
Lit 4 candle(s) for Abuela.

$ python ofrenda.py ask "Abuela" "What did she cook on Sundays?"
On Sundays, she made caldo de res.

$ python ofrenda.py ask "Abuela" "Where was she born?"
The ofrenda does not hold that memory yet. Would you like to add it?

That last answer is the one I care about. The story never said where she was born, so the tool says it doesn't know. It doesn't guess.

How I read the theme

An ofrenda is the altar families build for the dead: photos, marigolds, a favorite dish, a song. The point is that someone is remembered in specific detail. A ghost in a machine sounded to me like the opposite risk. A chatbot that speaks as your grandmother is eerie, and it will say things she never said.

So the assistant here is called the keeper, and the rule is written into its system prompt: it reads memories aloud, it never speaks as the person, and if the memories don't cover the question it says so.

How it works

A story goes through a short pipeline:

Gemma splits the story into separate memories, returned as JSON.
Each memory is embedded and stored in a local SQLite file.
When you ask something, the question is embedded, the closest memories for that person are retrieved, and Gemma answers from those only.

One story above became four stored memories:

- She burned the first tortilla on purpose for the house.            ["cooking"]
- She sang boleros while washing dishes, always singing Sabor a Mi.  ["song"]
- On Sundays, she made caldo de res.                                 ["food"]
- Nobody was allowed to start eating before she sat down.            ["ritual"]

Search is a plain cosine-similarity scan filtered by person. That is enough for a family's worth of stories, and it means one person's memories can't leak into another's answers.

The three pieces I used

Gemma, through Ollama. Both the splitting and the answering run on a local Gemma model. For my demo run I used gemma4:e2b, and the default in the repo is gemma3:4b. Any tag Ollama can serve works. Embeddings come from nomic-embed-text.

Temporal. Saving a story is two steps that depend on a local model, and local models are slow to load and sometimes fail. I wrapped the steps in a Temporal workflow with a retry policy. On my first real run, Ollama returned an HTTP 500 while the model was still loading. Temporal retried the step, and the command finished with all four memories saved. Without the workflow I would have lost the story and seen a stack trace. The activity also falls back to storing the original text as one memory if Gemma returns invalid JSON, so a bad parse can't drop a story.

Sentry Agent Tracing. Every ask is traced with Sentry's AI span names: an agent span for the keeper, a tool span for the memory lookup, and chat and embedding spans with model name and token counts. Tracing only turns on if you set a SENTRY_DSN. One decision here mattered to me: prompts and answers are not attached to traces unless you set OFRENDA_TRACE_CONTENT=1. Family stories shouldn't end up in a monitoring dashboard by accident.

Why open models matter for this

Stories about someone who died are about as private as data gets. With this setup the model, the embeddings and the database all live on your machine. It works with the internet off. The memories are one .db file you own, so there's no service that can change its pricing or shut down and take them with it. I couldn't build this on a hosted API and feel good about it, and that is the real reason I used open models, not cost.

What it doesn't do

It's text only, with no audio recording or transcription. The database isn't encrypted. A small local model can still misread a story, so it's worth checking that what was stored matches what you meant. Search is a linear scan, which is fine for a family and wrong for a million memories.

Demo

Live demo (recorded session): axololtos/ofrenda_demo

Ofrenda runs on a local model through Ollama, so it can't be hosted as a live app without sending people's stories to a server, which is the thing I built it to avoid. That page is a preview: a replay of a real run from my laptop, with the same commands and the same output. To use the real thing, run it locally with the steps below.

Try it

Code and setup steps are here: github.com/axololtos/ofrenda

You need Python, Ollama and the Temporal dev server. The README has the exact commands.

Prize categories: Best Use of Gemma, Best Use of Temporal, Best Use of Sentry Agent Tracing.