Cutting LLM Calls in Incident Triage: Sliding Windows, Log Reduction, and a pgvector Cache

A Kafka-to-WebSocket pipeline where the model is the last resort, not the first step.

The problem

The obvious way to add an LLM to log monitoring is to send logs to a model and ask what is wrong. That breaks quickly. Most log lines are not errors. During an outage, the same error repeats thousands of times with only timestamps, UUIDs and counters changing. A raw stack trace with framework frames is mostly noise. And the same database timeout will show up on orders today and inventory tomorrow, so you pay for the same diagnosis again.

I built telemetry-incident-dashboard around one rule: every step before the model must be cheaper than the model, and must be able to stop the pipeline.

It is a working prototype. I'll say clearly later what has and hasn't been exercised.

Architecture

flowchart TD
    A[Apps / log shipper] --> K[Kafka topic app-logs]
    K --> C[Spring Boot consumer<br/>virtual threads]
    C -->|every log| W[STOMP /topic/logs]
    C -->|ERROR only| D[Per-service sliding window<br/>10+ errors in 60s]
    D -->|below threshold| X[Stop: no tokens]
    D -->|burst| R[LogPayloadReducer]
    R --> E{Exact fingerprint<br/>in cache?}
    E -->|yes| H[CACHE_EXACT]
    E -->|no| V{pgvector cosine<br/>>= 0.90?}
    V -->|yes| S[CACHE_SEMANTIC]
    V -->|no| L[LLM call via LangChain4j]
    L --> P[(PostgreSQL + pgvector<br/>diagnosis_cache)]
    H --> I[(incidents hypertable)]
    S --> I
    L --> I
    I --> A2[STOMP /topic/alerts]

Stack: Java 21, Spring Boot 3.3, Apache Kafka (KRaft), PostgreSQL with TimescaleDB and pgvector, LangChain4j with OpenAI (gpt-4o-mini and text-embedding-3-small by default), and a React + Vite dashboard.

Step 1: detect bursts without a model

Each service gets its own ErrorWindow. It holds the errors from the last 60 seconds, fires when the count reaches the threshold (default 10), and then stays quiet for a cooldown (default 120 seconds).

synchronized List<LogEvent> record(LogEvent event, long nowMs, long windowMs, int threshold,
                                   long cooldownMs, int maxBuffered) {
    errors.addLast(new Stamped(nowMs, event));
    while (!errors.isEmpty() && nowMs - errors.peekFirst().arrivalMs() > windowMs) {
        errors.pollFirst();
    }
    while (errors.size() > maxBuffered) {
        errors.pollFirst();
    }
    if (errors.size() < threshold || nowMs - lastAlertMs < cooldownMs) {
        return List.of();
    }
    lastAlertMs = nowMs;
    return errors.stream().map(Stamped::event).toList();
}

Info, debug and warning logs never get here. They are counted and streamed to the dashboard, capped at 200 non-error events per second. A malformed Kafka record is counted and dropped. The listener never rethrows, so one bad record can't stall a partition.

Visual instruction: Capture this exact code section in VS Code using the OneDark-Pro theme. Use a readable font size and crop unnecessary editor UI.

The cooldown matters. Without it, a long outage would trigger analysis on every incoming error once the window is full.

Step 2: shrink the burst

LogPayloadReducer turns a burst into a short context string:

• strips ANSI codes, ISO timestamps, UUIDs and hex addresses
• drops framework frames (java.base/, jdk.internal., Tomcat, Spring AOP and CGLIB, JUnit)
• keeps the top 12 stack lines and notes how many were cut
• groups identical errors by fingerprint and prints them as xN
• caps the whole thing at 2,000 characters and 5 distinct errors

The fingerprint is a SHA-256 of the text after volatile parts are normalised. Numbers become <n>, so Timeout after 3001ms and Timeout after 9999ms hash the same:

private static String normalize(String s) {
    return NUMBER.matcher(scrub(s)).replaceAll("<n>");
}

For the 15-error connection-pool burst in sample-data/02-db-timeout-burst.ndjson, the README reports the model input going from 8,415 characters (about 2,100 tokens) to 506 characters (about 127 tokens). That is one synthetic burst and the token count is a four-characters-per-token estimate, so treat 94% as an illustration, not a benchmark.

The reduced output looks like this:

service=orders errors=15 distinct=1
[1] x15 ERROR HikariPool-1 - Connection is not available, request timed out after 30001ms
java.sql.SQLTransientConnectionException: HikariPool-1 - Connection is not available, request timed out after 30001ms
    at com.zaxxer.hikari.pool.HikariPool.createTimeoutException(HikariPool.java:696)
    at com.zaxxer.hikari.pool.HikariPool.getConnection(HikariPool.java:197)
    at com.shop.orders.Repository.save(Repository.java:48)
    at com.shop.orders.Service.handle(Service.java:71)

Step 3: a two-level cache

LogAnalysisEngine.resolve goes cheapest first.

Exact match. One UPDATE ... RETURNING both bumps the hit counter and fetches the stored diagnosis:

List<String> exact = jdbc.queryForList(
        "UPDATE diagnosis_cache SET hits = hits + 1 WHERE fingerprint = ? RETURNING diagnosis",
        String.class, fingerprint);

Semantic match. If there's no exact hit, embed the context and ask pgvector for the nearest row. A cosine similarity of 0.90 or more counts as a hit. This is what lets a database timeout on inventory reuse the diagnosis made for orders:

SELECT id, diagnosis, 1 - (embedding <=> ?::vector) AS similarity
FROM diagnosis_cache
WHERE embedding IS NOT NULL
ORDER BY embedding <=> ?::vector
LIMIT 1

The table uses an HNSW index with vector_cosine_ops. The schema is 1536 dimensions to match text-embedding-3-small. If you change the embedding model, you change docker/init.sql and EMBEDDING_DIMENSIONS together. The engine checks the returned vector length and skips the vector cache on a mismatch instead of failing.

LLM call. Only on a miss. The prompt asks for under 150 words in three headings (root cause with confidence, impact, fix) and says not to invent details. The answer and its embedding are written back, so the next occurrence is free.

Visual instruction: Capture this exact code section in VS Code using the OneDark-Pro theme. Use a readable font size and crop unnecessary editor UI.

Engineering decisions

Single-flight for identical bursts. Two services can hit the same failure at once. diagnoseOnce keeps a ConcurrentMap<String, CompletableFuture<Diagnosis>> keyed by fingerprint. The first caller does the work; the rest join() its future. That avoids paying for duplicate LLM calls in the gap before the first answer lands in the cache.

Never throw from the pipeline. If the cache query, the embedding call, the database or the LLM fails, the incident is still recorded with source FALLBACK and the raw condensed telemetry attached. A flaky dependency degrades the diagnosis, not the stream.

Virtual threads. Each Kafka consumer loop and each analysis task runs on a Java 21 virtual thread. Analysis is blocking I/O (HTTP to a model, JDBC), which is the case virtual threads fit. The analysis is also submitted off the consumer thread so a slow model call can't hold up log ingestion.

Timescale for history. Cached diagnoses live in diagnosis_cache. Every incident, including cache hits, goes into an incidents hypertable with its source, similarity and tokens used, so you can see how often the model is actually needed.

Testing

There are two unit test classes, five tests:

• ErrorWindowTest: fires at the threshold, respects cooldown, and expires old errors from the window.
• LogPayloadReducerTest: truncates stacks to 12 lines plus a marker, collapses 200 repeated errors to x200 under the 2,000-character budget, and gives two events with different numbers and UUIDs the same fingerprint.

Nine NDJSON scenarios in sample-data/ cover normal traffic, bursts (DB timeout, NPE, OOM, upstream 502, JWT failures), a similar failure on a second service, malformed lines, and a below-threshold case.

Limitations

I'd rather list these than have you find them:

• The successful LLM path (real diagnosis written to the cache and then served as a hit) has not been run against a funded OpenAI account. The fallback path, reducer, window logic and the dashboard demo have been run.
• The dashboard's live mode hasn't been tested against the backend.
• Kafka offsets auto-commit, so a crash mid-analysis can lose that burst.
• No auth on the WebSocket endpoint. No cache eviction. Single Kafka broker.
• No throughput or latency benchmarks.

Try it

cp .env.example .env            # optionally set OPENAI_API_KEY
docker compose up -d            # Kafka (KRaft) + TimescaleDB with pgvector
set -a; source .env; set +a
mvn spring-boot:run             # http://localhost:8080, STOMP at /ws

cd dashboard && npm install && npm run dev   # http://localhost:5173

scripts/send-sample-logs.sh sample-data/02-db-timeout-burst.ndjson

Without an API key the app still starts, and bursts are stored as FALLBACK. There is also a browser-only demo that simulates the same rules with hand-written diagnoses and is labeled as a simulation. Click "Play the story" to see an LLM call, a below-threshold burst, a similar-error hit and an exact hit.

What's next

Not implemented yet, but the obvious gaps: manual offset commits after analysis, cache eviction, authentication on the WebSocket, and a real run of the LLM path with measured cache hit rates.

---

Code is on GitHub: axololtos/telemetry-incident-dashboard

If it's useful, a star helps. Issues and PRs are welcome (there's a CONTRIBUTING guide). Follow me here on DEV for more write-ups like this.