DEV Community: Peter

Why our on-prem log analytics appliance doesn't ship with an LLM

Peter — Sat, 29 Aug 2026 17:30:41 +0000

Disclosure: I'm one of the builders of Keel (Infranexis), a commercial product. This is an engineering write-up of a design decision, not a sales post — read it with that bias in mind, and poke holes in the comments. There's a public, no-signup demo link at the bottom if you'd rather just click around.

Every log analytics vendor now has an AI assistant. Nearly all of them work the same way: your query — and often your log data — leaves your network, transits the vendor's cloud, hits a model API, and the answer comes back.

If you run infrastructure in a bank, a hospital, a defense contractor, or a government network, that architecture is a non-starter. It doesn't matter how good the assistant is; the data can't leave. The teams with the strictest compliance requirements — arguably the teams that need investigation help the most, because their on-call rotations are stretched thin — are exactly the ones locked out of the AI feature wave.

We built Keel on a different premise: the model comes to the data, not the other way around.

The constraint that shaped everything

Keel is a log analytics appliance — a self-contained VM image (OVA, VHDX, QCOW2) that runs entirely inside your network on vSphere, Hyper-V, or KVM/Proxmox — and the same QCOW2 image boots on OpenStack (Glance/Nova) and OpenShift Virtualization (KubeVirt), which is where a lot of sovereign and regulated private clouds actually live. It ingests via OpenTelemetry (OTLP gRPC 4317 / HTTP 4318), syslog (514), and Fluentd (24224). It has to work on a network with no internet route at all, because for a meaningful share of our customers that's not a preference — it's policy.

Air-gap seeps into every design decision, including ones that have nothing to do with AI. Licenses are signed with RS256 and verified offline against a published public key, so an isolated node can validate its entitlements without ever calling home. Updates ship as signed bundles the appliance verifies before applying. In connected mode the appliance's entire outbound surface is one narrow path to the license portal — and in air-gapped mode, zero.

That constraint rules out every convenient way to add AI features. No hosted model APIs, obviously. No "hybrid" mode where anonymized queries go out — anonymizing log data is a research problem, not a checkbox. And no baking fixed model weights into the appliance, because then customers are stuck with our choice forever.

What we settled on: Keel orchestrates, you own the inference. Under Settings → AI Configuration you point the appliance at whatever backend your security posture allows: an Ollama instance on your network for fully air-gapped operation, or — if your policies permit hosted models — your own Anthropic or OpenAI key, or any OpenAI-compatible endpoint (vLLM included). You can configure a primary and a secondary backend with a timeout, so if the primary is unreachable the appliance fails over automatically. When a better open-weights model ships next quarter, you swap the endpoint. No appliance update needed.

What the AI actually does

Three capabilities, all running against your chosen endpoint:

1. Natural-language search. You ask "failed logins from the VPN gateway in the last six hours" and Keel translates it into a structured query against the log store, shows you the query it generated, and runs it. The translation is the LLM's job; the execution belongs to the query engine — a column-oriented store (ClickHouse) with hot local disk and an optional S3-compatible cold tier that stays queryable. The model proposes, the engine disposes, and you can always see and edit what it proposed.

2. Automatic root cause analysis. When an alert arrives — from Alertmanager, PagerDuty, Datadog, or a webhook — Keel opens an incident and investigates before a human picks it up:

 alert sources          alert pipeline      incident        AI RCA            notification channels
 ───────────────  ──►   ────────────►  ──►  ─────────  ──►  ───────────  ──►  ─────────────────────
 Alertmanager /                                             LLM + RAG +       Slack / Teams / email /
 Datadog /                                                  correlated        PagerDuty / webhook
 PagerDuty /                                                logs
 webhook

The RCA isn't a model free-associating over raw text. The investigation is grounded two ways: the logs correlated around the alert window, and a retrieval layer — a vector index (Qdrant) over the runbooks and documentation you've uploaded to the appliance's knowledge base. The model reasons over what your logs show and what your own docs say the system is supposed to do, and delivers a plain-English finding to Slack, Teams, PagerDuty, or email.

3. The Platform Assistant. A chat interface that answers questions about your own platform — and can use tools: search logs, check disk usage, look up incidents. Which brings us to the part we consider the most important design decision in the product.

What the AI deliberately can't do

The assistant is read-only by construction. It has no tools that change or delete anything — not configuration, not data, not pipelines. This isn't a policy filter asking the model to behave; the mutating tools simply don't exist in its toolbox.

This matters for a reason that goes beyond caution: prompt injection through your own logs. A log analytics AI reads whatever your infrastructure ingests — and an attacker can often control what gets logged (a User-Agent string, a crafted username, a request path). If a hostile string in a log line ever tricked the model, the blast radius must be a wrong answer, not a changed system. Capability-level read-only means the worst case is bad prose. Humans keep the pager and the keyboard.

The same philosophy applies to failure: if your LLM endpoint is down or you never configure one, Keel degrades to a complete, conventional log platform — ingest, pipelines, search, dashboards, alerting, incident tracking, all fully functional. AI is a layer, not a dependency.

The unglamorous parts

The hard engineering wasn't the prompts. It was making all of this operable by a team that does not want to become a log-platform team: one VM image containing the collectors, the application engine, ClickHouse, the configuration store, the vector index, TLS termination, and a failover manager — with a guided setup wizard in front and a 3-node HA cluster mode (floating VIP, replicated stores, automatic promotion of a healthy node) when you need it. The appliance discipline is the actual product. The model is a guest.

Why an appliance, in 2026?

Because the alternative for these teams is grim: keep paying per-GB SaaS prices for a platform their auditors are increasingly unhappy about, or run a self-managed ELK cluster that eats 10–20 engineer-hours a month and will never answer a question in English.

Flat pricing falls out of the architecture almost by accident: since the appliance runs on your hardware and your disk, we don't meter your gigabytes. The license is a flat annual fee banded by ingest volume ($6K/yr up to 5 GB/day, $14.8K up to 50 GB/day), every feature at every tier, and retention is bounded by the storage you attach — which your compliance team will appreciate more than we can.

Try it / tear it apart

If you'd rather click around than take my word for it, there's a public live demo —
no signup, opens in your browser:
demo-box.keel.infranexis.com .

Architecture docs, the security overview, and honest comparisons against Splunk and self-managed ELK — including the parts where they're still ahead of us — are at keel.infranexis.com.

If you run logs in a network where the data can't leave, I'd genuinely like your critique in the comments — especially from folks who've had to solve this the hard way.

The Spot Instance That Killed Our Payments Service (And Why It Took Us 47 Minutes to Find It)

Peter — Sun, 26 Apr 2026 03:09:54 +0000

It started at 1:49 AM.

PagerDuty fired — payments-service entering CrashLoopBackOff, 3 replicas simultaneously. On-call engineer paged. I joined the incident bridge 4 minutes later.

By 2:36 AM, we had the fix deployed. 47 minutes of debugging for a 2-line YAML change.

This is the postmortem. Not of the incident itself — those exist internally — but of the investigation. Every wrong turn, every wasted minute, and the exact signals that eventually cracked it.

## The First 10 Minutes: The Obvious Wrong Answer

When pods crash simultaneously right after a deployment, the deployment is guilty until proven innocent. That's the right instinct most of the time. So the first 10 minutes were spent here:

  kubectl rollout history deployment/payments-service -n production
  kubectl describe deployment/payments-service -n production

The last deployment had gone out at 7:52 PM — over 6 hours earlier. The pods had been healthy for 6 hours since that deploy. This should have ruled out the deployment immediately, but we didn't internalize
it fast enough. We spent another 5 minutes reviewing the deployment diff anyway, looking for a subtle config change that could have caused a delayed failure.

Nothing.

Time lost: ~12 minutes.

## The Next 15 Minutes: Log Archaeology

With deployment ruled out, we went to logs.

  kubectl logs payments-service-7d9f8b-xkp2q --previous -n production
  kubectl logs payments-service-7d9f8b-mn4lw --previous -n production

The logs showed the service starting up normally, attempting a Redis connection, and then… nothing. Process killed. No error, no panic, no stack trace. Just termination.

This looked like an OOMKill at first. So we checked resource limits:

  kubectl describe pod payments-service-7d9f8b-xkp2q -n production | grep -A5 Limits

Memory usage was at 180Mi against a 512Mi limit. Not OOM.

We pulled metrics from Prometheus looking for a memory spike. Nothing unusual.

Time lost: ~15 more minutes. Now 27 minutes in.

## Minute 27: The Event Log (Should Have Started Here)

This is the moment in every postmortem where I think: why didn't we start with events?

  kubectl get events -n production --sort-by='.lastTimestamp' | tail -30

There it was:

  LAST SEEN   TYPE      REASON    OBJECT                        MESSAGE
  2m          Warning   BackOff   pod/payments-service-7d9f8b   Back-off restarting failed container                                                                                                            
  4m          Warning   Unhealthy pod/payments-service-7d9f8b   Liveness probe failed: Get "http://10.0.1.12:8080/healthz": context deadline exceeded (Client.Timeout exceeded while awaiting headers)

Liveness probe failure. The probe was timing out.

But here's where we made our second mistake: we assumed the liveness endpoint itself was broken. Maybe the service wasn't starting correctly. Maybe there was a dependency it couldn't reach. We spent another
10 minutes trying to curl the healthz endpoint manually, exec'ing into a running pod to see if the endpoint responded.

It did. The endpoint worked fine.

Time lost: ~10 more minutes. Now 37 minutes in.

## Minute 37: The Node Event Nobody Checked

One of the engineers on the call said something offhand: "Wait, when did this start? 1:49? Can you check if anything happened on the node around then?"

  kubectl get events -n kube-system --sort-by='.lastTimestamp' | grep -E "Node|node"

  47m   Normal   NodeReady    node/ip-10-0-1-5   Node ip-10-0-1-5 status is now: NodeReady
  49m   Normal   Starting     node/ip-10-0-1-5   Starting kubelet                                                                                                                                               
  52m   Normal   NodeNotReady node/ip-10-0-1-5   Node ip-10-0-1-5 status is now: NodeNotReady

A node had cycled at 1:47 AM — 2 minutes before the crash loop started.

That node was running redis-cache-0.

Redis had restarted when the node recycled. And Redis, spinning up fresh, takes about 3-4 seconds before it starts accepting connections.

The payments-service liveness probe hits /healthz, which internally checks Redis connectivity. The probe has a 2-second timeout. Redis is taking 3-4 seconds to warm up. The probe fails. Kubernetes kills

the pod. Repeat.

The deployment was innocent. The pods were healthy. The liveness probe was doing exactly what it was configured to do. It was just configured too aggressively for the actual startup behavior of its

dependencies.

The fix:

  livenessProbe:
    httpGet:
      path: /healthz
      port: 8080
    timeoutSeconds: 10        # was: 2
    initialDelaySeconds: 15   # was: 0

Deploy. Pods stabilize. Incident resolved at 2:36 AM.

## The Real Postmortem: Why Did This Take 47 Minutes?

The fix was 2 lines of YAML. The investigation took 47 minutes. That ratio is the actual problem.

Here's the map of where the time went:

| Phase | Time Spent | Why |

|---|---|---|

| Deployment investigation | 12 min | Correct first instinct, but over-indexed |
| Log archaeology | 15 min | Logs showed the symptom (killed process) not the cause |
| Healthz endpoint testing | 10 min | Found the mechanism but not the root cause |

| Node event discovery | 7 min | The actual signal — found last |
| Diagnosis + fix | 3 min | Trivial once cause was known |

The signals that cracked it were all there from minute zero:

k8s events showing Liveness probe failed: context deadline exceeded
Node events in kube-system showing the recycle at 1:47 AM
Redis pod restart time correlating with the node event
Deployment history showing the last deploy was 6 hours prior (early exoneration)

If you had read these four things in that order, the investigation is 5 minutes, not 47. The problem is that these signals live in three different places, and under pressure at 2 AM, humans don't naturally
start with the most diagnostic view. We start with the most familiar one (logs) and dig deeper instead of wider.

## What Changes With Automated Investigation

We've been building Causa partly because of incidents exactly like this one.

When this alert fired, Causa's investigation loop would have:

Received the PagerDuty signal
Pulled the k8s events for the affected pods — immediately seeing the liveness probe timeout
Correlated the pod restart timestamps against node events in kube-system — finding the 1:47 recycle
Identified redis-cache-0 as running on the recycled node
Checked recent deployments and exonerated them (6 hours ago, no correlation)
Run eBPF traces to confirm Redis connection latency in the 3-4 second range

And posted this to Slack in under 60 seconds:

  ROOT CAUSE: Liveness probe timeout (2s) on payments-service is failing
  because redis-cache-0 requires 3-4s to accept connections after the                                                                                                                                           
  spot instance recycle at 01:47 UTC.

  EVIDENCE:       
  • Node ip-10-0-1-5 recycled at 01:47 UTC (2 min before crash loop start)
  • redis-cache-0 was on ip-10-0-1-5 — restarted at same time                                                                                                                                                   
  • Liveness probe timeout: 2s — insufficient for Redis warmup
  • Last deployment: 6h ago — NOT correlated                                                                                                                                                                    

  FIX: Increase livenessProbe.timeoutSeconds to 10, add initialDelaySeconds: 15                                                                                                                                 

  CONFIDENCE: High (4 corroborating signals)

That's not guesswork and it's not a summary of the alert we already saw. It's the actual investigation — reading the right signals in the right order, without the human bottleneck.

The on-call engineer still makes the call. They still deploy the fix. But they do it with the full picture in front of them in minute one, not minute 47.

## Three Changes We Made After This Incident

1. Start every investigation with events, not logs.

kubectl get events --sort-by='.lastTimestamp' is now the first command in our runbook. Logs show what happened to the process. Events show what Kubernetes did about it. Start wider, then drill down.

2. Node events are infrastructure events — check kube-system.
Pod-level debugging often misses infrastructure-level causes. If you're not checking kubectl get events -n kube-system, you're missing a category of signals.

3. Timestamp correlation before hypothesis formation.
Before you start testing a hypothesis (broken deployment, bad code, OOM), check when things happened. The exact timing often rules out 80% of likely causes before you investigate them.

## Final Word

The best SREs I know aren't necessarily the fastest debuggers in isolation. They're the ones who've internalized a sequence — who know which signals to read in which order, and who don't get anchored on

the first hypothesis.

That sequence can be automated. Not to replace the engineer's judgment, but to surface the right information before the judgment is needed.

If your team runs Kubernetes in production and you've had an incident that took longer than it should have, Causa is worth 5 minutes of your time. Free tier, one Install command, works with whatever alerting you already have.

Have a war story like this one? I'd genuinely like to hear it — drop it in the comments.