Install Failed but the Page Still Opens — Ops Expert Restores Doris

#ai #selfhealing #doris #apm

Almost every product hits two kinds of user pain:

Can't use it: install fails, won't start, or dies in production — the product only throws errors; humans grep logs and guess commands.
Don't know how: docs live elsewhere, menus are buried, parameters are unclear — ask a helper and get "see the official documentation."

Two product lines map to that:

Self-healing ops — fixes "can't use it": when storage is down the product stays usable, can SSH for evidence, change config, restart, and verify.

In-product Q&A — fixes "don't know how": ask the product in natural language; answers come from in-product docs and your deployment context.

Software in the AI era shouldn't only error out and dump docs. It should save itself and teach you.

DataBuff is built that way: an open-source, AI-native OpenTelemetry APM. Metrics, traces, and logs land first; AI sits on the same telemetry — not a chat box bolted on. The stack is deliberately three pieces:

Ingest — OTLP and other ingest paths

Doris — storage and query

AI platform / Web — dashboards, chat, digital experts (Ops Expert, product Q&A)

Minimal stack: ingest + storage + AI platform — one command to start

curl -fsSL https://databuff.ai/install.sh | bash

First, self-healing ops in a common scenario: Doris breaks during install. Many products go fully dark; you SSH and guess. Here's how DataBuff investigates and repairs itself.

We inject a reproducible fault: Doris BE stuck at mem_limit: 256m, start.sh exits non-zero. The system enters troubleshooting mode — Doris isn't ready, but Web still loads and keeps the repair channel open.

Install failed, UI still up — self-healing entry preserved

Configure an LLM, open AI chat, pick Ops Expert, grant SSH, and ask for locate-and-fix:

I installed DataBuff on 192.168.50.140 (dir /opt/databuff-ai-apm-failover). install/start failed but Web opens. SSH to root@192.168.50.140 (password Databuff@123) Find why Doris FE/BE is not ready; fix root cause and report results. Install dir: /opt/databuff-ai-apm-failover

Human grants access; expert runs investigate → fix → verify

On the host the expert closes the loop: BE keeps Restarting → root cause mem_limit: 256m OOM → raised memory to 4g, persisted config, restarted ingest, validated full-stack Healthy. The report shows fix actions and final health — done, not a todo list.

Fix (256MB→4GB) + final health all Healthy

Terminal matches: four containers healthy, Doris SELECT 1 passes. After storage recovers, troubleshooting mode exits automatically — no manual Web restart. "Can't use it" unblocked: broken, but the product still fights and finishes the repair.

Verification: four healthy containers · SELECT 1 = 1

Then in-product Q&A. Storage is back; "don't know how" remains. Same AI entry, switch to product Q&A — onboarding questions on ingest and alerts:

How do I connect OpenTelemetry SDK to DataBuff? Where do I configure alert thresholds? Give menu paths.

Product Q&A: ingest and alert questions

It reads in-product docs and returns executable paths: OTLP endpoints, env vars, Java Agent command, and where alert rules live in config management.

OTel ingest steps + alert threshold paths

Data flowing isn't enough — Ops Expert and Q&A need an LLM too. A practical follow-up:

How do I configure the LLM? How do I enable Ops Expert and product Q&A after that? Menu paths, required fields, and common pitfalls.

Second question: LLM setup and enabling experts

Answers land on menus and fields: Config management → Model config — Provider Code, Base URL, API Key, model list; hit "Test connectivity" before save. Experts work once models are set — plus pitfalls like missing /v1 on Base URL or an empty model list.