Giving Kubernetes Dashboard a Graph Brain: How We Integrated FalkorDB for Autonomous SRE

The 2:00 AM Kubernetes Outage

Every engineer who has managed a production Kubernetes cluster knows this scenario:

An alert fires. Pods are crash-looping or throwing 504 Gateway Timeout errors. You open the Kubernetes Dashboard and start frantically jumping between tabs:

• Tab 1: Pods (they are crash-looping with exit code 137).
• Tab 2: Services (which service selects these pods?).
• Tab 3: Ingresses (did the routing rule or port change?).
• Tab 4: ConfigMaps & Secrets (did a credential update break the database connection?).
• Tab 5: RBAC (does the pod's service account have permission to talk to the API?).

In a modern microservices cluster running hundreds of pods, resources do not fail in isolation. A failure in Pod A is almost always caused by an upstream service, an expired secret, or an invalid selector three hops away.

Yet traditional dashboards present the cluster as a set of disconnected inventory tables.

Even worse, when engineers paste error logs into generic AI assistants like ChatGPT, the AI hallucinates. It doesn't have a map of your cluster. It blindly tells you to "restart the pod" or "increase replica memory"—completely unaware that restarting will cause cascading timeouts across three upstream microservices.

Kubernetes is fundamentally a graph.

To solve cluster troubleshooting, we decided to give the Kubernetes Dashboard an active relational nervous system. Here is the engineering story of Pilot Kubernetes, built for the Graph Hacks hackathon (Agents That Act on Connected Data track).

---

The Big Picture: From Flat Tables to a Live Knowledge Graph

Instead of building a toy standalone CLI, we forked the Go + Angular Kubernetes Dashboard repository and embedded FalkorDB directly into the cluster.

Live K8s State (API Server) ──┐
                              ├──► FalkorDB Graph (6379/TCP) ◄──► Autonomous AI SRE Agent
Live Telemetry (Logs/Metrics) ─┘     (In-Memory Cypher Engine)         (Multi-Hop Graph Tools)

FalkorDB continuously maintains a property graph of the cluster. When an issue occurs, our Autonomous SRE Agent doesn't read a flat dump. It executes openCypher graph queries as internal tools to discover:

1. Upstream Root Causes: Traversing 1 to 4 hops upstream to see what prerequisite failed.
2. Downstream Blast Radius: Finding every microservice and dependent workload that will break before taking any action.
3. Traceable Fixes: Delivering explainable root causes with exact Cypher verification queries and 1-click YAML patches.

---

Key Architectural Decisions

1. Why Fork the Official Kubernetes Dashboard?

We could have started an empty repo from scratch with a toy Python script. But production operators do not want another throwaway CLI.

By building inside the official Kubernetes Dashboard codebase:

• We inherited battle-tested client-go authentication, token handling, and multi-namespace security.
• We preserved the clean Angular Material interface enterprise teams are already familiar with.
• We proved that graph intelligence can be seamlessly retrofitted into existing, mature cloud-native tools.

2. Why FalkorDB Over Traditional Graph Databases?

Kubernetes state is ephemeral and high-velocity. Pods scale up, fail, and restart in seconds. Heavy, Java-based graph databases introduce massive memory footprints and slow startup times that make in-cluster deployment impractical.

FalkorDB was the clear choice:

• Redis Protocol Speed: Built on Redis in C, it delivers sub-millisecond Cypher query execution.
• In-Memory Efficiency: A single lightweight container (falkordb/falkordb:v4.4.0) runs comfortably with minimal CPU and memory requests (100m CPU / 200Mi RAM).
• openCypher Expressiveness: Cypher makes writing complex, variable-length path traversals (<-[*1..4]-) intuitive and readable.

3. Tool Selection Over Blind Prompting

Instead of stuffing thousands of lines of raw logs into an LLM context window (which leads to token limits and hallucinations), our agent uses a Tool-Augmented Graph RAG pattern implemented in modules/api/pkg/ai/agent.go:

• falkordb_explore(resource): Grabs immediate 1-hop topology neighbors.
• falkordb_blast_radius(resource, maxHops=4): Discovers upstream dependencies and downstream dependents.
• falkordb_diagnose_service(service): Traces (:Service)-[:SELECTS]->(:Pod)-[:EMITTED]->(:LogEvent).
• falkordb_attack_paths(namespace): Uncovers transitive RBAC privilege escalation chains.

4. Zero-Hassle Helm Packaging

Rule 10 of Graph Hacks requires easy local or live deployment. We packaged FalkorDB's Deployment, PVC, and Service directly into the official Helm chart templates (charts/kubernetes-dashboard/templates/falkordb/).
One helm upgrade --install command deploys the Web UI, API, and FalkorDB together with zero external database configuration.

---

How It Works Under the Hood

1. The Continuous Topology Syncer (pkg/graph/syncer.go)

A dedicated background goroutine runs inside the Go API pod. Every 60 seconds, it fetches cluster state via k8s.io/client-go and runs atomic Cypher MERGE queries:

MERGE (n:Pod {name: $name, namespace: $namespace})
SET n.status = $status, n.ip = $ip, n.nodeName = $nodeName

It maps relationships across all architectural tiers:

• (:Service)-[:SELECTS]->(:Pod)
• (:Ingress)-[:ROUTES_TO]->(:Service)
• (:Deployment)-[:MANAGES]->(:Pod)
• (:Pod)-[:RUNS_ON]->(:Node)
• (:Pod)-[:MOUNTS_CONFIG]->(:ConfigMap)
• (:Pod)-[:USES_ACCOUNT]->(:ServiceAccount)
• (:ServiceAccount)-[:BOUND_BY]->(:RoleBinding)-[:GRANTS]->(:Role)

2. Live Telemetry & Log Ingestion (pkg/graph/telemetry.go)

A secondary syncer ingests live CPU usage rate (cpuMilli) and memory consumption (memoryMi) directly onto Node and Pod nodes.

Simultaneously, it tails pod logs, filters warning and error lines, and creates rolling LogEvent nodes:

(:Pod {name: "api-7d4db"}) -[:EMITTED]-> (:LogEvent {level: "ERROR", message: "dial tcp: connection refused"})

Old log events are automatically culled after 30 minutes, keeping FalkorDB fast and bounded.

---

Production Cypher Queries in Action

1. Multi-Hop Blast Radius Calculation

When an operator investigates a degraded service, FalkorDB calculates both upstream causes and downstream impact in a single traversal:

// Downstream Blast Radius: What workloads will break? (1 to 4 hops)
MATCH (n:Service {name: $name, namespace: $namespace})<-[*1..4]-(dependent)
RETURN DISTINCT dependent.name, dependent.namespace, labels(dependent), dependent.status;

// Upstream Dependencies: What prerequisites does this service need?
MATCH (n:Service {name: $name, namespace: $namespace})-[*1..4]->(dependency)
RETURN DISTINCT dependency.name, dependency.namespace, labels(dependency), dependency.status;

2. Uncovering Transitive RBAC Attack Paths

Kubernetes RBAC is notoriously difficult to audit. A pod's ServiceAccount might appear unprivileged, but it can inherit cluster-admin permissions through nested RoleBindings:

MATCH (sa:ServiceAccount {namespace: $namespace})-[:BOUND_BY]->(b)-[:GRANTS]->(r)
WHERE (r:Role OR r:ClusterRole) AND r.sensitive = 'true'
RETURN sa.name AS serviceAccount, 
       b.name AS binding, 
       r.name AS role, 
       labels(r)[0] AS roleKind;

---

⏱️ The 2:00 AM Bug: Feature-by-Feature, How We Solve It in Under 2 Minutes

Let's walk through an actual incident scenario:

The Alert: @service/checkout-api is throwing 504 Gateway Timeout errors and child pods are in CrashLoopBackOff.

Here is how each feature in Pilot Kubernetes turns a 45-minute nightmare into a 2-minute fix:

Minute 0:00 – Feature 1: Interactive 5-Tier Topology Canvas (/ai/traffic)

• Traditional Way: You log into the dashboard, open the Pods tab, see 200 pods, search for "checkout", click on it, check events, go to Services tab, check endpoints... 15 minutes wasted.
• With Pilot Kubernetes: You open the Topology Canvas. The cluster is automatically organized into 5 clear tiers: Ingress ➔ Service ➔ Workloads ➔ Pods ➔ Config/Storage. In 1 click, you isolate the checkout-api stack. You immediately see the red degraded indicator on the pod and notice its bezier connection to the database pg-cluster is glowing red.

Minute 0:30 – Feature 2: Multi-Hop Blast Radius Engine (pkg/graph/result.go)

• Traditional Way: In panic mode at 2 AM, engineers are tempted to immediately restart pods or roll back deployments. Doing this blindly often takes down other downstream services.
• With Pilot Kubernetes: You click Blast Radius on checkout-api. FalkorDB runs a 1-to-4 hop Cypher traversal:

  MATCH (n:Service {name: 'checkout-api'})<-[*1..4]-(dependent) RETURN dependent;

In 3 milliseconds, FalkorDB warns you: 3 dependent microservices (payment-gateway, order-processor, and mobile-api) depend directly on this service. You now know exactly what is at risk and avoid making blind changes that cause cascading outages.

Minute 1:00 – Feature 3: Live Telemetry Graph & Log Correlation (pkg/graph/telemetry.go)

• Traditional Way: You run 5 terminal tabs with kubectl logs -n prod checkout-api-7d8f -f --tail=500 trying to parse thousands of lines of noisy JSON logs.
• With Pilot Kubernetes: The telemetry syncer has already filtered out the noise and created rolling LogEvent nodes in FalkorDB linked directly to the pod:

  (:Pod {name: "checkout-api-7d8f"}) -[:EMITTED]-> (:LogEvent {level: "ERROR", message: "FATAL: password authentication failed for user 'app'"})

The graph correlates CPU/memory spikes with the exact database authentication error timestamp.

Minute 1:30 – Feature 4: Autonomous AI SRE Agent (pkg/ai/agent.go)

• Traditional Way: You ask ChatGPT: "Why is my pod crashing with code 137?" ChatGPT gives a generic 500-word essay suggesting memory leaks, JVM flags, or node pressure.
• With Pilot Kubernetes: In the AI Diagnostics Studio, you type: @service/checkout-api is failing. What is the root cause and blast radius?

The agent doesn't guess. It autonomously executes FalkorDB tools:

1. falkordb_explore(@service/checkout-api)
2. falkordb_diagnose_service(@service/checkout-api)
3. falkordb_blast_radius(@service/checkout-api)

The agent reveals the real truth:

"The pod is crash-looping because an automated Secret rotation 10 minutes ago updated the Secret db-credentials, but the checkout-api Deployment template was still pointing to the legacy Secret key POSTGRES_PASSWORD_V1. The pod cannot authenticate with PostgreSQL and exits upon boot."

Minute 2:00 – Feature 5: 1-Click YAML Remediation Patch

• The Fix: The agent doesn't just explain the bug; it provides:
  1. The exact Cypher query to verify the relationship in FalkorDB.
  2. The exact kubectl command to inspect the Secret.
  3. A ready-to-apply YAML patch updating the deployment secret key reference.

You apply the patch. The new pod boots, reconnects to PostgreSQL, and the FalkorDB topology canvas turns green. Incident resolved in under 2 minutes.

Bonus – Feature 6: AI Security & RBAC Attack Path Analyzer (/ai/security)

• If the 2 AM failure is a 403 Forbidden authorization failure, FalkorDB traces the RBAC path:

  MATCH (sa:ServiceAccount {name: 'checkout-sa'})-[:BOUND_BY]->(b)-[:GRANTS]->(r)
  RETURN r.rules;

It immediately spots missing verbs on required resources and generates a hardened, least-privilege Role manifest on the spot.

---

The "Rule 3" Test: Why FalkorDB is Essential

Hackathon Rule 3 states:

"The graph has to do real work. If the product still works once FalkorDB is taken out, the graph is decoration rather than the thing the product runs on."

In Pilot Kubernetes, if you take FalkorDB down:

1. Blast Radius Calculations Halt: The API has no graph traversal engine to calculate dependency paths.
2. The AI SRE Agent Loses Its Vision: Tool execution fails. The agent cannot inspect graph neighborhoods and cannot ground its diagnoses.
3. RBAC Attack Path Finding Stops: Multi-hop permission chains cannot be evaluated with flat API calls.
4. Telemetry Correlation Breaks: Ingress-to-Service-to-Pod-to-LogEvent correlation fails.

FalkorDB is not a visualization cache; it is the reasoning engine the product runs on.

---

Quickstart: How to Run It in 60 Seconds

You can deploy the entire stack—including the Web UI, API, and in-cluster FalkorDB—with a single Helm command:

1. Deploy via Helm

# Clone the repository
git clone https://github.com/Niteesh57/Pilot-Kubernetes.git
cd Pilot-Kubernetes

# Deploy Pilot Kubernetes with embedded FalkorDB
helm upgrade --install pilot-dashboard ./charts/kubernetes-dashboard \
  --namespace kubernetes-dashboard \
  --create-namespace \
  --set falkordb.enabled=true \
  --set falkordb.persistence.enabled=true

2. Access the Dashboard

Forward local traffic to the web container:

kubectl port-forward -n kubernetes-dashboard \
  svc/pilot-dashboard-kubernetes-dashboard-web 8080:8000

Open http://localhost:8080 in your browser.

3. Verify FalkorDB Connectivity

Test that FalkorDB is actively synchronizing cluster topology:

curl -s http://localhost:8080/api/v1/graph/status
# Returns: {"connected":true,"graphName":"kubepilot","nodeCount":142,"edgeCount":318}

---

Conclusion

Agents are only as good as the context they work with.

By giving our SRE agent FalkorDB's connected graph, decisions become explainable, verifiable, and free of hallucinations. An operator can see the exact graph path that led to a diagnosis, inspect the blast radius before applying a fix, and execute remediation with confidence.

• GitHub Repository: https://github.com/Niteesh57/Pilot-Kubernetes
• Live Documentation: https://niteesh57.github.io/Pilot-Kubernetes/
• Hackathon Track: Agents That Act on Connected Data
• License: Apache 2.0

Have questions about our graph schema or Go syncer? Check out the repo or drop a comment below!