DEV Community: Guatu

The 6-Layer Memory Architecture I Run for Claude Code

Guatu — Fri, 17 Apr 2026 14:15:45 +0000

I started where most people start: a single CLAUDE.md at the root of every repo. It worked for a few weeks. Then it started failing in the same boring way every time. The file grew past 200 lines and instructions started getting ignored. The agent re-learned the same infrastructure facts every session. I'd find myself pasting the same context into the chat again, then opening a doc to copy a command I'd already told Claude about three times that week.

So I kept adding layers. Six months later there are six of them. Last week I ripped out the two that didn't earn their keep, sanitized the rest, and pushed the whole thing as a public reference implementation at github.com/futhgar/agent-memory-architecture.

This post is the honest tour — what each layer does, what I got wrong, what I'd skip if I were starting fresh today.

The layers

Session start (always loaded)
├── Layer 1  Auto-memory (tool-provided persistence)
├── Layer 2  System instructions (CLAUDE.md / .cursorrules)
└── Layer 3  Path-scoped rules (load conditionally on file path)

On-demand retrieval (lazy)
├── Layer 4  Wiki knowledge base (markdown + [[wikilinks]])
├── Layer 5  Semantic vector search (Qdrant + nomic-embed-text)
└── Layer 6  Cognitive memory with activation decay (MSAM / Zep / Letta)

The composition is deliberate. Layers 1-3 sit in context every session so the agent starts knowing how to behave. Layers 4-6 are called on demand when the agent needs a specific fact — cheap index lookup (Layer 4) first, semantic search fallback (Layer 5) when the keyword index misses, cognitive memory (Layer 6) only when you need temporal dynamics that flat files can't express.

Most teams stop at Layer 2. Some go to Layer 4 — Karpathy's "LLM Wiki" insight that a disciplined wiki with good cross-references outperforms naive RAG at near-zero operational cost. I kept going because the homelab had enough breadth that even the wiki started missing on keyword lookups, and because session-specific learnings — "we tried X, it failed because Y" — didn't belong in permanent wiki articles but shouldn't evaporate either.

What went wrong along the way

CLAUDE.md bloat. The first mistake. I kept adding "this one more thing" to ~/.claude/CLAUDE.md and watched the agent ignore the back half. Anthropic's Claude Code memory docs explicitly say under 200 lines per file. Take that seriously. Every line above the threshold is making the lines below it less effective.

Using a vector store for things a keyword index would have solved. I set up a Qdrant claude-code-memory collection early and started dumping session learnings into it. Six months in, it had 451 points and most of them were never retrieved. The wiki could have solved 95% of what I was using it for. I still use the vector store for session-to-session learnings — but I no longer recommend reaching for it before Layer 4 is properly curated.

Treating path-scoped rules as optional. The .claude/rules/*.md pattern lets you write "when editing kubernetes/**, load these K8s conventions" rules that don't eat tokens when you're writing Python. Before I moved K8s conventions from the monolithic project CLAUDE.md into .claude/rules/kubernetes.md, my baseline context load was ~500-800 tokens higher for every session, whether or not I was editing K8s. That's real money — not the dollar kind, the context-window-budget kind that directly reduces how much the user can use for work.

Cognitive memory before I needed it. I set up MSAM (a custom ACT-R-inspired memory system with activation decay) for three months before I had a single use case that actually needed temporal dynamics. It was cool, but skipping to Layer 6 before Layers 4-5 are mature is the classic over-engineering trap. If I started fresh today, I'd stop at Layer 4 for at least a month and only add Layers 5-6 once the wiki's limitations were obvious from actual use.

Forgetting to validate the "fixes." Mid-project I realized my MSAM MCP integration was silently broken — the wrapper path in .claude.json pointed to a file that didn't exist. Every "use MSAM for this" instruction in CLAUDE.md had been a dead letter for weeks. The lesson: when you configure a memory system, test the round-trip (store → recall) before trusting that it works. Configuration isn't validation.

What's actually in the repo

The repo is opinionated and template-heavy, not a framework. You fork it or cherry-pick pieces; you don't install it and "run" it.

templates/global/CLAUDE.md and templates/project/CLAUDE.md — sanitized starting points, under 60 lines each
templates/rules/ — path-scoped rule examples for kubernetes, terraform, dockerfiles, and wiki editing
templates/memory-files/ — YAML-frontmatter templates for project / reference / feedback memory files
scripts/rebuild-memory-index.py — audits memory files for orphans, stale content, oversized files, and credential leaks
scripts/build-wiki-graph.py — generates an interactive force-directed graph of your wiki's [[wikilinks]] (or use Cosma for a more polished rendering)
scripts/msam-mcp-wrapper.py — a FastMCP wrapper for cognitive memory if you go that far
scripts/check-sanitization.sh — pre-publish scanner for secrets, IPs, and personal data if you fork this

There's a one-line installer:

curl -sSL https://raw.githubusercontent.com/futhgar/agent-memory-architecture/main/bootstrap.sh | bash -s -- --layer=2

That auto-detects your agent (Claude Code, Cursor, or Aider), backs up any existing files, and drops in the templates. It has a --dry-run flag because I wouldn't blind-trust someone else's curl-bash either. See docs/getting-started.md for the decision tree on which layer to install when.

What to skip

Honestly: Layer 6 unless you already know you need it. The cognitive memory layer is the most opinionated and least-validated part of the system. MSAM is a research-grade tool. Zep and Letta are the production alternatives. All three require infrastructure, monitoring, and conceptual work. If your wiki (Layer 4) + vector search (Layer 5) aren't yet exhausting your team's patience, Layer 6 is premature.

Also: don't cargo-cult the whole stack. The repo's docs/getting-started.md has a decision tree — "is your CLAUDE.md over 200 lines? yes → try Layer 3; no → stay at Layer 2." Most teams should stop at Layer 4. The repo exists so you can see what the whole road looks like, not because everyone should walk it.

Why open-source it

Two reasons. One is pure: the pattern is genuinely useful and I didn't see it written down anywhere in its full form. There are a lot of "here is my CLAUDE.md" repos and a lot of research on isolated pieces (RAG, knowledge graphs, vector stores), but the composition — which layers to use together, in what order, with what tradeoffs — was pattern I had to piece together from running it.

The other reason is less pure: I run Guatu Labs, and we help companies implement AI agent infrastructure. A reference implementation that people can actually read is a better pitch than anything I'd write on a services page. If this saves you a week of research, and later you need help rolling something similar out in your org, you know where to find me.

Building Karpathy's LLM Wiki: A Production Homelab Implementation

Guatu — Mon, 13 Apr 2026 18:15:03 +0000

I tried to run Karpathy's LLM Wiki on my Proxmox homelab cluster and spent three days debugging why the front-end wouldn't load. The error log said 502 Bad Gateway, but the backend was running and the API was reachable. It turned out the problem was in the reverse proxy configuration. I'd missed a single line in the Nginx config that was required for WebSockets to work properly.

This isn't the first time I've run into an issue where the documentation says one thing and the reality is something else. That's why I'm writing this — to show you exactly what I did, what went wrong, and how I fixed it.

If you're running a multi-node Kubernetes cluster with Proxmox, or you're trying to set up a production-like AI agent environment in your homelab, this is for you. I'll walk you through the exact steps I used to deploy Karpathy's LLM Wiki in a way that mirrors real-world production setups, with the gotchas and workarounds that actually matter.

What I Tried First

I started by cloning the LLM Wiki repository and running the demo setup with Docker Compose. It worked locally, but when I tried to scale it up to Kubernetes, I hit a wall. The first thing that broke was the persistent storage. I assumed the default Kubernetes emptyDir would work, but the wiki needed to persist data across restarts. I tried Longhorn, but the initial setup didn't account for the way the LLM Wiki uses SQLite — it required a specific file permission that wasn't set in the default PVC.

Then I tried using the initContainers approach to set the right permissions, but that didn't work either. I ended up having to modify the StorageClass and PVC definitions to include fsGroup and runAsUser settings that matched the SQLite process. That was a pain, but it was necessary.

Next, I tried to set up the reverse proxy with Traefik, which is what I use in production. But again, the documentation didn't mention that the LLM Wiki's WebSockets needed a specific configuration. I spent a few hours trying to figure out why the chat interface was broken, until I found a single line in the Traefik config that I had to add: proxy-set-headers: "Connection: keep-alive".

The Actual Solution

Here's what I ended up with. This is a production-ready setup that mirrors real-world environments, not just a demo.

Prerequisites

I'm assuming you have:

A Proxmox cluster with Kubernetes installed
A working Kubernetes cluster with at least 2 nodes
A working Traefik ingress controller
A working Longhorn storage system

Deploying the LLM Wiki

I used Helm to deploy the LLM Wiki, but I had to modify the values file to match the specific needs of the application. Here's what my values.yaml looked like:

# values.yaml
ingress:
  enabled: true
  hosts:
    - host: "wiki.example.com"
      paths:
        - path: "/"
          pathType: Prefix
  annotations:
    traefik.ingress.kubernetes.io/router.middlewares: "default-websocket-middleware"
    traefik.ingress.kubernetes.io/router.tls: "true"

storage:
  enabled: true
  class: "longhorn"
  accessModes:
    - ReadWriteMany
  size: "10Gi"

Then I created the middleware for WebSockets in Traefik:

# traefik-middleware.yaml
apiVersion: traefik.containo.us/v1alpha1
kind: Middleware
metadata:
  name: default-websocket-middleware
spec:
  webSocket:
    upGrade: true

I also had to add the proxy-set-headers configuration to the Traefik IngressRoute, which I did by modifying the ingressRoute spec in the Helm chart:

# ingressroute.yaml
apiVersion: traefik.containo.us/v1alpha1
kind: IngressRoute
metadata:
  name: wiki-ingress
spec:
  entryPoints:
    - web
    - websecure
  routes:
    - match: Host(`wiki.example.com`)
      kind: Rule
      services:
        - name: wiki
          port: 80
      middlewares:
        - name: default-websocket-middleware
  tls:
    passthrough: true

For the storage, I used Longhorn with a ReadWriteMany access mode. That was tricky because SQLite doesn't support ReadWriteMany by default, but I was able to get it working by using a Longhorn volume with a Filesystem type. That's something the documentation didn't mention.

Configuring the Application

The LLM Wiki itself had a few configuration options I had to set. I added the following environment variables to the deployment spec:

env:
  - name: WIKI_DB_URL
    value: "sqlite:///wiki.db"
  - name: WIKI_HOST
    value: "wiki.example.com"
  - name: WIKI_PORT
    value: "80"

I also had to make sure the SQLite file had the correct permissions. I used a initContainer to set the right ownership:

initContainers:
  - name: init-sqlite
    image: busybox
    command: ["sh", "-c", "touch /wiki.db && chown 1001:1001 /wiki.db"]
    volumeMounts:
      - name: wiki-data
        mountPath: /wiki.db

This was necessary because the SQLite process runs as user 1001, and without the right ownership, it wouldn't be able to write to the database.

Why It Works

The key to getting this to work was understanding the specific requirements of the LLM Wiki and how they interacted with Kubernetes and Longhorn.

Persistent Storage: SQLite requires a specific file permission that wasn't set in the default PVC. I had to use an initContainer to set the right ownership and make sure the volume was mounted correctly.
Reverse Proxy Configuration: The WebSockets in the LLM Wiki required a specific Traefik middleware to work properly. That's something the documentation didn't mention, but it was necessary for the chat interface to function.
Longhorn Configuration: The LLM Wiki uses SQLite, which doesn't support ReadWriteMany by default. I had to use a Longhorn volume with a Filesystem type to make it work.

These are the kinds of gotchas that don't appear in the documentation, but they're critical to getting the application running in a real-world environment.

Lessons Learned

This was a learning experience, and here's what I'd do differently next time:

Start with a Minimal Setup: I should have started with a minimal setup and then added complexity incrementally. That would have helped me identify the issues earlier.
Use Real-World Tools: I should have used the same tools I use in production — like Traefik and Longhorn — from the start. That would have saved me time debugging compatibility issues.
Test the WebSockets Early: I should have tested the WebSockets early on to make sure they worked before deploying the application. That would have saved me from a lot of frustration.

I also learned that the documentation doesn't always cover the edge cases. For example, the LLM Wiki's documentation didn't mention the need for a specific Traefik middleware for WebSockets. That's something that's critical to getting the application running, but it's not something you'd find in the documentation.

If you're trying to run the LLM Wiki in a production-like environment, I'd recommend using the same tools you use in production — like Traefik and Longhorn — and testing the WebSockets early on. That will help you avoid a lot of the gotchas I ran into.

Finally, I'd like to mention that if you're looking for help with AI agent orchestration, Kubernetes infrastructure, or industrial IoT systems, I'm available for consulting. You can find more information at https://guatulabs.com/services.

Proxmox API Tokens: Bash History Expansion and the ! Character

Guatu — Mon, 13 Apr 2026 12:15:16 +0000

I spent 45 minutes trying to figure out why my Proxmox API token kept getting exposed in logs. Turns out, it wasn't a security hole — it was Bash history expansion eating my token string and spitting it back in plaintext.

The thing: Bash history expansion interprets ! as a history reference. When you paste an API token into a script or command line, any ! in the token gets expanded to a previous command. This is a problem when your token contains ! — which some Proxmox API tokens do.

# This will fail because Bash tries to expand the ! in the token
curl -u "user:my-token!123" https://proxmox.example.com/api2/json/nodes

# Fix it by quoting the token or escaping the !
curl -u "user:my-token\!123" https://proxmox.example.com/api2/json/nodes

The fix is simple but easy to miss: quote the token or escape any ! characters. I learned this the hard way after seeing my-token!123 show up in my shell history and logs after a failed API call.

Done: Bash history expansion is a gotcha for anyone using ! in API tokens — quote or escape them to avoid surprises.

AMD iGPU Stealing Your RAM: UMA Frame Buffer on Headless Servers

Guatu — Mon, 13 Apr 2026 10:15:17 +0000

I had 16 GB of RAM in my Proxmox host. I expected to see 16 GB. What I got was 11 GB.

The first time I noticed it was during a reboot. The Proxmox web UI said 11 GB installed. The free -h command showed 11 GB. The BIOS reported 16 GB. Something was missing 5 GB. Not a kernel module. Not a driver. Not a config. It was the UMA frame buffer, reserved by the AMD iGPU.

I had a headless server, no display, no GPU passthrough, no need for the integrated graphics. But the UMA frame buffer was still allocating memory. Every time the system booted, 5 GB was carved out for the iGPU, even though it had no use for it.

The fix was in the BIOS. I had to disable UMA frame buffer allocation. But I didn't find that setting right away. I tried looking for "graphics" in the BIOS, found a section called "Advanced Chipset Configuration," and there it was: "UMA Frame Buffer Size." Set to "Auto" by default. I changed it to "Disabled" and rebooted. The RAM came back.

I should have known this would happen. I've seen it before in other Proxmox nodes. The docs say nothing about it. The forums have a handful of posts about it, buried in threads about GPU passthrough or memory leaks. No one says, "Hey, if you're running a headless server with an AMD iGPU, you might be losing 5 GB of RAM."

I've been running this setup for months now. I've got a couple of other Proxmox nodes with the same hardware. I've applied the same fix on all of them. It's not a major issue, but it's one of those things that feels like a waste of resources. 5 GB is a lot for a headless server that's not even using the GPU.

If you're running Proxmox on hardware with an AMD iGPU and you're not using it for anything — no passthrough, no display, no headless GPU workloads — check the UMA frame buffer setting. It's easy to miss, but it's there. You'll save memory. You'll save confusion. And you'll avoid that moment when you think your system is broken, but it's just the iGPU eating your RAM.

You can find the setting in the BIOS under "Advanced Chipset Configuration." It's called "UMA Frame Buffer Size." If you're not using the integrated graphics, set it to "Disabled." You can also add nomodeset to your kernel command line if you want to avoid the iGPU altogether. But the BIOS setting is cleaner and more permanent.

I've seen this come up in discussions about Proxmox, Kubernetes, and even in some homelab setups with bare metal. It's not a common issue, but it's a real one. And it's one of those gotchas that doesn't make it into the official documentation. You have to dig for it. You have to know what to look for.

If you're running a headless server with an AMD iGPU and you're seeing unexplained memory shortages, check the UMA frame buffer. It's not a bug. It's a feature. And it's one you can turn off if you don't need it.

Agent Credential Management: Two-Tier Service Accounts for Secure AI Agent Workflows

Guatu — Fri, 10 Apr 2026 04:15:25 +0000

I spent three days trying to get a multi-agent system to talk to a Kubernetes API endpoint. Every time I used the default service account, the agent would hit a 403 and lock out. I was using the right permissions, the right roles, the right RBAC rules. It wasn’t until I implemented a two-tier service account system that the agents finally stopped throwing errors. It’s not just about having the right permissions , it’s about structuring them in a way that isolates the agent’s access and limits its blast radius.

If you're running AI agents in Kubernetes, especially ones that interact with external systems or sensitive data, this is a pattern you should consider. This isn't just about security , it's about making sure your agents fail safely and don't accidentally break your entire infrastructure if they're compromised.

I first tried using the default service account for all my agents. It worked fine for a while, but as I scaled out to more agents and more workflows, I started seeing odd behavior. The agents would occasionally call APIs with incorrect credentials, or worse, they'd silently fail without any logs. I checked the RBAC policies, adjusted them, and kept getting the same issues. It was like the agents were leaking credentials or being intercepted by something in the cluster.

At one point, I even tried using a single dedicated service account for all agents, which I thought would give them more consistent permissions. But that only made things worse. Now every agent had the same access, and when one got compromised, the whole system was at risk. I realized I needed a way to isolate each agent’s credentials while still maintaining some level of central control.

The solution came in the form of a two-tier service account system. Instead of giving each agent its own service account with direct access to the cluster, I created a central service account with limited permissions that acted as a "proxy" for all the agent workloads. Each agent then had its own "child" service account, which was granted access only through this central account. This way, I could manage permissions at the central level without having to update every agent individually when something changed.

Here’s how I set it up. First, I created the central service account with a role that had access to the necessary resources but nothing more:

apiVersion: v1
kind: ServiceAccount
metadata:
  name: agent-proxy
  namespace: agent-system
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: agent-proxy-role
  namespace: agent-system
rules:
- apiGroups: [""]
  resources: ["pods", "services", "configmaps"]
  verbs: ["get", "list", "watch"]
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: agent-proxy-binding
  namespace: agent-system
subjects:
- kind: ServiceAccount
  name: agent-proxy
  namespace: agent-system
roleRef:
  kind: Role
  name: agent-proxy-role
  apiGroup: rbac.authorization.k8s.io

Then, for each agent, I created a separate service account that was bound to the central account using a RoleBinding that referenced the central account’s role. This gave each agent its own identity while still limiting their access to only what was necessary:

apiVersion: v1
kind: ServiceAccount
metadata:
  name: agent-worker-1
  namespace: agent-system
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: agent-worker-1-binding
  namespace: agent-system
subjects:
- kind: ServiceAccount
  name: agent-worker-1
  namespace: agent-system
roleRef:
  kind: Role
  name: agent-proxy-role
  apiGroup: rbac.authorization.k8s.io

Each agent’s deployment was configured to use its own service account, and I made sure to set the automountServiceAccountToken to true so that the agents could access the token automatically. This way, the agents had their own credentials but were still limited by the permissions defined in the central role.

This approach has a few key benefits. First, it isolates each agent's access, so a compromise in one agent doesn’t automatically give access to all others. Second, it simplifies permission management , you can update the central role once and all agents inherit the new permissions. And third, it makes auditing easier , if an agent does something unexpected, you can trace it back to its specific service account.

This isn’t a perfect solution, though. There are tradeoffs. Setting this up requires more configuration than just using a single service account, and it adds some complexity to the system. You also have to make sure that the central account doesn’t have more permissions than it needs , otherwise, you end up with the same security risks you were trying to avoid.

What I would do differently is to automate this process more. I had to manually create each service account and role binding, which was tedious and error-prone. If I were to do this again, I’d set up a Kubernetes operator or a script that could generate the necessary resources based on a list of agent names. That would reduce the risk of mistakes and make it easier to scale.

Another thing I’d consider is adding more fine-grained permissions. Right now, the central role gives access to pods, services, and configmaps, but that might be more than some agents need. Going forward, I’d look into creating more specific roles for different types of agents , for example, one role for agents that only need to read pods and another for agents that can modify services. This would make the system more secure and more efficient.

I also wish I had implemented some kind of token rotation or short-lived credentials for the agent service accounts. Right now, the tokens are long-lived, which means if an agent’s credentials are ever compromised, the attacker has a long window to do damage. Adding a system that rotates the tokens periodically would help reduce that risk, even if it adds some complexity.

In the end, the two-tier service account system worked well for my needs. It gave me the security I wanted without sacrificing the flexibility I needed. It wasn’t the easiest setup to get going, but the tradeoff was worth it , I now have a more secure and manageable agent system. If you're managing AI agents in Kubernetes, I'd recommend considering this approach for your credential management.

Pod Disruption Budgets: Why kubectl drain Gets Stuck on Longhorn

Guatu — Thu, 09 Apr 2026 01:56:30 +0000

I spent three hours trying to drain a node running Longhorn, only to watch kubectl drain freeze with no progress or error. It felt like the command was waiting for something that would never come.

What I expected was a smooth drain — pods evicted, node marked as unavailable, and everything moving on. No drama. No hanging. Just a clean maintenance operation.

What actually happened was a silent stall. The drain command would not proceed past the point where it started trying to evict the Longhorn manager pod. It would sit there, stuck, with no message. After some digging, I realized it was a Pod Disruption Budget (PDB) blocking the eviction. Longhorn has a PDB in place that ensures at least one manager pod is always running, and that PDB was preventing the eviction from completing.

The fix came in two parts: first, I had to adjust the PDB to allow the drain to proceed. Second, I had to make sure the kubectl drain command used the right flags. Here's what I did:

kubectl drain <node-name> \
  --ignore-daemonsets \
  --delete-emptydir-data \
  --grace-period=30 \
  --timeout=300s \
  --force

I also updated the PDB to allow at least one manager pod to be evicted during the drain, while still ensuring availability:

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: longhorn-pdb
spec:
  minAvailable: 1
  selector:
    matchLabels:
      app.kubernetes.io/instance: longhorn

This change allowed the drain to proceed without blocking. The key takeaway was that PDBs are not just a safety net — they're a potential roadblock if not configured with maintenance in mind.

This matters a lot if you're managing a Kubernetes cluster with Longhorn. Every time you want to do maintenance, scale down, or upgrade, you're going to run into this. You need to know how PDBs interact with your storage layer, and how to configure them to allow safe operations. If you don't, you'll be stuck trying to figure out why kubectl drain is hanging — and it won't be fun.

I learned my lesson the hard way. I'm now making sure every PDB in my cluster is reviewed during maintenance planning. I've also added documentation to our team's Kubernetes guides about this specific scenario. If I had known earlier, I would have configured the PDB with more flexibility from the start.

If you're running Longhorn and doing any kind of node maintenance, take a moment to check your PDBs. They might be the reason your kubectl drain is getting stuck. And if you're not sure where to start, I recommend looking at the Longhorn PDBs and asking: "What's the worst that could happen if one of these pods goes away during a drain?" The answer will tell you what minAvailable should be.

For more on how to handle storage in Kubernetes, check out my post on Kubernetes Storage on Bare Metal: Longhorn in Practice. It dives deeper into how Longhorn fits into larger infrastructure setups.

Equipment Health Scoring: How One Number Made My Operators Stop Checking the Dashboard

Guatu — Thu, 09 Apr 2026 01:54:34 +0000

Operators in my last plant used to ignore dashboards — not because they didn’t care, but because the data was too noisy, too fragmented, and too abstract. I've seen this pattern again and again: a sea of metrics with no clear signal. But when I implemented a simple health scoring system that aggregated temperature, vibration, and pressure into a single number between 0 and 100, that changed everything. Now, operators check that number daily, and it’s the only metric they ask about during shift handovers.

The key was making the score intuitive, real-time, and tied directly to the equipment’s state. I built it using a simple weighted formula that reflects the relative importance of each sensor. Temperature and vibration are more sensitive to wear and tear, so I gave them higher weight. Pressure, while important, was less of a red flag unless it spiked suddenly. This isn’t perfect — it’s not an ML model, it’s a rule-based system — but it works for my use case and it’s easy to maintain.

Here’s how I implemented it in Python as part of a Node-RED flow that aggregates data from multiple sensors:

# equipment_health.py
import json
from flask import Flask, request, jsonify

app = Flask(__name__)

# Weights for each sensor type
WEIGHTS = {
    "vibration": 0.4,
    "temperature": 0.4,
    "pressure": 0.2
}

def calculate_health_score(data):
    # Normalize each metric to a 0-1 scale
    normalized = {}
    for sensor, value in data.items():
        if sensor in WEIGHTS:
            # Example: linear normalization between 0 and 100
            # This would be replaced with domain-specific logic
            normalized[sensor] = max(0, min(1, (value - 20) / 80))  # 20-100 range

    # Calculate weighted score
    score = 0
    for sensor, weight in WEIGHTS.items():
        if sensor in normalized:
            score += normalized[sensor] * weight

    # Map to 0-100 range
    return round(score * 100, 2)

@app.route("/health", methods=["POST"])
def health():
    data = request.json
    score = calculate_health_score(data)
    return jsonify({"health_score": score})

if __name__ == "__main__":
    app.run(host="0.0.0.0", port=5000)

This is a basic example, but it shows the core idea: take raw data, normalize it, apply weights, and return a single score. The scoring function can be as simple or complex as needed. For example, I’ve used exponential decay for vibration data or applied thresholds based on historical failure data.

Operators now see this score on a Grafana dashboard with color-coded thresholds (red for <40, yellow for 40–60, green for 60+), and it's the only metric they ask about. The real win isn’t the algorithm, though — it’s that the score is actionable. When it drops below 40, they know to investigate. When it hits 20, they know it’s time to schedule maintenance. It's not perfect, but it’s what they check every day.

Infrastructure as Code, but Automated: OpenTofu and GitHub Actions

Guatu — Thu, 09 Apr 2026 01:54:21 +0000

I once spent three hours debugging a "successful" pipeline that had actually failed to deploy a critical security group update because I had set continue-on-error: true in a shell script step. The logs said green, the UI said green, but my actual infrastructure was still wide open to the internet. It is a specific type of dread that only hits when you realize your automation is lying to you.

If you are managing even a modest cluster of VMs or a few bare-metal nodes, you eventually hit a wall where manual tofu apply commands from your laptop become a liability. You start worrying about which version of the binary you're running, whether your local state is out of sync with the remote, and if you accidentally left a sensitive variable in your shell history.

This is the problem for anyone moving from "I run scripts" to "I manage systems." Whether you are orchestrating Kubernetes nodes on Proxm/x or managing cloud resources, the goal is to move the source of truth from your terminal to a version-controlled workflow.

What I Tried First

My first instinct was the "Lazy Engineer" approach. I just wanted a GitHub Action that ran tofu apply whenever I pushed to main. I didn't bother with a plan phase, a pull request review, or even a remote backend. I just pointed a runner at my state file and hoped for the best.

It was a disaster.

I pushed a change to a networking module that had a small typo in a CIDR block. Because there was no "Plan" step to inspect, the runner immediately started destroying and recreating the primary network interface. My entire lab went dark. I couldn't even SSH into the nodes to fix the mistake because the automation had nuked the route I was using.

I also learned the hard way that running apply directly on a push is dangerous. Without a decoupled plan step that attaches to a Pull Request, you lose the ability to peer-review the impact of your changes. You aren't reviewing code; you are reviewing side effects.

The Actual Solution

A real automation pipeline needs three distinct stages: Validation, Planning, and Deployment. You want the Plan to happen when a Pull Request is opened, and the Apply to happen only after that Plan has been merged into your main branch.

Here is the architecture I use for my infrastructure projects.

1. The Backend Configuration

First, you cannot use local state. If the GitHub runner dies or the workspace is wiped, your infrastructure is orphaned. You need a remote backend—S3, GCS, or even a specialized state server.

# backend.tf
terraform {
  backend "s3" {
    bucket         = "my-infra-state-bucket"
s    key            = "network/terraform.tfstate"
    region         = "us-east-1"
    dynamodb_table = "terraform-lock"
  }
}

2. The GitHub Actions Workflow

The workflow is split into two jobs: plan (triggered on PR) and apply (triggered on push to main). I use the opentofu/setup-opentofu action because it handles the binary installation cleanly.

name: Infrastructure CI/CD

on:
  push:
    branches: [ main ]
  pull_request:
    branches: [ main ]

jobs:
  plan:
    runs-on: ubuntu-latest
    steps:
      - name: Checkout Code
        uses: actions/checkout@v4

      - name: Setup OpenTofu
        uses: opentofu/setup-opentofu@v1
        with:
          version: 1.6.0

      - name: Configure AWS Credentials
        uses: aws-actions/configure-aws-credentials@v4
        with:
          aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
          aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS/ACCESS_KEY }}
          aws-region: us-east-1

      - name: Tofu Init
        run: tofu init

      - name: Tofu Plan
        id: plan
        run: tofu plan -no-color > plan.txt

      - Upload Plan Artifact
        uses: actions/upload-artifact@v4
        with:
          name: tfplan
          path: plan.txt

  apply:
    needs: plan
    if: github.event_name == 'push' && github.ref == 'refs/heads/main'
    runs-on: ubuntu-latest
    steps:
      - name: Checkout Code
        uses: actions/checkout@v4

      - name: Setup OpenTofu
        uses: opentofu/setup-opentofu@v1
        with:
          version: 1.6.0

      - name: Configure AWS Credentials
        uses: aws-async/configure-aws-credentials@v4
        with:
          aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
          aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
          aws-region: us-east-1

      - name: Tofu Init
        run: tofu init

      - Name: Tofu Apply
        run: tofu apply -auto-approve

Note: In a production-grade setup, you would actually download the tfplan artifact from the previous job and run tofu apply tfplan. This ensures that the exact changes you reviewed in the PR are the ones being applied, rather than a fresh plan that might have drifted in the minutes since the PR was merged.

Why It Works

This setup works because it enforces a "Review-First" culture. When you open a PR, the plan job runs. You can look at the GitHub Actions logs, see exactly which resources are being added, changed, or destroyed, and then leave comments on the PR.

The separation of plan and apply creates a gate. The apply job is guarded by a conditional check: if: github.event_name == 'push'. This prevents accidental deployments from feature branches.

Using a remote backend with locking (like DynamoDB) is the secret sauce for stability. If two developers try to run a pipeline at the same time, OpenTofu will see the lock and fail the second job rather than corrupting the state file. This is the difference between a professional deployment and a lucky one.

If you are managing more complex environments, such as AI agent deployments that require specific GPU-backed compute nodes, this level of automation is non-negotiable. You cannot afford to manually tweak instance types or disk sizes when your workload depends on specific hardware availability.

Lessons Learned

If I could go back and rewrite my first few workflows, here is what I would change:

Never use auto-approve on a PR. It defeats the purpose of the plan. Only use it in the apply job where the human review has already happened in the PR stage.
Validate your logic before you run it. I have seen pipelines pass because a grep command failed to find a string, but the actual deployment failed. Always check the exit codes of your custom scripts. If you are using an automation tool like n8n to trigger these workflows, ensure your error-handling logic is explicit. A failed check should result in an immediate exit 1.
Secrets management is a minefield. Do not pass secrets as environment variables in your YAML if you can avoid it. Use the official provider actions (like aws-actions/configure-aws-credentials) which handle the heavy lifting of credential injection securely.
The "Drift" problem is real. Automation only works if you actually use it. The moment you manually change a setting in a web console or via the CLI, your OpenTofu state is out of sync. I have learned to treat any manual change as a "broken build" that needs to be codified immediately.

Automating infrastructure is not about making things "faster"—it is about making them predictable. When you can trust that a push to main will do exactly what the PR promised, you stop being a firefighter and start being an engineer.

Attention Residuals: How Kimi Is Rethinking Transformer Depth

Guatu — Tue, 07 Apr 2026 18:12:06 +0000

Every transformer you've ever used stacks layers with a dead-simple formula: take the input, add the layer's output, move on. x + layer(x). Fixed weight of 1. No questions asked.

The Kimi team at Moonshot AI just published a paper that asks: what if that's been wrong the whole time?

The Problem Nobody Talks About

Standard residual connections accumulate layer outputs with equal weight. Layer 1 contributes the same as layer 47. The hidden state grows without bound as you stack more layers, and each individual layer's contribution gets diluted into the noise.

This is called PreNorm dilution — and it gets worse the deeper your model goes. At 100+ layers, the early layers are essentially screaming into a hurricane. Their signal is there, mathematically, but it's buried under the sum of everything that came after.

For most of transformer history, we've papered over this with normalization tricks. RMSNorm, LayerNorm, various pre-norm and post-norm arrangements. They help. They don't solve the root cause.

What Attention Residuals Actually Do

The Kimi team's solution is elegant: replace the fixed x + layer(x) with softmax attention over all preceding layer outputs.

Instead of blindly accumulating, each layer looks back at every layer before it and decides — with learned, input-dependent weights — how much of each previous layer to carry forward.

Think of it like this: in standard residual connections, you're stuffing every letter you've ever received into one folder, in order, and hoping the important ones float to the top. Attention Residuals let each layer read through the folder and pick out exactly the letters that matter for the current task.

The key properties:

Input-dependent: The aggregation weights change based on what the model is processing, not fixed at training time
Depth-selective: Layer 50 might pull heavily from layer 3 and layer 48, but ignore everything in between
Learned: The attention mechanism over layers is trained end-to-end with the rest of the model

Block AttnRes: Making It Practical

The naive version — attending over every single preceding layer — would be computationally brutal. O(n²) in the number of layers, which matters when you're stacking 80+ of them.

Their practical solution is Block AttnRes: partition layers into blocks and attend over block-level representations instead of individual layer outputs. Same principle, much lower overhead.

They combine this with a two-phase computation strategy and cache-based pipeline communication. The result is a drop-in replacement for standard residual connections that adds minimal training cost.

The Results

They integrated Attention Residuals into the Kimi Linear architecture — 48 billion total parameters with 3 billion activated (a sparse mixture-of-experts setup). Pre-trained on 1.4 trillion tokens.

The improvements show up in two places:

Training stability: More uniform output magnitudes and gradient distribution across network depth. The deep layers aren't drowning in accumulated noise anymore.

Downstream performance: Consistent improvements across standard benchmarks (MMLU, GSM8K, TriviaQA). Not dramatic jumps — this isn't a "we beat GPT" paper. It's a "we found a better foundation to build on" paper.

The scaling law experiments are the most interesting part. The gains from Attention Residuals hold as you scale up model size. That's the signal that this is a fundamental architectural improvement, not a trick that works at one scale.

What This Means For You

If you're running inference on pre-trained models — which most of us are — you don't need to do anything. If Kimi or future models adopt AttnRes, you get the benefit automatically when those weights are published.

If you're fine-tuning or training from scratch (at research scale), this is worth paying attention to. The technique is architecture-level — you'd need to modify the model definition, not just swap a config flag.

The broader implication: we're still finding meaningful improvements in basic transformer plumbing. The residual connection hasn't changed since ResNet in 2015. Eleven years of "good enough" just got challenged with a clean, principled alternative.

For the AI agent systems we build, better base models mean better reasoning at every layer of the stack. An agent that can more effectively utilize deep network representations makes fewer mistakes in multi-step planning. The compounding effect is real.

The Bottom Line

Attention Residuals won't change your workflow tomorrow. But they represent the kind of foundational research that makes next year's models meaningfully better — not through scale alone, but through smarter architecture.

The Kimi team showed that a component we've taken for granted since 2015 still had room for improvement. That's the kind of finding that ages well.

Paper: Attention Residuals — Kimi Team, Moonshot AI.

NVIDIA Container Toolkit: Why the Default Runtime Matters

Guatu — Thu, 02 Apr 2026 20:46:15 +0000

I spent two hours trying to debug why my AI agent container couldn’t find the GPU. The error was cryptic, just a "device not found" in the logs, and I had no idea why. I had installed the NVIDIA Container Toolkit, configured containerd, and even tried running the container with --gpus all — nothing worked.

What I expected was for the container to launch with full GPU access, as I had done this before on a different setup. The container should have detected the GPU, initialized the libraries, and started training my model without a hitch.

What actually happened was that the container was using the default OCI runtime, which wasn’t the NVIDIA runtime. As a result, the NVIDIA libraries weren’t loaded, and the GPU wasn’t accessible. The container didn’t fail outright — it just silently missed the GPU, and the AI agent couldn’t proceed.

The fix came after I remembered to run nvidia-ctk runtime configure --runtime=containerd --set-as-default and restart containerd. That command sets the NVIDIA runtime as the default for all containers, not just those that explicitly request it. Without this step, even if you use --gpus all, the container might still run on the default runtime, which is usually runc or crun, not nvidia.

To make sure the configuration sticks, I added the following to my containerd config under [plugins."io.containerd.grpc.v1.cri".containerd.runtimes]:

nvidia:
  runtime_type: "nvidia"

Then I restarted containerd and verified with:

nvidia-ctk runtime configure --runtime=containerd --set-as-default
systemctl restart containerd

I also made sure that in Kubernetes, the NVIDIA device plugin was running with the correct runtime class. I updated the DaemonSet for the NVIDIA device plugin to explicitly set runtimeClassName: nvidia, which is crucial in newer Kubernetes versions where the default runtime isn’t automatically set.

Why does this matter? If you're running AI agents, LLMs, or any GPU-dependent workloads, and you forget to set the NVIDIA runtime as the default, you’ll run into silent failures. Containers might start, but they won’t see the GPU. Worse, you might not even get an error — just a model that doesn’t train or a container that exits with no clear reason.

This is especially critical in Kubernetes environments where the NVIDIA device plugin relies on the runtime being set correctly. If it’s not, the device plugin won’t register the GPU, and your cluster will report zero GPU capacity.

If you're using Proxmox and running containers with GPU access, or if you're deploying AI agents in a Kubernetes cluster, always make sure the NVIDIA runtime is set as the default. You can do this with the nvidia-ctk tool, and it’s a simple but crucial step. Otherwise, you’ll be chasing cryptic errors and wasted compute time.

For more on how to avoid GPU passthrough gotchas in VMs, check out this post on GPU passthrough on Proxmox. If you're deploying AI agents in Kubernetes, this guide on building multi-agent systems might also help.

Kubernetes Storage on Bare Metal: Longhorn in Practice

Guatu — Thu, 02 Apr 2026 20:45:24 +0000

I spent nearly a week trying to get Kubernetes storage working on my bare metal cluster before I finally figured out the right combination of Longhorn settings, node labels, and storage classes that made it stable. Turns out, most of the pain came from assumptions I made about how storage should work—not the tool itself.

If you're running Kubernetes on physical hardware and need persistent storage—especially for databases, media servers, or AI agents that need state—this is for you. You’ll find the gotchas I hit, the config I used, and why it works better than most other bare-metal storage setups I've tried.

My setup involved a small 3-node Kubernetes cluster, all running on bare metal. I wanted to run PostgreSQL, some AI agents, and a few stateful services like Nextcloud and MinIO. The cluster was deployed with Kubespray and Proxmox as the hypervisor. I tried using hostPath for a while, but that was a nightmare when nodes died or storage filled up.

I tried a few other tools first—Rook-Ceph, RBD with Ceph, and even some DIY iSCSI solutions. None of them clicked the way I needed them to. Rook-Ceph was too heavy, and RBD required a full Ceph cluster. iSCSI was flaky and required a dedicated storage node, which I didn’t want to add. I needed something simple, self-hosted, and easy to manage—and that’s when I turned to Longhorn.

I assumed that Longhorn would be a drop-in solution. I followed a few tutorials and tried to deploy it as-is. The first problem was that my nodes weren’t labeled properly for storage scheduling. I didn’t set node.kubernetes.io/role=worker or longhorn.io/host-attached-storage, so the storage manager wouldn’t assign volumes correctly.

Then I tried to create a PVC and it failed with ProvisioningFailed. I thought it was a storage class issue, but it turned out my disks weren’t labeled correctly in Longhorn. I had three drives on each node, but I forgot to set them to used—Longhorn wasn’t picking them up for volume placement.

I also tried using numberOfReplicas: 3 on a 3-node cluster, which sounded logical. But when I tried to write data to a volume, it started failing because it couldn’t replicate across all three nodes. I had to manually adjust the replicas and found that a 1 replica was actually more stable in my setup.

Here’s how I got Longhorn working reliably. This is based on a 3-node Kubernetes cluster with Proxmox VMs, each with a single storage disk. You can adjust the number of replicas and storage classes based on your cluster size and reliability needs.

Step 1: Install Longhorn via Helm

First, install Longhorn with Helm. I used the official chart from the Longhorn repo.

helm repo add longhorn https://charts.longhorn.io
helm repo update
helm install longhorn longhorn/longhorn --namespace longhorn --create-namespace

This will deploy the Longhorn manager, engine, and UI components.

Step 2: Label Your Nodes

Longhorn uses node labels to determine where to place volumes. Make sure your nodes are labeled properly.

kubectl label nodes <node-name> longhorn.io/host-attached-storage=true
kubectl label nodes <node-name> node-role.kubernetes.io/worker=true

Repeat for all worker nodes. This tells Longhorn that the node has storage available and is suitable for scheduling.

Step 3: Configure Storage Disks in Longhorn UI

Open the Longhorn UI (usually at http://<your-cluster-ip>:30000) and navigate to the Nodes section. For each node, you'll see a list of disks. Make sure each disk you want to use is marked as used.

You can set the disk to used directly in the UI. This tells Longhorn that the disk is available for volume creation.

Step 4: Create a Storage Class

Now create a storage class in Kubernetes. I used the following for a single-replica setup (ideal for 3-node clusters with minimal redundancy but better performance):

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: longhorn-slow
parameters:
  numberOfReplicas: "1"
  staleReplicaTimeout: "2880" # 48 hours
  fromBackup: ""
provisioner: driver.longhorn.io
reclaimPolicy: Delete
volumeBindingMode: Immediate

Apply it with:

kubectl apply -f longhorn-slow.yaml

You can also create a longhorn-fast class with numberOfReplicas: 2 if you want more redundancy.

Step 5: Create a PVC

Now create a PersistentVolumeClaim that uses the storage class you just defined.

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: longhorn-pvc
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 50Gi
  storageClassName: longhorn-slow

Apply it:

kubectl apply -f longhorn-pvc.yaml

And then you can use it in a pod like this:

apiVersion: v1
kind: Pod
metadata:
  name: longhorn-pod
spec:
  containers:
    - name: app
      image: nginx
      volumeMounts:
        - name: longhorn-volume
          mountPath: /usr/share/nginx/html
  volumes:
    - name: longhorn-volume
      persistentVolumeClaim:
        claimName: longhorn-pvc

Apply that and you should have a pod with a persistent volume backed by Longhorn.

Longhorn works well on bare metal because it abstracts away the need for a full storage cluster. It doesn’t require a shared filesystem or a dedicated storage node—just one or more storage disks per node. The key to stability is labeling your nodes correctly and configuring the storage class with the right number of replicas.

Longhorn uses a distributed engine to manage volume replication and snapshotting. When you write data to a volume, Longhorn creates a block device that mirrors the data across replicas. The more replicas you have, the more redundant the data is—but also the more resources it uses.

By setting numberOfReplicas: 1, I was able to get better performance and avoid the issues I had with multi-replica setups in a small cluster. Longhorn’s snapshot system is also great—it allows you to take snapshots and roll back to them if needed, which is especially useful for databases or stateful apps.

Another thing that helps is the staleReplicaTimeout setting. This tells Longhorn how long to wait before removing a stale replica that’s no longer in sync. I set it to 2880 (48 hours) to give myself time to investigate issues before cleanup.

Here’s what I learned after running this setup for a few weeks:

Don’t assume all nodes can host storage: I had one node that wasn’t labeled properly, and that caused volumes to fail to schedule. Make sure all nodes are labeled correctly.
Replicas = redundancy, not performance: Setting numberOfReplicas: 3 in a 3-node cluster didn't improve performance—it just caused more overhead and failures. Stick to 1 or 2 replicas unless you really need it.
Storage classes matter: I had a few PVCs that failed because I didn’t use the right storage class. Make sure your pods are using the correct class for their use case.
Snapshots are powerful but not automatic: Longhorn snapshots work well, but you have to manage them manually unless you set up an external tool like Velero or Kubernetes Event Watcher.
Disk labels are important: If your disks aren’t marked as used in Longhorn, it won’t schedule volumes on them. That was a big gotcha I hit early on.

Overall, I’m happy with how Longhorn works on bare metal. It’s lightweight, reliable, and doesn’t require a full storage cluster. It’s not perfect—there are some quirks regarding scheduling and snapshot management—but for a small to medium-sized cluster, it’s one of the best options I’ve found.

Helm fullnameOverride: Naming Sanity in ArgoCD

Guatu — Thu, 02 Apr 2026 20:45:19 +0000

ArgoCD is great for GitOps, but when you start deploying multiple Helm charts, naming collisions become a real pain. I’ve spent hours digging through logs trying to figure out why a deployment failed because the wrong service was being referenced. That’s where fullnameOverride comes in — it’s not just a feature, it’s a lifeline for keeping your Kubernetes resources uniquely identifiable.

The key is to explicitly set fullnameOverride in your ArgoCD app spec. This ensures that Helm uses a predictable name for your resources, preventing clashes between different apps or environments. Without it, you end up with names like myapp-abc-xyz, which are fine in isolation but disastrous when you scale.

Here’s how to do it in your ArgoCD app definition. This example sets a custom name for a Helm chart, ensuring it doesn’t clash with any other deployments:

spec:
  source:
    repoURL: https://charts.bitnami.com/bitnami
    chart: redis
    targetRevision: 18.1.1
    helm:
      valueOverrides:
        fullnameOverride: "redis-cache-prod"

This change ensures that every deployment of this chart gets the same consistent name, making it easier to manage, troubleshoot, and reference in other services or configurations. It also helps with resource discovery in ArgoCD, avoiding the confusion that comes with ambiguous names.

Don’t underestimate the power of a well-named resource — it’s the difference between a smooth deployment and a debugging nightmare.

DEV Community: Guatu

The 6-Layer Memory Architecture I Run for Claude Code

The layers

What went wrong along the way

What's actually in the repo

What to skip

Why open-source it

Links

Building Karpathy's LLM Wiki: A Production Homelab Implementation

What I Tried First

The Actual Solution

Prerequisites

Deploying the LLM Wiki

Configuring the Application

Why It Works

Lessons Learned

Proxmox API Tokens: Bash History Expansion and the ! Character

AMD iGPU Stealing Your RAM: UMA Frame Buffer on Headless Servers

Agent Credential Management: Two-Tier Service Accounts for Secure AI Agent Workflows

Pod Disruption Budgets: Why kubectl drain Gets Stuck on Longhorn

Equipment Health Scoring: How One Number Made My Operators Stop Checking the Dashboard

Infrastructure as Code, but Automated: OpenTofu and GitHub Actions

What I Tried First

The Actual Solution

1. The Backend Configuration

2. The GitHub Actions Workflow

Why It Works

Lessons Learned

Attention Residuals: How Kimi Is Rethinking Transformer Depth

The Problem Nobody Talks About

What Attention Residuals Actually Do

Block AttnRes: Making It Practical

The Results

What This Means For You

The Bottom Line

NVIDIA Container Toolkit: Why the Default Runtime Matters

Kubernetes Storage on Bare Metal: Longhorn in Practice

Step 1: Install Longhorn via Helm

Step 2: Label Your Nodes

Step 3: Configure Storage Disks in Longhorn UI

Step 4: Create a Storage Class

Step 5: Create a PVC

Helm fullnameOverride: Naming Sanity in ArgoCD