DEV Community: Sabith KS

Evidence-Gated RCA — Prove, Then Narrate

Sabith KS — Wed, 08 Jul 2026 17:00:00 +0000

Evidence-gated multi-plane RCA — fixed DAG, structural evals, and token-aware tool loops for production agent workflows.

Just realized the JSON pain has been already solved thanks to github.com/kaptinlin/jsonrepair Shout out to @kaptinlin

Sabith KS — Fri, 03 Jul 2026 18:41:46 +0000

Sabith KS

Jul 1

Why We Chose Go for Our AI Agent Platform (When Everyone Else Picked Python)

#go #aiagents #architecture #llm

1 min read

You Can’t Debug What You Can’t See — Observability for AI Agents

Sabith KS — Wed, 01 Jul 2026 16:00:00 +0000

Traditional APM can’t tell you why your agent spent $4.72 asking the same question three times.

We’ve been running AI agents in production for 4 months. The hardest part isn’t building them — it’s understanding what they’re doing when they go wrong. Agents don’t crash with stack traces. They loop, hallucinate, burn tokens, and produce plausible-looking output that’s subtly wrong.

Here’s the observability stack we built to see inside.

Why Standard Monitoring Falls Short

Standard application monitoring answers questions like:

Is the service up? (health check)
How fast are responses? (latency P50/P99)
Are there errors? (error rate)

Agent monitoring needs to answer:

Why did this task cost $12 when it usually costs $0.50?
Why did the agent call the same tool 7 times?
Did the agent actually do what it said it did?
Which model is best for this task type?
Is the agent learning, or is it making the same mistakes?

These are fundamentally different questions. You can’t answer them with Prometheus counters and Grafana dashboards alone.

The Three Pillars for Agents

1. Traces — The Session Timeline

Every agent session produces a trace. Not an APM trace — an agent trace that captures the full decision history:

Session: "Investigate production latency spike"
├─ LLM Call #1: Plan generation (tokens: 1200 in, 340 out, $0.004)
├─ Tool: web_search("production latency monitoring") → 3 results
├─ LLM Call #2: Analyze results (tokens: 2100 in, 890 out, $0.008)
├─ Tool: run_shell("kubectl top pods -n production") → approved, 1.2s
├─ Sub-agent: "Check database metrics"
│ ├─ LLM Call #3: Sub-plan (tokens: 800 in, 200 out, $0.002)
│ ├─ Tool: run_shell("psql -c 'SELECT * FROM pg_stat_activity'")
│ └─ LLM Call #4: Analysis (tokens: 3400 in, 1200 out, $0.012)
├─ LLM Call #5: Synthesize findings (tokens: 4200 in, 1800 out, $0.018)
└─ Tool: send_message("Root cause: connection pool exhaustion...")

Total: 5 LLM calls, 3 tool calls, 1 sub-agent, $0.044, 47 seconds

We use Langfuse as our trace backend. Every LLM call, tool execution, and sub-agent delegation is a span. Traces nest — sub-agent traces are children of the parent trace.

Trace delivery is non-blocking — we use OpenTelemetry’s batch exporter pipeline so tool execution is never waiting on a synchronous HTTP POST to the tracing backend. The OTel SDK buffers spans in memory and flushes them periodically. On pod shutdown, a graceful drain hook flushes remaining spans before the process exits. If the trace backend is temporarily unreachable, spans are dropped after the buffer fills — you lose telemetry, not availability.

2. Costs — The $4.72 Question

Token costs are the unit economics of agents. We track:

Per-session:
  Total cost $0.044
  Input tokens 11,700
  Output tokens 4,430
  Model breakdown:
    claude-sonnet: $0.038 (4 calls)
    gemini-flash: $0.006 (1 call, efficiency task)

Per-agent (daily):
  sre-copilot: $12.40 (28 sessions)
  security-analyst: $3.20 (7 sessions)
  dev-assistant: $18.90 (42 sessions)

Why this matters: An agent that loops — calling the same tool repeatedly because it can’t make progress — burns tokens geometrically. A 10-iteration loop on Claude Sonnet costs 10× a single call. Without cost monitoring, you discover this when the invoice arrives.

Proactive guardrails: Reactive alerting alone isn’t fast enough — a tight exception loop in a parallel agent can burn through dollars in seconds before a Slack webhook fires. So cost management is layered: hard iteration caps (MaxIterations), per-tool call budgets (ToolBudgets), and loop detection middleware that blocks identical consecutive calls all act as pre-flight circuit breakers in the execution pipeline. Alerts are the second line of defense, not the first.

Alerting: We alert when a single session exceeds 3× the rolling average cost for that agent. This catches the slower-burning anomalies — hallucination spirals, model routing errors, and gradually accumulating context — that slip past the hard limits.

3. Audit — The Immutable Record

Every tool call, governance decision, and memory operation is logged to an append-only NDJSON file:

{"ts":"...","event":"tool_call","tool":"run_shell","args":{"cmd":"kubectl get pods"},"decision":"auto_approved","middleware_ms":2}
{"ts":"...","event":"tool_result","tool":"run_shell","status":"success","output_bytes":1247,"redacted":false}
{"ts":"...","event":"hitl_pending","tool":"kubectl_apply","request_id":"abc123"}
{"ts":"...","event":"hitl_approved","request_id":"abc123","approver":"sabith","latency_s":6.2}
{"ts":"...","event":"memory_store","type":"episodic","goal":"investigate latency","confidence":0.82}

The audit trail is PII-redacted. Tool outputs that contain sensitive data are sanitized before logging. You can trace what happened without exposing credentials.

The Diagnostic CLI

We built genie doctor (think brew doctor) — a diagnostic command that checks agent health:

$ genie doctor

✅ Model connectivity: claude-sonnet, gemini-flash (2/2 reachable)
✅ Vector store: qdrant at localhost:6334 (healthy, 1,247 vectors)
✅ HITL: 0 pending approvals
✅ Memory: 42 episodic memories, 12 skills, 8 notes
⚠️ Langfuse: connected but 3 failed trace uploads in last hour
❌ MCP server "datadog": connection refused
   → Last successful connection: 2 hours ago
   → Try: npx -y @datadog/mcp-server --check

One command tells you if the agent’s dependencies are healthy. No digging through logs, no checking 5 dashboards.

Trace Analysis — Automated Session Reviews

Raw traces are useful for debugging individual sessions. But with 50+ agents running hundreds of sessions daily, you can’t review them all manually.

We built a trace analyzer that produces automated session breakdowns:

## Session Analysis: sre-copilot (session-abc123)

**Request:**"Why is the API slow?"
**Duration:** 47s | **Cost:** $0.044 | **Tools:** 3 | **LLM calls:** 5

### Efficiency Assessment
- ✅ No tool loops detected
- ✅ Sub-agent completed successfully
- ⚠️ High input token count on call #5 (4,200 tokens)
  → Consider: Trim sub-agent output before synthesis

### Cost Breakdown
| Call | Model | Tokens (in/out) | Cost |
|------|-------|-----------------|------|
| Plan | claude-sonnet | 1,200 / 340 | $0.004 |
| Analyze | claude-sonnet | 2,100 / 890 | $0.008 |
| Sub-plan | gemini-flash | 800 / 200 | $0.002 |
| Sub-analyze | claude-sonnet | 3,400 / 1,200 | $0.012 |
| Synthesize | claude-sonnet | 4,200 / 1,800 | $0.018 |

The analyzer runs on every trace. Anomalous sessions (loops, high cost, tool errors) are flagged for human review.

Prometheus Metrics

For real-time dashboards and alerting, we export metrics:

# Tool call success/failure rates
toolwrap_tool_call_total{tool="run_shell",outcome="success"} 142
toolwrap_tool_call_total{tool="run_shell",outcome="failure"} 3

# Per-agent session costs
agent_session_cost_usd{agent="sre-copilot"} 0.044

# HITL approval latency
hitl_approval_latency_seconds{quantile="0.5"} 6.2
hitl_approval_latency_seconds{quantile="0.99"} 45.1

# Semantic router classification
semantic_router_classification_total{route="operations",tier="L1"} 89
semantic_router_classification_total{route="jailbreak",tier="L0"} 2

These feed into standard Grafana dashboards for the operations team. They don’t replace Langfuse traces — they complement them with real-time alerting.

A warning on cardinality: Keep Prometheus labels low-cardinality. Labels like tool="run_shell" or agent="sre-copilot" are safe — they have bounded values. Never put unique identifiers like session_id or run_id into Prometheus labels. A production system running thousands of agent sessions daily will cause a high-cardinality explosion that bloats memory and crashes the Prometheus server. Leave per-session details to your tracing backend (Langfuse) or structured logs.

What We Monitor

What	How	Alert Threshold
Session cost	Langfuse traces	> 3× rolling average
Tool loop	Middleware counter	> 2 identical consecutive calls
HITL latency	Prometheus histogram	> 5 minutes (approval stale)
Model errors	Prometheus counter	> 5% error rate in 5 minutes
Vector store health	`genie doctor` cron	Connection refused
MCP server health	Heartbeat check	3 missed heartbeats
Token burn rate	Daily aggregation	> 2× daily budget
Audit file growth	Filesystem monitor	> 1GB/day (potential loop)

Lessons Learned

Cost is your canary. Sudden cost spikes almost always indicate a bug — loops, model routing errors, or unbounded context accumulation. Alert on cost first, debug second.
Traces are for debugging, metrics are for alerting. Don’t try to alert on traces (too detailed) or debug with metrics (too aggregated). Use both.
Audit PII-redaction is non-negotiable. Your audit trail will be queried during incident reviews. If it contains credentials or PII, your observability tool becomes a liability.
Build a diagnostic CLI. genie doctor saves more time than any dashboard. One command, all dependencies, clear pass/fail.
Automate trace analysis. You can’t review 200 sessions a day manually. Let the analyzer flag anomalies; humans review the flags.

What observability tools do you use for your agent platform? I’m especially interested in cost monitoring and loop detection approaches. Find me on GitHub or LinkedIn.

🚀 We’re building AI-powered SRE at StackGen. If you’re tired of 3 AM pages and want AI agents that triage incidents, run diagnostics, and draft RCA reports — check out ai.stackgen.com and try our new SRE offering.

The HITL Paradox — When Human Approval Makes Agents Worse

Sabith KS — Wed, 01 Jul 2026 14:00:00 +0000

Human-in-the-loop (HITL) is supposed to make agents safer. Put a human between the agent and the dangerous action. Simple.

In practice, HITL has a paradox: too much approval kills productivity, too little kills safety, and the wrong amount creates a false sense of security.

We deployed HITL for our agent runtime and watched three failure modes emerge. Here’s what happened and how we fixed each one.

Failure Mode 1: Approval Fatigue

Our first HITL deployment required approval for every tool call. Shell commands, web searches, memory reads — everything needed a human click.

Within two days, operators were auto-approving everything without reading the details. The approval popup became muscle memory: see popup → click approve → continue.

The data: We tracked approval latency. In week 1, operators spent an average of 8 seconds reviewing each request. By week 2, it was under 2 seconds. They weren’t reviewing — they were dismissing.

Why this is worse than no HITL: Operators now believe they have a safety net. They don’t. The safety net is a rubber stamp. But everyone — operators, managers, auditors — thinks the system is reviewed because “human approval is required.”

The Fix: Risk-Based Classification

We classified tools into three tiers:

[hitl]
# Never needs approval — safe, read-only, or internal
always_allowed = ["web_search", "memory_*", "read_*", "discover_skills", "note"]

# Needs approval — can modify external state
# (This is the default for any tool not in always_allowed)

# Never allowed — blocked entirely, regardless of approval
denied_tools = ["bash", "shell_*"]

An important subtlety: these lists match tool names, not shell command strings. denied_tools = ["bash"] blocks the tool named bash from being invoked at all — it doesn’t do regex matching against command arguments passed to run_shell. String-level blocklisting on shell primitives (e.g., blocking “rm” as a substring) is fundamentally unsafe — any sufficiently creative LLM can bypass it via base64 encoding, variable interpolation, or aliasing. Instead, the HITL gate operates at the tool invocation boundary : run_shell as a whole requires human approval, and the human sees the full command in the approval request. If you need granular command-level control, the right approach is typed Go API clients (e.g., a Kubernetes Go client with RBAC) instead of raw shell access.

Only state-modifying tools require approval. Read-only operations auto-approve. Destructive tool names hard-block regardless of approval.

Result: Approval requests dropped by 70%. Operators now see 3-5 requests per task instead of 20+. Each request is meaningful — they actually read them.

Failure Mode 2: The `always_allowed = ["*"]` Escape Hatch

Some teams set always_allowed = ["*"] to skip all approvals. They’d been burned by approval fatigue and decided HITL wasn’t worth the friction.

This defeats the entire purpose of governance. An agent with always_allowed = ["*"] can execute any tool without review — including shell commands on production servers.

The Fix: Guardrails on the Guardrails

We added warnings when always_allowed contains wildcards:

⚠️ Warning: always_allowed contains "*" — all tools will bypass 
HITL approval. This includes run_shell, kubectl, and other 
state-modifying tools. Are you sure?

The real safeguard is the denied_tools list — even when always_allowed = ["*"], any tool in denied_tools is hard-blocked. So teams that want minimal friction can set always_allowed = ["*"] while keeping the most dangerous tool names denied:

[hitl]
always_allowed = ["*"]
denied_tools = ["bash", "shell_*"]

This gives teams the fast workflow they want while maintaining hard gates on the most dangerous operations.

Failure Mode 3: Blocking on Approval Halts Everything

Early HITL was synchronous — the agent stopped working and waited for approval. If the operator was in a meeting, the agent sat idle for 45 minutes waiting for a click.

For a single approval, this is annoying. For a task requiring 5 approvals across different tools, the total wait time could exceed the task’s useful lifetime.

The Fix: Asynchronous Approval

HITL approval is now asynchronous:

Agent encounters a tool that requires approval
Stores the pending request in the database with a TTL (default: 30 minutes)
Sends a notification via the event bus (Slack, web UI, AG-UI protocol)
Continues working on other parts of the task
When approved, the tool executes and results flow back

The agent doesn’t block. If it has parallel sub-tasks, it works on those while waiting. If there’s nothing else to do, it waits — but the user sees a clear “waiting for approval” status, not a mysteriously silent agent.

A note on state drift: Asynchronous approval introduces a classic distributed systems risk — the environment state may change between when the agent formulated the tool call and when a human approves it 20 minutes later. We mitigate this with short approval TTLs (stale approvals auto-expire via a background reaper) and session-scoped approval caching that expires entries after 10 minutes, ensuring that long-deferred approvals don’t execute against a drifted environment without the agent re-evaluating.

Batch operations: Operators can view multiple pending requests at once via the ListPending API, grouped by tool name:

Pending approvals (3):
  [✅ Approve All] [❌ Reject All]

  🔧 run_shell: kubectl get pods -n production
  🔧 run_shell: kubectl describe pod api-server-7d8f
  🔧 run_shell: kubectl logs api-server-7d8f --tail=50

One important guardrail: the “Approve All” pattern works well for read-only investigation commands like the above. For state-modifying operations, each approval should be reviewed individually — otherwise you recreate the rubber-stamp problem at a higher abstraction level.

What happens on rejection? When a human rejects a tool call (with or without feedback), the middleware returns an ErrToolCallRejected error to the agent’s context. This isn’t a hard cancellation — the LLM receives the rejection as a tool error and can replan. If the human provided feedback (e.g., “use the staging cluster instead”), the agent sees it and can adjust its approach. This gives operators a conversational override, not just a binary approve/deny gate.

The Hidden Bug: HITL Bypass on Sub-Agents

This was a real security issue. When our agent delegated to sub-agents via ReAcTree, the sub-agent’s tools were bound directly from the registry — without the HITL middleware wrapper.

A sub-agent could run run_shell without approval, even though the parent agent required it.

Why it happened: The sub-agent tool binding was written before HITL existed. When we added HITL, we wrapped the parent’s tools but forgot the sub-agent delegation path.

The fix: All tool binding — parent, sub-agent, plan-step, fallback — goes through the same ToolWrapSvc middleware chain. One path. One governance stack. No exceptions.

The lesson: When you add a governance layer, you must audit every tool execution path. The path you forget is the one that gets exploited.

What Good HITL Looks Like

After three iterations, here’s our current model:

Tool Type	Behavior	Example
Read-only	Auto-approve	`web_search`, `memory_search`, `read_file`
Informational	Auto-approve	`discover_skills`, `list_pods`
State-modifying	Require approval	`run_shell`, `commit_code`, `create_pr`
Destructive	Hard deny	`rm -rf`, `kubectl delete namespace`, `DROP TABLE`
Memory writes	Exempt (not state)	`memory_manage`, `note`

The exemption for memory writes is important. Memory tools modify the agent’s internal state, not external systems. Requiring approval for every memory_manage call would trigger approval fatigue without adding safety — the agent is only modifying its own notes.

Metrics That Matter

Track these to know if your HITL system is working:

Approval latency — If it drops below 3 seconds, operators aren’t reading requests
Approval rate — If it’s above 95%, you’re probably approving too aggressively
Rejection rate — If it’s below 1%, either your agent is perfect or nobody is paying attention
Time-to-abandon — How long before operators set always_allowed = ["*"]

Our current numbers: ~6 second average approval latency, 88% approval rate, 7% rejection rate, 5% auto-expired (operator didn’t respond in time).

Lessons Learned

Less approval is more safety. Fewer, higher-signal approval requests get more attention than constant popups.
Classify tools by risk, not by category. Not all shell commands are dangerous. kubectl get pods is read-only; kubectl delete pod is not.
Make approval asynchronous. Synchronous blocking kills agent productivity and operator patience.
Audit every tool path. HITL that applies to 90% of tool calls creates a false sense of security. The 10% that bypasses it is where the risk lives.
Memory tools are not external state. Don’t require approval for internal memory operations — it’s noise that drowns out real signals.

How does your team handle the approval fatigue problem? I’d love to hear about alternative approaches. Find me on GitHub or LinkedIn.

🚀 We’re building AI-powered SRE at StackGen. If you’re tired of 3 AM pages and want AI agents that triage incidents, run diagnostics, and draft RCA reports — check out ai.stackgen.com and try our new SRE offering.

Pensieve — Memory Management for AI Agents That Actually Forget

Sabith KS — Wed, 01 Jul 2026 12:00:00 +0000

Your agent remembers everything. That’s a bug, not a feature.

We’ve all seen it: you give an agent a task, it retrieves 40 “relevant” context chunks from a vector store, stuffs them into a 128K context window, and produces a response that’s technically accurate but practically useless — because 35 of those chunks were irrelevant, stale, or contradictory.

We built a memory system called Pensieve that handles this differently. Instead of “remember everything and search later,” Pensieve manages four distinct memory types , with automatic decay, importance scoring, and self-pruning. This post walks through the architecture, the algorithms, and the production lessons.

Why Naive RAG Fails for Agents

RAG (Retrieval-Augmented Generation) works brilliantly for Q&A systems. You have a corpus of documents, you embed them, and you retrieve the most similar chunks for a user question.

Agents are different:

Agents generate memories at runtime. Every task produces new experiences — what worked, what failed, what the user corrected. The corpus grows with every interaction.
Agent memories have temporal relevance. “The staging API was down” was true yesterday. Retrieving it today makes the agent avoid an API that’s working fine.
Agent memories have quality variation. Successful task completions, failed attempts, hallucinated outputs, and user corrections all go into the same store. Quality varies wildly.
Agents need structured recall, not just similarity. “What skills do I have for Kubernetes troubleshooting?” is a different retrieval mode than “find text similar to ‘pod crashloopbackoff’.”

Standard RAG treats all chunks equally — same embedding, same retrieval, same ranking. Agents need curated, time-aware, quality-gated memory.

The Four Memory Types

Pensieve manages four distinct memory stores, each with different lifecycle and retrieval semantics:

┌──────────────────────────────────────────────────────────┐
│ Agent Memory │
├──────────────┬──────────────┬──────────┬─────────────────┤
│ Working │ Episodic │ Notes │ Skills │
│ Memory │ Memory │ │ │
├──────────────┼──────────────┼──────────┼─────────────────┤
│ Session │ Goal-keyed │ Cross- │ Reusable │
│ blackboard │ experiences │ session │ procedures │
│ │ │ facts │ │
├──────────────┼──────────────┼──────────┼─────────────────┤
│ Lifetime: │ Lifetime: │ Lifetime:│ Lifetime: │
│ Single task │ Decays over │ Until │ Until │
│ │ ~2 weeks │ deleted │ deprecated │
├──────────────┼──────────────┼──────────┼─────────────────┤
│ No embedding │ Vector + │ Key- │ Semantic │
│ (key-value) │ weighted │ value │ search │
│ │ retrieval │ lookup │ │
└──────────────┴──────────────┴──────────┴─────────────────┘

1. Working Memory — The Session Blackboard

Working memory is a key-value store scoped to a single task execution. When a parent agent delegates to multiple sub-agents via ReAcTree, working memory is the shared blackboard:

Parent: "Investigate production outage"
  ├─ Sub-agent 1: writes working_memory["log_analysis"] = "OOM killer triggered at 14:32"
  ├─ Sub-agent 2: writes working_memory["metric_summary"] = "Memory usage spiked from 2GB to 8GB"
  └─ Parent reads both entries to synthesize RCA

Working memory dies when the task completes. It’s not persisted. Think of it as function-scoped variables.

2. Episodic Memory — Experiences with Expiration Dates

Episodic memory stores what happened during past tasks — the goal, the approach, the outcome, and what was learned. It’s the agent’s autobiography.

The critical design choice: not all episodes are worth remembering.

// Store episodes with explicit status tagging
if !output.looksLikeError() {
    episodic.Store(ctx, Episode{
        Goal: goal,
        Trajectory: trajectory,
        Status: EpisodePending, // promoted to Success on user validation
        Importance: scoreImportance(ctx, goal, trajectory),
    })
}

This is status-aware storage. Episodes start as pending and are promoted to success only after user validation (e.g., a 👍 emoji reaction). Failed tasks are stored separately with verbal reflections (more on this below). Raw error traces never pollute the memory — only synthesized lessons.

(In the ReAcTree bugs post, I mentioned that naively storing failures poisoned the agent. That’s true for raw execution traces — saving a 50-line stack trace of a timeout pollutes context. We evolved this: we still drop the raw failure log, but we pass the event to a FailureReflector to synthesize a concise, one-sentence lesson, which we safely store with a failure status and a verbal reflection.)

Retrieval Scoring

Episodic memories are ranked using a three-signal weighted score that combines semantic similarity, temporal recency, and importance:

final_score = 0.4 × cosine_similarity + 0.3 × recency + 0.3 × importance

recency = e^(-0.01 × hours_since_created) // exponential decay, [0,1]
importance = importance_score / 10 // normalized from 1-10 to [0,1]

A memory from 1 hour ago has a recency score of ~0.99. A memory from 1 week (168 hours) ago has a recency score of ~0.19. After ~2 weeks, the recency component effectively drops to zero.

Why three signals instead of two? With only recency and importance (the naive approach), you get a mathematical problem: if recency dominates (say, 60% weight), then any memory older than 4-5 days will always be outranked by a completely routine memory from the last hour — even a critical production incident. By adding cosine similarity as the largest component (40%), the system surfaces memories that are semantically relevant to the current task first, then uses recency and importance as tiebreakers. A week-old production incident retrieves strongly when the current task involves a similar failure mode.

Why this matters: An agent that investigated a DNS issue last week shouldn’t treat that experience the same as investigating the same issue 5 minutes ago. But it also shouldn’t forget a critical production incident just because a routine health check happened more recently. The three-way weighting handles both cases.

Importance Scoring

When an episode is stored, a lightweight LLM call scores it 1-10:

"Deployed a hotfix to production under time pressure" → 9/10
"Ran a routine health check, everything was green" → 2/10

Unscored episodes (importance = 0) receive a neutral 0.5 weight so they don’t dominate or disappear from retrieval.

3. Notes — Cross-Session Persistence

Notes are simple key-value pairs that persist across sessions:

notes["user_preference_timezone"] = "US/Pacific"
notes["team_oncall_rotation"] = "PagerDuty schedule ID: P123ABC"
notes["k8s_cluster_prod"] = "us-east-1, EKS 1.31, 47 nodes"

Notes don’t decay. They’re for facts that change rarely and apply broadly. The agent manages its own notes — it can create, read, update, and delete them as tool calls.

4. Skills — Reusable Procedures

Skills are structured documents that describe how to do something — step-by-step procedures with failure handling:

# Skill: Kubernetes Pod Crashloop Triage

## Steps
1. Get pod status: `kubectl get pods -n {namespace} | grep CrashLoopBackOff`
2. Check pod events: `kubectl describe pod {pod_name} -n {namespace}`
3. Read last 100 log lines: `kubectl logs {pod_name} -n {namespace} --tail=100`
4. Check resource limits vs actual usage
5. Check if recent deployments changed the image or config

## Common Causes
- OOM kills → check memory limits
- Missing config/secrets → check configmap/secret mounts
- Image pull failures → check registry access

Skills are stored on the filesystem and indexed in a vector store for semantic discovery. When an agent starts a task, it searches for relevant skills and loads them into context.

The Self-Pruning Agent

Here’s where it gets interesting. In most agent frameworks, you (the developer) manage memory — you decide what to store, what to retrieve, how much context to inject.

In Pensieve, the agent manages its own memory budget. It has memory management tools:

Tool	Purpose
`memory_search`	Semantic search across episodic memories
`memory_manage`	Save, update, or delete memories
`note`	Read or write persistent notes
`read_notes`	List all notes
`discover_skills`	Search for relevant skills
`load_skill`	Load a skill into working context

The agent’s system prompt includes instructions to manage its context proactively:

“Before starting work, search memory for relevant past experiences and available skills. If your context is getting large, offload resolved information to notes and prune completed sub-task context.”

Why this works: The agent knows what information it needs for the current step better than any static retrieval algorithm. By giving it memory management tools, we let it curate its own context window.

Why this is risky: An agent can delete useful memories or fail to store important ones. We mitigate this with audit logging — every memory operation is logged to an immutable audit trail, so we can reconstruct what happened. Sub-agents get a restricted view: they can read and search memory, but writes flow through the parent agent’s middleware stack with the same governance controls as any other tool call.

The Learning Loop

After every completed task, a background process evaluates whether the experience is worth remembering as a reusable skill :

Task completes
  → Novelty scoring (LLM rates 1-10)
  → If novelty ≥ 7:
      → Distill into structured skill document
      → Semantic dedup check (similarity ≥ 0.8 → merge or skip)
      → Store to filesystem + vector index

This is post-session skill distillation. The agent doesn’t learn during the task — it learns after, asynchronously, without blocking the user.

Example: An agent successfully triages a Redis connection storm for the first time. The learning loop:

Scores it 8/10 novelty (agent hasn’t handled Redis issues before)
Distills the approach into a skill document
Checks if a similar skill exists (none found)
Stores it as dynamic_skills/redis_connection_storm_triage.md

Next time a Redis issue comes up, the agent finds this skill via discover_skills and follows the documented procedure.

Failure Learning

Successes teach you what to do. Failures teach you what to avoid. We capture both.

When an agent fails a task (timeout, too many errors, explicit failure status), a FailureReflector generates a verbal reflection:

Goal: "Scale the production database replica set"
Status: Failed
Reflection: "Attempted to modify replica count without checking 
if the cluster was in maintenance mode. The API returned 403 
Forbidden. Next time, verify cluster status before making 
scaling changes."

These failure reflections are stored as episodic memories with a ⚠️ prefix. When the agent encounters a similar task, it retrieves both successful experiences and past failures:

## Relevant Experience
✅ Successfully scaled Redis cluster by updating replica count 
   after confirming maintenance window (2 days ago)

⚠️ Failed to scale database replica set — didn't check 
   maintenance mode first, got 403 (5 days ago)

The agent sees what worked and what didn’t. This is inspired by Reflexion — verbal reinforcement without weight updates.

Daily Wisdom Consolidation

Individual episodic memories accumulate. Over time, retrieving them all is expensive and noisy.

Once per day, an EpisodeConsolidator reads recent episodes and summarizes them into wisdom notes — concise bullet-point lessons:

## Consolidated Lessons (July 1, 2026)

- When scaling database replicas, always check cluster 
  maintenance mode status first (learned from failed attempt)
- Redis connection storms usually indicate connection pool 
  exhaustion in the application, not Redis server issues
- PagerDuty incident creation requires the service_id field; 
  use the pd_service_list tool to find it first

Wisdom notes are injected into the agent’s system prompt as a ## Consolidated Lessons section, capped at the 2-3 most recent notes. They provide distilled experience without the noise of individual episodes.

To prevent the wisdom section from growing unboundedly, the consolidator naturally limits itself: it retrieves a small, fixed window of recent wisdom notes per prompt injection. Older notes still exist in storage but aren’t injected — they’ve served their purpose by informing the agent during their active window, and the lessons they encode are either still relevant (and get re-learned) or have become stale. The consolidation job also deletes the raw episodes it summarized from the vector store, so the episodic memory stays clean.

The PII Problem

Agent memories contain user conversations, tool outputs, API responses — all potentially containing PII (names, emails, IP addresses, tokens).

We run PII redaction before persisting any memory:

// All memory storage goes through PII redaction
func (s *Store) Save(ctx context.Context, req SaveRequest) error {
    req.Content = pii.Redact(req.Content)
    req.Goal = pii.Redact(req.Goal)
    return s.backend.Save(ctx, req)
}

The pii.Redact() function strips emails, bearer tokens, API keys, and other sensitive patterns before persistence.

There’s a domain-specific tension here: our product is an SRE copilot, and in SRE contexts, IP addresses and hostnames are critical telemetry. If the agent remembers “the outage was caused by a rogue pod on node [REDACTED],” the memory is functionally useless for future debugging. We handle this by allowlisting internal RFC-1918 subnets and using deterministic tokenization for external addresses — 203.0.113.42 consistently becomes [EXT_IP_A] across memories, so the agent can still learn network topology patterns without storing regulated PII in the vector store.

Architecture Summary

┌─────────────────────────────────────────────────────┐
│ Agent Prompt │
│ ├── Consolidated wisdom (injected automatically) │
│ ├── Episodic memories (retrieved per-goal) │
│ ├── Skills (loaded on demand) │
│ └── Notes (read via tool call) │
├─────────────────────────────────────────────────────┤
│ Memory Management Tools │
│ memory_search, memory_manage, note, │
│ discover_skills, load_skill │
├─────────────────────────────────────────────────────┤
│ Storage Layer │
│ ├── Vector store (Qdrant) — embeddings │
│ ├── Filesystem — skills, notes │
│ └── PII redaction — applied before persist │
├─────────────────────────────────────────────────────┤
│ Background Processes │
│ ├── Learning loop — skill distillation │
│ ├── Wisdom consolidation — daily digest │
│ └── Memory decay — exponential time-based │
└─────────────────────────────────────────────────────┘

Lessons Learned

Memory is a data quality problem. Treat it like a database — validate before insert, enforce schema, handle duplicates.
Temporal decay is essential. Agents operate in changing environments. Yesterday’s truths can be today’s hallucinations.
Let agents manage their own context. Static retrieval algorithms can’t know what the agent needs at each step. Give it memory tools and let it curate.
Separate memory types for different lifetimes. Working memory (seconds), episodic (weeks), notes (permanent), skills (permanent until deprecated). Mixing lifetimes causes stale context pollution.
Failure memories are as valuable as success memories — but they must be clearly labeled. An agent should learn “don’t do X” without concluding “X is impossible.”
PII redaction is non-negotiable. Agents process sensitive data. Memory stores are search targets. Unredacted PII in a vector store is a compliance incident waiting to happen.

The Comparison

Feature	Standard RAG	LangGraph Memory	Mem0	Pensieve
Memory types	1 (chunks)	2 (checkpointer + Store)	Centralized (user/session)	4 (working, episodic, notes, skills)
Temporal decay	None	Manual / custom	Implicit (managed platform)	Exponential (configurable λ)
Quality gates	None	Manual / custom	Manual / custom	Status-aware + importance scoring
Failure learning	None	Manual / custom	None	Reflexion-style verbal reflections
Self-pruning	None	Manual / custom	None	Agent-managed via tools
Skill distillation	None	None	None	Post-session, novelty-gated
PII redaction	Manual	Manual	Manual	Automatic before persist

How does your agent handle memory? I’m especially interested in approaches to temporal decay and memory quality. Find me on GitHub or LinkedIn.

🚀 We’re building AI-powered SRE at StackGen. If you’re tired of 3 AM pages and want AI agents that triage incidents, run diagnostics, and draft RCA reports — check out ai.stackgen.com and try our new SRE offering.

Why We Chose Go for Our AI Agent Platform (When Everyone Else Picked Python)

Sabith KS — Sat, 20 Jun 2026 17:00:00 +0000

Every AI framework is in Python. We built ours in Go. Here's why we'd do it again — and when you shouldn't.

Originally published at Production Notes

DEV Community: Sabith KS

Evidence-Gated RCA — Prove, Then Narrate

Just realized the JSON pain has been already solved thanks to github.com/kaptinlin/jsonrepair Shout out to @kaptinlin

Why We Chose Go for Our AI Agent Platform (When Everyone Else Picked Python)

You Can’t Debug What You Can’t See — Observability for AI Agents

Why Standard Monitoring Falls Short

The Three Pillars for Agents

1. Traces — The Session Timeline

2. Costs — The $4.72 Question

3. Audit — The Immutable Record

The Diagnostic CLI

Trace Analysis — Automated Session Reviews

Prometheus Metrics

What We Monitor

Lessons Learned

The HITL Paradox — When Human Approval Makes Agents Worse

Failure Mode 1: Approval Fatigue

The Fix: Risk-Based Classification

Failure Mode 2: The always_allowed = ["*"] Escape Hatch

The Fix: Guardrails on the Guardrails

Failure Mode 3: Blocking on Approval Halts Everything

The Fix: Asynchronous Approval

The Hidden Bug: HITL Bypass on Sub-Agents

What Good HITL Looks Like

Metrics That Matter

Lessons Learned

Pensieve — Memory Management for AI Agents That Actually Forget

Why Naive RAG Fails for Agents

The Four Memory Types

1. Working Memory — The Session Blackboard

2. Episodic Memory — Experiences with Expiration Dates

Retrieval Scoring

Importance Scoring

3. Notes — Cross-Session Persistence

4. Skills — Reusable Procedures

The Self-Pruning Agent

The Learning Loop

Failure Learning

Daily Wisdom Consolidation

The PII Problem

Architecture Summary

Lessons Learned

The Comparison

Why We Chose Go for Our AI Agent Platform (When Everyone Else Picked Python)

Failure Mode 2: The `always_allowed = ["*"]` Escape Hatch