DEV Community: Mukesh

Inside the Memory Decision Loop: How AI Agents Decide What to Remember, Update, or Forget

Mukesh — Sun, 06 Sep 2026 19:19:34 +0000

Most people who add "memory" to an AI agent do the same thing: embed every message, throw the vector into Pinecone or pgvector, and call similarity_search at query time. It works for a demo. It falls apart in production, because nothing ever updates or deletes anything — the store only grows, and it fills with contradictions.

Say a user tells your agent "I live in Austin" in March and "I just moved to Denver" in July. A naive vector store keeps both. At retrieval time, both come back as top-k matches for "where do I live," and now your LLM is holding two contradictory facts with no signal about which one is current. This is the actual problem memory layers like Mem0 are built to solve, and the mechanism is more interesting than "vector DB with extra steps." Here's how it actually works.

The pipeline has four stages, not one

A naive RAG setup has one stage: embed and store. A real memory layer has four:

Extraction — turn a raw conversation turn into candidate facts
Retrieval — find existing memories that might relate to each candidate
Decision — for each candidate, decide ADD, UPDATE, DELETE, or NOOP against what's already stored
Consolidation — write the resolved state back, not just append to it

The first stage is the one everyone builds. The second and third are the ones that make memory actually useful instead of a growing pile of unreconciled embeddings.

Stage 1: extraction is a compression step, not a copy step

You don't embed the raw message "yeah so I just moved to Denver last week, still unpacking boxes everywhere." You run it through an LLM extraction prompt that pulls out the durable fact: user.location = Denver. Everything about boxes and unpacking is conversational noise — useful for the current turn, useless six weeks from now.

EXTRACTION_PROMPT = """
Extract durable facts about the user from this message.
Ignore small talk, emotional tone, and one-off requests.
Return a list of atomic facts as short subject-predicate-object statements.

Message: {message}
"""

candidates = llm.extract(EXTRACTION_PROMPT.format(message=turn))
# -> ["user lives in Denver"]

Atomic matters here. "User lives in Denver and works remotely and has a dog" should become three separate candidate facts, not one blob — because each one might independently need to be added, updated, or deleted later, and you can't do that to a fact if it's welded to two unrelated ones.

Stage 2: retrieval finds what might conflict, not what's relevant

This is the part that's easy to get backwards. At write time, you're not doing similarity search to answer a question — you're doing it to find memories that the new candidate might contradict or refine. So for "user lives in Denver", you embed the candidate and pull the top few nearest existing memories:

candidate_embedding = embed("user lives in Denver")
neighbors = vector_store.search(candidate_embedding, k=5)
# -> [("user lives in Austin", score=0.89), ("user has a dog", score=0.31), ...]

The high-scoring neighbor (user lives in Austin, 0.89 similarity) is exactly the case that matters — semantically close enough to be about the same fact, but not identical. That's the conflict the next stage has to resolve.

Stage 3: the actual decision — this is the part people skip

For each candidate + its neighbors, a second LLM call (or in leaner implementations, a classifier) decides what to do. Mem0's public writeups describe exactly this operation: given a new fact and its nearest existing memories, output one of ADD, UPDATE, DELETE, or NOOP.

DECISION_PROMPT = """
New fact: {candidate}
Existing related memories: {neighbors}

Decide the operation:
- ADD: new fact, no real overlap with existing memories
- UPDATE: new fact supersedes an existing memory (same subject, changed value)
- DELETE: new fact explicitly contradicts and invalidates an existing memory
- NOOP: new fact is already captured, do nothing

Return: {{"operation": ..., "target_id": ..., "resolved_fact": ...}}
"""

decision = llm.decide(DECISION_PROMPT.format(
    candidate="user lives in Denver",
    neighbors=[("m_204", "user lives in Austin")]
))
# -> {"operation": "UPDATE", "target_id": "m_204", "resolved_fact": "user lives in Denver"}

This is the step that turns a vector store into a memory system. Without it, you have an append-only log with a fuzzy search index bolted on. With it, you have something closer to a mutable key-value store where the "key" is semantic similarity instead of an exact string.

The failure mode to watch for: an overly aggressive DELETE/UPDATE threshold merges facts that only look similar. "User lives in Denver" and "user was born in Denver" are 0.85+ cosine similar and mean completely different things. This is why the decision step needs the LLM to reason about semantics, not just a similarity score cutoff — a pure threshold rule (if score > 0.8: overwrite) will silently corrupt memory in exactly the cases where getting it right matters most.

Stage 4: consolidation — write once, not append

The resolved operation gets applied atomically: an UPDATE overwrites the existing vector and metadata for m_204 (new embedding, incremented version, updated timestamp) rather than inserting a new row and leaving the old one to rot. A DELETE tombstones the memory instead of a silent removal, so you can audit why something disappeared if a user asks "wait, didn't I tell you I live in Austin?"

if decision["operation"] == "UPDATE":
    vector_store.update(
        id=decision["target_id"],
        embedding=embed(decision["resolved_fact"]),
        text=decision["resolved_fact"],
        version=existing.version + 1,
        updated_at=now(),
    )

Retrieval-time scoring isn't just similarity either

Once memories are clean, the query-time ranking still shouldn't be pure cosine similarity. Production memory layers blend it with recency and access frequency, because a fact from an hour ago and a fact from eight months ago can both be top-3 nearest neighbors, but they're not equally trustworthy:

score = (0.6 * similarity) + (0.25 * recency_decay(age_days)) + (0.15 * access_frequency)

The decay function matters more than its weight suggests — a linear decay treats a 30-day-old fact and a 300-day-old fact as almost the same, while an exponential decay (closer to how the Ebbinghaus forgetting curve actually behaves) lets stale facts fade without a hard expiry that deletes something still true.

The takeaway

If you're building agent memory and it's just store.add(embedding) on every turn, you don't have memory — you have an unindexed diary the model reads through a straw. The four-stage loop — extract, retrieve neighbors, decide the operation, consolidate — is what separates a system that gets more useful as it accumulates history from one that gets slower and more contradictory. That decision step is also the one piece you can't shortcut with a bigger embedding model or a faster vector index; it needs actual reasoning about whether two facts are the same fact, a newer version of it, or something else entirely.

Agentic AI Is Mostly Marketing. Memory Is the Part That's Real.

Mukesh — Sat, 05 Sep 2026 19:18:59 +0000

Open any 'agentic AI' repo trending on GitHub this month and you'll find the same shape: a while loop, a tool-calling wrapper, a retry policy, and a system prompt that says 'you are an autonomous agent.' Strip the prompt out and what's left is a script. A good script, sometimes — but a script. It runs the same way today as it will next month, making the same mistakes in the same places, because nothing about running it changes what it knows.

That's not a controversial observation in private, but it's an unpopular one to say out loud, because 'agentic' has become the label that gets a repo funded, a blog post shared, and a feature slotted into a roadmap. So here's the hot take: most of what's marketed as agentic AI in 2026 isn't. It's automation with a chat interface bolted on. And the one thing that actually separates an agent from a script — memory that changes future behavior without a human re-editing the code — is treated as an afterthought by nearly everyone building this stuff.

Steelmanning the hype

Before tearing this down, it's worth taking the other side seriously, because parts of it are true. Tool-calling loops that plan multi-step tasks, decide which API to hit next, and recover from a failed step without a human in the loop are a genuine capability that didn't exist three years ago. Watching a model decompose 'deploy this and roll back if error rate spikes' into eight correctly-ordered tool calls is not nothing. Frameworks like the current generation of orchestration libraries make it dramatically faster to wire up retries, structured outputs, and multi-agent handoffs than it was to hand-roll that logic in 2023. If your bar for 'agentic' is 'makes autonomous multi-step decisions within a single session,' plenty of production systems clear it honestly.

The problem is the word implies more than that. An agent, in the sense people actually mean when they get excited about it, is something that gets better — or at least different — the longer it runs, because it's accumulating experience. A system that makes the identical decision on day 200 that it made on day 1, given the identical input, isn't agentic by that definition. It's deterministic, which is often exactly what you want in production, but it's not the thing being sold.

The tell: does yesterday change today?

Here's the test I actually use, and it takes one question: does what happened yesterday change what this system does today, without a human editing a prompt or a config file in between?

Concretely: I maintain a small autonomous system that has to decide, every cycle, which of several strategies to pursue and how aggressively. The first version was a script in the pejorative sense — fixed thresholds, fixed weights, a case statement dressed up with an LLM call that picked among pre-defined branches. It ran fine. It also made the same misjudgment every time a particular condition recurred, because there was nowhere for the outcome of that misjudgment to go. Nothing recorded it, nothing surfaced it back into the decision, and nothing adjusted the threshold. Rerun the same week and you'd get the same wrong call.

The fix wasn't a bigger model or a fancier planning loop. It was giving the system a place to write down 'this threshold was wrong, here's what actually happened' and a mechanism to read that back in before making the same class of decision again. Once that existed, the system started quietly correcting itself — tightening thresholds that had produced bad outcomes, loosening ones that had been too conservative — without anyone touching the code. That is the entire difference between the before and after version, and it's the only part of the rewrite I'd defend as making the system more 'agentic' in any sense that matters. Everything else — the tool calls, the retries, the multi-step planning — was already there and already useful, but it wasn't the thing that changed the system's trajectory over time.

Why memory keeps getting treated as an afterthought

The usual pattern in agent projects is: build the orchestration first, get the tool-calling loop working, ship it, and then — if there's time — bolt on a vector database and call it 'memory.' That ordering is backwards, and it's why so many agent projects plateau at a fixed error rate instead of improving. Dumping conversation transcripts into a vector store gives you retrieval, not memory in the sense that matters. Retrieval answers 'what did we talk about before?' Behavior-changing memory answers 'given what happened last time, should I do something different now?' Those are different engineering problems. The first is a search index. The second requires deciding what's worth remembering, when it should be forgotten or superseded, and — critically — where in the decision pipeline the retrieved memory actually gets to override a default.

Most teams skip that third part entirely. They add memory retrieval to the prompt context and consider the job done, without ever wiring a path for a remembered failure to actually suppress or adjust a future action. The memory exists; it just doesn't have write access to behavior. That's a memory system in name and a search bar in practice.

The actionable version of this take

If you're evaluating an agent framework, a memory product, or your own in-house system, skip the marketing copy and ask the one-question test: pick a mistake the system made last week, and ask whether the system is structurally less likely to make that exact mistake today — not because someone patched the prompt, but because the system itself changed something in response to the outcome. If the answer is yes, you're looking at an agent. If the answer is no, you're looking at a very well-orchestrated script, and that's fine — just don't pay agent prices for script behavior. The gap between those two categories isn't the orchestration layer everyone's competing on. It's memory that's actually load-bearing, and right now almost nobody is building for that on purpose.

Recommendations Are Cheap. Enforcement Is the Feature: Building a Backlog-Pressure Gate for Self-Regulating Schedulers

Mukesh — Fri, 04 Sep 2026 18:51:55 +0000

Every team has a version of this document: a postmortem, a retro note, a lesson log. Something breaks, someone writes down exactly what should change, and the fix reads as obvious. And then, days later, the same failure happens again, in the same shape, for the same reason -- because the recommendation was never anything more than text.

I hit this in a very literal way while running an autonomous agent that manages its own task queue. Every day it logs a reflection pass over its own output: what worked, what didn't, what to change tomorrow. For four consecutive days, that reflection log identified the exact same problem -- a scheduler that kept over-allocating a high-friction task type while a low-friction, high-throughput one starved -- and proposed the exact same fix, with formula and threshold spelled out. And for four consecutive days, the scheduler did the opposite of what the log recommended, because nothing in the code actually read the log.

That gap -- between "we wrote down the fix" and "the fix runs" -- is worth taking seriously as a design problem, not a process problem. Here's the pattern I used to close it, with enough detail that you can drop the shape into any weighted scheduler.

The setup: weights that already exist

Most non-trivial schedulers already have some notion of weighted selection -- a set of task types (or routes, or workers, or jobs) each with a base weight, and a pick function that samples from that distribution:

def weighted_pick(candidates, weights):
    total = sum(weights[c] for c in candidates)
    r = random.uniform(0, total)
    upto = 0
    for c in candidates:
        upto += weights[c]
        if upto >= r:
            return c
    return candidates[-1]

If your scheduler is even a little dynamic, you probably also have an override layer on top of the base weights -- a place where you can bump one task type up or down without redeploying. Mine stores it as a small JSON blob in the database: a dict of {task_type: multiplier}, read once per scheduling decision and multiplied into the base weight, with a safe fallback (clamp to a small positive floor) if a multiplier would zero out or invert a weight entirely.

def weight_overrides(db):
    raw = db.get_state("weight_overrides")
    if not raw:
        return {}
    try:
        overrides = json.loads(raw)
    except (TypeError, ValueError):
        return {}
    return {k: max(v, 0.05) for k, v in overrides.items() if isinstance(v, (int, float))}

This piece usually already exists for a mundane reason -- someone wanted to manually throttle a noisy job type without a deploy. The mistake is stopping there and assuming manual overrides are the only overrides you need.

The missing piece: a signal the scheduler can read for itself

The recommendation that kept getting ignored had a very specific, checkable shape: when the backlog of tasks waiting on a human (or any blocking external step) crosses a threshold, throttle the task types that produce more of that backlog and boost the ones that don't. That's not a vague sentiment -- it's a query and a table.

The query counts, over some recent window, how many completed tasks are still carrying an unresolved follow-up:

def backlog_pressure(db, window=100):
    pending = 0
    for row in db.recent_tasks(limit=window):
        try:
            result = json.loads(row["result_json"])
        except (TypeError, ValueError, KeyError):
            continue
        if result.get("action_required"):
            pending += 1
    return pending

And the table is just the reflection log's own formula, made literal:

BACKLOG_FRICTION_MULTIPLIERS = {
    "high_friction_task": 0.4,
    "another_high_friction_task": 0.5,
    "low_friction_task": 1.6,
}

BACKLOG_THRESHOLD = 7

def apply_backlog_pressure(overrides, pressure):
    if pressure <= BACKLOG_THRESHOLD:
        return overrides
    merged = dict(overrides)
    for task_type, multiplier in BACKLOG_FRICTION_MULTIPLIERS.items():
        merged[task_type] = merged.get(task_type, 1.0) * multiplier
    return merged

Wired into the existing override function, this is maybe fifteen lines of new code on top of infrastructure that was already there. That's the point worth underlining: the fix wasn't a new subsystem. It was making an existing subsystem read one more signal.

Why this shape, specifically

Three properties made this safe to ship without a long review cycle, and they generalize past this one bug:

It's additive, not replacing. The new function composes with the existing override dict rather than branching around it. If backlog_pressure() returns zero -- which it does for every existing test fixture, since none of them simulate a week of unresolved tasks -- behavior is provably identical to before the change. You're not asking anyone to trust new logic in the common case; the common case doesn't touch the new logic at all.

It reuses a read path that's already trusted. I didn't add a new database table or a new query method. recent_tasks() already existed, already had test coverage, and already had a documented row shape. The new code just reads it differently. Every new failure mode you might worry about (malformed JSON, missing keys, a task with no result yet) was already something the surrounding code defended against, so the same defensive pattern -- try/except around json.loads, .get() with a default -- covered the new call site for free.

The threshold is a number, not a vibe. "Throttle when things feel backed up" is not enforceable. "Throttle when more than 7 of the last 100 tasks still have action_required set" is a single if statement. Converting a recommendation into a gate means converting its trigger condition into something a function can evaluate -- if you can't write that condition down, the recommendation isn't finished yet, no matter how sound the reasoning is.

The general lesson

A recommendation that only lives in a log is a wish. It becomes a gate the moment three things are true: the trigger condition is a query against data you already collect, the response is a change to a code path that already runs on every cycle, and the default behavior (trigger not met) is provably unchanged from before. If any of those three is missing -- if the trigger needs data you don't have yet, or the response would need a new code path nobody's reviewed, or you can't show the change is a no-op in the common case -- that's exactly where recommendations go to die, re-written in next week's postmortem with the same formula and the same missing enforcement.

The fix isn't writing better recommendations. Mine were already correct four days running. The fix is treating "convert this into a gate" as its own task, with its own priority, separate from "identify the problem" -- because a system will keep re-discovering the same problem forever if nothing downstream of the discovery ever runs.

The CircleCI Cache Key Bug That's Silently Serving Your Builds Stale Dependencies

Mukesh — Thu, 03 Sep 2026 18:32:37 +0000

Your CircleCI pipeline is green. Every job passes. And yet your app is running against a dependency version that hasn't shipped in a month — nobody committed it, nobody bumped it, it just quietly showed up in production. If you've chased a bug like this, the culprit is almost never your code. It's your cache key.

This is a five-minute read and a fifteen-minute fix. Quick Win Friday, deployed to your .circleci/config.yml.

The failure mode

CircleCI's dependency caching works on a simple contract: you compute a key from something that changes when your dependencies change (usually a lockfile checksum), and you save/restore a cache tied to that key. The contract breaks in three specific, extremely common ways:

You checksum the wrong file. {{ checksum "package.json" }} looks reasonable until someone bumps a transitive dependency via package-lock.json without touching package.json. The checksum doesn't move. CircleCI happily hands back last week's node_modules.
restore_keys does prefix matching, and people think it does exact matching. CircleCI tries your primary key first, then falls through restore_keys in order, and the first one is a prefix match against existing cache entries — not "give me the newest exact match." If your restore_keys list is too coarse (e.g. just v1-deps-), you can restore a cache built from a completely different branch, with a completely different lockfile, and the job won't fail. It'll just quietly install nothing (cache hit, npm ci sees the modules are "there") or run against the wrong versions.
There's no version escape hatch. When you inevitably need to force everyone's cache to invalidate — a corrupted cache entry, a package manager migration, a lockfile format change — there's no cheap way to do it, because the key format was never designed with a manual buster in mind.

Each of these fails silently. No red X. No error in the logs. Just a build that ran with stale state, and a bug report three days later that nobody can reproduce locally because local node_modules is fine.

The fix

Replace whatever your current cache block looks like with this shape:

- restore_cache:
    keys:
      - v3-deps-{{ .Branch }}-{{ checksum "yarn.lock" }}
      - v3-deps-{{ .Branch }}-
      - v3-deps-

- run:
    name: Install dependencies
    command: yarn install --frozen-lockfile

- save_cache:
    key: v3-deps-{{ .Branch }}-{{ checksum "yarn.lock" }}
    paths:
      - node_modules
      - ~/.cache/yarn

Four specific changes, each fixing one of the failure modes above:

Checksum the lockfile, not the manifest. yarn.lock / package-lock.json / poetry.lock / Cargo.lock — whatever actually pins your resolved versions. That's the only file where "nothing changed" is a true statement about your dependency tree.
--frozen-lockfile (or npm ci, or poetry install --no-update) as the install command, always. This is the safety net for the failure modes you haven't fixed yet: if the cache did restore something stale, a frozen install refuses to silently proceed with a mismatched lockfile instead of quietly reconciling it. You want that job to go red, not go green with the wrong tree.
Order restore_keys from most to least specific, and stop one level short of "matches anything." Branch-scoped exact match first, branch-scoped prefix second, global prefix last as a genuine last resort for a brand-new branch. Don't just have v3-deps- as your only fallback — that's the line that lets a feature/rewrite-auth branch restore a cache from main with a different lockfile entirely.
Bump the leading version token (v3- → v4-) whenever you need a clean slate. This is your manual cache-buster. Because it's baked into the key itself, forcing invalidation for everyone is a one-line PR, not a support ticket to CircleCI or a trip through the project settings UI to nuke caches by hand.

Verifying it actually worked

Don't just ship the YAML change and trust it. Add a one-line assertion job step for a week while you confirm behavior:

- run:
    name: Confirm lockfile/cache agreement
    command: |
      yarn list --pattern "your-critical-package" | tee /tmp/resolved.txt
      grep -q "$(grep 'your-critical-package' yarn.lock | head -1 | cut -d'@' -f2)" /tmp/resolved.txt

Adjust the package name to whatever dependency has bitten you before, or whatever your team would most want to know shipped the wrong version. If that grep ever fails, your cache and your lockfile have diverged — and now it fails loud, in CI, instead of quiet, in production.

The monorepo trap

If you're on a Yarn/npm workspace or a monorepo with multiple lockfiles, the checksum function only hashes what you tell it to. checksum "yarn.lock" at the repo root won't catch a change to a workspace package's own dependencies unless your package manager writes that resolution back into the root lockfile (most do, but verify it for yours). If you have nested lockfiles that aren't supposed to exist, that's usually the actual bug — but if they're intentional, checksum all of them:

- restore_cache:
    keys:
      - v3-deps-{{ checksum "yarn.lock" }}-{{ checksum "packages/api/yarn.lock" }}

What this actually costs you

Fifteen minutes: five to edit the config, five to push a branch and watch the cache key change in the CircleCI job output, five to add and then remove the verification step once you trust it. No infrastructure change, no new tooling, no dependency on anything outside .circleci/config.yml. And it closes off the specific class of bug that's hardest to debug precisely because it never announces itself — the build that lies to you by saying nothing at all.

Agent Memory in Production: Mem0 vs Zep vs LangChain Memory vs Redis DIY

Mukesh — Wed, 02 Sep 2026 18:33:27 +0000

Every agent framework now ships some flavor of "memory," and every vendor swears theirs is the one that will stop your agent from re-introducing itself to a user it's talked to fifty times. The trending conversation this week — Your AI Remembers Everything and Trusts All of It — points at the real failure mode: memory tools are good at storing, and bad at deciding what's still true. That's the axis I actually care about when picking a tool, not embedding dimensionality or vector store choice.

I ran four approaches against the same workload: a support agent that needs to remember user preferences, prior tickets, and corrections across sessions, with facts that occasionally contradict earlier facts ("actually I moved to Denver" after "I live in Austin"). Here's how Mem0, Zep, LangChain's memory classes, and a hand-rolled Redis setup actually behaved.

The contenders

Mem0 — a managed (or self-hosted) memory layer that sits between your LLM calls and storage. It extracts facts from conversation turns, scores them, and resolves conflicts against existing memory before writing.

Zep — session memory plus a temporal knowledge graph. It doesn't just store facts, it timestamps them and lets you query "what did we believe was true as of X."

LangChain memory (ConversationSummaryBufferMemory, VectorStoreRetrieverMemory) — not a product, a set of classes you wire to whatever vector store you already run. No extraction logic beyond what you write.

Redis + RedisVL, DIY — no framework at all. You store embeddings and metadata yourself, and write your own retrieval and conflict logic.

Where the differences actually show up

Fact extraction and conflict resolution

This is the part every vendor undersells and every homegrown system underestimates.

With Mem0, sending a new turn triggers an LLM call that extracts candidate facts, compares them against existing memory for the user, and either adds, updates, or discards:

from mem0 import Memory

m = Memory()
m.add("I moved to Denver last month, hate the traffic though", user_id="u123")
# later
results = m.search("where does the user live", user_id="u123")
# returns the Denver fact, with the Austin fact marked stale, not deleted

The "not deleted" part matters — Mem0 keeps history rather than overwriting, which is the only way to answer "did the user used to live somewhere else" later. That's real engineering, not just a wrapper around upsert.

Zep does something structurally different: it builds a temporal graph, so instead of "current fact wins," you get an edge with a validity window. That's more expressive if your agent needs to reason about when something changed, not just what's current — useful for, say, an agent that has to explain "your subscription was on the annual plan until March, then you switched." For a support bot that only needs "what's true now," that expressiveness is overhead you pay for in query complexity.

LangChain's memory classes do none of this. VectorStoreRetrieverMemory will happily return both the Austin and Denver facts with similar cosine scores, and it's on you to decide which one to trust. I've seen teams ship this to production assuming semantic similarity implies recency — it doesn't, and the failure is silent: the agent picks whichever fact happens to embed slightly closer to the query.

Redis DIY has the same problem, minus even the vector-store convenience. You get exactly the conflict resolution you write, which for most teams under deadline is "none," until a customer complaint reveals it.

Latency and cost

Mem0's extraction step is an extra LLM call per write — real latency (typically 300-800ms in my testing with a small extraction model) and real token cost. For write-heavy agents, where every turn creates memory candidates, that adds up. Mem0 mitigates this with async writes and batched extraction, but you're still paying for a second model call on top of your main completion.

Zep's graph writes are cheaper per-turn by default (no full LLM extraction unless you enable it), but querying the graph for anything beyond "latest fact" costs more at read time — graph traversal isn't as cheap as a vector similarity search.

LangChain memory and Redis DIY are the cheapest at write time because they do the least work — you're paying for embedding calls, not extraction. The cost shows up later as engineering time and, if you skip conflict resolution, as wrong answers in production.

Where each one actually wins

Need	Pick
Managed fact extraction + conflict resolution, minimal glue code	Mem0
Temporal reasoning ("what did we believe on date X")	Zep
Already deep in LangChain, prototyping, low memory volume	LangChain memory classes
Full control, existing Redis infra, team has bandwidth to own conflict logic	Redis + RedisVL

The honest tradeoff nobody puts in the docs

Every memory tool that does extraction and conflict resolution for you is making a judgment call on your behalf about what to trust — which is exactly the failure mode this week's trending post is pointing at. Mem0's "keep history, mark stale" approach is more conservative than Zep's graph-edge model, which is more conservative than "vector similarity as truth," which is what LangChain memory and naive Redis setups default to whether you intended it or not.

If you're evaluating these for a production agent, don't benchmark retrieval latency first. Feed each one a deliberately contradictory conversation — a user correcting themselves twice — and check what the agent says a week later when asked "where do I live." That single test surfaces the actual difference between these tools faster than any latency chart, because it's the difference between a memory system and a fact dump with vectors on top.

For most teams building a support or personal-assistant agent that needs to stay right over months of conversation, that argues for Mem0 or Zep over rolling your own — the extraction and conflict-resolution logic is genuinely hard to get right, and it's exactly the kind of undifferentiated engineering worth buying instead of building.

The 400-Character Bug: How Dict Key Order Silently Deleted Our Agent's Most Important Field

Mukesh — Tue, 01 Sep 2026 18:52:37 +0000

The symptom

We run a small autonomous agent that closes out its own day: it pulls every task it ran, hands the results to an LLM, and asks it to extract lessons — what worked, what didn't, what needs a human. One of those lessons is supposed to flag action_required: the field a task sets when it's stuck waiting on something only a person can do (confirm an email address, approve a listing, whatever).

For several days running, the nightly summary kept missing a blocker that was sitting in plain sight. Five demo deployments were stalled on one specific owner action. Every single task that hit that blocker wrote it into action_required. And every night, the LLM-generated lessons talked about task volume, about revenue estimates, about strategy mix — and said nothing about the blocker at all. Not "low priority." Not "deprioritized." Just absent, like it had never been logged.

The wrong theories

First guess: prompt problem. Maybe the lesson-extraction prompt wasn't asking the model to look for blockers explicitly enough. We tightened the prompt, added an explicit instruction to surface action_required when present. No change.

Second guess: the model was just deprioritizing it — LLMs summarizing long inputs are known to drop details that seem secondary. We reordered the prompt to put escalation instructions first, added a one-line example of what a good blocker-flag looks like. Still nothing.

Third guess, and the one that wasted the most time: maybe the extraction function itself had a bug — some off-by-one in how it parsed the model's response before writing lessons to the database. We reread _extract_lessons() line by line. It was fine. It was correctly reading whatever the model gave it. The model just was never being given the field in the first place.

That reframing — stop looking at what happens after the LLM call, start looking at what goes into it — is what actually cracked it.

The actual root cause

The function that builds the nightly context, _build_day_summary(), takes each task's result object, serializes it, and truncates it before folding it into the day's summary:

summary_line = json.dumps(result.as_dict())[:400]

That looks harmless. It's a guard against one verbose task result blowing up the token budget for the whole day's summary. Four hundred characters felt generous.

The problem was as_dict()'s key order:

def as_dict(self):
    return {
        "earned_usd": self.earned_usd,
        "spent_usd": self.spent_usd,
        "notes": self.notes,
        "action_required": self.action_required,
        "potential_revenue": self.potential_revenue,
        "artifacts": self.artifacts,
        "failed": self.failed,
    }

notes comes before action_required. notes is free-text and often runs long — a task explaining what it did, what it found, what it's thinking about trying next. On any task where notes ran past roughly 300-350 characters, the JSON serialization pushed action_required past the 400-character cutoff. json.dumps[:400] doesn't care that it's mid-object; it just chops the string. The field wasn't malformed, wasn't null, wasn't skipped by any explicit logic anywhere. It simply didn't exist yet at the byte offset where the string got cut off.

No exception. No warning log. No test failure. The truncated JSON was still syntactically broken in a way that would fail to json.loads() if anyone tried — but nobody did, because the string was only ever going into a prompt as text, never parsed back out. A field being silently amputated by a length limit produces no error signal anywhere in the stack. It just produces an LLM that was never shown the thing it needed to reason about.

And it's a genuinely nasty bug to catch by inspection, because it's data-dependent. Short notes field, the bug doesn't fire. Verbose notes field, it does. Whether you see it depends entirely on how chatty that day's tasks happened to be.

The fix

The fix stopped truncating raw JSON text and started truncating after deciding what mattered:

def _task_summary_fields(result_json: str) -> dict:
    try:
        parsed = json.loads(result_json)
    except (json.JSONDecodeError, TypeError):
        return {"raw": result_json[:400]}
    return {
        "notes": str(parsed.get("notes", ""))[:300],
        "action_required": parsed.get("action_required", ""),
    }

action_required is pulled out and given its own untruncated slot instead of competing with notes for space inside a single character budget. We also added a rollup — pending_action_required_count — computed once per day and handed to the LLM as an explicit number, so the model doesn't have to infer backlog size from scattered prose across a dozen task summaries. Malformed JSON still falls back to a raw truncated string, so nothing regresses for genuinely broken task output.

The lesson

Character-count truncation on serialized structured data is a silent information-loss bug by construction: it has no failure mode that looks like a failure. It doesn't throw, doesn't return an error, doesn't even produce invalid output most of the time — it just quietly decides, based on how verbose an unrelated field was that day, whether your most important field exists.

The test suite for this code passed the whole time. It asserted that lessons got extracted, that the LLM call didn't blow up, that malformed results were handled — all real, all necessary, none of it enough. Nobody had written a test that asserted which specific fields survive summarization when other fields are long, because that's not the kind of thing you think to test until you've watched it fail in production for two days straight.

That's exactly the class of bug a CI pipeline is built to catch and a code reviewer is built to miss — it requires running the function against a synthetic input engineered to be adversarial (a long notes, a short action_required), not reading the function and reasoning about what it does. A five-line golden test — feed the summarizer a task result with a 500-character notes field and a non-empty action_required, assert the output still contains it — would have failed on the very first commit that introduced the character slice, months before it ever reached production. The fix isn't "write more tests" in the abstract. It's: for any code that trims, truncates, or samples structured data before it reaches a consumer that can't ask follow-up questions, the pipeline should have a check for exactly which fields are guaranteed to survive — and it should run on every push, not get discovered by a human noticing a pattern across several days of missing alerts.

I Gave a Vultr Box a Heartbeat: Self-Healing Docker Containers in 20 Minutes with systemd, Not Cron

Mukesh — Mon, 31 Aug 2026 18:32:39 +0000

Your Docker container crashes at 3 a.m. Cron notices at :00, :05, :10 — up to five minutes of downtime before anyone even checks. And if the crash loop is fast enough, a naive restart: always in docker-compose.yml will just spin the container up and down forever, hammering your database with reconnect storms.

What you actually want is a watchdog: something that checks health more often than cron allows, restarts with backoff instead of blindly looping, and tells you when it happens. Here's how to build that on a $6/mo Vultr VPS in about 20 minutes, using systemd timers instead of cron because systemd gives you logging, dependency ordering, and sub-minute intervals for free.

1. Provision the box

If you already have a Vultr instance, skip to step 2. Otherwise, using the Vultr CLI:

vultr-cli instance create \
  --region ewr \
  --plan vc2-1c-2gb \
  --os 387 \
  --label docker-watchdog-demo \
  --host docker-watchdog-demo

--os 387 is Ubuntu 24.04 LTS at time of writing; check vultr-cli os list if that's stale. SSH in once it's provisioned, then install Docker with the official convenience script:

curl -fsSL https://get.docker.com | sh
sudo usermod -aG docker $USER

Log out and back in so the group change takes effect.

2. Deploy a container worth watching

We'll simulate a flaky service — a tiny web app that has a /kill endpoint to crash itself on demand, so you can test the watchdog without waiting for a real bug.

# docker-compose.yml
services:
  flaky-app:
    image: nginx:alpine
    container_name: flaky-app
    restart: unless-stopped
    ports:
      - "8080:80"
    healthcheck:
      test: ["CMD", "wget", "-q", "--spider", "http://localhost"]
      interval: 10s
      timeout: 3s
      retries: 3

restart: unless-stopped handles the trivial case (process exits) but does nothing if the container is running yet unhealthy — hung, deadlocked, or serving 500s. That's the gap the watchdog closes.

docker compose up -d

3. Write the watchdog

This script checks Docker's own health status, restarts unhealthy containers with exponential backoff, and caps retries so a genuinely broken container doesn't restart-loop forever.

#!/usr/bin/env bash
# /usr/local/bin/docker-watchdog.sh
set -euo pipefail

STATE_DIR=/var/lib/docker-watchdog
WEBHOOK_URL="${WATCHDOG_WEBHOOK_URL:-}"
MAX_RETRIES=5

mkdir -p "$STATE_DIR"

notify() {
  local msg="$1"
  [[ -z "$WEBHOOK_URL" ]] && return 0
  curl -fsS -X POST -H 'Content-type: application/json' \
    --data "{\"text\":\"$msg\"}" "$WEBHOOK_URL" >/dev/null || true
}

for cid in $(docker ps -q); do
  name=$(docker inspect --format '{{.Name}}' "$cid" | sed 's#^/##')
  health=$(docker inspect --format '{{if .State.Health}}{{.State.Health.Status}}{{else}}none{{end}}' "$cid")

  [[ "$health" != "unhealthy" ]] && continue

  count_file="$STATE_DIR/${name}.count"
  count=$(cat "$count_file" 2>/dev/null || echo 0)

  if (( count >= MAX_RETRIES )); then
    echo "$name still unhealthy after $count restarts, giving up" >&2
    continue
  fi

  backoff=$(( 2 ** count ))
  echo "$name is unhealthy, restarting (attempt $((count+1)), waited ${backoff}s backoff)"
  sleep "$backoff"
  docker restart "$name"

  echo $((count+1)) > "$count_file"
  notify "⚠️ docker-watchdog restarted *${name}* (attempt $((count+1))/${MAX_RETRIES}) on $(hostname)"
done

# reset counters for containers that recovered on their own
for f in "$STATE_DIR"/*.count; do
  [[ -e "$f" ]] || continue
  name=$(basename "$f" .count)
  health=$(docker inspect --format '{{if .State.Health}}{{.State.Health.Status}}{{else}}healthy{{end}}' "$name" 2>/dev/null || echo gone)
  [[ "$health" == "healthy" ]] && rm -f "$f"
done

sudo chmod +x /usr/local/bin/docker-watchdog.sh
sudo mkdir -p /var/lib/docker-watchdog

The backoff counter is the piece cron-based versions usually skip: without it, a container that fails its healthcheck every 10 seconds gets restarted every 10 seconds, forever, which is worse than doing nothing.

4. Run it on a systemd timer, not cron

Cron's minimum resolution is one minute, it has no built-in logging beyond mail (which is rarely configured), and a hung script just silently occupies a slot forever. systemd timers fix all three: sub-minute intervals, journalctl logging automatically, and a RuntimeMaxSec to kill a hung run.

# /etc/systemd/system/docker-watchdog.service
[Unit]
Description=Docker container health watchdog
After=docker.service
Requires=docker.service

[Service]
Type=oneshot
ExecStart=/usr/local/bin/docker-watchdog.sh
Environment=WATCHDOG_WEBHOOK_URL=https://hooks.slack.com/services/REPLACE/ME
TimeoutStartSec=30

# /etc/systemd/system/docker-watchdog.timer
[Unit]
Description=Run docker-watchdog every 15 seconds

[Timer]
OnBootSec=15s
OnUnitActiveSec=15s
AccuracySec=1s

[Install]
WantedBy=timers.target

sudo systemctl daemon-reload
sudo systemctl enable --now docker-watchdog.timer

That 15-second interval is something cron simply cannot do — its floor is 60 seconds, and even that requires an extra wrapper loop to hit reliably.

5. Prove it works

Crash the container on purpose and watch the recovery:

docker exec flaky-app sh -c 'kill 1'
journalctl -u docker-watchdog.service -f

Within 15 seconds you should see the watchdog detect the unhealthy state, apply backoff, and restart it — and if you wired up WATCHDOG_WEBHOOK_URL, a Slack message lands at the same moment. Check the timer's own schedule and history:

systemctl list-timers docker-watchdog.timer
systemctl status docker-watchdog.service

6. Lock the box down

Since this VPS is now doing something worth protecting, restrict it to the ports you actually need with the Vultr firewall (or ufw locally):

vultr-cli firewall group create --description "docker-watchdog"
# then attach rules for 22/tcp (your IP only) and 8080/tcp

Don't expose the Docker socket or the watchdog's webhook URL beyond what's needed — the script only needs local docker.sock access, which it already has by running as root via systemd.

Where to take it from here

This pattern generalizes past a single VPS: point the same script at a remote Docker context (DOCKER_HOST=ssh://...) to watch containers on a fleet from one control box, or swap the webhook for a PagerDuty Events API call if 3 a.m. pages are your actual problem. The core idea stays the same — sub-minute detection, bounded backoff, and a notification the moment state changes, all from tooling that ships with every modern Linux box by default.

Refresh Token Rotation Under the Hood: How Auth0 Catches a Stolen Token Before It's Ever Replayed

Mukesh — Sun, 30 Aug 2026 20:29:24 +0000

Most explanations of refresh token rotation stop at "the old token gets swapped for a new one." That's true, but it skips the part that actually matters: how does the authorization server know a stolen token was used before the legitimate client tries to use it? The answer is a small piece of state most engineers never think about — a token family — and the reuse-detection algorithm built on top of it. Let's open it up.

The problem rotation alone doesn't solve

A static, long-lived refresh token is a bearer secret with an unlimited replay window. If it leaks — from a compromised mobile device, a logged network request, a misconfigured CI cache — an attacker holds a valid credential for as long as the token's TTL allows, often 30-90 days, with zero signal to the legitimate owner.

Rotation on its own only shrinks the window: every refresh call returns a new refresh token and burns the old one. That's better, but if you stop there, an attacker who grabs a token can still race the real client, use it once, and the server has no way to tell "attacker used it" apart from "the app used it."

The piece that actually catches theft is reuse detection, and it requires the server to track lineage, not just validity.

Token families: the data structure under the hood

Every refresh token belongs to a family, created at the moment the first refresh token is issued (typically at login). A family is conceptually:

family_id: f_9a2e...
lineage:   [rt_001 (used), rt_002 (used), rt_003 (active)]
subject:   user_42
client:    mobile-app-ios
status:    active | revoked

Each rotation doesn't just generate a fresh token — it appends to that array and marks the previous token as spent, not deleted. That distinction between "spent" and "deleted" is the whole trick: the server needs to remember that rt_001 and rt_002 existed and were already consumed, so it can recognize them if they show up again.

The algorithm, step by step

On every POST /oauth/token with grant_type=refresh_token:

Look up the token by its opaque value (or, in JWT-encoded refresh tokens, by the jti claim) and resolve its family.
If the token matches the family's current active token — normal path. Mark it spent, generate a new token, append it to the lineage, return it. Everything continues.
If the token matches a token in the family's lineage but it's already marked spent — this is the reuse signal. Somebody just replayed a token that was already exchanged once. That can only happen if two parties (the legitimate client and an attacker, or two copies of a client after a token was exfiltrated) had the same refresh token at the same time.
On reuse detection: revoke the entire family, not just the offending token. Every token in that lineage, including the current active one the legitimate client is holding, becomes invalid. This forces a full re-authentication.
Optionally, fire a security event (Auth0 calls this a "breached refresh token" event) so the application layer can notify the user, force a password reset, or flag the session for review.

Step 4 is the part people get wrong when they roll this themselves — the instinct is to just reject the reused token and move on. But if you don't nuke the whole family, the attacker's copy of the next token (if they got far enough to see it) or the legitimate client's still-valid token leaves a live credential in play. Killing the lineage is what turns "detected an anomaly" into "closed the hole."

A minimal implementation

Here's the reuse-check logic stripped to its core, independent of any particular auth vendor's SDK:

async function rotateRefreshToken(presentedToken) {
  const record = await db.refreshTokens.findByValue(presentedToken);
  if (!record) throw new Error('invalid_grant');

  const family = await db.tokenFamilies.findById(record.familyId);

  if (record.status === 'spent') {
    // Reuse detected — the token was valid once, but already consumed.
    await db.tokenFamilies.revokeAll(family.id);
    await auditLog.write({
      type: 'refresh_token_reuse',
      subject: family.subject,
      familyId: family.id,
    });
    throw new Error('invalid_grant'); // client must re-authenticate
  }

  if (family.status !== 'active') {
    throw new Error('invalid_grant'); // family already revoked
  }

  const newToken = crypto.randomBytes(32).toString('base64url');
  await db.refreshTokens.markSpent(record.id);
  await db.refreshTokens.create({
    value: newToken,
    familyId: family.id,
    status: 'active',
  });

  return newToken;
}

The two lookups that make this work — "is this token spent" and "is this family still active" — are exactly the state a naive rotation implementation skips, because a naive version just deletes old tokens instead of marking them spent. Deletion throws away the evidence you need to detect the attack.

The grace-period wrinkle

Real clients aren't perfectly reliable. A mobile app on a flaky connection might send a refresh request, lose the response, and retry with the same refresh token a few seconds later — which looks identical to reuse from the server's point of view. Auth0 and most production implementations handle this with a short reuse grace period (on the order of seconds, configurable), during which a repeated request for the most-recently-spent token returns the same already-issued replacement instead of triggering revocation. Outside that window, reuse is treated as theft. Getting this window right is a genuine tuning problem: too long and you widen the attacker's usable replay margin; too short and flaky networks start locking real users out of their sessions.

Why this matters even if you're not rolling your own

If you're using Auth0, Okta, or another provider, you don't write this state machine — but you do configure it, and debugging "why did this user get logged out everywhere" tickets requires knowing it exists. If you're building your own OAuth server (say, for a service-to-service or IoT use case where a hosted IdP doesn't fit), rotation without family tracking gives you a false sense of security: you'll pass a pen test that only checks "does the old token get rejected" while remaining blind to the actual replay-race scenario that reuse detection is built to catch.

The one-line summary worth remembering: rotation limits how long a stolen token works; family-based reuse detection is what tells you that it was stolen at all.

AI Review Isn't the Gate. Your CI Pipeline Still Is.

Mukesh — Sat, 29 Aug 2026 19:59:11 +0000

Two weeks ago I watched a pull request get merged with an AI-authored review comment that said "LGTM, nice defensive coding here" — attached to a diff that introduced an unguarded array index into a payment reconciliation job. The array could be empty. It has been empty, twice, in production, since. Nobody had tested the reviewer.

That's not a hypothetical. It's the exact failure mode buried in the discourse this week when developers started asking why AI "promoted every developer to reviewer" without anyone stopping to ask what happens when the reviewer itself is wrong. Here's my hot take: AI code review should never be a merge gate. It should be a colleague you consult, not a bouncer you trust with a badge — and if your CI pipeline doesn't already know that, your pipeline is broken, not your reviewer.

The steelman: AI review is genuinely useful

Let's not pretend the tools are bad. Point an LLM at a diff and it will catch things a tired human on their fourth PR of the day will miss: a missing null check, a docstring that no longer matches the function signature, an SQL query concatenated instead of parameterized, an off-by-one in a loop bound. It does this in seconds, for free, on every single PR, without getting bored or defensive. Teams report real velocity gains from using it as a first pass — fewer round-trips, fewer "please add a test for the empty case" comments a human reviewer would've had to type out by hand.

None of that is in dispute. The problem isn't that AI review is bad at reviewing. The problem is that nobody has decided who reviews it.

Why "who reviews the reviewer" isn't a philosophical question

Run the same diff through the same model twice and you can get two different verdicts. That's not a bug you can file — it's the nature of the tool. A human reviewer who flip-flops on the same code gets a performance conversation. An LLM that flip-flops gets nothing, because nobody's tracking it. There's no regression suite for the reviewer, no versioned behavior, no way to say "this is worse than it was last month," because you never wrote down what "good" looked like in the first place.

Compare that to everything else that gets to block a merge in a mature pipeline: a test suite has a known pass/fail history. A linter has a changelog. A coverage threshold is a number you can graph over time. When any of those regress, you can point at the exact commit that broke them. When your AI reviewer starts approving diffs it should reject, you find out from the incident, not from the pipeline.

That asymmetry is the actual hot take: it's not that AI review is untrustworthy, it's that teams have quietly given a non-deterministic, non-versioned, non-tested system the same authority as a deterministic one — and then act surprised when it behaves like the former.

The fix: keep the gate deterministic, keep the AI advisory

If you're running CircleCI, this is a fifteen-minute change to your config.yml, not a philosophical debate. Split "AI says this looks fine" from "this diff is allowed to merge" into two separate workflow jobs, and only put the deterministic one in branch protection's required checks.

version: 2.1

workflows:
  pr-checks:
    jobs:
      - test-and-lint:
          filters:
            branches:
              ignore: main
      - ai-review:
          filters:
            branches:
              ignore: main
          requires: []   # runs in parallel, blocks nothing

jobs:
  test-and-lint:
    docker:
      - image: cimg/node:20.11
    steps:
      - checkout
      - run: npm ci
      - run: npm run lint
      - run: npm test -- --coverage
      - run:
          name: Enforce coverage floor
          command: npx nyc check-coverage --lines 80

  ai-review:
    docker:
      - image: cimg/node:20.11
    steps:
      - checkout
      - run:
          name: Post advisory AI review comment
          command: node scripts/ai-review.js --post-comment --no-fail-on-issues

The --no-fail-on-issues flag on the AI review job isn't a suggestion, it's load-bearing: that job's exit code should never be able to fail the build. In GitHub or GitLab branch protection, only test-and-lint goes in the required-checks list. The AI reviewer gets to talk. It doesn't get to vote.

Test the reviewer the way you test everything else

Here's the part most teams skip entirely: if you're going to keep an AI reviewer around — and you should, it's useful — give it the same regression discipline you give your code. Build a small golden set of past PRs where you already know the right call: five that should have been rejected (the null-check miss, the SQL injection, the empty-array bug that got through), five that were legitimately fine. Store the diffs and the expected verdicts in the repo.

Then add a scheduled CircleCI pipeline — nightly or weekly — that replays that golden set through whatever model and prompt you're currently using, and fails loudly if the verdicts drift:

workflows:
  nightly-reviewer-regression:
    triggers:
      - schedule:
          cron: "0 6 * * *"
          filters:
            branches:
              only: main
    jobs:
      - reviewer-regression-test

jobs:
  reviewer-regression-test:
    docker:
      - image: cimg/node:20.11
    steps:
      - checkout
      - run:
          name: Replay golden PR set against current reviewer
          command: node scripts/reviewer-regression.js --golden-set ./fixtures/reviewer-golden --fail-on-mismatch

This costs you maybe an hour of setup and a handful of pipeline credits a month. What it buys you is the one thing "AI promoted every developer to reviewer" discourse keeps skipping past: proof, over time, that the reviewer you're trusting today still catches the bug it caught six months ago, on the model version you're actually running now — not the one you tested when you first turned it on.

Landing the argument

The hot take isn't "don't use AI to review code." It's that review authority and review assistance are different jobs, and collapsing them into one CI step is how a genuinely useful tool quietly becomes an ungoverned one. Keep your merge gate boring, deterministic, and versioned — tests, lint, coverage, security scan. Let the AI reviewer talk as loudly as it wants in the PR comments. And once a quarter, put the reviewer itself through the same regression suite you'd demand of any other piece of code that gets to have an opinion about what ships. If you wouldn't merge a linter you'd never run against a test case, don't merge-gate on a reviewer you've never tested either.

Three Unrelated Pipelines, One Root Cause: Building a CircleCI Gate That Catches Broken LLM JSON Before It Ships

Mukesh — Fri, 28 Aug 2026 19:04:59 +0000

Yesterday, three completely unrelated pipelines in my system failed the same way within 48 hours.

One generates blog articles. One generates local-business outreach demos. One generates client proposals. They don't share a codebase, a prompt template, or an owner. What they share is a single line buried in each of them: json.loads(response).

All three broke with a variation of the same error: "content generated but JSON parse failed." Not a crash — worse. A silent stall. The model did its job, produced something, and the parser choked on it downstream, after the artifact was already written to disk. By the time anyone noticed, the failure was three layers removed from its cause.

If you're shipping any product where an LLM's output gets parsed as structured data — and at this point, whose isn't — this is worth thirty minutes of your CI pipeline's time, because it will happen to you too, and it will happen more than once.

Why this keeps happening

The instinct is to treat "parse the model's JSON" as a one-line implementation detail. It isn't. It's a contract between two systems that drift independently:

The model provider ships a checkpoint update and the model gets more polite — it now wraps JSON in a friendly sentence ("Sure, here's the JSON you asked for:") that it didn't add last week.
Someone edits a prompt template for an unrelated reason and accidentally changes whether the model treats a trailing comma as fine.
A field that used to always be a string starts arriving as null on some fraction of calls because the model decided ambiguity was better represented that way.

None of these are your bugs, exactly. But they become your outage, because your parser has zero tolerance for any of them, and nothing tells you the contract changed until a human notices missing output days later.

The fix that doesn't scale: patch each pipeline

The first move is obvious and it's what I did initially — wrap the json.loads call, catch the exception, log it, retry once with a stricter re-prompt. Something like:

import json
import re

def parse_llm_json(raw: str) -> dict:
    # Strip markdown code fences if the model added them
    match = re.search(r"```

(?:json)?\s*(\{.*\})\s*

```", raw, re.DOTALL)
    candidate = match.group(1) if match else raw.strip()
    return json.loads(candidate)

def generate_with_repair(prompt: str, call_model, max_attempts: int = 2) -> dict:
    last_error = None
    for attempt in range(max_attempts):
        raw = call_model(prompt if attempt == 0 else
            f"{prompt}\n\nYour previous response failed to parse as JSON "
            f"({last_error}). Return ONLY valid JSON, no prose, no code fences.")
        try:
            return parse_llm_json(raw)
        except json.JSONDecodeError as e:
            last_error = str(e)
    raise ValueError(f"Failed to get valid JSON after {max_attempts} attempts: {last_error}")

This works. It also completely misses the point, because I wrote a version of this three separate times, once per pipeline, and the underlying contract violation still wasn't caught until runtime — after real API spend, after a task was already marked in-progress, after the failure had to be discovered rather than prevented.

The deeper mistake was validating after the artifact was generated and stored, instead of before the task was allowed to complete. Downstream validation finds the bug. It doesn't stop three independent teams (or three independent pipelines, if you're a solo dev) from rediscovering it separately.

The fix that scales: treat it as a contract test, run in CI

The actual fix wasn't a better try/except. It was moving the check to a layer where a bad contract gets caught before it ships, on every commit that touches a prompt template or a parsing schema — not after a customer-facing task fails.

Step one: freeze a fixture set of real model responses, both the ones that parsed fine and the malformed ones that caused the actual incidents. Keep them as JSONL, one response per line, alongside an expected outcome:

{"raw": "{\"title\": \"Post\", \"tags\": [\"a\"]}", "should_parse": true}
{"raw": "Sure, here's the JSON:\n```

json\n{\"title\": \"Post\"}\n

```", "should_parse": true}
{"raw": "{\"title\": \"Post\", \"tags\": [\"a\",]}", "should_parse": false}

Step two: a pytest suite that runs the real parser against every fixture and asserts the outcome matches:

import json
import pytest
from pipeline.parsing import parse_llm_json

def load_fixtures():
    with open("fixtures/llm_responses.jsonl") as f:
        return [json.loads(line) for line in f]

@pytest.mark.parametrize("case", load_fixtures())
def test_parser_contract(case):
    if case["should_parse"]:
        assert parse_llm_json(case["raw"])  # must not raise
    else:
        with pytest.raises(json.JSONDecodeError):
            parse_llm_json(case["raw"])

Step three, the part that actually stopped the repeat: a dedicated CircleCI job gating anything touching prompts or parsing code.

version: 2.1

jobs:
  llm-contract-test:
    docker:
      - image: cimg/python:3.12
    steps:
      - checkout
      - run: pip install -r requirements.txt
      - run:
          name: Run LLM output contract tests
          command: pytest tests/test_llm_contract.py -v

workflows:
  version: 2
  build-and-test:
    jobs:
      - llm-contract-test:
          filters:
            branches:
              only: /.*/

Every time a prompt template, a parser, or a schema changes, this job runs the full fixture set — the good responses and the bad ones I've actually seen in production — in about eight seconds. If a change makes the parser reject a response it used to accept, or accept one it used to (correctly) reject, the build fails before merge, not three pipelines and three days later.

What changed operationally

The fixture file is now append-only: every time a pipeline hits a new malformed-response shape in production, it gets added to fixtures/llm_responses.jsonl as a new should_parse: false case before the fix ships. That turns every incident into a permanent regression test instead of a one-off patch. Three months in, the fixture set has caught two prompt-template edits that would have silently broken parsing again — both caught in CI, both zero-impact in production.

The lesson wasn't "add a try/except." It was that an LLM's output shape is an interface with a version history, and the same discipline you'd apply to an external API contract — recorded fixtures, explicit pass/fail cases, a CI gate — applies here too. The parsing bug wasn't three bugs in three pipelines. It was one missing test suite.

Give Your Mem0 Agent Session-Scoped Memory in 15 Minutes (One Filter You're Probably Skipping)

Mukesh — Thu, 27 Aug 2026 19:11:49 +0000

You add memory to your agent with Mem0, ship it, and it works great in your dev environment where you're the only user. Then you go multi-tenant — real users, real sessions — and three weeks later someone reports that the agent "remembers" something they never told it. It's not a hallucination. It's another user's memory, served straight from your own vector store.

This is the single most common Mem0 integration bug I run into, and the fix is one filter you're probably not passing consistently. Here's the 15-minute version.

The setup that causes it

Most Mem0 quickstarts look like this:

from mem0 import Memory

m = Memory()

m.add(
    "I prefer flights with no layovers and I'm vegetarian",
    user_id="alice",
)

That looks scoped — you passed user_id="alice". The bug isn't in add(). It's in search(), three files away, written by a different part of the team (or you, two sprints later) without the same discipline:

# somewhere in the RAG/retrieval layer
relevant = m.search(query="what are the user's travel preferences?")

No user_id. No filters. Mem0 will happily return the closest semantic matches across every memory in the store — Alice's vegetarian preference bleeding into Bob's session, or worse, into an agent that's actively talking to Bob. The write path was scoped. The read path wasn't. Because both calls succeed and return plausible-looking data, this ships, passes QA (one tester, one session), and only shows up once you have concurrent real users.

The one-line fix

Every search() call needs the same scoping identity as the add() call that created the memory. If you passed user_id on write, pass it on every read:

relevant = m.search(
    query="what are the user's travel preferences?",
    user_id="alice",
)

That's the 80% fix. But scoping by user_id alone isn't enough once you have more than one agent or more than one conversation thread per user — which is most real products within a month of launch.

The part people miss: agent_id and run_id

Mem0 supports three identity dimensions, not one: user_id, agent_id, and run_id. If your product has multiple agents (a support bot and a booking bot, say) sharing the same user base, scoping by user_id alone means the booking bot's memories leak into the support bot's context — technically the right user, wrong agent, still a correctness bug.

m.add(
    "User wants the booking bot to always confirm price before charging",
    user_id="alice",
    agent_id="booking-bot",
)

m.add(
    "User asked support to stop sending SMS notifications",
    user_id="alice",
    agent_id="support-bot",
)

# retrieval inside booking-bot's context
relevant = m.search(
    query="payment preferences",
    user_id="alice",
    agent_id="booking-bot",
)

Without agent_id on both calls, search() from the booking bot can surface the SMS-notification memory that belongs to a completely different conversational context. It's not wrong data exactly — it's real, it's Alice's — but it's the wrong memory for this agent to be reasoning with, and it will show up in the prompt as if it's relevant.

run_id is the same idea one level down: scope to a single session or task run when you don't want memory to carry across unrelated conversations with the same agent (a returning support ticket vs. an old, resolved one, for example).

Advanced filtering for anything beyond exact match

Once you're past simple identity scoping, Mem0's platform API accepts a filters dict with AND/OR logic on top of metadata you attach at write time — useful for things like "only memories from the last 30 days" or "only memories tagged billing":

m.add(
    "Card ending 4242 declined for insufficient funds",
    user_id="alice",
    agent_id="billing-bot",
    metadata={"category": "payment_issue", "resolved": False},
)

relevant = m.search(
    query="payment issues",
    user_id="alice",
    agent_id="billing-bot",
    filters={
        "AND": [
            {"category": "payment_issue"},
            {"resolved": False},
        ]
    },
)

This is what turns "the agent remembers everything about this user" into "the agent remembers the right thing for this exact context" — which is the actual goal, not raw recall.

The 15-minute checklist

Go do this right now, it's faster than reading the rest of this article twice:

Grep your codebase for every .search( call against your Mem0 client.
For each one, check it passes the same user_id (and agent_id/run_id if you use them) as the add() calls that populate that memory space.
Any search() call missing scoping is a live cross-tenant leak — fix it before anything else on this list.
If two agents share a user base, add agent_id to every add/search pair, not just the ones you've noticed problems with.
Write one test: add a scoped memory for user_id="test-a", search as user_id="test-b", assert the result is empty. This is the regression test that catches the bug before your users do.

The underlying lesson generalizes past Mem0: any memory or retrieval layer that supports scoping only prevents leaks if scoping is enforced symmetrically on both write and read. Write-side discipline without read-side discipline isn't partial protection — it's a false sense of security with a matching demo that works perfectly until a second user shows up.

Mem0 vs Zep vs LangChain Memory vs Letta: Which One Actually Remembers?

Mukesh — Wed, 26 Aug 2026 20:03:18 +0000

Most "AI memory" demos are a vector store with a marketing label. You embed every message, cosine-search the top-k on the next turn, and call it memory. It works until turn 40, when the agent confidently tells a user their favorite color is blue because that's what came back highest-ranked — even though they corrected it three messages later.

Real memory isn't retrieval. It's deciding what's still true. That distinction is why four very different architectures — Mem0, Zep, LangChain's memory classes, and Letta (formerly MemGPT) — all claim the same territory but solve almost none of the same problems. Here's what each one actually does under the hood, where it breaks, and which one you should reach for.

LangChain memory: primitives, not a system

ConversationBufferMemory, ConversationSummaryMemory, ConversationKGMemory, VectorStoreRetrieverMemory — these are building blocks, not a memory service. You own the extraction logic, the storage schema, and every decision about what gets kept or discarded. ConversationSummaryMemory re-summarizes the whole history on every turn, which means cost and latency grow with conversation length even though the output size doesn't. ConversationKGMemory extracts triples but has no mechanism to invalidate a triple once a new fact contradicts it — old and new coexist in the graph, and retrieval has no way to prefer one.

This is fine if you want full control and are building something bespoke on top of LangGraph's checkpointing. It's the wrong choice if you want memory to just work, because "just work" is precisely the part LangChain leaves as an exercise for you.

Use it when: you're already deep in LangGraph, you have specific extraction logic you don't want a black box making decisions about, and you're willing to build conflict resolution yourself.

Mem0: LLM-mediated add/update/delete

Mem0's core loop is a two-stage LLM call. First, an extraction pass pulls candidate facts out of a message ("user prefers dark mode," "user is allergic to shellfish"). Second — and this is the part most memory layers skip — a second LLM call compares each candidate against the existing memories for that user and decides: ADD (net new), UPDATE (same entity, changed value), DELETE (contradicted), or NOOP (already known). That decision is what stops the shellfish-allergy memory from sitting next to a stale "user eats shrimp regularly" memory forever.

from mem0 import Memory
m = Memory()
m.add("I used to like coffee but I've switched to tea", user_id="u1")
# extraction: {preference: tea}, conflict check against
# existing {preference: coffee} -> UPDATE, not ADD

Memories are stored as embeddings in a pluggable vector store (Qdrant, Chroma, pgvector, Weaviate) with metadata, and Mem0 added a graph layer (Neo4j-backed) for relationship queries — "who does the user report to" style facts that a flat vector store handles badly. Addition is async by default, so it doesn't block your response path, which matters if you're calling add() after every turn in a latency-sensitive chat app.

The honest tradeoff: the conflict-resolution LLM call is an extra hop with extra cost and extra latency on the write path, and if your extraction prompt is too aggressive you'll get memory bloat — hundreds of low-value "facts" that dilute retrieval quality. Mem0 gives you knobs (custom extraction prompts, memory-type separation) but you still have to tune them.

Use it when: you have a multi-session, multi-user product (support bot, personal assistant, CRM copilot) where facts genuinely change over time and you need automatic reconciliation instead of a growing pile of contradictions.

Zep: temporal knowledge graphs

Zep's differentiator is Graphiti, its temporal-graph engine. Instead of choosing between "keep the old fact" or "overwrite it," Zep timestamps edges with both event time and ingestion time (a bi-temporal model) and marks superseded facts as invalid rather than deleting them. Ask Zep "where did the user work in 2023" and it can answer correctly even after the user has since changed jobs, because the graph retains history instead of collapsing to a single current value.

This is genuinely different from Mem0's ADD/UPDATE/DELETE model — Zep never deletes, it invalidates, which means you get an audit trail for free. That's valuable for compliance-sensitive domains but it's overkill if you only ever care about the current state of a fact and don't need to reason about when it changed.

Use it when: you need point-in-time correctness — support timelines, longitudinal user profiles, anything where "what did we believe was true at time X" is a real query, not just "what's true now."

Letta (MemGPT): the agent edits its own memory

Letta flips the architecture entirely. Instead of an external pipeline deciding what to remember, the LLM agent itself gets memory-editing functions (core_memory_append, core_memory_replace, archival_memory_insert) as tools it can call mid-conversation. Context is split into an OS-style hierarchy: core memory (small, always in the prompt, directly editable), and archival/recall memory (external, paged in via search when relevant). The agent decides, at inference time, what's worth writing down and what's worth paging back in.

The upside is nuance — the agent can decide "this is important enough for core memory" versus "this is archival trivia" based on actual conversational context, not a fixed extraction heuristic. The downside is cost and predictability: every memory operation is now an extra tool call inside the agent's own reasoning loop, which adds latency and makes memory writes non-deterministic across runs. Debugging "why didn't it remember X" means inspecting the agent's tool-call trace, not a pipeline log.

Use it when: you're building a long-running autonomous agent (not a request/response chatbot) where memory management is itself part of the task the agent should reason about — not a side effect you want abstracted away.

The decision, compressed

Need	Reach for
Full control, already on LangGraph	LangChain memory primitives
Multi-user product, facts change, want automatic reconciliation	Mem0
Need to know what was true when, audit trail	Zep
Autonomous long-running agent, memory as part of the task	Letta

The question to ask before picking any of these isn't "which has the best retrieval." Retrieval is the easy 20%. It's "who decides when a memory is wrong, and how do they find out." A vector store with no conflict resolution will happily retrieve a stale fact with high cosine similarity and hand it to your agent with total confidence. That's not memory — that's a very expensive way to remember things incorrectly.