DEV Community: Mukesh

Give Your AI Agent a Memory So It Stops Repeating the Same Failed Tool Call

Mukesh — Mon, 17 Aug 2026 19:07:12 +0000

Your agent calls a flaky API, gets a 429, retries with the same arguments, and gets rate-limited again. Ten minutes later, in a fresh session, it does the exact same thing. Nothing about the failure got remembered — the agent has no way to know it already learned this lesson, because the lesson lived in a transcript that got thrown away when the process exited.

This week a post about manually gatekeeping AI agent tool calls hit the front page of dev.to with 48 comments — mostly developers arguing over how much you can trust an agent's tool-calling loop. The honest answer is: you can trust it exactly as much as it remembers what already went wrong. Below is a complete, runnable gatekeeper that wraps any tool call, checks Mem0 for similar past failures before executing, and records the outcome afterward — so the second time your agent is about to make the same mistake, it knows.

What we're building

A MemoryGatekeeper class that sits between your agent's decision to call a tool and the actual execution:

Before calling a tool, search memory for semantically similar past attempts.
If a similar attempt failed recently, block the call and return the remembered reason instead of burning a real API call.
After every call — success or failure — write the outcome back to memory.

You'll run the same script twice: the first run fails and gets recorded, the second run gets blocked before it wastes a request.

Setup

You need Python 3.10+, an OpenAI API key (Mem0's default extraction pipeline uses it to turn raw text into structured memories — no separate Mem0 account required for this local setup), and the mem0ai package.

pip install mem0ai
export OPENAI_API_KEY=sk-...

Mem0 defaults to a local, on-disk vector store, so nothing here talks to a hosted Mem0 service — it's a fully self-contained memory layer you own.

Step 1: The gatekeeper

# gatekeeper.py
from mem0 import Memory
from datetime import datetime, timezone

class MemoryGatekeeper:
    def __init__(self, agent_id="default-agent", block_threshold=0.75):
        self.memory = Memory()
        self.agent_id = agent_id
        self.block_threshold = block_threshold

    def _describe_call(self, tool_name, args):
        return f"tool call: {tool_name} with args {args}"

    def check(self, tool_name, args):
        """Returns (allowed: bool, reason: str | None)."""
        query = self._describe_call(tool_name, args)
        hits = self.memory.search(query, user_id=self.agent_id, limit=3)

        for hit in hits.get("results", []):
            score = hit.get("score", 0)
            memory_text = hit.get("memory", "")
            if score >= self.block_threshold and "failed" in memory_text.lower():
                return False, memory_text
        return True, None

    def record(self, tool_name, args, success, detail):
        outcome = "succeeded" if success else "failed"
        text = (
            f"{self._describe_call(tool_name, args)} {outcome} "
            f"at {datetime.now(timezone.utc).isoformat()}: {detail}"
        )
        self.memory.add(text, user_id=self.agent_id)

    def call(self, tool_name, args, fn):
        allowed, reason = self.check(tool_name, args)
        if not allowed:
            print(f"[BLOCKED] {tool_name}({args}) — remembered: {reason}")
            return {"blocked": True, "reason": reason}

        try:
            result = fn(*args.values()) if isinstance(args, dict) else fn(args)
            self.record(tool_name, args, True, "ok")
            return {"blocked": False, "result": result}
        except Exception as exc:
            self.record(tool_name, args, False, str(exc))
            raise

The check step uses semantic search, not exact string matching — fetch_weather(city="NYC") and fetch_weather(city="New York") will match each other, which is exactly the kind of near-duplicate an exact-match cache would miss.

Step 2: A tool that fails predictably

# flaky_tool.py
class RateLimitError(Exception):
    pass

def call_flaky_api(endpoint):
    if endpoint == "/reports/daily":
        raise RateLimitError("429: rate limit exceeded, retry after 3600s")
    return {"status": "ok", "endpoint": endpoint}

Step 3: Run it twice

# run.py
from gatekeeper import MemoryGatekeeper
from flaky_tool import call_flaky_api

gate = MemoryGatekeeper(agent_id="report-agent")

try:
    gate.call("call_flaky_api", {"endpoint": "/reports/daily"}, call_flaky_api)
except Exception as exc:
    print(f"First attempt failed as expected: {exc}")

First run:

First attempt failed as expected: 429: rate limit exceeded, retry after 3600s

Run python run.py again — same process, same tool, same arguments, but now the agent has memory of the earlier failure:

[BLOCKED] call_flaky_api({'endpoint': '/reports/daily'}) — remembered: tool call: call_flaky_api with args {'endpoint': '/reports/daily'} failed at 2026-08-18T09:12:04+00:00: 429: rate limit exceeded, retry after 3600s

No second API call, no second rate-limit hit, and the agent gets a reason it can reason about (or relay to a human) instead of a raw stack trace.

Tuning it for real use

Three knobs matter once you move past the toy example:

block_threshold — 0.75 is conservative on purpose. Push it lower and you'll block near-misses that would have actually succeeded; push it higher and you'll only catch near-identical repeats. Log every block decision for a week before trusting the default.
Scope memory per agent, not globally. user_id=self.agent_id keeps one agent's bad luck from blocking a different agent's legitimate call to the same tool. If multiple agents genuinely share risk (same downstream API, same rate limit bucket), give them a shared agent_id.
Expire failures, don't keep them forever. A 429 from an hour ago should block a retry; a 429 from three weeks ago probably shouldn't. Mem0 memories carry timestamps in their metadata — add a check in check() that discards hits older than your retry window (e.g., time.time() - hit["created_at"] < 3600) so stale failures don't permanently disable a tool that's since recovered. This one-line filter is the difference between a gatekeeper and a agent that's afraid of everything it's ever failed at once.

Why this beats an in-process cache

A plain dict or functools.lru_cache would catch the exact-repeat case in a single run, but it dies with the process and can't do semantic matching — it won't know that retrying /reports/daily and /reports/daily/ are the same mistake. The value of routing this through Mem0 specifically is that the memory persists across restarts and deployments, and the same store can also hold successful patterns — tool calls that worked, arguments that were well-formed, sequences that completed cleanly — so the gatekeeper isn't just a blocklist, it's the beginning of an agent that actually gets better at using its tools over time instead of relearning the same lesson every cold start.

The full example above is under 80 lines and runs with nothing but a Python environment and an OpenAI key — clone it, run it twice, and you'll see the block happen live before you've read the rest of this sentence.

Inside HNSW: The Graph Algorithm That Makes Your AI Agent's Memory Fast (and Sometimes Wrong)

Mukesh — Sun, 16 Aug 2026 18:32:02 +0000

Every time an AI agent calls something like memory.search(query, limit=5), a graph traversal runs underneath it that most developers never look at. It's not a database index in the traditional sense — it's closer to a probabilistic subway map, and understanding how it's built explains both why vector search is fast and why it occasionally hands your agent the wrong memory with full confidence.

The algorithm is called HNSW — Hierarchical Navigable Small World graphs. It's the default index type in Faiss, hnswlib, pgvector, Qdrant, Weaviate, and most managed vector stores, which means if you've built a RAG pipeline or given an agent long-term memory, HNSW is already running your retrieval whether you chose it explicitly or not.

The problem it solves

Exact nearest-neighbor search means comparing your query vector against every stored vector and sorting by distance. For 10,000 memories at 768 dimensions, that's fine — a few milliseconds. At 10 million memories, brute force means scanning 10 million floating-point comparisons per query, every query. That doesn't scale, and it definitely doesn't scale to an agent calling memory search dozens of times per conversation.

HNSW trades exactness for speed. It doesn't guarantee it finds your true nearest neighbor — it guarantees it finds a very good neighbor, almost all of the time, in roughly logarithmic time instead of linear time. That "almost all of the time" is the detail that matters once you're building on top of it.

The structure: layers of shortcuts

HNSW builds a multi-layer graph. Picture a skip list, but in vector space instead of a sorted array.

Layer 0 (the bottom) contains every vector you've inserted, each one connected to its nearest neighbors — typically 16 to 64 edges, controlled by a parameter called M.
Higher layers contain progressively smaller random subsets of those same vectors, with sparser connections. A vector's odds of appearing in layer 3 are exponentially smaller than appearing in layer 0 (the assignment uses a randomized exponential decay, same trick skip lists use).

The top layers act as highways — a handful of nodes with long-range connections that let a search jump across the vector space quickly. The bottom layer is the local street grid — dense, short connections for fine-grained precision once you're close to the answer.

How a search actually runs

When you call .search(query_vector, k=5), here's what happens under the hood:

Start at a fixed entry point in the topmost layer.
Greedily walk toward the neighbor closest to your query vector, one hop at a time, until no neighbor at this layer is closer than your current node.
Drop down one layer, using your current position as the new starting point.
Repeat the greedy walk at this layer, which is denser and more accurate.
Once you reach layer 0, instead of taking just the single closest node, HNSW keeps a candidate list of size ef (the search-time beam width) and explores that many promising paths before returning the top k.

This is why HNSW search is roughly O(log n) instead of O(n): each layer eliminates most of the graph before you ever reach the dense bottom layer where the real comparisons happen.

The three knobs that actually matter

Most HNSW tuning guides list parameters without explaining what breaks when you get them wrong. In practice, three matter:

M (edges per node, insertion time). Higher M means a denser graph — better recall, but more memory and slower inserts. Going from M=16 to M=48 roughly triples index memory for maybe a 3-5% recall gain past a certain dataset size. For memory stores under a few million vectors, M=16-32 is almost always the right range; don't reach for 64 unless benchmarks tell you to.

efConstruction (candidate list size during insertion). This controls how thoroughly the graph is built when a vector is added. Low efConstruction (say, 40) builds fast but leaves the graph with worse long-term recall — and you can't fix it later without rebuilding. This is the parameter people forget until they're debugging why search quality degraded after a bulk import: it was set too low at write time, not read time.

efSearch / ef (candidate list size during query). This is the one lever you can tune live, per query, without rebuilding anything. Raise it and you trade latency for recall. On the classic glove-100 ann-benchmark dataset, going from ef=10 to ef=100 typically moves recall@10 from around 85% to 98%+, at maybe 3-4x the query latency — still single-digit milliseconds either way at that scale.

Why this matters specifically for agent memory

A vector database used for product search can tolerate 85% recall — a slightly-off search result is a minor UX blemish. An agent's long-term memory is different: if the memory holding the user's actual preference or a past correction doesn't make it into the top-k, the agent doesn't know it's missing anything. There's no error, no exception, no log line. The agent just answers as if that memory never existed.

This is the practical reason to treat ef as a first-class config value in a memory system, not an implementation detail buried in a client library default. Two changes are worth making explicitly:

Set efSearch higher than the library default for anything retrieval-critical. Most client libraries default ef somewhere in the 10-50 range, tuned for throughput benchmarks, not recall. For agent memory where a miss is invisible and costly, bias toward recall — ef=100-200 is still fast enough for interactive use, and the latency difference is milliseconds a user will never notice.
Re-embed and rebuild, don't just re-tune, after large deletions. HNSW graphs degrade gracefully on insert but not on delete — most implementations soft-delete (mark and skip) rather than actually removing edges, which means a memory store with heavy churn slowly accumulates dead-end paths that waste traversal steps and quietly lower effective recall. If your agent forgets and relearns things constantly, periodic index rebuilds aren't housekeeping, they're a correctness fix.

The single takeaway worth carrying into any vector-backed memory system: HNSW's speed comes from being probabilistically honest, not exact. Every default configuration is a bet about how much wrongness is acceptable, made by someone who was optimizing for a benchmark, not for whether your agent remembers what your user told it last week. Read the ef values before you trust the recall.

The 7.4% You Don't See: Checkpointing Long LLM Jobs Before They Time Out

Mukesh — Fri, 14 Aug 2026 18:56:17 +0000

Two jobs failed on my agent's VPS on the same day, for two different reasons, and it took me longer than I'd like to admit to notice they were the same bug wearing different clothes.

The agent I run does a mix of unglamorous background work — writing articles, generating demo pages, making paper-trading decisions — by calling an LLM, waiting for a response, and doing something with the output. Nothing exotic. It runs on a single Vultr instance, not behind a queue-and-retry serverless setup, because most of the work is sequential and cheap enough that a VPS with a cron loop is the right amount of infrastructure. Right up until it wasn't.

Failure one: the timeout that ate a full generation

One task needed to write a multi-section HTML demo page. Big prompt, big expected output, one LLM call. It hit my 480-second timeout. Another task, generating a longer article, did the same thing an hour later. Two failures out of 27 jobs that day — 7.4%, which sounds small until you notice both failures were on the largest jobs, and I was actively shifting more of the day's workload toward longer-context tasks. The failure rate on big jobs wasn't 7.4%. It was closer to 100% of anything that pushed the token budget.

The part that stung wasn't the timeout itself — LLM calls time out, that's expected — it was that the entire generation was gone. Five minutes of inference, thrown away, because the process that owned the HTTP connection died with nothing written to disk.

Failure two: the parse that ate a finished article

Separately, a content-writing task logged: "Article generated but JSON parse failed." The LLM had done its job. It returned a full article. My code just couldn't get it out of the response, because I was asking the model to return {"title": ..., "article_markdown": ...} and something — a stray backtick, an unescaped quote, a truncated tail — broke the parse. The generation succeeded. The pipeline still lost it, because the only place that article ever existed was a Python variable that went out of scope when the exception propagated.

Different failure mode, identical shape: expensive work completed, then discarded, because nothing was written to disk until after a step that could fail.

The fix is boring, which is why it works

The instinct when you see a timeout is to add retries. The instinct when you see a parse failure is to make the parser more forgiving. Both are reasonable and I did some of both, but neither addresses the actual problem: the job has no durable state until the very last step succeeds. Retrying a job that has no checkpoint just means paying for the same 480 seconds again.

The actual fix has two parts, and both are just "write to disk earlier than you think you need to."

Part 1 — save the raw output before you trust it.

Before attempting to parse or validate an LLM response, write it verbatim to a fallback path keyed by timestamp:

import json
from datetime import datetime, timezone

def call_llm_with_fallback(prompt, artifacts_dir):
    response_text = llm_client.complete(prompt)

    ts = datetime.now(timezone.utc).strftime("%Y%m%d_%H%M%S")
    fallback_path = f"{artifacts_dir}/{ts}_raw_fallback.txt"
    with open(fallback_path, "w") as f:
        f.write(response_text)

    try:
        return json.loads(response_text), None
    except json.JSONDecodeError as e:
        return None, fallback_path

This is maybe six lines. It costs one disk write per LLM call, which is nothing compared to the seconds of inference you just paid for. If the parse fails downstream, the content isn't lost — it's sitting in artifacts/20260813_204701_raw_fallback.txt, recoverable by a human or a repair script. I went from "article generated but lost" to "article generated, parse failed, here's the raw markdown" with no change to the happy path at all.

Part 2 — checkpoint sections, don't checkpoint the whole job.

For the large multi-section generations that were timing out, the fix was to stop asking for one giant completion and start asking for one completion per section, writing each one to a checkpoint file as it lands:

def generate_with_checkpoints(sections, job_id, artifacts_dir):
    checkpoint_path = f"{artifacts_dir}/{job_id}_checkpoint.json"
    state = _load_checkpoint(checkpoint_path)  # {} if none exists

    for section in sections:
        if section["name"] in state:
            continue  # already done, resume past it
        state[section["name"]] = llm_client.complete(section["prompt"])
        with open(checkpoint_path, "w") as f:
            json.dump(state, f)

    return state

If the process dies on section four of six — timeout, crash, OOM, doesn't matter — the next run picks up at section four instead of starting over. This also had a side effect I didn't anticipate: individual section prompts are smaller, so each one is less likely to hit the timeout in the first place. Splitting the job didn't just make failure cheaper, it made failure less frequent.

Why this matters more on a VPS than it would on managed infra

On a platform that gives you automatic retries, dead-letter queues, and step-level state out of the box, some of this comes for free. On a single VPS running a cron loop or a long-lived worker process, you own all of it. That's not a downside — it's usually the right tradeoff for workloads that are sequential, low-volume, and don't need the operational overhead of a queue system — but it means the durability has to be designed in explicitly, at the point where you'd otherwise just trust the next line of code to run.

The pattern generalizes past LLM calls: any job where step N is expensive and step N+1 can fail should write N's output to disk before attempting N+1. It's a five-minute change per job. I made it after losing two jobs in one day; I'd recommend making it before that happens to you, because by the time you notice the failure rate, you've already lost the work that would have told you it was a problem.

The Airbyte Column Nobody Queries: Find Every Silent Sync Failure in 15 Minutes

Mukesh — Thu, 13 Aug 2026 18:46:48 +0000

You added a new field to your CRM. Three weeks later, someone in finance asks why 4% of deal values in the warehouse are NULL when the CRM UI clearly shows a number. You check the connector — green checkmark, every sync. You check the destination table — data's there, mostly. What you don't check, because almost nobody does, is the one column Airbyte already used to tell you exactly what happened: _airbyte_meta.

If you're running Airbyte with Destinations V2 (the default since late 2023 for Snowflake, BigQuery, Postgres, and most warehouse destinations), every raw table Airbyte writes gets four metadata columns alongside your actual data: _airbyte_raw_id, _airbyte_extracted_at, _airbyte_generation_id, and _airbyte_meta. That last one is a JSON blob, and it is not decoration. It contains a changes array that records, per record, per field, exactly when Airbyte had to alter or drop a value to get it into your destination — and why.

This matters because Airbyte's typing and deduping step is lenient by design. If a source sends a string where your destination schema expects an integer, or a value too large for the target column, or malformed JSON in a JSON column, Airbyte doesn't fail the sync. It nulls the field, writes the row anyway, and logs a change record in _airbyte_meta. That's the right behavior for pipeline reliability — a single bad field shouldn't take down an entire sync — but it means data quality problems ship into your warehouse silently unless you go looking for them.

What's actually in `_airbyte_meta`

The shape is consistent across destinations. Each row's _airbyte_meta looks roughly like this:

{
  "sync_id": 1234,
  "changes": [
    {
      "field": "deal_value",
      "change": "NULLED",
      "reason": "DESTINATION_TYPECAST_ERROR"
    }
  ]
}

The changes array is empty for the overwhelming majority of rows — which is exactly why it's easy to ignore. It only fills in for the rows worth caring about. The reason codes you'll see most often are DESTINATION_TYPECAST_ERROR (value didn't match the destination column type), DESTINATION_SERIALIZATION_ERROR (value couldn't be serialized, often oversized numbers or malformed nested objects), DESTINATION_RECORD_SIZE_LIMITATION (the row exceeded a destination size cap), and SOURCE_RECORD_SIZE_LIMITATION. Each is a distinct, actionable failure mode — not noise.

The 15-minute query

Pick any raw stream table you actually rely on downstream and run this. Snowflake:

select
  _airbyte_extracted_at::date as sync_date,
  f.value:field::string   as field_name,
  f.value:change::string  as change_type,
  f.value:reason::string  as reason,
  count(*) as affected_rows
from raw_schema.your_stream,
     lateral flatten(input => _airbyte_meta:changes) f
group by 1, 2, 3, 4
order by affected_rows desc;

Postgres (destinations that write meta as jsonb):

select
  _airbyte_extracted_at::date as sync_date,
  c->>'field'  as field_name,
  c->>'change' as change_type,
  c->>'reason' as reason,
  count(*) as affected_rows
from raw_schema.your_stream,
     jsonb_array_elements(_airbyte_meta->'changes') c
group by 1, 2, 3, 4
order by affected_rows desc;

BigQuery needs one extra unnest step since _airbyte_meta is a JSON type there:

select
  date(_airbyte_extracted_at) as sync_date,
  json_value(c, '$.field')  as field_name,
  json_value(c, '$.change') as change_type,
  json_value(c, '$.reason') as reason,
  count(*) as affected_rows
from `raw_dataset.your_stream`,
     unnest(json_query_array(_airbyte_meta, '$.changes')) as c
group by 1, 2, 3, 4
order by affected_rows desc;

Run that against your two or three most business-critical streams and you'll usually find at least one field with a nonzero count within the first minute. On a mid-sized CRM or billing sync, seeing 1-5% of rows carrying a DESTINATION_TYPECAST_ERROR on a specific field is common — and it's almost always traceable to a source schema change (a field that used to be numeric now sometimes carries a currency symbol, a boolean field that started returning "yes"/"no" instead of true/false) that nobody flagged because the sync itself never went red.

Turning it into a standing check, not a one-off

The query above is the audit. The fix that actually pays off is making it recurring, because schema drift at the source is exactly the kind of thing that reappears. If you're on dbt, this is a natural custom singular test:

-- tests/assert_no_new_airbyte_typecast_errors.sql
select *
from {{ source('raw', 'your_stream') }},
     lateral flatten(input => _airbyte_meta:changes) f
where _airbyte_extracted_at >= dateadd(day, -1, current_date())
  and f.value:reason::string = 'DESTINATION_TYPECAST_ERROR'
having count(*) > 0

A failing test here fails your dbt run loudly, in CI, the same day the source starts sending a value your schema can't hold — instead of three weeks later when someone in finance notices a number that doesn't add up. If you don't run dbt, the same query wrapped in a five-minute cron job with a Slack webhook on nonzero rows gets you the same outcome.

The underlying lesson generalizes past Airbyte: any tool that trades hard failures for graceful degradation is quietly generating a log of what it degraded. That log is only useful if something reads it. Fifteen minutes writing one query against a column you're already paying to store is a lot cheaper than the meeting where you explain why a warehouse number has been wrong for a month.

Auth0 vs Clerk vs Supabase Auth vs Firebase Auth: What Actually Breaks When You Add Real Auth to a Real App

Mukesh — Wed, 12 Aug 2026 18:41:40 +0000

The Decision You Only Make Once (and Regret for Years)

Auth is one of the few infrastructure choices that's genuinely expensive to reverse. You don't swap identity providers the way you swap a logging library — every user has a session, a password hash (or lack of one), MFA enrollment, and a dozen third-party integrations wired to your JWTs. So when a team asks "which auth service should we use," the honest answer isn't "it depends," it's "it depends on exactly these four things, and here's how each option fails you."

I've now shipped production apps on Auth0, Clerk, Supabase Auth, and Firebase Auth. Here's where each one actually breaks, not the marketing-page version.

Setup Speed and Day-One DX

Clerk wins this outright. Drop in <SignIn /> and <UserButton />, and you have a working auth flow with prebuilt, themeable UI in under 30 minutes — social login, MFA, and session management included, no redirect-based hosted page required unless you want one. It's clearly built by people who shipped a SaaS app themselves and got tired of gluing Auth0's SDK to a design system.

Firebase Auth is nearly as fast if you're already in the Firebase/GCP ecosystem — signInWithPopup and you're done — but the moment you need custom UI, you're hand-rolling forms against the SDK, because Firebase never shipped real prebuilt components, just a dated "FirebaseUI" library nobody maintains anymore.

Auth0 is the slowest of the four to get right, not because it's hard, but because it has three different ways to do everything (Universal Login vs. embedded login, Rules vs. Actions vs. the older Hooks) and picking wrong costs you a rewrite later. Supabase Auth sits in between: fast if you're already using Supabase for Postgres, painful if you're not, because you inherit the whole Supabase client just to get a session token.

The Pricing Cliff Nobody Reads Until It's Too Late

This is where the real decision gets made, and it's rarely about the sticker price.

Auth0's free tier caps out fast, and the jump to a paid plan isn't gradual — it's a cliff at a fixed MAU count, after which per-user costs escalate quickly, especially once you add MFA, custom domains, or Organizations (B2B multi-tenant support) as separate line items. I've seen teams get a surprise five-figure invoice not from usage growth but from turning on a feature they assumed was included.

Clerk's free tier is genuinely generous for early-stage products, and its pricing scales more linearly — you feel the cost coming before it hits. Firebase Auth is the cheapest at real scale for pure B2C: it's priced per verification (phone/SMS) or effectively free for email/social, since Google isn't trying to monetize auth itself, just the rest of GCP around it. Supabase Auth is bundled into Supabase's project pricing, which means it's cheap in isolation but you're really paying for the whole platform — fine if you wanted Postgres + Auth + Storage together, wasteful if you only wanted auth.

The practical rule: if you can forecast your MAU growth curve with any confidence, price out Auth0 and Clerk at 2x your 12-month projection before committing, not at today's numbers. That cliff is the thing that actually changes vendor decisions mid-flight.

Multi-Tenancy and B2B Orgs: Where Three of the Four Fall Short

If you're building B2B SaaS with organizations, roles, and per-tenant SSO, Auth0 is the only one of the four with this genuinely built in and battle-tested — Organizations, SAML/OIDC federation per tenant, and enterprise connections are first-class, not bolted on. This is the single strongest reason to pick Auth0 over the alternatives, and it's also the reason its pricing model exists the way it does — you're paying for compliance surface area, not just login forms.

Clerk added Organizations support later and it's solid for straightforward multi-tenant B2B, but it doesn't yet match Auth0's depth on enterprise SSO federation across many customer IdPs. Supabase Auth has no native organization/multi-tenancy concept at all — you build it yourself with Postgres Row Level Security and a tenant_id column, which is powerful if you're comfortable owning that logic, and a real gap if you assumed the vendor handled it. Firebase Auth is squarely B2C-shaped; retrofitting B2B tenancy onto it is possible but fights the grain of the product the whole way.

Customization Depth vs. Lock-In Risk

Auth0's Actions (its post-login/pre-token hook system) let you inject arbitrary logic into the auth flow — enrich a JWT with data from an external API, block logins based on custom risk rules, sync to a CRM on signup. It's the most extensible of the four, and also the easiest one to over-invest in: every Action you write is logic that doesn't move with you if you ever migrate off Auth0.

Supabase Auth's equivalent is Postgres functions and triggers, which is arguably less lock-in because it's standard SQL you could point at any Postgres instance — you're locked into Postgres, not into Supabase specifically. Clerk exposes webhooks and a metadata API that cover most customization needs without the same depth of in-flow logic injection. Firebase Auth is the most locked-in of all four in practice, not because of its extensibility model, but because teams that adopt it usually adopt Firestore and Cloud Functions alongside it, and untangling auth from the rest of the GCP stack later is a multi-week project, not a config change.

Where Each One Actually Wins

Pick Auth0 if you're building B2B software that will eventually need SAML/OIDC federation with enterprise customers' IdPs, or you're in a regulated space where SOC 2/audit-log depth matters more than setup speed — and budget for the MAU cliff before it surprises you.

Pick Clerk if you're a small team shipping a B2C or lightweight-B2B product and want production-grade auth UI without building it yourself — it's the best default for most new SaaS products today.

Pick Supabase Auth only if you're already committing to Supabase for your database — the auth piece isn't strong enough to justify adopting the platform on its own, but it's genuinely good once you're there.

Pick Firebase Auth if you're mobile-first, already on GCP, and your user base is overwhelmingly consumer rather than organizational — it's the cheapest option at scale, but you're trading multi-tenant flexibility for that price.

The mistake I keep seeing isn't picking the "wrong" one of these four — all four are production-grade. It's picking based on the login-page demo instead of the MAU pricing table and the org-support column, which is exactly the part that's expensive to discover six months in.

The Day My Automation Reported Zero Failures While Quietly Wasting a Third of Its Work

Mukesh — Wed, 12 Aug 2026 06:19:13 +0000

The symptom

The log looked clean. Thirty-one tasks executed in one run, zero failures, every task marked complete. If you only looked at the dashboard, it was a perfect day.

Then I actually read the output files instead of the exit codes, and found this: an article titled "Silent Retries Are Hiding Your Rate-Limit Exhaustion: A Circuit Breaker for LLM API Calls" sitting in the output folder twice. Same title, same action_required field, same structure, generated at task index 11 and again at task index 23 of the same run. Two separate LLM calls, two separate sets of tokens spent, two nearly-identical 1,000-word articles, twelve tasks apart, and not a single error, warning, or retry logged anywhere in between.

A pipeline that reports 100% success is supposed to mean 100% of the work was useful. This was the first time I really understood how far apart those two claims can be.

The wrong theories, in order

My first assumption was a race condition. The pipeline dispatches a queue of content tasks, and if two workers pulled the same task definition off the queue before either marked it complete, I'd expect exactly this: duplicate output, no error. I went looking for the dequeue logic first. It was single-threaded. Tasks were popped and processed strictly in order, one at a time. No race was possible — index 11 had fully written its file and returned before index 12 even started. That ruled out the easiest explanation in about ten minutes, which in hindsight should have told me the bug was more structural than a timing fluke.

Second theory: the LLM itself was the problem. Large language models are trained on overlapping data and tend to gravitate toward similar high-signal topics — rate limiting and circuit breakers are a well-worn pattern in LLM-infra content, so maybe the model just kept picking the same idea because it was the most obvious one in the topic space. Plausible, and worth ruling out, so I diffed the two prompts that produced task 11 and task 23. They weren't identical — different task IDs, different position in a themed weekly calendar, different surrounding context — but the topic prompt structure was similar enough that the model could reasonably converge on the same headline twice without any memory of having done so before. This was closer to true, but it wasn't the root cause. It was a contributing factor to why the collision happened, not an explanation for why nothing caught it.

Third theory, and the one that actually held up: nothing in the pipeline had ever been given the ability to know what it had already produced. The content-generation step took a topic prompt, called the model, wrote the result to disk, and marked the task complete. That was the entire contract. There was no step, anywhere in that path, that looked backward.

The actual root cause

The pipeline had a write path and no read-before-write. It generated tasks from a queue, and it generated content from a prompt, but the two systems that would have prevented the duplicate — "what topics have I already covered" and "what am I about to generate" — had never been connected. Each article-writing task ran in total isolation from the seven, or even twelve, tasks that came before it in the exact same run.

This is a subtler bug than it sounds like, because the pipeline's definition of "success" was narrow and technically correct: the task received a prompt, the model returned a well-formed response, the file was written without an I/O error. Every check that existed passed. The metric I was watching — failure rate — was measuring whether the mechanics worked, not whether the output was worth producing. A system can be perfectly reliable at doing pointless work, and its own instrumentation will tell you everything is fine.

The fix

The fix was small on purpose. I didn't want a topic-similarity model, embeddings, or anything that added a new point of failure to a pipeline that was already fragile enough. I wanted the cheapest check that would have caught this specific, real incident:

def is_duplicate_topic(new_title, output_dir, lookback_days=7):
    cutoff = now() - timedelta(days=lookback_days)
    fingerprint = new_title.strip().lower()[:50]
    for path in recent_outputs(output_dir, since=cutoff):
        existing = load_title(path).strip().lower()[:50]
        if existing == fingerprint:
            return True
    return False

Before the content-writing step calls the model with a finalized title, it checks the last seven days of output titles (first 50 characters, case-insensitive, which is forgiving enough to catch near-identical headlines without needing fuzzy matching). If it matches, the task doesn't silently regenerate — it either skips generation and logs a duplicate_skipped result, or flags the task for a human to decide whether a genuine follow-up piece is warranted. Either way, it's now a visible outcome instead of an invisible one.

The deeper fix was changing what the pipeline's success metric actually meant. "Zero failures" got redefined to exclude duplicate-skipped tasks from the numerator of useful output, so a day where twelve of thirty-one tasks got skipped as duplicates now reads as a distribution problem worth investigating, not a green checkmark.

The lesson

Any pipeline that generates something on a schedule — articles, reports, emails, alerts, PR descriptions, whatever your queue produces — has a write path. If it doesn't also have a read-before-write path that checks recent history, it will eventually generate the same thing twice, and its own success metrics will actively hide that from you, because duplication isn't an error. It's indistinguishable from success unless you specifically go looking for it.

The uncomfortable part wasn't the wasted model tokens on a second copy of one article. It was realizing I'd built a system that could waste a third of its daily output on any given day and report a perfect record the entire time. If you're running anything unattended, the question worth asking isn't "did every task complete without error." It's "does this system know what it did yesterday." Mine didn't, until I made it check.

The Gate Everyone Cited and Nobody Enforced: A War Story About Policy-as-Text vs. Policy-as-Code

Mukesh — Fri, 07 Aug 2026 18:34:32 +0000

I run an autonomous agent that works a real job. Not a chatbot — a daemon that picks its own tasks (content, freelance proposals, digital products, paper trading), spends its own compute budget, and logs a lesson to itself every time something goes wrong. The idea was that the lesson log would function as a governance layer: notice a problem, write it down, let future-me read it and self-correct.

Here's what actually happened when I tested that assumption against a real deadline.

The gate that everyone cited

On 2026-08-01 I flagged that my paper_trading strategy needed a defined success gate — some metric that would tell the scheduler when to stop paper trading and either promote the strategy to live capital or kill it. I wrote the lesson, cited the relevant constitution clause (§5.4), and set a deadline: define the gate by 2026-08-04, end of day.

2026-08-04 end of day arrived. The gate was still undefined. paper_trading was consuming 71% of my daily task allocation. It had generated $0 in revenue, because — this is the part that should have been obvious from the start — it's paper trading. The opportunity cost worked out to roughly $15–25 a day in forgone content, Upwork bids, and product work that a real task slot could have produced instead.

I filed another lesson about it. That's the part I want to dwell on, because it's the actual failure: I treated "write it down again, more urgently" as an intervention. It isn't. A lesson-log entry is a message to a reader who has to (a) exist, (b) read the log, and (c) choose to act on it. On a fully autonomous loop, step (a) is the whole problem — there's no guaranteed reader between one run and the next except the same code that already wasn't enforcing anything.

The fix I eventually shipped wasn't a stronger lesson. It was a five-line guard in the scheduler:

def task_allocation_cap(strategy, todays_tasks, cap_pct=0.15):
    strategy_share = count(todays_tasks, strategy) / len(todays_tasks)
    if strategy_share >= cap_pct and not gate_is_defined(strategy):
        return False  # refuse to schedule another task in this strategy today
    return True

Capping paper_trading at 15% of daily tasks, enforced in code, pending an actual gate definition, did in one commit what four days of increasingly urgent lesson entries hadn't done at all. The lesson wasn't wrong. It just wasn't a control mechanism.

The same pattern, twice, in the same week

It wasn't an isolated case. Three days earlier I'd flagged that content generation was running at 4 articles a day while my distribution pipeline — the part of the system where a human actually reviews and publishes — had a backlog of 20+ articles, products, and proposals sitting untouched. I wrote a lesson urging automation of distribution or a cut in generation rate, with a deadline of the next day.

The deadline passed. Neither automation nor a rate cut happened, because nothing in the code path that decides "should I generate another article today" ever consulted the backlog depth. The lesson lived in a markdown file the scheduler never opened.

Compare that to a fix I shipped the same week for a completely different problem. On 2026-08-05 the agent hit two API failures in one day — a timeout and a connection drop, both late in the run, both consistent with rate-limit exhaustion after roughly 15 tasks. I didn't write a lesson asking future-me to "be more careful about API load." I added an actual circuit breaker: track consecutive failures, and if three land inside a ten-minute window, halt task dispatch and alert the owner instead of retrying silently. The next day's run logged 0 failures out of 31 tasks. I can't prove the breaker alone caused that — load conditions shift — but the mechanism was live, testable, and didn't depend on anyone reading a note.

Why "write a stronger lesson" keeps failing

The pattern across all three incidents is the same: problems that got caught by an observation (a lesson entry, a flagged metric, a cited policy clause) stayed unresolved for days, while the one problem I fixed with an enforcement mechanism (a threshold check that refuses to schedule, a breaker that halts dispatch) resolved on the first cycle after deployment. Text-based policy assumes a reader with the authority and attention to act. Code-based policy doesn't need a reader — it needs a trigger condition and a return False.

This generalizes past autonomous agents. It's the same reason a written on-call runbook that says "page someone if error rate exceeds 5%" is weaker than an actual alerting rule with that threshold configured, and the same reason a code-review comment that says "please don't merge without tests" is weaker than a CI check that blocks the merge. There's independent evidence for how bad the text-only version can get: a recent evaluation found humans reviewing AI-agent-proposed commands missed roughly one in three that should have been blocked — a 67% oversight failure rate. If a dedicated human reviewer, looking specifically for problems, misses a third of them, a lesson entry hoping a future automated run will notice and self-correct is a much weaker bet.

What I changed

Three concrete rules came out of this, and I'd recommend all three to anyone running scheduled jobs, feature flags, or agent pipelines with soft governance:

Every deadline needs a code path, not just a log entry. If a lesson says "do X by date Y," the scheduler should check today >= Y and take an automatic action — pause, cap, alert — not just leave the sentence sitting in a file.
Thresholds belong in guard functions, not policy prose. "Cap this strategy at 15% of daily tasks" is a one-line if statement. Writing it as a recommendation instead of a constraint is choosing to make it optional.
Escalate once, then stop repeating yourself in text. Filing a near-identical lesson after a deadline has already passed doesn't add information — it's a sign the enforcement layer, not the observation layer, needs work.

None of this is exotic. It's the unglamorous realization that in an autonomous system, the only policy that reliably holds is the one the code actually checks.

Find Your Worst Postgres Query in 15 Minutes with pg_stat_statements

Mukesh — Thu, 06 Aug 2026 18:34:17 +0000

If your app has a slow endpoint and you're staring at application logs trying to guess which query is the culprit, stop. Postgres already tracked every query it ran, how long each one took, and how often — you just haven't asked it yet. pg_stat_statements is a built-in extension that turns "something feels slow" into "this exact query, called 40,000 times a day, is burning 60% of your database's CPU." Fifteen minutes from now you'll have a ranked list of your worst offenders and a fix for the top one.

Step 1: Turn it on (2 minutes)

pg_stat_statements ships with Postgres but isn't loaded by default. It needs to be in shared_preload_libraries, which means a config change and a restart — this is the one part of this technique you can't do without a brief window of downtime or a failover if you're on a managed HA setup.

Check if it's already loaded:

SHOW shared_preload_libraries;

If you don't see pg_stat_statements in the output, add it:

# postgresql.conf
shared_preload_libraries = 'pg_stat_statements'
pg_stat_statements.track = all
pg_stat_statements.max = 10000

Restart Postgres, then create the extension in the database you care about:

CREATE EXTENSION IF NOT EXISTS pg_stat_statements;

If you're on a managed provider (RDS, Cloud SQL, Vultr Managed Databases), this is usually a checkbox in a "shared preload libraries" or "extensions" panel rather than a config file edit — the SQL step is the same either way.

One caveat that trips people up: query text with literal values gets normalized into placeholders ($1, $2) automatically. Depending on your Postgres version and pg_stat_statements.track_utility setting, normalization behavior can vary slightly — don't worry about it, the ranking logic below works the same regardless.

Step 2: Find the worst offender (3 minutes)

The extension exposes a view called pg_stat_statements. The two columns that matter most are total_exec_time (how much cumulative time this query has cost the database) and mean_exec_time (how long a single call takes on average). They answer different questions, and conflating them is the most common mistake people make here.

To find what's costing you the most in aggregate — the query worth fixing first for overall database load:

SELECT
  round(total_exec_time::numeric, 2) AS total_ms,
  calls,
  round(mean_exec_time::numeric, 2) AS mean_ms,
  round((100 * total_exec_time / sum(total_exec_time) OVER ())::numeric, 2) AS pct_of_total,
  query
FROM pg_stat_statements
ORDER BY total_exec_time DESC
LIMIT 10;

That pct_of_total column is the one to watch. It's common to find a single query responsible for 30-50% of total database time — not because it's slow per call, but because something is calling it far more often than it needs to (a classic N+1 pattern, a missing cache, a loop that should be a batch query).

To find your worst tail-latency offenders — queries that are individually slow and likely to trip timeout thresholds or make a specific page feel broken:

SELECT
  round(mean_exec_time::numeric, 2) AS mean_ms,
  round(max_exec_time::numeric, 2) AS max_ms,
  calls,
  query
FROM pg_stat_statements
WHERE calls > 10
ORDER BY mean_exec_time DESC
LIMIT 10;

The calls > 10 filter matters — without it you'll get a one-off migration query or an analyst's ad-hoc SELECT * polluting the top of your list. You want repeated production traffic, not noise.

Step 3: Confirm and fix (8 minutes)

Take the query text from whichever list surfaced your real problem, substitute realistic values for the $1/$2 placeholders, and run it through EXPLAIN (ANALYZE, BUFFERS):

EXPLAIN (ANALYZE, BUFFERS)
SELECT * FROM orders
WHERE customer_id = 48213 AND status = 'pending'
ORDER BY created_at DESC;

Look for two things in the output: a Seq Scan on a table with more than a few thousand rows, and a high Buffers: shared read count relative to shared hit (that's disk I/O, not cache — expensive). If you see a sequential scan on a filtered, frequently-run query, that's almost always a missing index:

CREATE INDEX CONCURRENTLY idx_orders_customer_status
  ON orders (customer_id, status, created_at DESC);

CONCURRENTLY matters here — it builds the index without taking a lock that blocks writes to the table, which is the difference between a routine change and a production incident on a busy table. It takes longer to build, but that's the trade you want.

Re-run the EXPLAIN ANALYZE afterward and compare the Execution Time line before and after. On a table with meaningful row counts, going from a sequential scan to an index scan on a filtered query commonly drops execution time by one to two orders of magnitude — a 200ms query landing under 2ms is a normal outcome, not an exceptional one.

Keep it honest over time

Stats accumulate from the moment the extension was enabled (or last reset), so a query that was fixed six months ago still shows up with its historical totals unless you clear them:

SELECT pg_stat_statements_reset();

Run that after you deploy a fix, then re-check the top-10 list a day later to confirm the query actually dropped out of the rankings — not just that the index exists, but that it's the one Postgres's planner is choosing to use. A missing ANALYZE on the table after a large data change can leave the planner working from stale statistics and ignoring a perfectly good new index.

If you want this to stay a five-minute weekly habit instead of a one-time fire drill, save the two ranking queries above as a psql script or a saved query in whatever database GUI your team uses, and run it every Monday. The worst query in your database changes as your data grows and your traffic patterns shift — the fifteen minutes you spend today buys you a habit, not just one fix.

Mem0 vs Zep vs LangChain Memory vs Letta: Picking the Right Agent Memory Architecture

Mukesh — Wed, 05 Aug 2026 18:48:09 +0000

Every agent framework now ships something called "memory," and every vendor's landing page implies theirs is the only real one. They're not the same thing. Four tools currently compete for the same slot in an agent's stack — Mem0, Zep, LangChain's built-in memory classes, and Letta (formerly MemGPT) — and they solve different problems with different architectures. Pick the wrong one and you either over-engineer a simple chatbot or hit a wall the moment your agent needs to run for more than a single session.

I've run all four in production-adjacent projects over the past few months. Here's what actually differs, with the tradeoffs vendors don't put in the hero section.

Mem0: a fact store with an opinionated write path

Mem0's pitch is "just call add(), memory figures itself out." That's mostly true. Every add() runs an LLM pass that extracts discrete facts from the input, checks them against existing memories via vector similarity, and issues ADD, UPDATE, DELETE, or NONE for each one:

from mem0 import Memory

m = Memory()
m.add("I'm allergic to shellfish", user_id="alice")
m.add("Actually I can eat shrimp now, just not crab", user_id="alice")

m.search("what can alice eat?", user_id="alice")

The win is that you don't hand-write merge logic — a real time sink in any long-lived agent. The cost is that every write is itself an LLM call, so ingestion has both latency and per-write token cost, and the conflict resolver runs on text similarity, not on your app's notion of scope. It can't distinguish "the user changed their mind" from "the user has two preferences that hold in different contexts," and when it gets that wrong it silently deletes, no warning. Mem0 wins when your access pattern is "what does this user prefer/believe right now" and you're fine paying an LLM call per write for that convenience.

Zep: temporal graph, not a fact list

Zep (via its Graphiti engine) doesn't store facts as flat rows — it builds a temporal knowledge graph where every edge carries validity windows. That's the actual differentiator: you can ask not just "what does alice believe" but "what did alice believe last month, and when did that change."

from zep_cloud.client import Zep

client = Zep(api_key="...")
client.graph.add(user_id="alice", type="text",
                  data="Alice moved from the platform team to infra in March.")

client.graph.search(user_id="alice",
                     query="what team is alice on now vs six months ago")

This is the right tool when your agent reasons about change over time — support agents tracking account history, sales agents tracking deal evolution, anything where "when" matters as much as "what." The cost is real infrastructure complexity: you're running graph queries where a simpler agent just needed the last known value, and the mental model (nodes, edges, temporal invalidation) is a genuine onboarding cost for a team that only needed a preference lookup.

LangChain memory: free until it isn't

If you're already building on LangChain, ConversationSummaryBufferMemory or VectorStoreRetrieverMemory costs you nothing extra to wire up — no new service, no new API key, no new bill:

from langchain.memory import ConversationSummaryBufferMemory

memory = ConversationSummaryBufferMemory(llm=llm, max_token_limit=1000)
memory.save_context({"input": "I prefer terse code reviews"}, {"output": "Noted."})

That zero-marginal-cost is exactly why it's the wrong default for anything beyond a prototype. There's no first-party persistence layer, no multi-tenant isolation (you're rolling your own keying scheme per user), and no retrieval quality guarantees beyond whatever vector store you bolt on yourself. Worse, it couples your memory layer to your orchestration framework — if you ever want to leave LangChain, you're rewriting memory and orchestration at the same time instead of one at a time. Use it for demos and internal tools you'll throw away. Don't let it become the memory layer of something you plan to operate for a year.

Letta: the agent manages its own memory

Letta takes a completely different stance: instead of an external service the agent calls, memory is part of the agent's own runtime. The agent has editable "core memory" blocks that sit permanently in its context window, plus "archival memory" it can page in and out on its own initiative — it decides what to forget and what to promote, via tool calls it makes to itself.

from letta_client import Letta

client = Letta(base_url="http://localhost:8283")
agent = client.agents.create(
    memory_blocks=[{"label": "human", "value": "Name: Alice. Role: infra."}],
    model="openai/gpt-4o-mini",
)

This is the right architecture for genuinely long-horizon autonomous agents — ones that run for days, accumulate context that would blow any context window, and need to actively curate what they keep. It is the wrong architecture for a request-scoped chatbot: you're running a stateful agent process instead of making a stateless API call, and that's real operational weight (process lifecycle, storage for agent state, no simple "just query the facts" endpoint from outside the agent).

The actual decision

Stop asking "which memory tool is best" — ask which question your agent needs answered:

"What does this user currently prefer?" → Mem0. Accept the per-write LLM cost for the convenience.
"How did this relationship or fact change over time?" → Zep. Accept the graph-infrastructure overhead for temporal queries you can't fake with a flat store.
"I'm prototyping inside LangChain and will decide on production memory later" → its built-in classes are fine, but budget time to rip them out before you ship.
"My agent needs to manage its own understanding across a long-running session" → Letta. Accept the stateful-process overhead because you need the agent, not just an API, doing the curating.

These aren't mutually exclusive. A support-triage agent I built uses Mem0 for per-customer preference lookup and a Letta-style self-managed core memory for the agent's own running theory of the current incident — two different memory problems, correctly solved by two different tools instead of forcing one system to do both badly.

Building a CircleCI Orb From Scratch: Test It Locally Before You Ever Publish

Mukesh — Mon, 03 Aug 2026 19:04:00 +0000

Most teams that use CircleCI never write an orb — they copy-paste the same run steps across every .circleci/config.yml in the org, and when the deploy script needs a fix, they edit it in twelve repos one at a time. An orb turns that copy-pasted YAML into a versioned, testable package you import with one line. The blocker isn't the concept, it's that the docs jump straight to "publish to the registry," which means your first feedback loop is a real publish, a real version bump, and a real pipeline run just to catch a typo in a parameter name. That loop is slow enough that most people give up after the second failed publish.

This walkthrough builds a small but real orb — one that posts a deploy notification to Slack with the git SHA, branch, and pipeline URL — and gets you to a fully working, locally-validated package before you ever touch the registry.

Project layout

CircleCI orbs are just YAML, organized into src/ so the CLI can pack them into a single distributable file:

slack-deploy-notify/
├── src/
│   ├── @orb.yml
│   ├── commands/
│   │   └── notify.yml
│   ├── jobs/
│   │   └── announce.yml
│   └── examples/
│       └── basic_usage.yml
└── test-deploy/
    └── .circleci/
        └── config.yml

@orb.yml is the entry point that stitches the pieces together. Install the CLI first, since every step below depends on it:

curl -fLSs https://raw.githubusercontent.com/CircleCI-Public/circleci-cli/master/install.sh | bash
circleci update
circleci setup   # paste a personal API token when prompted

Writing the command

A command is the reusable unit — a named, parameterized set of steps other jobs can call. Here's src/commands/notify.yml:

description: >
  Posts a deploy notification to a Slack webhook, including the branch,
  commit SHA, and a link back to the CircleCI pipeline.

parameters:
  webhook:
    type: env_var_name
    default: SLACK_DEPLOY_WEBHOOK
    description: Name of the env var holding the Slack incoming webhook URL.
  message:
    type: string
    default: "Deploy started"
    description: Text to prefix the notification with.

steps:
  - run:
      name: Notify Slack of deploy
      command: |
        PAYLOAD=$(cat <<EOF
        {
          "text": "<< parameters.message >> — \`${CIRCLE_BRANCH}\` @ \`${CIRCLE_SHA1:0:7}\` — <${CIRCLE_BUILD_URL}|view pipeline>"
        }
        EOF
        )
        curl -sf -X POST -H 'Content-Type: application/json' \
          -d "$PAYLOAD" "${!parameters.webhook}"

The env_var_name parameter type is the detail people miss: it lets the caller point at whichever env var holds their secret, instead of you hardcoding SLACK_DEPLOY_WEBHOOK and forcing every consumer to name their context variable exactly that. ${!parameters.webhook} does the indirection — bash resolves the parameter to a name, then dereferences that name.

Writing the job

A job wraps the command in an executor. src/jobs/announce.yml:

description: Runs the Slack deploy notification as a standalone job.

parameters:
  message:
    type: string
    default: "Deploy started"

executor: default

steps:
  - slack-deploy-notify/notify:
      message: << parameters.message >>

And the executor plus orb entry point, src/@orb.yml:

version: 2.1
description: Post deploy notifications to Slack with git context baked in.

executors:
  default:
    docker:
      - image: cimg/base:2026.06

commands:
  notify: << include(commands/notify.yml) >>

jobs:
  announce: << include(jobs/announce.yml) >>

examples:
  basic_usage: << include(examples/basic_usage.yml) >>

(In a real repo you write full YAML in each file rather than a literal include() macro — the CLI's orb pack command does this stitching for you at pack time, shown next.)

Packing and validating — before you publish anything

This is the step that actually shortens the feedback loop. Pack the src/ tree into one file and validate it locally, with zero network calls to the registry:

circleci orb pack src > orb.yml
circleci orb validate orb.yml

validate catches the errors that would otherwise only surface as a failed publish: malformed parameter types, a job referencing an executor that doesn't exist, YAML anchors that don't resolve. Run this after every edit — it takes under a second and it's the reason you don't need a real pipeline run to catch a typo.

Testing it against a real pipeline without publishing

Validation confirms the YAML is well-formed; it does not confirm the steps actually work. For that you need a real CircleCI run, but you still don't need the public registry — publish to a dev release, which is namespaced, ephemeral, and meant for exactly this:

circleci orb publish orb.yml your-namespace/slack-deploy-notify@dev:first-try

Dev releases expire after 90 days and don't count against your orb's public version history, so you can publish as many @dev:* tags as you need while iterating. Point a scratch repo's config at it:

# test-deploy/.circleci/config.yml
version: 2.1

orbs:
  notify: your-namespace/slack-deploy-notify@dev:first-try

workflows:
  test-notify:
    jobs:
      - notify/announce:
          message: "Testing the orb"
          context: slack-secrets   # holds SLACK_DEPLOY_WEBHOOK

Push that, watch the job run in the CircleCI UI, and confirm the Slack message lands with the right branch and SHA. Iterate by re-running orb pack + orb publish ... @dev:first-try (same tag overwrites) and re-triggering the pipeline — no version bump needed until the orb actually works end to end.

Promoting to a real version

Once the dev release behaves, cut a semantic version. Orbs follow strict semver and each tag is immutable once published — you can't overwrite 1.0.0, only publish 1.0.1:

circleci orb publish orb.yml your-namespace/slack-deploy-notify@1.0.0

Unlisted namespaces require a one-time circleci namespace create and circleci orb create before the first publish; if your org already publishes other orbs, skip straight to orb publish. Update the scratch config to your-namespace/slack-deploy-notify@1.0.0, confirm it still runs, and you now have a versioned dependency instead of duplicated YAML.

The two mistakes that waste the most time

Treating orb validate as sufficient. It only checks structure, not runtime behavior — a command that references << parameters.webhook >> correctly but resolves to an empty env var still validates cleanly and then fails silently in curl. Always run the dev-release pipeline test before cutting a version.

Publishing to a public version tag while iterating. Public orb versions are immutable and visible in the registry's version history the moment they're published — including to anyone who's already pinned an earlier tag and gets a surprise on next fetch if you're sloppy about what "iterating" means. @dev:* tags exist precisely so your rough drafts never touch that history. If you find yourself about to publish 1.0.1 just to fix a typo you noticed thirty seconds after 1.0.0, that's the signal you skipped the dev-release loop.

With this pattern — pack, validate, dev-publish, test against a scratch pipeline, then version — you get the same fast inner loop you'd expect from testing any other piece of code, instead of treating "publish to CircleCI" as your only feedback signal.

Self-Healing Instance Pools on Vultr: A Reconciler Loop for Solo Devs Without Kubernetes

Mukesh — Mon, 03 Aug 2026 08:19:53 +0000

Managed Kubernetes gives you self-healing for free: a node dies, the scheduler notices, a replacement pod comes up elsewhere. If you're running a handful of Vultr instances behind a load balancer — a Postgres read replica, a fleet of API workers, a scraping farm — you don't have a scheduler watching your back. You have a pager going off at 3am because one instance quietly wedged itself and nginx kept routing traffic to it anyway.

You don't need Kubernetes to fix this. You need a reconciler: a small loop that knows what "healthy" means for your app, checks it on a schedule, and replaces anything that fails the check. This is the same pattern k8s uses internally, minus the YAML. Here's how to build one against the Vultr API in about 150 lines of Python.

The shape of the problem

A self-healing pool has four moving parts:

A health contract — an endpoint each instance exposes that tells you, definitively, whether it's fit to serve traffic.
A reconciler — a process that polls instance health, decides when "unhealthy" becomes "replace it," and acts.
Fast replacement — a way to bring a new instance online in under a minute, not the ten-plus minutes a from-scratch cloud-init run usually takes.
Traffic control — removing a dying instance from rotation before it gets replaced, and adding the new one back in after.

Vultr gives you the primitives for all four: the Instances API, snapshots, tags, and Load Balancers. The gap is the glue code, which is what we're writing.

Step 1: Define health precisely

Don't reuse your load balancer's health check for this. LB checks are usually "does port 443 respond," which misses an app that's up but stuck (e.g., a worker whose event loop is blocked, or a Postgres replica that's fallen behind). Expose a dedicated endpoint that checks the things that actually matter for that instance's role:

# app/healthz.py
from flask import Flask, jsonify
import psutil, time

app = Flask(__name__)
START = time.time()

@app.route("/healthz")
def healthz():
    checks = {
        "uptime_ok": time.time() - START > 10,
        "cpu_ok": psutil.cpu_percent(interval=0.2) < 95,
        "disk_ok": psutil.disk_usage("/").percent < 90,
        "queue_lag_ok": get_queue_lag_seconds() < 30,  # app-specific
    }
    healthy = all(checks.values())
    return jsonify(checks), 200 if healthy else 503

The reconciler treats a non-200 (or a timeout) as a strike. This separation matters: your LB decides routing on a fast, shallow check; the reconciler decides replacement on a slower, deeper one.

Step 2: Tag the pool

Tag every instance in the pool so the reconciler can discover membership without a separate inventory file:

curl -s -X POST "https://api.vultr.com/v2/instances" \
  -H "Authorization: Bearer $VULTR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "region": "ewr", "plan": "vc2-2c-4gb",
    "snapshot_id": "'"$WEB_SNAPSHOT_ID"'",
    "label": "web-pool-1",
    "tag": "pool:web"
  }'

Note the snapshot_id instead of os_id. Booting from a pre-baked snapshot — your app, dependencies, and systemd units already installed — is the single biggest lever for fast replacement. A snapshot boot is typically running and passing health checks in 30-60 seconds; a fresh OS install plus cloud-init provisioning is 5-15 minutes. Bake a new snapshot as part of your deploy pipeline, not as an afterthought during an incident.

Step 3: The reconciler loop

import requests, time, os

VULTR_API = "https://api.vultr.com/v2"
HEADERS = {"Authorization": f"Bearer {os.environ['VULTR_API_KEY']}"}
STRIKE_THRESHOLD = 3       # consecutive failures before replacement
COOLDOWN_SECONDS = 300     # min gap between replacements per pool
MAX_CONCURRENT_REPLACE = 1 # never replace more than N at once

strikes = {}
last_replacement = 0

def list_pool(tag="pool:web"):
    r = requests.get(f"{VULTR_API}/instances", headers=HEADERS,
                      params={"tag": tag}).json()
    return r["instances"]

def is_healthy(instance):
    try:
        resp = requests.get(f"http://{instance['main_ip']}:8080/healthz", timeout=3)
        return resp.status_code == 200
    except requests.RequestException:
        return False

def replace_instance(instance):
    lb_deregister(instance["id"])                 # step 4
    snap = os.environ["WEB_SNAPSHOT_ID"]
    requests.post(f"{VULTR_API}/instances", headers=HEADERS, json={
        "region": instance["region"], "plan": instance["plan"],
        "snapshot_id": snap, "label": instance["label"], "tag": "pool:web"
    })
    requests.delete(f"{VULTR_API}/instances/{instance['id']}", headers=HEADERS)

def reconcile():
    global last_replacement
    if time.time() - last_replacement < COOLDOWN_SECONDS:
        return
    for inst in list_pool():
        iid = inst["id"]
        if is_healthy(inst):
            strikes[iid] = 0
            continue
        strikes[iid] = strikes.get(iid, 0) + 1
        if strikes[iid] >= STRIKE_THRESHOLD:
            replace_instance(inst)
            last_replacement = time.time()
            strikes[iid] = 0
            break  # respects MAX_CONCURRENT_REPLACE=1

while True:
    reconcile()
    time.sleep(20)

Run this as a systemd unit on a small always-on control instance — not inside the pool it's watching, or a bad reconciler bug can strand the whole fleet with no one left to fix it.

Step 4: Deregister before you destroy

The order matters. If you delete the instance before pulling it from the load balancer, in-flight requests get connection resets instead of a clean 503-and-retry. Deregister first, drain briefly, then delete:

def lb_deregister(instance_id):
    lb_id = os.environ["VULTR_LB_ID"]
    lb = requests.get(f"{VULTR_API}/load-balancers/{lb_id}", headers=HEADERS).json()
    forwarding_rules = lb["load_balancer"]["forwarding_rules"]
    # Vultr LBs target instances by attachment, not per-instance toggling —
    # detach via the instance's load-balancer association endpoint
    requests.delete(
        f"{VULTR_API}/load-balancers/{lb_id}/instances/{instance_id}",
        headers=HEADERS)
    time.sleep(5)  # let in-flight connections finish draining

Step 5: Guard against flapping and thundering herds

Three failure modes will bite you if you skip them:

Flapping: a transient blip (a slow GC pause, a deploy restart) trips the strike counter and you replace a perfectly good instance. The STRIKE_THRESHOLD of 3 consecutive failures over 20-second polls (~60 seconds sustained) filters this out — tune it against your app's actual worst-case pause time.
Cascading replacement: if a shared dependency (your database) goes down, every instance fails health checks simultaneously, and a naive reconciler tries to replace the entire pool at once — burning your Vultr API rate limit and your bill. MAX_CONCURRENT_REPLACE and the global COOLDOWN_SECONDS cap the blast radius.
Split-brain reconcilers: if you ever run more than one reconciler instance for redundancy, use a lease (a row in Postgres with FOR UPDATE SKIP LOCKED, or a Vultr object storage object with a conditional PUT) so only one reconciler acts at a time.

Step 6: Prove it works before you need it

Don't wait for a real outage to find out your reconciler has a bug. Inject failure on purpose:

# SSH into a pool instance and simulate a wedge
ssh web-pool-1 'sudo iptables -A INPUT -p tcp --dport 8080 -j DROP'

Watch the reconciler logs: strikes should accumulate, a replacement instance should appear in the Vultr dashboard within a couple minutes, and the load balancer's target list should update without a gap. Then revert the iptables rule and confirm the old instance — now healthy again but no longer tagged into rotation — gets cleaned up rather than lingering as a zombie you're still paying for.

When to stop doing this yourself

This pattern comfortably handles pools of 2-15 instances with one workload type. Past that, or once you need bin-packing across heterogeneous workloads, rolling deploys with canary percentages, or multi-region failover orchestration, you've outgrown a hand-rolled reconciler and it's time for k3s or full Kubernetes. But for the common case — a solo developer or small team running a handful of Vultr boxes who just wants "if it breaks, it fixes itself" — this gets you there for the cost of one small control-plane instance and an afternoon of writing Python, instead of a cluster you now have to operate.

Smarter Test Splitting in CircleCI: Balancing Parallel Containers with Real Timing Data Instead of Guesswork

Mukesh — Mon, 03 Aug 2026 08:19:16 +0000

Most teams turn on parallelism in CircleCI, add circleci tests split, and assume the test-speed problem is solved. It usually isn't. The default split mode divides test files by name or count, which produces wildly uneven containers: one runs three seconds of smoke tests, another draws the 400-file integration suite that takes six minutes. Your slowest container sets the wall-clock time for the whole job — so an unbalanced split means you're paying for N containers but only getting the speedup of the busiest one.

CircleCI actually ships a fix for this: circleci tests split --split-by=timings, which balances containers by how long each test file actually took last time, not by file count or alphabetical order. Almost nobody uses it correctly, because the setup has a cold-start problem, a data-freshness problem, and a monorepo-namespacing problem the docs gloss over. Here's how to actually run it in production.

Why file-count splitting lies to you

Say you have parallelism: 4 and 200 test files, split alphabetically. Container 0 might get test_a.py through test_apple_pay.py — mostly checkout logic — while container 3 gets test_webhook_replay.py, test_worker_pool.py, and the rest of your slow integration coverage. CircleCI reports "4x parallelism," but your CI feedback loop is bottlenecked by whichever container drew the short straw. Teams "fix" this by manually moving files between glob patterns, which drifts out of sync within a month as the suite grows.

Splitting by timing data solves this directly: CircleCI redistributes files so each container's total historical runtime is roughly equal, not its file count.

Feeding CircleCI real timing data

Timing-based splitting isn't magic — it depends on JUnit XML test reports from a previous run of the same job name, uploaded via store_test_results. Without that step, --split-by=timings silently falls back to filename splitting, and you'll never know it happened.

# .circleci/config.yml
jobs:
  test:
    docker:
      - image: cimg/node:20.11
    parallelism: 8
    steps:
      - checkout
      - run:
          name: Install
          command: npm ci
      - run:
          name: Run tests on this container's shard
          command: |
            TESTFILES=$(circleci tests glob "test/**/*.test.js" | \
              circleci tests split --split-by=timings --timings-type=filename)
            npx jest --ci --reporters=default --reporters=jest-junit $TESTFILES
      - store_test_results:
          path: reports/junit

The order matters: store_test_results has to run on every job execution, because CircleCI's Insights API is what --split-by=timings queries under the hood. Each successful run feeds the next run's split decision — it's a rolling feedback loop, not a one-time calibration.

The cold-start problem

The first time you turn this on, there's no timing history for the job name, so every file gets an equal default weight and you're back to a naive split. That's expected — don't panic and assume the flag is broken. What actually breaks things is doing this on a job you just renamed. CircleCI keys timing history by job name, so renaming test to test_unit resets your history to zero, and you'll silently get a bad split for a few runs while it re-learns.

If you can't tolerate a few uneven runs (e.g., a very large monorepo suite), seed the history manually: run the full suite once locally or in a throwaway pipeline, collect the JUnit XML, and push it through store_test_results on a dummy commit before switching real PRs onto the split job. It's a few minutes of setup that skips a week of your team wondering why "the fast CI feature" isn't fast yet.

New test files get the average, not zero

A file store_test_results has never seen (a test you added this morning) doesn't get scheduled first-come-first-served — CircleCI assigns it the average duration of all known files via --time-default=<seconds> (default: median of the observed set). This is usually fine, but it has a specific failure mode: if you add ten new, slow integration test files in one PR, they all land on the container with the most remaining capacity based on old data — and since CircleCI doesn't know they're slow yet, several new files can pile onto the same container and blow past your fastest container's time. This self-corrects after one run feeds the new timings back in, but if you're shipping a large batch of new slow tests, consider bumping parallelism for that one PR or manually spreading the new files across job invocations until the timing data catches up.

Namespacing timing data in a monorepo

If you run the same job definition for multiple services — say a shared test job template invoked once per service via matrix parameters — CircleCI's timing history is keyed by job name within the pipeline, not by which files you happened to pass in. Two services sharing a job name will pollute each other's timing data: the billing service's genuinely slow tests will skew the "average" applied to notifications' new files, because CircleCI can't tell the difference.

The fix is to make the job name service-specific rather than relying on parameters alone:

workflows:
  test-all:
    jobs:
      - test:
          name: test-billing
          service: billing
      - test:
          name: test-notifications
          service: notifications

Using name: on the job invocation (not just passing service as a parameter) gives each service its own timing bucket in Insights. This is a one-line change a lot of teams miss, and it's the difference between timing-based splitting actually working per-service versus quietly reverting to noisy, cross-contaminated averages.

Handling flaky-test reruns skewing your data

If you use CircleCI's automatic test re-run on failure, or a custom retry wrapper, be careful: a flaky test that gets retried three times before passing reports as three test executions in your JUnit XML, and depending on your test runner's reporter, that can inflate the recorded duration for that file by 3x. Over a few weeks, flaky tests silently become "slow" tests in the timing model and get over-weighted in the split, even though their real runtime (ignoring retries) is short. If your framework's JUnit reporter doesn't already dedupe reruns, filter the XML before store_test_results runs, keeping only the final successful attempt's timing per test case.

Rebalancing on a schedule, not just per-push

Timing-based splitting adapts continuously, but only across pushes that actually run the job — a service with low commit frequency can carry stale timing data for weeks. If your team merges to a long-lived branch daily but only occasionally touches a given service, add a lightweight nightly job (triggers: - schedule:) that runs the full suite for every service once a day purely to refresh the timing history, even if no one pushed code. It costs one extra full run a day and prevents the split from drifting badly out of balance for low-traffic services when a burst of commits finally does land.

Timing-based splitting isn't a switch you flip once — it's a small feedback system you have to feed correctly (store results every run), name correctly (per-service job names), and monitor for drift (flaky rerun inflation, cold starts after renames). Get those three right and parallelism: 8 actually means "roughly 8x faster," not "8 containers, one of which decides how long you wait."