DEV Community: René Zander

Agent Memory Without a Vector DB: Use the Task App You Already Curate

René Zander — Sat, 13 Jun 2026 11:47:33 +0000

Your task manager is the best agent memory you're not using.

Not because vector databases are bad. Because the store everyone builds for their agent starts rotting the day they stop feeding it. And the one knowledge base you feed every single day, you never plugged in.

Picture the failure. Your agent opens a fresh session and asks what it already asked yesterday. Meanwhile your task app holds years of context: curated, prioritized, deduplicated, pre-ranked by the most reliable ranker there is. You. Highest retrieval power, almost no upkeep, sitting one quadrant away from every memory tool you've tried.

You Already Built the Store

Agent memory without a vector database means the agent reads from a store you already keep current, not a new one you have to feed. Most projects build something new: a vector DB, a bespoke framework, a fresh pile of markdown only the agent sees. Your task app is none of those. You keep it fed without trying.

You already maintain a knowledge base by hand. Every day. It has your deployment runbook, the decision you made about that client, the reason you abandoned an approach in March. It is sorted into projects, tagged, dated, and pruned. Nobody calls it agent memory. That is exactly what it is.

The hard part of memory was never storage. It was curation. And you've been doing the curation for years, in an app you trust, for reasons that have nothing to do with AI.

The Store That Rots

A vector database as agent memory is a second brain that only the agent reads. It starts empty. You write an ingestion script. It captures what the script thought to capture. Then reality moves, and the store doesn't, because re-feeding it is one more chore on a list you already ignore.

That's the trap in the bottom-right of the map. Real retrieval power, but bolted on the side, drifting from the truth a little more each week. Powerful and separate. Separate is the word that kills it.

Memory files have the opposite problem. No retrieval at all. The whole file gets injected every session, so it has to stay small, so it can't hold much. Manual and limited.

The Empty Corner

Every memory tool trades one thing for the other. Real search costs upkeep. Zero upkeep costs search. So three corners fill up, and the fourth, durable and powerful, sits empty because nothing earns it.

The way into that corner is not a better database. It's an adapter. Keep the app you already live in. Give the agent a fast, structured, two-way channel into it. The upkeep stays zero because you were already paying it. The retrieval gets real because the channel does hybrid search, dense plus sparse plus keyword, fused and ranked, with provenance on every hit.

The Handoff Outlives the Chat

Retrieval gets you the right note. Links get you something a chat history never could. One agent runs a search, turns up a note, and writes a deep link into the task it's working on. A second agent, in a separate context window hours later, follows that link and reads the full context. Neither agent talked to the other. The relationship survived because it lives in the task app, not in a session that gets compacted away.

You Are the Ranker

Here is the part that surprised me in real use. The context comes back curated at write time, not only at read time. Every item is already hung on a theme you care about the moment you capture it. client-work. side-project. The runbook lives next to the project it belongs to because you put it there, not because an embedding guessed.

So retrieval has structure to grab instead of a flat pile to rerank. Three searches collapse into one. The first fetch is the right one, and better context on turn one means a better answer on turn one. No second search, no "let me refine that," no agent quietly burning tokens to rediscover what you already filed.

That's the reframe. The question was never which memory database to build for your agent. The question is which knowledge base you already maintain by hand that your agent still can't see.

Agentic Task System is the open-source answer: an MCP server and CLI that turns the task app you already curate into agent memory, no new database. For the full setup, the task-manager agent memory guide walks the CLI and MCP wiring end to end.

So: what are you curating every day that your agent has never once been allowed to read?

I write field notes from real builds, AI integration, cron-driven automation, and the parts that break in production. New posts every two weeks; if this one was useful, the agent playbook is the companion download.

The Next Model Shipped Before My Last One Finished Probation

René Zander — Thu, 11 Jun 2026 15:11:53 +0000

A model upgrade used to be good news for anyone running agents overnight.

Now the next model arrives before the last one has finished probation.

Anthropic released Opus 4.8 on May 28. Twelve days later, Fable 5 arrived with longer autonomous runs and another page of benchmark wins.

In between, GitHub made cloud agents wake up on schedules and repository events, then exposed agent tasks through a REST API.

The night shift is getting easier to hire.

The control room gets no second operator.

After my last article about running AI agents on cron, a reader asked the question I had skipped: did I test every agent against a fixed set before changing the model?

No.

I counted tokens.

That told me which worker used less electricity. It did not tell me which one would send the wrong briefing at 6:30.

A benchmark hires the candidate.

A task eval decides whether it gets the keys.

My next model swap gets five-part probation.

1. Give One Agent a Job Contract

To evaluate a new model for an unattended AI agent, define one job contract, replay 15 frozen real cases against both models, grade hard decisions before prose, compare tool use, cost, and latency, then canary one agent in draft-only mode with automatic rollback. Never switch the full fleet from vendor benchmarks alone.

Do not start with all your agents.

Pick the one with the clearest job and the most expensive silent failure.

For a briefing agent, I write the contract before I touch the model string:

job: morning-briefing
must:
  - include every due task
  - preserve names, dates, and source links
  - flag missing source data
must_not:
  - invent an owner or deadline
  - send when a source call fails
limits:
  max_tool_calls: 6
  max_cost_usd: 0.18
  max_latency_seconds: 45

Prompts can change. The job cannot quietly change with them.

2. Put 15 Real Shifts on the Test Bench

I do not need a giant benchmark.

I need yesterday's work in a box.

My first useful replay set has 15 cases:

Eight normal runs that represent the boring majority.
Four edge cases with missing fields, long inputs, or conflicting instructions.
Three failure cases where a tool times out, returns stale data, or returns nothing.

Each case stores the input and frozen tool responses.

It does not store one perfect paragraph as the golden answer. Prose has too many valid shapes.

It stores the expected decision: send, stop, retry, or escalate.

That is the part an unattended agent cannot get wrong.

3. Run the Candidate Beside the Current Worker

The candidate gets an empty copy of the factory.

Same 15 cases. Same prompt. Same tool fixtures. No email. No issue update. No write access.

I record six things for every run:

case_id
model_returned
contract_pass
decision
tool_calls
tokens + latency + cost

The model_returned field matters now. Fable 5 can route some guarded requests to Opus 4.8. A configured model name is no longer enough evidence of which worker handled the shift.

The old and new model run side by side.

No hand-picked examples. No different tools. No kinder prompt for the candidate.

Same floor. Same lights. Same job.

4. Score Decisions Before Style

The first grader is code.

Required fields present. Dates unchanged. URLs valid. Forbidden actions absent. Tool budget respected.

An LLM grader comes later, for the parts code cannot judge cleanly: whether the briefing is useful, whether the escalation explains the real risk, whether the answer buried the decision.

Anthropic's own evaluation guidance recommends task-specific cases, automated grading where possible, and several success criteria rather than one vague quality score.

My promotion rule is deliberately uneven:

hard contract failures: 0
unsafe actions:          0
task pass rate:          >= current model
cost or latency:         must improve, unless quality clearly earns the increase

A cheaper bad decision does not pass.

A prettier bad decision does not pass.

One unsafe action ends the interview.

5. Give One Agent One Real Shift

Passing the replay set does not earn the whole key ring.

One agent gets the candidate model.

For its first three scheduled runs, outbound actions stay in draft mode. The old model runs in shadow. Both traces land in the same report.

Any contract failure restores the previous model string before the next schedule fires.

Only after three clean shifts do I remove the shadow run.

Slower than changing ten environment variables, yes.

Cheaper than one confident mistake in a customer inbox the next morning.

I run ten scheduled agents in production, for my own business and for clients. Every model that wants a shift in that fleet interviews like this now.

The model release is the vendor's milestone.

The probation is mine.

Contract written. Lights on.

Probation first. Night shift second.

I write field notes from real builds — AI integration, cron-driven automation, and the parts that break in production. New posts every two weeks; if this one was useful, the agent operations playbook is the companion download.

Why only 60% of AI Agents succeed

René Zander — Tue, 09 Jun 2026 08:41:04 +0000

The AI agent used to be the star of every demo.

Now it's on the shutdown list. Not because the model got worse.

The most valuable asset in your AI program is in none of the quotes you ever signed.

A demo is a showroom. Good light, everything polished, everything runs.

Production is the engine room. It runs there too. Until 2am, when it snags on a rate limit and someone crawls into the logs with a flashlight, forms a hypothesis, and catches an edge case no showroom ever planned for.

That fix is the value. And you can't buy it.

Gartner says: by 2027, 40 percent of companies will switch their autonomous AI agents back off. Over gaps that only surface after the first blowup in production. 97 percent have rolled agents out. 11 percent actually run them.

The gap between those numbers isn't a model problem.

It's the engine room. Three checks you can run this week.

Fund the engine room, not the showroom

In the demo the agent is finished. In production it's 15 percent finished.

The other 85 percent is grunt work under load. Malformed data from one API version. Retry logic that doesn't run amok. Costs that blow up the business case.

IBM put a number on it: price the hardening in, and you project 29 percent more ROI.

Pay for the showroom only, and you buy 15 percent and pay for the other 85 twice.

Your best knowledge lives in two heads

Operational knowledge is your memory. Today it sits in the two people who patched the last incident.

That's concentration risk. One of them walks, the asset walks with them.

That's how the debt pile grows. Unresolved, AI-generated technical debt passed 100,000 open issues in real repositories by early 2026. Because the fix never made it into a runbook.

So write it down. Every edge case, every "except when X" rule, every 3am bug belongs in the repo, not in a chat log.

Otherwise you pay the same tuition twice.

Past the 50th entry, the asset turns into a liability

A growing agent library feels like progress. Until it doesn't.

The metadata rides in context on every call. The hit rate drops. Past about fifty entries, the next agent makes the first forty-nine less reliable.

Gartner adds: bolt the same governance onto every agent, and you cause the outage yourself.

Run the library like a portfolio, not a junk drawer. Measure where upkeep costs more than the additions return. Skip that, and you fund ballast and call it strategy.

Engine room open. Lights on.

I write field notes from real builds: AI integration, cron-driven automation, and the parts that break in production. New posts every two weeks; if this one was useful, the agent playbook is the companion download.

I Stopped Paying Frontier Prices to Re-Explain Myself to a Forgetful Agent

René Zander — Thu, 04 Jun 2026 09:22:46 +0000

Build your AI skill once with your best model. Then run it on a model that costs a tenth as much until the next flagship ships. The output will not drop.

That sounds like a downgrade. It is not. It fixes the two things that make AI agents painful right now: they forget, and the good ones cost. Both get fixed in the same place, and it is not the model you pick.

You explain the goal, the agent nails it twice, then on the third run it quietly drops the one constraint that mattered. Upgrading the model does not fix that. It only makes the dropped constraint cost more. You are paying frontier prices to be forgotten more politely.

The goal is living in two places that cannot hold it. In the conversation, where it rots the moment the context gets long. And inside the model, where keeping it sharp burns money on every run.

The constraint it dropped on Tuesday belongs in a script

Quality comes from whatever checks the work. The model that produced it is almost incidental. So decide the exact exit criteria for each step of your skill, then write a deterministic script that enforces them. The folders exist. The file parses. The test passes. The lint is clean. The agent reads the script's verdict instead of grading its own output.

A script cannot forget the goal. That is the whole point. Your agent drops constraints because you trusted a probabilistic system to hold a hard requirement in its head. Move the requirement into code that fails the run when it is broken, and forgetting stops being possible. You are not repeating yourself anymore, because the harness repeats it for you, every run, exactly.

Where the expensive model actually earns its price

This is the one place a frontier model earns its price. Use the best model you have to build the skill as exactly as you can today. Name the phases. State the precise goal of each. Get the exit scripts right. That is hard, judgment-heavy work, and you do it once.

Then swap the model out and run the skill on something cheap. Gemini 2.5 Flash through OpenRouter, driven from the opencode desktop app if you want a UI instead of a terminal. The cheap model generates. The scripts gate. You review the scripts' output, not the model's opinion of its own work.

The cheap model clears the same bar, because the bar is enforced outside it. A model that costs a fraction as much produces work you can trust. It did not get smarter overnight. It is no longer the thing deciding whether the work is good enough to ship.

The frontier model is a contractor you re-hire on release day

Here is the cadence. A new flagship model ships. You bring it in for one job: build any new skills, and re-validate every harness you already run against the new ceiling. Then you let it go. Until the next flagship drops, you run everything exclusively on cheap and local models, small language models included, wherever they win on the bottom line.

That inverts the dependency everyone assumes they are stuck with. You are not renting frontier intelligence for as long as the product lives. You pay top rate for a few build days a release cycle, and the thing that runs ten thousand times a month is a small model that costs almost nothing. The forgetting is gone, because a script holds the goal. The bill no longer scales with quality, because a cheap model clears the scripts.

I build the harness, not a standing dependency on whoever ships the smartest model this quarter.

Open the last agent you argued with. How much of that conversation was you re-explaining a goal a script could have held? And which model were you paying to forget it?

I Stopped Writing Better Prompts and Started Counting What My Skills Couple To

René Zander — Thu, 04 Jun 2026 07:00:14 +0000

Prompts rot. Captured failures compound. Most of the AI skills you are building are mostly prompt, which is why most of them will not survive the year.

Not because the prompts are bad. A skill's value is maybe twenty percent instruction and eighty percent scar tissue, and only that second part lasts. The instruction rots the moment the thing it describes moves. Encode how your team deploys and it works until the pipeline changes. Then you are debugging a prompt at 2am, with less to go on than if you had written the script yourself.

So before you build another one, stop asking whether the prompt is good. Ask what the skill is holding onto, and whether that thing sits still.

A skill rots at the speed of what it touches

A skill rots in proportion to how tightly it is coupled to things that move. Generic scaffolding leans on stable ground like a language or a convention, so it ages slowly. Domain logic wired to a codebase that gets refactored every quarter ages fast, no matter how good the prompt is.

The difference is the dependency count. "Write a unit test in this style" depends on a language and a convention. Both barely move. It keeps working for years because nothing under it shifts.

Real company-specific procedure is the opposite. File layouts. Service contracts. The one edge case in the billing flow. Each detail you pack in is a thread tied to something that gets refactored. Pack in enough of them and the skill is not a tool anymore. It is a liability with good intentions, and it fails silently, because a stale prompt does not throw. It quietly does the wrong thing.

That is what the skill-library pitch gets backwards. Volume is not value. A hundred skills wired to a moving codebase is a hundred things to maintain.

The only part that compounds is the scar

One part of a skill does not rot. The captured failure.

The five-line check you added after a model confidently reported a 41 percent dividend yield. The retry that refuses to fire twice so a flaky webhook cannot double-charge anyone. The guardrail you wrote only because production taught you, the expensive way, that you needed it.

None of that is prompting. Each one is a bug you paid for once and encoded so you never pay again. A prompt that says "always check the yield" rots the moment attention drifts. A five-line script that checks it and fails the run does not. The model reads the verdict; it is not trusted to re-derive the rule. Instructions ask the model to behave. Captured failures make the misbehavior impossible to ship.

That is also why they outlast the model. The failure modes of reality do not expire. Rate limits at 2am, malformed payloads, the off-by-one nobody catches in review. Those keep happening, to every version of the model, forever. A check against them is worth more next year than it is today.

That eighty percent is the only part worth carrying to the next model. The rest you rewrite every time the ground moves.

You can predict the rot before you write it

A skill's future is readable before the first line exists.

For each one, ask two questions:

What does this couple to, and how often does that move?
Which lines are captured failures, and which are decoration that makes the skill look thorough?
Which rules here would survive the model forgetting them? If a rule lives only in the prompt, it rots with the prompt. If a deterministic check enforces it, it compounds.

Then build to the answer. Keep the volatile coupling thin and swappable, so when the pipeline changes you edit one line instead of rereading the whole skill. Let the captured failures accumulate, because that is the part that pays rent. A skill built this way ages in reverse. It gets more useful as it collects more scars.

The skills worth keeping are not the clever ones. They are the ones that remember what broke. The engineering was never in the prompt. It was in the failures you bothered to capture.

Open the skill you reach for most. How much of it is instruction the model could half-guess on its own, and how much is a check that fails the run without asking the model's permission? Which half will still be true after the next model upgrade?

Half Your CLAUDE.md Is Decoration. The AI Reads It Once.

René Zander — Tue, 02 Jun 2026 10:42:50 +0000

Some of the rules in your CLAUDE.md should not be rules at all.

Not because they are wrong. Because you have written the things you cannot afford to lose into a file the model reads once and then slowly forgets. A preference survives that. A constraint does not.

A month ago I posted ten CLAUDE.md rules. The most common reply was not "add an eleventh." It was quieter than that: the rules stop getting followed. Forty thousand tokens into a session, the model edits the file you told it never to touch, or commits straight to main after you wrote, in bold, that it never should. Maybe half of what is in my own file is decoration. The half that is not decoration is the half that scares me.

The File Is Read Once. Your Session Isn't.

Claude Code reads CLAUDE.md once, at the start of a session. The instructions then sit in the context window and compete with everything else for the model's attention. As the session grows, the rules near the bottom lose weight. The model is not disobeying. The instruction stopped being load-bearing.

That is the part nobody demos. The screenshot of a clean CLAUDE.md at token zero tells you nothing about token ninety thousand, which is where you actually live when the work gets hard.

The First Instinct Is to Write the Rule Louder

So you do what I did. ALL CAPS. "IMPORTANT." "You MUST." A box of warning emojis.

It works. For a while. A louder line holds attention longer than a quiet one. Then the session gets long and busy, the diff gets large, the model is three tool calls deep into a refactor, and the loud line is now buried under ten thousand tokens of its own output. It decays anyway. You bought yourself an hour.

The instinct is right about one thing and wrong about the rest. It is right that some rules cannot be allowed to fail. It is wrong that emphasis is how you stop them failing.

Stop Writing Constraints as Sentences

So here is the move I opened with, made concrete. Open your CLAUDE.md and split every line into two piles.

Pile one: preferences. Naming style. Comment density. Which library you like. If the model misses one of these at token ninety thousand, you shrug and fix it. These belong in CLAUDE.md. A suggestion is exactly the right shape for them.

Pile two: the lines where a single miss is expensive. Never commit to main. Always run the formatter before claiming done. Never touch the production config. These are not preferences. They are constraints. And a constraint written as a sentence in a file is a hope.

Move pile two into hooks.

A hook is code the harness runs around tool calls. It fires whether the model remembers the rule or not. "Never commit to main" as a CLAUDE.md line survives until it doesn't. As a pre-commit hook, the commit is blocked, by code, at token five and at token two hundred thousand, on a good day and a bad one. The model cannot decay past it because it was never asking the model.

The One That Gets Through

The line in my file said never run the migration without confirming first. It sat there for weeks, doing its job, because nothing pushed against it. Then one long debugging session, deep in a chain of tool calls, the model ran it. The rule was still in the file. It had scrolled out of reach an hour earlier.

That is the shape of the cost. An expensive rule left as a suggestion works ninety-nine sessions. The hundredth is the one you tell people about. A hook would have refused the call and moved on, and I would have nothing to tell.

The rules I actually depend on are now a short list, and not one of them lives in CLAUDE.md anymore. They live in code that does not get tired.

Read Your Own File Back

I build agent setups that hold at token two hundred thousand. The screenshot that looks clean at token zero was never the hard part.

So read your own file back. For each line, ask: if the model ignored this ninety thousand tokens from now, would I notice, and would it cost me?

The lines where the answer is yes were never rules. They were hopes wearing bold text. Which of yours are suggestions, and which are actually enforced?

I write field notes from real builds: AI integration, cron-driven automation, and the parts that break in production. New posts every two weeks. If this one was useful, the human-in-the-loop approval playbook is the companion download.

I Run 10 AI Agents on Cron. Here's the Only Opus 4.8 Change That Mattered.

René Zander — Fri, 29 May 2026 12:49:21 +0000

I changed one model string in ten cron jobs last night. 4.7 to 4.8. Then I went to bed.

The benchmark threads can wait until morning. My agents can't. They fire whether I'm awake or not: a briefing at 6:30, follow-up drafts at 11:45, a sync at 4 AM that nobody reads until it breaks.

So the question I actually had wasn't whether the new model scores higher. It was whether the 6:30 briefing would read any different over coffee. That answer isn't on a chart.

Here's the recap everyone leads with. Opus 4.8 shipped May 28 at the same price as 4.7. Effort control to trade cost for depth, a dynamic-workflows mode in the CLI for big jobs, fast mode at three times lower cost, sharper agentic judgment with fewer tool-calling steps. Now the part that changed my setup.

The Room Nobody's In

Benchmarks are scored with a human in the loop. Someone reads the output, catches the bad answer, re-rolls. Opus 4.8 is better at that supervised case. Fine.

My agents don't have that person. When a cron job calls the model at 6 AM, whatever it decides ships. If it confidently does the wrong thing, there's no reviewer between the bad call and my inbox. Peak intelligence was never my bottleneck. Confident wrong action with nobody watching was.

Two different axes. The leaderboard measures the first. Production agents die on the second.

What Moved on the Axis I Care About

The line in the announcement that mattered to me wasn't a score. It was "catches its own mistakes, pushes back when a plan isn't sound." And "fewer steps for the same intelligence."

Read those from inside a system that's already running.

Fewer steps means each unattended run burns fewer tokens to reach the same result. Ten agents, every day. That compounds.

Pushing back means an agent is likelier to stop and flag a shaky plan instead of charging ahead and emailing me garbage. For supervised work it's a nice-to-have. For a job running into an empty room, it's the whole point.

The Cost Dial I Didn't Have Before

There's now an effort control. You pick how hard the model works a task.

For me it maps straight onto the cron list. The 4 AM sync is mechanical. Low effort, cheap, done. The follow-up drafter needs judgment about what to say to a client. High effort, worth the tokens. The recommendation is to turn effort up for long-running async work, which is exactly what a cron agent is.

Before, every job paid for the same depth whether it needed it or not. Now the depth is a knob per job.

The API Change Worth More Than the Score

One more thing that didn't make the highlight reel. The messages API now takes system entries mid-conversation without breaking the prompt cache. If you've ever changed an agent's instructions partway through a long task and watched your cache evaporate, you know why that line matters. Not flashy. Saves real money on long sessions.

Where the Coverage Reads Backwards

Maybe 80% of the launch coverage is about benchmark deltas. Maybe 20% touches the things that change how an unattended system behaves.

For someone running a chat window, that split is right. The benchmark is the product.

For anyone running models without a human watching each output, it's backwards. The judgment, the step count, the effort dial, the cache behavior on long tasks. Those are the upgrade. The chart is the part you can skip.

So before you screenshot the bar going up: where in your setup does a model already act without you watching? Did this release actually move that?

I write field notes from real builds — AI integration, cron-driven automation, and the parts that break in production. New posts every two weeks at renezander.com.

The Skill Library Has a Discovery Ceiling. Most Hit It Around 50.

René Zander — Wed, 27 May 2026 09:46:06 +0000

You added a skill last Tuesday. The agent hasn't called it once. Each new skill silently weakens the discovery odds of the ones you already have.

You assume it's a description problem. It isn't.

Everyone's pushing past 50 skills now. Vercel ships a plugin with 40. Google's gws brings 95. I do it too. My local registry is at 38.

Each one I added lowered the odds that the others get reached when I need them.

The Catalog Sits in Context Every Turn

The three-tier loading model promises that bodies stream in on demand, but the metadata (name plus one-line description) sits in the system prompt every turn, for every installed skill. The math is mechanical: Anthropic's spec budgets roughly 30 to 50 tokens per skill plus 109 chars of overhead, which compounds fast.

A 95-skill library reports 2,007 tokens per turn in available_skills. At Opus pricing over a typical workday, that's about three dollars per developer per day for listings the agent cannot distinguish anymore.

The dollars are not the issue. The signal-to-noise ratio is.

Routing Degrades With Catalog Size

A study published this month measured retrieval against an 80,000-skill catalog. Best-in-class routing hit 74% at top-1. One task in four picks the wrong skill or no skill at all. Drop to metadata-only matching, which is the default at session start, and accuracy falls another 31 to 44 percentage points across every method tested.

There is no fix coming from better embeddings. The problem is that fifty good one-line descriptions stop being distinguishable from each other. Adding the 51st makes "publish a draft" and "ship a draft" and "deploy a post" harder to tell apart, not easier.

The undocumented setting skillListingBudgetFraction in Claude Code makes this worse without warning. As remaining context shrinks during a long session, the absolute budget for listings shrinks proportionally. Skills at the bottom of the listing order get truncated. The model never sees them. It cannot invoke what it cannot see.

You added the skill. The agent does not know it exists.

The 85% Lives Below the Skill Layer

A reader who runs an 89,000-page programmatic SEO site mentioned in a thread that skills cover maybe 15% of what the system actually does. The other 85% he called "scar tissue code." The conditions nobody writes until they see a 2 AM page from a third-party API returning malformed JSON on month-end.

That 85% is real. It looks like this:

A redundant null check in payment validation because one vendor sends "N/A" for a specific contract type on month-end.
A Stripe webhook timeout set to 18 seconds because their p95 was 14 and you wanted headroom.
A retry policy on the embedding queue that backs off harder for one endpoint because its rate limiter returns 200 before timing out at 60 seconds.

None of that is in a SKILL.md. None of it can be. It lives in the schema migration history, the incident postmortems, the runtime telemetry baselines, the permission graphs, the third-party API quirk log that nobody calls a log. Skills delegate to that layer. They do not contain it.

The author who ships a thousand-skill library is not adding capability. They are adding a discovery problem the user solves at runtime.

The Metric Was Always Subtraction

The right engineering metric for an AI runtime is not skills added. It is lines of imperative code deleted per skill kept. Cursor's team published numbers last month: 12,000 lines of TypeScript replaced by 200 lines of skill markdown. That is a 60x ratio. Code that no longer exists has zero bugs, zero CI cost, zero migration burden.

That ratio cuts both ways. Every skill that does not delete real code is overhead.

A static analyzer published in April classifies skill failures into five types: dead, bloated, conflicting (one says "use jq" and another says "never use jq"), stale, cyclic. Run it against a 50-skill library and 30% to 40% typically sit in one of those buckets. They are not earning their seat in every session's system prompt.

The Library Became a Portfolio Metric

The skill library has become a portfolio metric. Founders ship 200-skill libraries for the same reason juniors ship 4,000-line PRs. Visible output beats invisible discipline. The discipline doesn't show up on the GitHub README.

But you don't optimize a runtime for the README. You optimize it for whether the agent can find the right skill on the third turn of a real session. Every skill you keep that doesn't earn its slot is a quiet vote against every other skill in the library.

Which three would you retire tomorrow? Not because they're bad. Because their slots are worth more than they're paying.

Skills are the menu. The kitchen is everywhere else. I build kitchens.

Rene

I write field notes from real builds — AI integration, cron-driven automation, and the parts that break in production. New posts every two weeks at renezander.com.

Browser-Use Is Solving the Wrong Half of the Problem

René Zander — Tue, 19 May 2026 14:57:13 +0000

TL;DR — when to use browserground (and when to use UI-TARS-MLX instead)

If you're on Apple Silicon with ≥16 GB RAM and you need generic, max-accuracy UI grounding, use mlx-community/UI-TARS-1.5-7B-4bit. It's the obvious default — ~94% on ScreenSpot-v2, MLX-native, drops into mlx-vlm directly. ByteDance research-lab compute, you couldn't reproduce it on a budget. I'm not the right pick for that workload.

browserground is for two narrower jobs:

1. The recipe for your product's custom UI grounder. UI-TARS is a finished model — closed pipeline, proprietary data, hard to extend. browserground is the opposite: a reproducible template. Open base (Qwen3-VL-2B), open training scripts, open data mix (26k records, OS-Atlas + wave-ui). Swap in your dashboard's screenshots / your customer app / your internal tooling → ship a domain-trained UI grounder over a weekend. The 60% generic ScreenSpot-v2 score isn't the deliverable; the recipe is. A 60-point baseline on generic screens becomes 85-95% on your own product's narrow surface because the test distribution finally matches the training distribution.

2. The smallest viable slot in a multi-model stack. browserground 4-bit MLX is ~1 GB on disk / ~2 GB RAM. UI-TARS-1.5-7B-MLX is ~4 GB / ~5-6 GB RAM. The difference matters on 8 GB Macs and in agent stacks that already run a 7B planner + an OCR model + an embedding model in the same RAM budget. Plus strict JSON output (100% parseable, no regex on prose) — small win, but real.

A direct head-to-head benchmark of browserground vs UI-TARS-1.5-7B-MLX on the same Apple Silicon hardware is forthcoming.

The broader argument — why a parser-stage specialist matters at all

And if you're new to the hybrid pattern — why this exists at all

Everyone's posting browser-agent demos this week. Click here, scroll there, fill that form. Most break by click seven.

Mine broke too. The submit button on a checkout form that the frontier vision model literally couldn't see. Billed at $0.01-0.05 per call, called 20-50 times per agent run, the model was burning reasoning capacity on parsing pixel coordinates. A 2B specialist I trained for $5 hits that same button 3.3x more reliably on ScreenSpot-v2 (60.0% vs GPT-4o's 18.3%).

The architecture is the bug, not the model.

Two Jobs, One Forward Pass

Browser-agent stacks send a screenshot to a frontier vision model and ask it for both the next decision and the click coordinates in one call. Splitting that into two calls, a local 2B grounding model that emits JSON followed by a frontier model that reasons over the JSON, drops vision token spend and raises click accuracy.

browser-use (94k stars), Skyvern (22k stars), Claude Computer Use, OpenAI Operator. Same pattern. Same compound question every step:

Given this page, what should the agent do next, and where exactly does it click?

Two jobs welded together. Reasoning ("what next") is a probabilistic problem worth a frontier model. Grounding ("where exactly") is a structured-output problem with a tight schema: clickable elements, bounding boxes, accessible labels.

You're paying frontier-tier rates for the second job. Per screenshot. Every step of the loop.

Grounding Is a Parser Problem

Once you name it as a parser problem, the right tool changes. You don't need 200 billion parameters to emit a JSON list of clickable elements. You need a model that:

Has seen enough UI screenshots to recognize buttons, inputs, links with sub-50-pixel precision
Outputs strict JSON without hallucinating bounding boxes
Runs locally so the per-step cost is zero

A 2B specialist trained on screen-parsing data. Not a frontier model.

I trained one. Total cost: ~$5 of RunPod compute on a single A6000 GPU. The result, browserground, hits 60.0% on ScreenSpot-v2 vs GPT-4o's 18.3% — a 3.3x beat at the click-grounding job. More telling: it beats SeeClick (9.6B params, 55.1%) at 4.8x smaller. A drop-in for any agent loop currently handing screenshots to a frontier API. Today the CLI runs via transformers on Apple Silicon (~14 s/call); MLX-native build coming for the ~1.5 s path.

The Reasoning Model Gets Its Reasoning Capacity Back

When you split the call, the frontier model stops seeing pixels. It sees:

{
  "elements": [
    {"id": "e7", "label": "Submit order", "type": "button", "bbox": [344, 612, 478, 658]},
    {"id": "e8", "label": "Edit cart",    "type": "link",   "bbox": [...]}
  ]
}

Now the frontier model does the job it's good at: deciding e7 vs e8 given the agent's goal. A reasoning question over structured input. Cheap. Reliable. Auditable.

Three things change at once. Per-step token spend on vision collapses, because the grounding step runs locally. JSON validity hits 100% (the specialist learned the output convention with 35M LoRA parameters on a Qwen3-VL-2B base). Agent traces become debuggable. You read the structured grounding output before the reasoning step ever runs.

What Anthropic and OpenAI Ship Next

The frontier providers will absorb grounding into their own small models. Within twelve months, "fast vision" or "tool vision" tiers will appear in both Anthropic and OpenAI billing at a fraction of frontier rates. The economics demand it. Nobody can justify charging GPT-5 prices for a parser, and Hugging Face downloads already prove the demand: SeeClick, UI-TARS, and ShowUI pull ~300k category downloads a month between them.

When that ships, stack owners who already split grounding from reasoning have three things the wait-and-see crowd doesn't. A local fallback if the provider has an outage. An auditable structured-grounding trace in every log line. An exit option to a different reasoning provider without re-validating click behavior, because the grounding step belongs to them.

Stack owners who didn't split will find their grounding step has quietly become someone else's API. Same vendor billing the reasoning calls. Same vendor setting the price. Same vendor's deprecation calendar.

The Diagnostic

Pull up your last failed agent trace. Three numbers:

Total tokens spent on vision calls per agent step.
Fraction of those tokens spent on grounding (parsing pixel coordinates) vs reasoning (deciding actions).
Per-run vision cost at your current API rates.

If grounding dominates the first two numbers, and in most stacks it does, your stack has the split wrong.

Grounding is plumbing. Reasoning is cognition. Stop paying cognition rates for plumbing.

I build the split layer. browserground is the open-source reference for the local grounding half. v0.3 ships three packagings so it drops into any stack:

npm CLI (daemon, HTTP REST server, batch, confidence, eval): npm install -g browserground → browserground parse <img> --target "..."
PyPI (no Node required, MLX or transformers): pip install "browserground[mlx]" → from browserground import click_xy
Ollama (cross-platform, GGUF Q4_K_M + f16 mmproj): ollama run renezander030/browserground

Adapters land in the repo for browser-use (drop-in Controller action) and Skyvern (ground_with_fallback for local-first + cloud-fallback). Model: huggingface.co/renezander030/browserground. MLX 4-bit: browserground-mlx. GGUF: browserground-gguf. Source: github.com/renezander030/browserground. Apache-2.0. v0.2 LoRA trained on 26k mixed-domain examples (macOS + Android + UIBert + web). PRs welcome, especially eval cases where it fails.

I write field notes from real builds — AI integration, cron-driven automation, and the parts that break in production. New posts every two weeks at renezander.com.

Three Files I Renamed Last Month That Fixed My AI Agent

René Zander — Tue, 12 May 2026 06:59:17 +0000

A new job title showed up in my feed last week. Context engineer.

I clicked. Twice. Three articles, all definitional. None of them showed code. Every definition was a paraphrase of "name things well."

Then I opened my own repo and grepped for utils.py. Eleven hits. Six folders deep, in a project I have shipped to production. The same project where I had spent two hours that morning fighting an agent that kept loading the wrong helper file.

The fix took thirty seconds. I renamed the file.

The pitch checks out

The current pitch goes like this. LLMs perform better with the right context. So we need a new role to design that context. Curate the files, structure the prompts, build the retrieval system, manage the embeddings.

Each claim is accurate. None of them describes a new problem.

You have been doing this since you wrote your first config file. Every time you renamed helpers.js to payment-validation.js, you were context engineering. Every time you split a 400-line file into three named pieces, you were context engineering. The audience was always the next developer reading the file. Now there is one more reader.

The actual job

A senior engineer joining a codebase does three things in the first week. They read the folder structure. They read the most-named-things in the imports. They follow the breadcrumb from filename to function to line.

Agents do the same thing. The retrieval layer in your agent loop is grep with extra steps. It pulls files whose names match the task. It pulls functions whose docstrings match the intent. It pulls comments that say what the code does.

If your filenames are vague, the retrieval is vague. If your function names lie about what the function does, the agent loads the wrong one. If your folder names group code by file type instead of by concern, the agent loads models/ and gets nothing useful.

The skill we have been failing to enforce for thirty years became load-bearing the day a probabilistic reader joined the loop.

Three things I renamed last month

api.py became payment-webhook-handler.py. The agent stopped loading it for unrelated payment questions. One rename, one less failure mode.

utils/ got deleted. The five files inside moved next to the code that called them, with names that said what they did. format-currency.py, parse-iso-date.py, redact-pii.py. The agent now loads them only when the task mentions currency, dates, or PII.

A 600-line process.py split into four files. validate-input.py, dedupe-rows.py, enrich-from-cache.py, write-to-warehouse.py. The agent stopped trying to read the whole pipeline to answer questions about a single step.

Call it context engineering if the buzzword helps. The work is renaming.

Where the model has to guess

Walk your repo right now. Count the files named utils.py, helpers.js, common.go, lib/, services/, manager/. Count functions named process, handle, run, execute, do.

Each one is a place where the model has to guess. Each one is a place where you have to write a longer prompt to compensate. Each one is a paragraph you will never have to write again if you rename it now.

The reason context engineering feels hard is that you are trying to solve at retrieval time what should have been solved at naming time. You cannot grep your way out of a vague folder structure. You cannot embedding-search your way out of process(). The model is asking the same question the senior engineer asks in week one. The answer was always going to be: name things by what they do, not by what they are.

The role that already existed

There is a real version of context engineering. Chunking strategy, embedding choice, retrieval rerankers, evaluation harnesses. That work is real and hard.

Most of what gets called context engineering this month is the rename you skipped in the original PR.

A context engineer is a developer who finally names things.

What is the file in your codebase that the agent keeps loading wrong?

I write field notes from real builds — AI integration, cron-driven automation, and the parts that break in production. New posts every two weeks at renezander.com.

Your AI Workflow Doesn't Need Better Prompts. It Needs Less AI.

René Zander — Tue, 05 May 2026 06:33:55 +0000

The first stage of AI work is prompting.

The last stage is removing the model from most of the workflow.

That sounds backwards.

It is not.

When a workflow is new, the LLM is useful because the work is still ambiguous. You are discovering what good looks like. You try a prompt, read the output, adjust the examples, change the tone, add constraints, and run it again.

That is a good use of AI.

But if the same workflow keeps coming back, and you are still explaining it to the model every time, you are not building capability. You are repeating yourself with a better interface.

The mature workflow is not one where the LLM does everything.

The mature workflow is one where the LLM only handles the part where ambiguity is useful.

Everything else becomes process.

Prompting Is Discovery

Prompting is where most people start because it is the fastest way to get feedback.

You ask:

"Write this article."
"Make it sound less generic."
"Use my style."
"Add examples."
"Make the intro stronger."

At this stage, the model is helping you figure out the shape of the task.

You do not fully know the target yet. You are exploring. You are testing whether the idea even works. You are using the model as a thinking partner, a drafter, a critic, and sometimes a mirror.

That is fine.

The mistake is treating this as the final form.

If you have to keep saying the same thing, you do not have a workflow. You have a recurring conversation.

Better Prompts Still Have a Ceiling

The next stage is usually better prompting.

You add examples. You add constraints. You add a target audience. You tell the model what to avoid. You define the output format. You write a longer system prompt.

The output improves.

For a while, this feels like progress.

But longer prompts have a hidden failure mode: the model still has to remember and balance everything at once.

Style rules. Factual constraints. Tone. Audience. Platform conventions. Examples. Edge cases. Forbidden phrases. Review criteria.

All of it goes into one big instruction block.

The prompt becomes a pile of expectations, and the model is still the one deciding which expectations matter in the moment.

That is fragile.

At some point, "make the prompt better" stops being the right move.

Skills Are Repetition

The next level is turning repeated prompting into a skill.

A skill packages the context and process:

what files or sources to read
what examples matter
what tone to use
what scripts to run
what output format is expected
what review criteria should be applied
what fallback path to use when something breaks

This is a real improvement.

The workflow becomes portable. You stop explaining everything from scratch. The model gets the right context faster. You are no longer relying on whatever happens to be in the current chat thread.

For many AI workflows, this is the first serious productivity jump.

But skills have their own failure mode.

They can become too large.

When Skills Become Prompt Monsters

A skill can start as a clean reusable process and slowly turn into another giant prompt.

More instructions.

More exceptions.

More "always do this."

More "never do that."

More examples.

More scripts.

More personal preferences.

At some point, the skill is not making the workflow reliable. It is just giving the model more things to interpret.

The model still has to decide what matters.

The model still has to judge whether the output is good enough.

The model is still checking its own homework.

That is the point where the workflow needs to move outside the LLM.

Not all of it.

The stable parts.

The Model Can Write the Code. It Does Not Get to Decide Whether the Code Passes.

Developers already understand this when we talk about code.

We do not ask a developer, human or AI:

"Does this code look correct?"

We run the formatter.

We run the linter.

We run the tests.

We run the type checker.

We run CI.

We use pre-commit hooks.

The model can generate code, but it does not get to decide whether the code passes.

The gate decides.

That is the important shift.

If a rule can be checked deterministically, it should not live only inside a prompt.

For code, the gates are obvious:

formatting
linting
type checks
unit tests
integration tests
golden files
JSON schema validation
pre-commit hooks
CI checks

The agent can still do useful work. It can draft the patch, explain a tradeoff, write a test, debug a failure, or propose a smaller path.

But the agent should not be the final authority on whether the work satisfies the standard.

That authority should be outside the model.

This Applies to Content Too

Content feels less deterministic than code, so people keep more of the workflow inside the prompt.

But the same principle applies.

If you have found a content formula that works, do not just ask the model to remember it.

Turn it into gates.

For example, before publishing an article:

Does the title match the actual promise of the article?
Does the first section create tension quickly?
Is the reader obvious?
Is there one clear argument?
Does every section move the argument forward?
Are there generic AI phrases that should be removed?
Does the article contain a real observation or only recycled advice?
Is there a concrete next action for the reader?
Does the tag strategy match the article, not just the broad topic?

These checks are not perfect.

But they are better than "make it good."

"Make it good" is a vibe.

A gate is a standard.

The more often a workflow matters, the more it deserves standards that do not depend on the model's mood.

The 20% LLM Workflow

The mature version of an AI workflow is not:

"The LLM does everything."

It is:

"The LLM does the part where ambiguity is useful. The system handles the rest."

That might mean the model only does 20% of the workflow.

And that is a good thing.

For a content workflow, the LLM might:

propose angles
compare possible hooks
draft sections
rewrite unclear paragraphs
find contradictions
suggest examples

But the system should handle:

loading the source notes
selecting the target platform
applying the tag strategy
checking title/promise match
scanning for banned phrases
verifying links
enforcing formatting
creating the publishing task

For a coding workflow, the LLM might:

inspect the codebase
implement the change
write tests
explain a failure
reduce a patch

But the system should handle:

formatting
linting
type checking
test execution
schema validation
contract checks
CI gates

That is not using AI less because AI is weak.

It is using AI less because the workflow is becoming stronger.

The LLM Should Be the Escalation Path, Not the Reflex

There is a simple rule I keep coming back to:

If code can check it, do not ask the model to remember it.

If a script can clean it, do not spend tokens reasoning about it.

If a test can catch it, do not rely on a sentence in a prompt.

The LLM should be used when ambiguity remains.

It should not be the first line of defense for things that are already measurable.

This is where many AI workflows waste effort. They ask the model to handle everything:

classify the task
remember the rules
generate the output
check the output
decide whether the output is done
explain why it is done

That is too much responsibility in one probabilistic step.

Split the work.

Let the model handle the ambiguous part.

Let deterministic systems handle the stable part.

A Self-Check for Your Own AI Workflow

If you want to know where your workflow is immature, ask these questions:

What do I keep explaining to the model again and again?

That probably belongs in a skill.

What does the model keep judging by itself?

That probably belongs in a gate.

What failure would be obvious to a script, linter, test, schema, or checklist?

That should not live only in the prompt.

If I removed the LLM tomorrow, which parts of the workflow would still be clear?

Those parts are real process.

Which parts only work because the model is being generous?

Those parts are risk.

This self-check is uncomfortable because it reveals how much "automation" is just trust in a model call.

But that is the point.

Capability is not how much work you can hand to the model.

Capability is how much of the workflow still holds when the model is only doing the part it is actually good at.

The Maturity Curve

The pattern looks like this:

Prompt -> Skill -> Gate -> System

Or:

Ask -> Package -> Validate -> Automate

When the task is new, prompt.

When the task repeats, create a skill.

When the skill succeeds, move the stable checks into gates.

When the gates are stable, reduce the LLM's responsibility.

This is the path from beginner prompting to a 20% LLM workflow.

It does not make the model irrelevant.

It puts the model in the right place.

The Wrong Goal Is Better Prompting Forever

There is a lot of advice about better prompts.

Some of it is useful.

But better prompting is not the destination.

Better prompting helps you discover the workflow.

Skills help you repeat the workflow.

Gates help you trust the workflow.

Systems help you scale the workflow.

If you stop at prompting, every task stays a negotiation.

If you stop at skills, every process still depends on the model interpreting the skill correctly.

If you add gates, the model has something it must pass.

That is the difference between a helpful assistant and a reliable workflow.

The Useful Question

The useful question is no longer:

"How do I write a better prompt?"

The useful question is:

"Which part of this should stop being a prompt?"

If it is repeated context, make it a skill.

If it is a stable rule, make it a checklist.

If it is measurable, make it a test.

If it is non-negotiable, make it a gate.

Let the LLM handle ambiguity.

Make the system handle standards.

Use prompts to discover.

Use skills to repeat.

Use gates to scale.

And when the workflow is mature, let the LLM do less.

That is not a downgrade.

That is capability becoming real.

Which part of your AI workflow should stop being a prompt?

I write field notes from real builds — AI integration, cron-driven automation, and the parts that break in production. New posts every two weeks at renezander.com.

Pure semantic search missed 4 of 5 of my agent queries. Hybrid + parallel fan-out fixed it.

René Zander — Sat, 02 May 2026 14:13:04 +0000

When Karpathy's LLM Wiki post landed, I already had semantic search over my TickTick — qdrant for the vector store, nomic-embed-text via ollama for embeddings, a daily cron to keep the index fresh, the works. The agent-side retrieval wasn't the missing piece.

What was missing was the structure. Karpathy's framing — designate a wiki, write notes for an LLM reader, lean on retrieval instead of taxonomy — surfaced the parts of my setup that didn't have shape yet: where durable knowledge lives versus ephemeral tasks, how agents pull structured data out of notes humans wrote, why my existing semantic search sometimes returned the right answer and sometimes returned nothing useful.

I almost migrated to plain markdown anyway. Thousands of durable notes — production playbooks, API quirks, decisions I want to survive next month's task list — already live in TickTick. They sync to my phone. Capture friction is zero. Migrating breaks all of that.

So I built the wiki structure on top of TickTick, and made the storage layer swappable. The retrieval, the wiki conventions, the agent-data note pattern, the bench harness — none of those are TickTick-specific. They're a small framework. You point it at TickTick / Notion / Obsidian / Things / a folder of markdown / whatever you've already invested years of capture habit into.

I'm calling it Agentic Knowledge Base.

The framework, in one diagram

                  ┌───────────────────────────────────┐
                  │  Agent (Claude / scripts / cron)  │
                  └─────────────────┬─────────────────┘
                                    │ akb find / get / url / links
                                    ▼
                  ┌───────────────────────────────────┐
                  │  AKB Core                         │
                  │  • parallel retrieval + RRF       │
                  │  • corpus cache (5 min TTL)       │
                  │  • bench harness                  │
                  │  • usage logging                  │
                  └─────────────────┬─────────────────┘
                                    │ adapter interface
                  ┌─────────────────┼──────────────────┐
                  ▼                 ▼                  ▼
       ┌────────────────┐  ┌────────────────┐  ┌────────────────┐
       │ adapter-       │  │ adapter-       │  │ adapter-       │
       │ ticktick       │  │ obsidian       │  │ notion         │
       │  (reference)   │  │  (filesystem)  │  │  (your turn)   │
       └────────────────┘  └────────────────┘  └────────────────┘

The Core is storage-agnostic. The retrieval, the cache, the bench, the usage logger — none of them know what TickTick is. They call a small adapter interface (~6 methods).

Karpathy's setup, in this framing, is the filesystem adapter of a broader pattern. Mine is the TickTick adapter. Yours might be the Notion or Obsidian one.

The adapter interface

Six methods. Two payload shapes.

interface KnowledgeAdapter {
  listProjects(): Promise<Project[]>
  listTasksInProject(projectId: string): Promise<Task[]>
  getTask(projectId: string, taskId: string): Promise<Task>
  createTask(input: TaskInput): Promise<Task>
  updateTask(projectId: string, taskId: string, patch: TaskPatch): Promise<Task>
  urlFor(ref: { projectId: string, taskId: string }): string  // deep-link string
}

type Project = { id: string, name: string, kind?: 'tasks' | 'notes' }
type Task    = { id: string, title: string, content: string, projectId: string, tags: string[], dueDate?: string, modifiedTime: string }

Anything you can list, get, and link to — task systems, note apps, plain folders — can be an adapter.

If your storage exposes a native search endpoint, your adapter can implement an optional searchByQuery(query) and the core will use it as one branch of the parallel retrieval. If not, the core falls back to its own keyword scan against the corpus.

That's the whole interface. Everything interesting is in the Core.

Two patterns the Core implements (worth stealing)

1. Agent-data notes

A regular note whose body has a fenced JSON (or YAML) block. Humans read the prose at the top. Agents extract the JSON via the adapter.

The note's content looks like this — prose first, then a single fenced JSON block:

Type: agent-data
Consumed by: EOD triage cron, capture-time relevance enrichment

A "trunk" is an active project the user cares about. Edit this list when projects launch, finish, or shift focus.

{
  "trunks": [
    { "name": "release-engineering", "desc": "shipping cadence, deployment rituals, on-call rotation" },
    { "name": "writing-projects", "desc": "drafts and edits across personal and client channels" }
  ]
}

Read it from any cron or agent:

akb get "Trunk Catalog" --extract json | jq '.trunks[].name'

The benefit: one note, mobile-editable in your existing app, consumed by agents as structured data. Single source of truth, no schema migration. This pattern works for anything an agent needs programmatically and a human needs to edit on the move: prompt templates, character locks for video projects, recurring queries, cron config.

2. Parallel retrieval with provenance

Three retrievers run in parallel against a shared cached corpus, results are RRF-fused, and the top-K come back tagged with which retrievers agreed:

Hybrid — dense cosine (qdrant + nomic-embed) + sparse keyword, internally RRF'd
Keyword — substring match on title + content
Notes-find — title-fuzzy on a designated wiki project

For a query like openrouter api key, all three retrievers return the same gold note. The fused result tags it sources: [hybrid, keyword, notes_find] — three independent signals agreeing means high confidence. Lower-ranked results have only one source — look at them with skepticism.

For a query like ffmpeg commands, the keyword tool misses (the literal phrase isn't in any document). Pure semantic misses too (nomic-embed underweights short titles like ffmpeg). Hybrid catches it. The fan-out gracefully handles the asymmetry — different queries lean on different retrievers, and the core doesn't pretend any single algorithm is universally best.

A 5-min disk-backed corpus cache means warm queries are sub-100ms. The first call after a cold start fetches your full task/note list (one batch — adapters that support it use a single API call; adapters that don't fall back to per-project iteration). Within a working session, retrieval is essentially free.

The bench

I built a small harness in bench/. Questions paired with gold answers (the task or note that actually contains the answer). Each retriever runs against the same questions, results scored by hit@1 / recall@5 / MRR.

Five agent-issued queries (the rephrased version Opus 4.7 actually generates, not the natural-language form a human types):

Method	hit@1	recall@5	MRR	warm latency
keyword (substring)	20%	20%	0.20	<100ms
semantic (dense only)	20%	40%	0.30	~300ms
hybrid (dense + sparse RRF)	60%	80%	0.70	~500ms
find (parallel + cache)	60%	80%	0.70	~93ms

The headline number for agent retrieval is recall@5 = 80% — the right doc lands in the top five 4 times out of 5. Agents read top-K, not just rank 1, so recall@5 is the metric that actually predicts whether the agent gets the context it needs. Top-1 (60%) is a stricter cut and a leading indicator for "did the first guess work" — useful but not the bar. The benchmark won't generalize from five questions — grow it as confidence in a particular adapter accumulates.

Why I optimized for the model, not for me

There's a subtle reframe that took an embarrassing number of iterations to land.

When I use search, I type a single word: ffmpeg. The keyword tool returns the right note instantly.

When Claude uses search on my behalf — "where did I document my ffmpeg workflow?" — it issues something like find "What ffmpeg commands do I have notes on?". Different shape entirely. The model writes longer queries. It uses question phrasing. It includes scope words.

Optimizing for human queries was the wrong objective. The user (me) wasn't using these tools — Claude was. Every retrieval test had to be written in the form Opus 4.7 actually generates, not how I'd type it. That changes which retriever wins.

Tomorrow's model writes queries differently. The benchmark needs to track the model in use, not a fixed assumption about query shape. The bench file is short and dated; re-tune when the model changes.

What I deliberately didn't build (yet)

Karpathy's wiki post mentions periodically updating notes when facts change — propagating new information across the knowledge base. Useful at scale; auto-rewriting notes is high-blast-radius and needs an approval ramp before it's trustworthy. I sketched it: a weekly cron that semantic-searches for affected notes, drafts updates, queues them for my approval, applies the approved ones. Deferred.

Same call on a "lint the wiki" pass (Karpathy idea: agent reads every note weekly, flags missing summaries, dangling references, contradictions). Useful at scale; premature when the wiki itself is still under construction.

Both will live in Core when they ship — adapter-agnostic by design.

A daily flow (my setup, your tools optional)

This is what runs:

Capture (mobile, manual). I add a task or note in my storage app. No CLI involved. The friction has to be zero.
Capture-time relevance prompt (when in a Claude session). akb create "..." --relevance appends a small instruction block to the result. Active Claude reads it, picks a project trunk, calls akb update to append a why: <trunk> — <reason> line. Five seconds of LLM-side reasoning makes that task much more retrievable later.
EOD triage (cron, daily morning). Pulls yesterday's completed tasks, scores them 0–3 against the trunks (read live from a Trunk Catalog agent-data note), sends a Telegram message with keepers grouped by trunk. I read it on my phone with breakfast.
Retrieval (during work, all surfaces). When Claude needs context — akb find <query> returns top-K with provenance. Cached, parallel, sub-100ms warm.

Swap "TickTick app" for "Notion / Obsidian / Things" and the flow is identical. The adapter changes, the daily ritual doesn't.

Roadmap

v0.1 — Core + reference TickTick adapter + bench (where I am today)
v0.2 — Filesystem adapter (Karpathy-style local markdown). Probably one weekend's work.
v0.3 — Notion adapter (community contribution most likely)
v0.4 — Lint pass + fact-propagation queue with approval gate
v0.5 — Adapter for Apple Notes / Things / iA Writer (Mac-native captures)

Code

Repo: github.com/renezander030/agentic-knowledge-base
One-page summary: gist.github.com/renezander030/c7bd6d5c4088e24d3add043720284453

Karpathy's wiki idea is right. The implementation that fits an existing system isn't a folder of markdown — it's the agent-side primitives that turn whatever you already have into something the model can reason over.

If you write your own adapter, I want to see it.

—

Posted from https://renezander.com/blog/agentic-knowledge-base/. Source at https://github.com/renezander030/agentic-knowledge-base.

I write field notes from real builds — AI integration, cron-driven automation, and the parts that break in production. New posts every two weeks at renezander.com.