DEV Community: Daniel Ferraz

Giving AI Coding Agents a Memory That Argues Back

Daniel Ferraz — Fri, 30 Oct 2026 21:09:36 +0000

I'm writing a series called "Becoming a Super Engineer": a software engineer since 2013, learning to hold software, data and AI under the same belt, and writing it down as it happens.

Chapter 1 ended with agents that only stopped repeating mistakes once a check failed on them.

This chapter is about the other half of that problem: what agents remember. It's about an open-source project of mine, a knowledge base the agents read before touching anything and write back to when they're done. It taught me a lot about working with agents :)

Two agents without memory, two different correct APIs. The one with memory gave back the fix we had recorded

The problem: every session starts from zero

Without persistent memory, an AI coding agent forgets everything between sessions.

Monday it spends 20 minutes figuring out how the auth middleware works. Tuesday, same question, same 20 minutes. Wednesday it debugs a bug you already fixed last week, from scratch.

My first pitch for a knowledge base was the obvious one: stop paying tokens to rediscover the same things.

A small experiment changed my mind. More on that later.

The real problem showed up in an experiment I had an AI agent run: three agents got the same money-rounding bug, in the same codebase.

All three fixed it correctly.

The two without the knowledge base came back with two different correct fixes. One added round_money(amount). The other added to_decimal, round_money(value) and percent_of(amount, pct).

The agent with the knowledge base rebuilt the fix we had already recorded: same API, same four call sites, same line numbers.

All three were right. Only one matched what the team already had.

That's the actual problem: drift. Each of these agents was smart enough to solve the problem, and each one solved it a little differently. Over months, that's how a codebase ends up with three ways to round money.

Each step moved a rule from "please do this" to "this fails if you don't"

The walkthrough: from asking nicely to failing loudly

The project is called knowledge-database. The shape is simple.

A folder of dated markdown entries, in four buckets:

explorations/: what is true about this system
solutions/: what we did, and how
errors/: what broke, and the trap that catches it next time
decisions/: what we chose, what we rejected, and who decided

Every entry cites its evidence as path:Lstart-Lend, so a claim points at real lines of code.

And one loop, on every non-trivial task: read first, do the work, write back.

Read first, do the work, write back. The check makes sure nobody skips step three

I'd been using this idea heavily in another setup before it went public. The public version, and making agents actually follow it, came together in twenty days.

Phase 1: conventions

The first versions were rules written in the agent instructions. Please read the KB first. Please record what you learned.

Recording started as a prompt at the end of every task ("want me to record this?"), and turned mandatory pretty quickly. I wanted to review what went into an entry, without approving every single one.

Everything in this phase depended on the agent choosing to behave.

Phase 2: enforce or delete

Then someone reviewed it from the outside and showed me the conventions were drifting. Index tags didn't match the entries. Entries marked verified had prose instead of proof. One decision used a hypothetical code comment as its evidence.

That review changed the project. The new rule was simple: enforce or delete.

Every hard rule got a stable ID (KB001 to KB015 today) and a test fixture that passes or fails. A rule without an ID isn't a rule anymore. It moves to a "guidance, not checked" section.

A few of my favorites:

verified requires a verification block with captured output. Comments like # should work get rejected
citations must point at files that exist, so a deleted or renamed file fails the check
a diff that touches code but not the knowledge base fails the check, and names the bucket you probably forgot

The checks run from a zero-dependency Python CLI, a pre-commit hook, CI, and a hook at the end of every agent session.

Phase 3: rules that never reach the agent don't count

A field failure showed the rules only loaded for one agent runtime, and only at the start of a session.

So now a short rules block (about 240 tokens) gets injected on every single prompt, and the same rules get planted in six agent instruction files, one per runtime. Install it mid-session and it works on the very next prompt.

Phase 4: the audit

Someone installed it fresh, on a workspace that wasn't a git repo, and read the docs like a new user. Thirteen findings.

The verdict hurt a little: everything mechanically enforced worked. Everything at the edges didn't.

The installer died halfway on a non-git folder. The rules told agents to run two scripts no install had ever copied. A staleness rule would start blocking every commit, 90 days after the entries were written, all on the same day.

All thirteen got fixed in one release, plus a setup wizard and a real upgrade path.

The part where my own benchmark humbled me

Then, in a second experiment, I had an agent measure the pitch itself. The control was the same task run three times without the knowledge base, recording each answer, the tokens and the time. Then the same task with the knowledge base in place.

The agent with the knowledge base cost more than the control.

Both causes were my fault. Nothing told the agent what "done" looks like when the answer already exists, so it wrote a new entry anyway. And the documented way to look things up was "read the index", which cost thousands of tokens before reaching the answer.

The fixes:

kb find "<keywords>" prints only the hits: 134 to 360 tokens instead of 2,360 to read the index
kb show <entry> prints just the useful part of one entry: 279 tokens instead of 709
a new rule, "reuse ends it": if an entry already answered the task, you refresh that entry and stop

Together, reusing an answer went from about 3,000 tokens to about 400 to 650.

And I rewrote the README. The old headline promised token savings that the measurements didn't support at small scale. Token savings still happen, but only when an answer gets read many times, or when the investigation behind it was expensive. The new headline is: your agents give the same answer twice, and can show their work.

Profiling found one more surprise: about 55ms of a 58ms lookup was just Python starting up. So now one command returns the hits and the top fix together, and the search reads a single index file instead of every entry.

My favorite bug came last. kb find returned the right entries, then added a note at the bottom: "(deep scan: no title/tag/summary match)".

An agent read "no match" as the verdict, ignored the verified entries listed right above it, and went off to re-derive the answer, which is exactly what the tool exists to prevent.

The fix was one line of text. The note now only says where the match came from.

How to give your agents a memory that holds

If you want to try this, with my tool or your own, this is what I'd keep:

Four buckets, dated entries, evidence as path:line. Explorations, solutions, errors, decisions. Every error entry records what broke and the trap that catches it next time.
Read first, write back, every time. And make "reuse ends it" explicit, or agents will write duplicates.
Make lookups cheap. A search that returns hits, and a "show me the fix" command. If finding the answer costs more than re-deriving it, agents will re-derive it.
Enforce or delete. Every rule you care about needs a check that fails. Rules agents only read will drift.
Deliver the rules every prompt. Start-of-session instructions get lost.
Give humans a window. The KB generates one offline HTML page where anyone can search entries and expand every citation into the actual source lines. No install.

And be honest about where it doesn't pay off: when the fact is one grep away, when the project is solo and short-lived, or when the code changes faster than you can re-verify entries. Stale memory is worse than none.

Chapter 2 lived mostly in the AI circle. Still heading for the middle :)

Soooo, what did this one teach me?

Looking back, almost every release came from outside feedback. A reviewer, a field report, a fresh install, an experiment, a profiler, an agent misreading my own tool.

So thank you to everyone who reviewed it, installed it fresh and broke it. It didn't always feel great in the moment, and it made the tool a lot better :)

And the direction never changed: take something that depends on an agent choosing correctly, and turn it into something that fails loudly instead.

That's the same lesson from Chapter 1, just one level up. There it was deploy traps. Here it's knowledge itself: a claim with a test attached holds up much longer than one nobody checks.

For the Super Engineer path, this was the AI side getting real for me. I think a big part of the work there is designing what the agent can't skip.

If you're feeling the AI fear right now, I believe this is a good place to start: you already know where your team's traps are. Write them down in a way an agent has to respect.

The project is open source: github.com/dcoferraz/knowledge-database. Issues and stories very welcome.

Let me know in the comments: how do your agents remember things today? Or do they?

What AI Agents Taught Me About Building Databricks Apps

Daniel Ferraz — Wed, 21 Oct 2026 21:19:53 +0000

What AI Agents Taught Me About Building Databricks Apps

The agents wrote most of the code. Everything around it was still on me, and I think that's good news

I've been building software since 2013.

Most of that time on the frontend: React, components, the part of the product people actually touch.

Then AI agents started writing code. A lot of it, and faster than me.

And I had the same question I think a lot of us have right now: if the code writes itself, what am I here for?

My answer so far is to make the job bigger. I'm working toward what I call a "Super Engineer": software engineering, data engineering and AI engineering under the same belt.

I'm sharing it while it happens, wrong turns included :)

This is one of those turns. We built a production Databricks App with AI coding agents doing a big part of the work. The code showed up fast. Getting it to run correctly took way longer.

And every one of those extra hours went into knowledge the agents didn't have.

The agent types faster than I ever will. So what do I bring to the table now?

The problem: the agent writes the code, the platform decides if it runs

Databricks Apps let you host a web app right next to your data.

Users sign in with the identity they already have, every query can run as that user, and Unity Catalog decides what each person can see. For an internal tool on top of governed tables, that's pretty close to ideal.

Ask an agent for "a React app with an Express API that edits these tables" and you'll have working code in an afternoon.

Whether it runs is a different story. That part depends on configuration: a bundle file, user API scopes, a warehouse binding and three separate permission systems.

Most mistakes there are one line long. Many fail silently, or with an error that points somewhere else -> and an agent that reads the error and searches the code will confidently fix the wrong thing.

Our app: React plus Node/Express in one container, reading and writing Unity Catalog tables through a SQL warehouse, deployed with a Databricks Asset Bundle and promoted from dev to sqa to prd through CI.

One request, five hops, and a trap waiting at almost every one of them

The walkthrough: where it broke

Wiring user identity

The design is on-behalf-of-user (OBO) authentication.

The Databricks proxy forwards the signed-in user's token in X-Forwarded-Access-Token. The API opens a SQL session with it on every request. The app's own service principal gets CAN_USE on the warehouse and nothing else.

First deploy: blank user names, empty tables.

hm.

The code was fine. The sql scope was sitting under config: in the bundle, where it's silently ignored. It belongs at the app level:

resources:
  apps:
    my_app:
      name: ${var.deploy_app_name}
      source_code_path: ./
      user_api_scopes: [sql]   # app level, not under config:

Declaring the scope only requests it. The workspace has to allow it and have user-token passthrough enabled, and the only real proof is the deployed app:

databricks apps get <app>   # effective_user_api_scopes must include "sql"

Configuring environments

We promoted to sqa and saw dev data.

A committed app.yaml was overriding the bundle's per-target settings, so every environment was running against the dev catalog. We deleted it and kept every environment value in the bundle targets.

Then a bare warehouse id made every query return 404 (the driver needs /sql/1.0/warehouses/<id>). And a catalog typo (_qa instead of _sqa) gave us empty screens and no error at all.

These days the app logs its effective catalog, schema and warehouse at boot. That one line catches this kind of bug in seconds.

Deploying

Our first deployment pulled the code from a git branch and never started: "App process did not start within 10 minutes."

A git snapshot only has tracked files, and our build output was gitignored. So we switched to the bundle and built at start:

command: ["sh", "-c", "npm ci && npm run build && node server/dist/index.js"]

Next one: the pipeline went green and the app didn't change.

With Asset Bundles, bundle deploy uploads the code and bundle run starts the new version. BUT our CI template only ran the second step for resource keys that matched its naming convention. Ours didn't.

databricks bundle deploy -t dev
databricks bundle run <app-resource-key> -t dev

The npm install had its own surprises: a corporate registry pinned in .npmrc that Databricks can't reach, an omit=optional setting that dropped native binaries, and a missing package-lock.json.

Opening it to users

Some users still saw nothing.

Three permissions have to line up: app CAN_USE (who can open the app), warehouse CAN_USE (who can run queries), and Unity Catalog grants on each table, including USE CATALOG and USE SCHEMA. Each one fails on its own, and different teams usually own them.

Our group check was also rejecting the right people. The workspace SCIM /Me endpoint leaves out account-level groups synced from the identity provider. Running SELECT is_account_group_member(:group) with the user's own token fixed it.

And the first request after a quiet period sometimes failed with "Failed to open new session" while the warehouse woke up (30 seconds to five minutes on a classic one). Nothing had run yet, so retrying with backoff was safe. We added retries and a small "warehouse is starting" banner, and moving to a serverless warehouse removed most of those errors.

What the agents couldn't learn from a wiki

Each of these failures was easy to fix and took hours to find.

The agents hit every single one. And they kept hitting them after we wrote the fixes down. They only stopped when a check failed on the mistake.

That changed how I see my own role. I think the most valuable work in this project was knowing where the traps were, and turning that into something the agent has to pass.

The solution: how you can skip the hard path

Set up four things before your agent writes its first line.

1. Ten rules in AGENTS.md

Put short rules in an AGENTS.md (or CLAUDE.md) at the repo root, where the agent reads them first:

Run every query as the user. Unity Catalog is the security boundary.
user_api_scopes: [sql] goes at the app level. Check effective_user_api_scopes after each deploy.
Three permissions must line up: app CAN_USE, warehouse CAN_USE, table grants.
Deploy through the bundle, never a git snapshot.
An app deploy is bundle deploy plus bundle run.
No app.yaml. Environment values live in bundle targets.
Normalize the warehouse path to /sql/1.0/warehouses/<id>.
Retry session-open errors with backoff, and tell users the warehouse is waking up.
Fake databases are for tests only.
Every deploy failure becomes a check.

2. A preflight check before the first pipeline

Turn each trap into a lint rule in a script that exits non-zero on errors: user_api_scopes under config:, git_source, an app.yaml with environment values, a missing lockfile, a pinned registry, omit=optional, a personal /Workspace/Users/... root path, secrets in plain env values, two targets sharing one workspace.

The script should also list what it can't check (secret bindings, scope passthrough, the three grants), so the agent tells you who owns each one.

3. A template that already follows the rules

Generating the app from a reviewed spec is way more reliable than asking an agent to remember forty rules.

Ours asks for the app name, the catalog, warehouse and groups per environment, and the entities and fields. Then it generates an app with OBO sessions, group checks, cold-start retries, auditable writes, a bundle with dev, sqa and prd targets, grant scripts for the data owner, and tests that pass on day one.

4. A fixed order for "I see no data"

Check app access, then the OBO scope, the warehouse, the Unity Catalog grants, the data, and only then the UI.

Most fixes in the first four are admin actions, so the agent should pass them to the owner and leave grants alone.

Soooo, where does this leave me?

After this setup, our deploys passed on the first run, and every "no data" report went through the same steps. The agents stopped asking "why won't it start?" and started asking "which admin owns this grant?"

This project touched all three parts of the path I'm on: a software app, on governed data, built with AI agents.

The agents did write most of the code. What made it work was understanding the platform underneath and writing that down in a way the agents had to follow.

I find that really encouraging. It's work I can learn, and I believe you can too.

If you're feeling that same fear about AI right now, I get it. I think there's still a lot of room for us. The job just looks a bit different.

Huge thanks to the team I built this with - you know who you are :)

Let me know in the comments: what are you learning next? Or which Databricks trap got you the worst?

PS: everything from this article lives in bricksmith, the skill I built so my agents stop re-learning these traps. If it saves you a deploy or two, a ⭐ helps a lot :)
github.com/dcoferraz/bricksmith