DEV Community: Aalishan

Long AI Chats Drift. I Built a Harness on Claude Code Instead

Aalishan — Sun, 15 Nov 2026 02:50:38 +0000

On the first day of a multi-day piece of work, the assistant knows every rule I gave it. By day three the thread is long, and the rules from that first morning sit somewhere in the middle of it. The middle of a long context is a bad place for anything you need a model to remember.

So I stopped treating the chat as the memory. I built cockpit, a local-first command system that runs inside Claude Code, and published a generalized copy as opencockpit (MIT license, set up as a template repository). Every claim about how the system works can be checked against that repo; the numbers and settings under "Lessons from daily use" come from my own copy.

Why long threads and naive retrieval fall apart

Two published results match what I kept running into. Lost in the Middle (Liu et al., TACL) found that models do best when the relevant information sits at the beginning or end of the input, and that performance degrades significantly when it sits in the middle of a long context, even for models built for long contexts. Chroma's Context Rot report tested 18 models and found performance consistently degrading as input length grows, even on simple tasks.

A multi-day thread is a long context that keeps growing, and the decisions that matter most are usually the old ones in the middle.

Naive retrieval doesn't fix this. Similarity search returns what looks like the query, not what is current. A decision and the later decision that replaced it look almost the same to an embedding, so retrieval can hand back the stale one with full confidence.

There is a quieter failure too: ledger bloat. If every change is appended to a log the model has to read, the log grows with every session, and the model has to work out which of many entries still holds. That is the same long-context problem, rebuilt by hand.

What cockpit does instead

Four habits, all visible in the repo.

Current state lives in small files, overwritten in place. RUNWAY, the domain that ships my website, keeps its state in data/runway/state.json, a single file the script rewrites whole:

def save_state(st): STATE.write_text(json.dumps(st, indent=1, ensure_ascii=False) + "\n", encoding="utf-8", newline="\n")

History lives in git, not in the context window. Every command that changes files ends with one commit and a push. When I need to know what changed last week, I read a git log, not an old thread.

Where I keep an append-only log, a script can fold it. The ideas backlog is an event log on purpose, because I want every version of an idea. The model doesn't read that raw log; scripts/ideas.py derives the current state from it. The format, from scripts/schemas.md:

{"ts": "...", "idea_id": "i-0001", "event": "edit", "data": {"set": {"gist": ""}, "note": ""}}
{"ts": "...", "idea_id": "i-0001", "event": "status", "data": {"from": "new", "to": "developing", "conversation": "<folder>", "urls": [], "note": ""}}

Skills stay small and load on demand. Each domain is one skill file. Claude Code keeps a short description of each skill in context and loads the full instructions only when a command needs them.

State contracts: the files the model must obey

There is no single spec file. The contract is spread over a few files, each with one job:

CLAUDE.md: the router, the folder contract, the global rules and the git rules.
COMMANDS.md: every command name. A skill change updates it in the same commit.
.claude/skills/<domain>/SKILL.md: each command step by step, including how it ends and what it commits.
scripts/schemas.md: the format of every data file.
State files such as data/runway/state.json and data/fourthwall/state.json.
The folder contract: inbox/ is the drop zone, raw/ keeps originals and is never deleted, data/ holds structured truth that only skills write, and out/ holds renders a script can always regenerate.

The rule I lean on most is the fourth one in CLAUDE.md:

4. Code does exact work. Arithmetic, dates, sorting, deadlines, hashes, rendering, and validation are done by scripts/. Never compute these in your head. Run the script and report its output.

Every command takes the same path:

you: chat, files, phone photos
 -> CLAUDE.md      routes the request, applies the global rules
 -> a skill        the command's steps, from start to its end condition
 -> scripts/       every number, date, hash, render and check
 -> data/          structured records, validated after every edit
 -> git            one commit and push per command

The model makes the judgment calls. The harness does everything that must be exact.

Domains and boundaries

Every domain except HELP has its own skill, its own folder under data/, its own section in scripts/schemas.md, and a script for its exact work. Among them:

EXTROVERT interviews me about an idea and edits the transcript into posts in my own voice. It also keeps the ideas backlog and versioned profile copy.
RUNWAY ships content and code to my own site repo: a preview branch first, live only after I approve it, then live tests against production.
FOURTHWALL documents the system itself and exports the public copy.
CHASSIS is an example domain: vehicle maintenance from the owner's manual and service slips, tracked against odometer readings.
HELP prints the command sheet.

The boundaries that matter most are enforced by code, not by the model's good intentions. A Claude Code hook runs the validator after every edit, from .claude/settings.json:

"PostToolUse": [
  {
    "matcher": "Edit|Write|MultiEdit",
    "hooks": [
      { "type": "command", "command": "python \"$CLAUDE_PROJECT_DIR\"/scripts/validate.py --hook" }
    ]
  }
]

A disclosure lint per audience (internal, resume, public) gates every resume render and every site publish, and a voice lint (banned phrases, emoji, British spellings, em dashes) runs on every post before I see it. FOURTHWALL's export runs five numbered leak checks before a public copy can ship, and any finding blocks it. The last two, from scripts/fourthwall.py:

report["4 deny list"] = deny
report["5 em dashes"] = dash

Posts that make claims about my work can go to a claim checker before they go out: a read-only subagent with only the Read, Grep and Glob tools that labels every claim supported, stretched or unsupported against my own records.

To be precise about one word: this is not a multi-agent system. CLAUDE.md says that "agent" here means a division of duties, not a separate process. There is one Claude Code session plus one read-only subagent. There is no telemetry either. The record is git.

Lessons from daily use

Latency comes from the model, not the harness. The validator runs after every single edit, and on my own data it takes about 0.4 seconds (365 ms, the median of five runs, Python startup included). I never notice it. What costs time is the model reading, so the cheapest speedup is giving it less to read.

Keep the context small on purpose. Scripts print short summaries and the model reports them. A status command prints a few lines a script computed, not the files behind them. A skill loads only when its command runs.

Deterministic checks beat careful prompting. opencockpit ships 66 synthetic tests in one file. By default, every RUNWAY deploy waits for production and then runs the site's live tests against it, and a test that passed only on its retry still counts as passed, but the report names it as flaky. When a rule matters, it becomes a check that fails loudly.

The human stays at the gates that can't be undone. Nothing reaches my live site until I approve its preview. By default, a public copy of cockpit ships only after I've seen the diff and the leak report and said yes. I've since recorded a standing rule in my own copy (auto_publish): when an export is clean and its own tests pass, Claude approves and publishes it without asking and tells me afterwards. The publish script still refuses anything but a clean, approved export. And the permission rules deny the plain forms of force push, hard reset and global config change.

Try it

Create your own private repo from the template, then:

git clone <your private repo> cockpit
cd cockpit
pip install -r requirements.txt
python tests/test_suite.py
claude

It ships with empty data and a fictional example user. Copy examples/persona/data/ into data/ and run /cockpit to try every command before adding your own records. The whole thing is about 4,000 lines of Python, scripts and tests together, small enough to read before you trust it.

The repo is github.com/aalishan001/opencockpit. If you build a domain of your own, I'd like to see it.

I Took a Memory Leak Arena to Google Ananta. It Got Picked

Aalishan — Sun, 15 Nov 2026 02:48:49 +0000

A pool of developers waiting for registration to pick up the esteemed, albeit temporary, ID tags, the ceiling about as high as the ambitions in the room. Possibly the only place I felt like I belonged, and had yearned for, for so long.

That was Saturday, 24 October 2026, at Google Ananta in Bengaluru, for FORK. BUILD. MERGE., hosted by GDG Cloud Bengaluru. I brought two open-source tools aimed at one problem: AI agents that share long-term memory, and leak it to each other.

The problem, in one hotel

Picture a hotel where every room key opens every door, and the only thing keeping you out of the next room is a polite sign on it. That is how a lot of agent memory works today. Every user's memories sit in one shared vector table, and the boundary between them is a WHERE clause the application has to remember to write, on every query, in every code path, for as long as the product lives.

Similarity search doesn't read the sign. It ranks by distance, not by owner, so a perfectly innocent question about your own roadmap can pull up a colleague's private note that happens to sound like it, and no attacker is needed for that to happen. The MemLeak paper measured this kind of leak in multi-tenant agent memory with no attacker involved at all. Add the ordinary bugs on top: a filter someone forgot, a user id read from the prompt instead of the session, a sub-agent reading memos above its clearance.

My position is that the boundary belongs in the database, not in the app. To be fair, an application filter is simple, and it is right almost every time. Almost is the problem, because the one query that forgets it is the one that leaks.

What I brought

adk-memshield is a drop-in memory service for Google's Agent Development Kit (ADK) that moves the ownership check into PostgreSQL. It uses FORCE ROW LEVEL SECURITY, a database role that cannot bypass it, and a per-transaction identity: the user from the ADK session, the tenant and clearance from the service's own config. The design goal: a query with a missing or wrong filter returns fewer rows, not someone else's rows. It is on PyPI as adk-memshield.

adk-leak-arena is the red-team harness that tries to break it. It runs two real ADK Runners side by side on the same seeded data, one on a plain pgvector table guarded by application filters and one on adk-memshield, attacks both with five cross-boundary scenarios that go from incidental overlap all the way to a query with no WHERE clause at all, plus one same-tier boundary scenario, and shows the rows each side returned with the PostgreSQL telemetry underneath, so you can watch the boundary hold or fail query by query. Its GitHub repo dates from event day, and it reached PyPI the day after, as adk-leak-arena.

What the numbers say, and how far they go

The arena's benchmark has six scenarios plus a benign control, 54 queries in all. Forty of them are cross-boundary attacks, eight in each of the scenarios S1 to S5, asked as one analyst with clearance 1, top three results per query.

Unprotected pgvector: 39 of the 40 attack queries returned at least one row the caller is not allowed to read (97.5%).
adk-memshield: 0 of 40.
The legitimate answer, where the caller had one, made the top three in 19 of 22 queries on the unprotected table and in 22 of 22 with adk-memshield. Leaked rows pushed it out once, and a spoofed owner filter returned only the victim's rows twice.

A rerun with gemma3:4b through Ollama gave the same leak and recall numbers. Good. They are measured on what the search returns, not on what the model says.

At 2,000 rows, the query plan shows where the work happens. RLS filters 2,000 rows down to the 45 the caller may read before any distance is computed; with fresh statistics, the scope index narrows to the tenant's 978 rows first.

Now the limits, because a security number without its limits is just marketing. The dataset is synthetic and hand-written, 36 rows across three tenants, which means these percentages show you failure modes you can reproduce on your own machine and argue with, not real-world leak rates. One scenario is a known boundary, too. If someone at your own clearance writes a poisoned note into shared memory you are entitled to read, RLS returns it, because the read is authorized. Row-level security decides who may read a row, not whether the row is true, and that part needs write approval or moderation in the application.

Every number above has its command, commit and raw output in the arena's BENCHMARKS.md. Both test suites pass on those commits: 8 integration tests for adk-memshield, 45 end-to-end tests for the arena.

The bug that looked like a pass

Two days before the event, an adversarial review I'd set my coding agents to run threw malformed requests at the arena's API. One of them hid a NUL byte inside a spoofed user id, in the scenario where the unprotected side takes the user id straight from a tool argument. It came back HTTP 200, with zero rows.

Zero rows is a plausible answer for a mangled user id, so on its face nothing looked wrong. Why was it a fake pass, then? Because the telemetry under the pane was empty and the container log held an exception: three reasonable layers had lined up. The database driver, psycopg, refused the parameter before it ever reached PostgreSQL, since text fields cannot hold a NUL byte. ADK's PreloadMemoryTool caught that exception, logged a warning and let the turn continue with no memory. The arena then classified an empty result and counted zero leaked rows. A crash, reported as a security pass.

The arena now reports that case as a retrieval error instead of a verdict, and rejects NUL bytes at the API before they get near SQL. Upstream, I opened google/adk-python#7723. It adds a raise_on_error option and an on_error callback to PreloadMemoryTool, so an application can fail closed, or at least find out. The defaults keep today's behavior, and the PR is open, waiting for review. One honest gap remains in my own tool: its leak percentages still count an errored query as not leaking, and the CLI and Markdown tables don't show the separate error count (the JSON report and the web console do). Every published run had zero errors.

The people

Krishna Mehta hosted and led the organizing. Pratik Singh, one of the judges, from GitLab, watched the leak-arena demo with good humor and stories from his own work, and we swapped notes on the week's CVE news.

The result

At the end, the organizers announced 10 projects, in no particular order, with goodie bags. Mine was one of them: one of 10 projects picked, out of 211 PRs raised that day (by my count).

adk-leak-arena had zero GitHub stars when I checked this morning, so for now this post is its entire marketing department.

Try it

If you run agents that share memory, add your own retrieval pattern to the arena as a target and attack it before someone else does.

adk-memshield: GitHub and PyPI
adk-leak-arena: GitHub and PyPI
The upstream PR: google/adk-python#7723
The deep-dive on why the gate belongs in PostgreSQL: Shared Agent Memory Leaks. Enforce the Boundary in PostgreSQL

Shared Agent Memory Leaks. Enforce the Boundary in PostgreSQL

Aalishan — Sun, 15 Nov 2026 02:42:09 +0000

Agents with long-term memory often keep it in one vector store shared by many users, and often by many agents. Ownership is a metadata column, and the boundary is a WHERE clause in application code. This post argues that the boundary belongs in the database instead. Why the database? Because it is the one layer every query has to pass through, including the ones nobody remembers writing. The post shows the PostgreSQL that does it, and the two open-source tools I built around it.

What the research measured

MemLeak (Mudgal et al., October 2026) studies multi-tenant personal agents that share one memory store. Under pooled same-team retrieval, incidental leakage reached 70–100% with no attacker involved. Adversarially crafted memories reached 90–100% top-k placement. Of three mitigations it tested (metadata filtering, ownership-aware embeddings, post-retrieval ownership gating), only the hard ownership gate, which drops every hit not owned by the querying user, consistently restored the clean baseline. It added about 1.4 ms of mean latency per query (12.4 ms to 13.8 ms, measured against a 10,000-memory pgvector HNSW index). The authors flag the scale themselves: 10 queries per condition on synthetic fixtures, so read the rates as a failure mode, not prevalence.

AgentLeak (El Yagoubi et al.) looks inside multi-agent systems. It defines seven leakage channels, including C1 (final output), C2 (inter-agent messages) and C5 (shared memory), and runs 1,000 scenarios across five production LLMs. In multi-agent mode, final-output leakage was 27.2%, but 68.9% of traces leaked through at least one of C1, C2 or C5: a 41.7-point gap that an audit of final outputs alone would miss. The paper reports that none of the four frameworks it evaluated offers a way to intercept inter-agent messages.

MemLeak's fix is an ownership gate, and my whole argument is about where that gate should run, because a gate that lives in application code only guards the code paths that remember to call it.

Threat 1: cross-user semantic bleed

Similarity search ranks by distance, not by owner. Three ordinary paths turn that into a leak:

Filters forgotten on secondary endpoints. The main search path filters by user. The "related memories" endpoint, the export job or the debug tool written later does not. In adk-leak-arena's incidental-overlap scenario, where the filter checks the tenant but not the user, leaked rows pushed the caller's own answer out of the top 3 on 1 of the 5 queries that had one.
Prompt-injected tool parameters. If a memory tool accepts a target_user_id and the model fills it in, injected text can make it target_user_id='cfo'. The WHERE clause is correct; the value is the attacker's.
Planted lures. A co-tenant writes private memories shaped like a victim's questions. In the arena's lure scenario (8 queries, top 3, synthetic data), lures that mirror the victim's question, which assumes an attacker who knows what will be asked, took rank 1 on 3 of 8 queries against a filter on tenant and clearance that skips the owner check.

Threat 2: lateral movement between agents

Multi-agent frameworks often share state freely. AgentLeak describes inter-agent messages passed along unfiltered and shared state visible to every agent. OpenAI's Swarm (experimental and educational by its own README, now replaced by the OpenAI Agents SDK) documents that on a handoff "the system prompt will change, but the chat history will not". The next agent inherits the whole conversation, and no access control between agents is documented.

That makes one compromised low-clearance agent dangerous in two directions. It can read up, pulling a clearance-3 memo out of shared memory. It can also write poison up, planting a memory labeled clearance 3 that higher-clearance agents will trust.

The classic models cover both. Bell-LaPadula's simple security property is no read up. Its write rule, no write down, protects confidentiality, which is a different goal. Poison written upward is an integrity problem, and the rule for that is Biba's: no write up. A memory table needs both.

The fix: make PostgreSQL the gate

Four pieces:

The memory service connects as a NOSUPERUSER NOBYPASSRLS role that does not own the table.
The table has ENABLE and FORCE ROW LEVEL SECURITY, so even its owner is subject to the policies.
Identity is bound per transaction with set_config(name, value, true). The true makes the value local to the transaction, so a pooled connection can't carry one user's identity into the next request.
Policies encode the rules: no read up on SELECT, no write up on INSERT. UPDATE and DELETE need policies of their own; without one, RLS lets them touch no rows.

ALTER TABLE memories ENABLE ROW LEVEL SECURITY;
ALTER TABLE memories FORCE ROW LEVEL SECURITY;

-- Read: your own rows, plus shared rows at or below your clearance (no read up).
CREATE POLICY mem_select ON memories FOR SELECT
USING (
  tenant_id = current_setting('app.tenant_id', true)
  AND (
    user_id = current_setting('app.user_id', true)
    OR (scope = 'tenant_shared'
        AND min_clearance <= NULLIF(current_setting('app.clearance', true), '')::int)
  )
);

-- Write: only as yourself, and shared rows only at or below your clearance (no write up).
CREATE POLICY mem_insert ON memories FOR INSERT
WITH CHECK (
  tenant_id = current_setting('app.tenant_id', true)
  AND user_id = current_setting('app.user_id', true)
  AND (scope = 'private'
       OR min_clearance <= NULLIF(current_setting('app.clearance', true), '')::int)
);

NULLIF is there because once a custom setting has been used on a connection, PostgreSQL reads it back as an empty string, not NULL. For the same reason, the identity columns carry CHECK (tenant_id <> '') and CHECK (user_id <> ''), so an empty setting can't match an empty owner. This is a minimal standalone demo; adk-memshield uses the same pattern with its own setting and role names, plus an app_name scope and owner-only UPDATE and DELETE policies.

The test that matters: a vector search with no WHERE clause, run as alice at clearance 1. Without RLS, the top 3 would be alice's note, another tenant's identical row and the CFO's private note; a clearance-3 board memo is just as close.

BEGIN;
SELECT set_config('app.tenant_id', 'acme', true),
       set_config('app.user_id', 'alice', true),
       set_config('app.clearance', '1', true);
SELECT user_id, scope, min_clearance, content
FROM memories
ORDER BY embedding <=> '[0.9, 0.1, 0.0]'
LIMIT 3;
COMMIT;

-- result (set_config's own row omitted):
 user_id |     scope     | min_clearance |          content
---------+---------------+---------------+---------------------------
 alice   | private       |             1 | alice: my 1:1 notes
 alice   | private       |             1 | alice: draft of my review
 hr      | tenant_shared |             1 | hr: holiday calendar

With the injected argument (WHERE user_id = 'cfo'), alice gets zero rows. With no identity bound, the query sees zero rows. An attempt to write a shared clearance-3 memory as alice, or a row owned by the CFO, fails with new row violates row-level security policy for table "memories".

A query with a missing or wrong filter should return fewer rows, never someone else's rows.

This is MemLeak's ownership gate, plus a clearance-labeled shared scope, moved out of application code and into the database. Anything that connects as the app role gets the same gate, including the forgotten endpoint and next year's debug script.

Where the trust boundary really is

All of this holds only if trusted middleware sets the identity from the authenticated session, never from a tool argument, and the agent never gets raw SQL. agent_app can call set_config itself. In the same run, a transaction that claimed to be the CFO at clearance 3 read the CFO's rows without complaint.

authenticated session (user, tenant, clearance)
        |
        v
trusted middleware      binds identity with set_config(..., true), per transaction
        |
        v
memory service          role agent_app: NOSUPERUSER NOBYPASSRLS, not the table owner
        |               (agent tools pass parameters, never SQL)
        v
PostgreSQL              FORCE ROW LEVEL SECURITY decides every row

Know the bypasses too: superusers and roles with BYPASSRLS; the table owner, unless the table has FORCE ROW LEVEL SECURITY; SECURITY DEFINER functions, which run with their owner's rights; and views without security_invoker, which also run with their owner's rights. In the demo, a view created by a superuser showed alice all 6 rows. The same view with security_invoker = true showed her 3.

Planner and performance

From adk-leak-arena's published runs, with the scale stated. Every command, commit and raw output is in its BENCHMARKS.md.

36 rows, 40 cross-boundary attack queries, top 3: the unprotected table returned at least one row the caller may not read on 39 of 40 queries; adk-memshield on 0 of 40. Latency was measured on Docker Desktop and is noisy, so no latency number from these runs is a claim.
2,000 rows per table: RLS filters 2,000 rows down to the 45 the caller may read before any distance is computed; with fresh statistics, the scope index narrows to the tenant's 978 rows first. Straight after the arena's bulk load, before ANALYZE, the same search is a sequential scan with the RLS filter, and it still computes distances for those 45 rows only. The unprotected search computes them for all 2,000 either way.

With no vector index, the policy predicate is just another filter, applied before the sort that computes distances, and with fresh statistics the planner can serve the tenant part of it from a B-tree index.

At large scale with an approximate index, this changes, and I haven't benchmarked it yet, so no numbers here. The pgvector README says that with approximate indexes, filtering is applied after the index scan, and an HNSW scan returns at most hnsw.ef_search candidates (40 by default). RLS is such a filter, so a query can return fewer than k rows. The answers are pgvector's iterative index scans (0.8.0 and later, SET hnsw.iterative_scan = strict_order or relaxed_order, bounded by hnsw.max_scan_tuples), or partitioning by tenant, so each partition's index holds only that tenant's rows and the post-index filter discards far fewer. Either way, measure recall at your own scale.

Honest limits

RLS stops cross-boundary reads and writes. It does nothing about same-tier poisoning: a clearance-1 writer can still plant a lure in a clearance-1 shared scope, and the demo accepts that insert. adk-leak-arena reports this as a boundary scenario of its own, outside the attack numbers. It needs content-level defenses, such as provenance on shared memories and review or quarantine before a shared write is trusted.

It also can't see memory-preload tools that swallow exceptions. Google ADK's preload memory tool catches any exception from the memory search, logs a warning and lets the turn continue without memory. Nothing leaks, but a broken identity binding looks like an agent with no memory instead of an error. I learned this the embarrassing way, when my own harness reported a crash as a clean security pass. I've since opened google/adk-python#7723, which would let an application fail closed (raise_on_error) or get a callback (on_error); the defaults keep today's behavior, and the PR is open and waiting for review. Until then, alert on that warning, but know that a missing identity returns zero rows with no error, so it logs nothing. Test the binding itself.

What the ecosystem needs

The failure modes are now measured. What the ecosystem needs is reproducible red-team harnesses and memory backends where the database enforces ownership, for frameworks like Google ADK and for local Gemma deployments. I've open-sourced one of each: adk-memshield, a drop-in ADK memory service built on the same pattern as the policies above, and adk-leak-arena, which attacks a plain pgvector table and adk-memshield side by side in real ADK Runners. A rerun with gemma3:4b through Ollama gave the same leak and recall numbers as the scripted agent. Both are on PyPI, and numbers at approximate-index scale come next.``