Agent Memory Is Just Files: Building a Memoryfield Your AI Agent Can Actually Use

Every agent I run has the same flaw, and it took me embarrassingly long to name it. It forgets. Not in the poetic sense. In the operational sense: my article pipeline runs three times a day, and every single run re-learns that the file has to be pushed to git before it will publish. The lesson exists. I have written it down in at least four places. The agent just never has it in context at the moment it matters.

Last year's answer to this was to bolt on a memory system. That usually meant one of two things: a vendor's built-in memory that mines your chats, or a serious piece of infrastructure. I have seen setups that need pgvector, a graph database, and a second LLM whose only job is deciding what is worth remembering. I run everything on a single rented VPS. None of that was ever going to survive contact with my server budget.

So when Cal Paterson's post Agent memory as a file format hit the Hacker News front page with 174 points this week, it read less like an idea and more like permission. His argument: the three popular kinds of agent memory all fail for the same reason. They treat memory as a process, when memory works better as data.

One disclosure before I go further. I had read about this pattern but never built it, so I built the smallest working version on my own server before writing this. Every command and every result below is from that run. The one part I faked is the embedding model, and I will flag exactly where and why.

What a memoryfield actually is

The whole idea fits in one directory listing. A memoryfield is a flat folder of Markdown pages, each with YAML frontmatter, plus one optional SQLite file that indexes them for search. Packed for sharing, it is a zip:

my-memories.memoryfield.zip
├── vps-renewal-playbook.md
├── agent-deploy-quirks.md
├── sqlite-gotchas.md
└── nomic-embed-text-v1.5.sqlite3

That is it. No daemon, no API server, no background process re-summarizing your conversations at 3 a.m. The spec, a draft v0.1 from August 2026, pins down the few rules that matter:

• Pages are UTF-8 Markdown with a .md extension. Prose, written by the agent, in the format agents write best.
• Frontmatter fields: title, uuid, created, updated should be present; summary is optional. The one hard number in the spec is the page size: a page should stay under 8,192 bytes, roughly 1,300 words, so a page always fits in an embedding.
• The layout stays flat. Pages cannot be nested in subdirectories. Non-Markdown files may ride along but are never indexed.
• The vector index is disposable. Filenames start with the embedding model's name, nomic-embed-text-v1.5.sqlite3 being the recommended default. Delete it and regenerate it from the Markdown at any time. The Markdown is the canonical data.

Paterson's framing of the frontmatter is that it is for humans skimming the folder. The agent gets the whole page anyway. It is the opposite of a schema-first design, and that is deliberate.

Why the graph-walking approach lost

The interesting part of the post is the post-mortem on the alternative. The obvious "smart" design, inspired by Karpathy's interconnected Obsidian-style wikis for agents, is a knowledge graph the agent walks link by link. Paterson tried it and the failure mode is concrete enough to quote:

• It is slow. If the answer is N links deep, the agent needs N+1 tool calls, each one a round trip through a model that bills by the token and pauses two to three seconds per call. My own agent's read tool behaves exactly this way, and I can confirm the cost: a multi-hop lookup through my session notes is visibly slower than a single search.
• It is unreliable. The agent judges relevance by link text and page titles, not page content. Relevant material that happens to have an unhelpful title never gets found. Paterson calls the incentive here "1990s-SEO-style page metadata hacking," which is exactly what it is.
• It pollutes context. Every hop drags the front page and a pile of irrelevant pages through the context window, and the agent comes back fixated on noise.

The memoryfield answer is to skip walking entirely: one semantic search call jumps straight to every relevant page, then the agent reads them all in parallel. Worst case, two tool calls, regardless of how deep the knowledge sits.

The four design decisions, translated

Prose, not chunks. RAG pipelines exist because legacy documents are hostile: 200-page PDFs that must be chunked, embedded, re-ranked, and hybrid-searched. But a memory is written by the agent itself, at the moment of learning, by a system that is fluent in Markdown. Chunks and double-summarization solve a problem memories do not have.

A semantic jump, not graph walking. Covered above, and it is the decision I was most skeptical of. My instinct was that keyword search would be enough at small scale. The spec's answer is that the index is a cache either way; start with nothing, add search when the folder grows past a hundred pages or so.

More model, less mechanism. A big custom API for memory means loading an interface maze into context. A file format means the agent uses whatever access pattern it already knows from training: grep, perl one-liners, even dropping inline CSV into a page and querying it with SQLite. Paterson reports seeing both in the wild. As models improve, they use the same boring files more cleverly. A fixed pipeline does not get that upgrade for free.

Open and transport-invariant. The canonical format is a zip, but the spec explicitly allows serving from a directory, S3, git, or plain HTTP. Paterson syncs his own fields with Syncthing and shares others over S3. The point is escape velocity from any one vendor's memory API.

Building the smallest working version

You do not need the official tooling to feel the shape of this. To make sure I understood the mechanics before recommending them, I wrote a minimal version in about sixty lines of Python: three real memories from my own operations, one SQLite table with an FTS5 full-text index, and a placeholder for the embedding.

A quick honesty note on that placeholder. The spec recommends nomic-embed-text-v1.5, a 270 MB embedding model that runs fine without a GPU. I did not want to install Ollama on the box mid-article, so for the demo I substituted a hashed-bag-of-words vector of fixed size. It is NOT a real embedding, and I did not use it for retrieval. The searches you will see run on SQLite's full-text search, which is genuinely part of the spec's spirit: the index is a convenience, not the canonical data. For your real field, install the real model.

The three pages, abridged from what I actually run:

---
title: "VPS renewal playbook"
created: '2026-08-14T09:00:00Z'
summary: What I check before renewing the rented VPS that runs my agents
---

The VPS bill lands on the 14th. Before renewing: check disk usage on /var
(images and logs are usually the culprit), confirm the backup cron actually
copied files in the last 24h, and re-run the restore drill.

The table schema is the whole storage layer:

CREATE TABLE pages(
  path TEXT PRIMARY KEY,
  title TEXT,
  created TEXT,
  summary TEXT,
  body TEXT,
  embedding BLOB        -- the stand-in vector, 16 floats
);

And the FTS5 index that makes search one tool call:

CREATE VIRTUAL TABLE pages_fts USING fts5(
  path UNINDEXED, title, summary, body);

Load the pages, then ask real questions. First one: where did I write about backups?

== search 'backup' (FTS) ==
  vps-renewal-playbook.md

Second: which page holds the deploy timing rules? This is the exact memory my article pipeline keeps re-learning, and the search surfaces it in one call:

== search 'deploy slot' (FTS) ==
  agent-deploy-quirks.md

Finally, the packing step, which is the transport story in one command:

== archive ==
  vps-renewal-playbook.md
  agent-deploy-quirks.md
  sqlite-gotchas.md
packaged OK

Three pages, two searches, one zip. Total runtime, well under a second. Total infrastructure: the Python standard library. That is the entire sales pitch, demonstrated rather than asserted.

What I would do differently on a real deployment

The demo skips four things that matter in production, so here is the honest punch list:

• Use a real embedding model. The demo's FTS search only matches literal keywords; "restore drill" would not surface the page that says "backup cron." A real vector index catches the meaning match, which is the entire reason the spec includes one. ollama pull nomic-embed-text and you are running the spec's recommended default.
• Add a save-time rule to your agent. The memory only exists if the agent writes it. The pattern I am adopting: after any session where I corrected the agent, it updates the relevant page's updated field and appends the lesson. No pipeline. A standing instruction in the agent's config file.
• Pin anything you did not write. A memoryfield you downloaded is untrusted input to your agent. The spec's zip format exists partly so you can sha256sum it and review pages before they ever reach a context window. Remember: there is still no reliable way for an agent to tell a good prompt from a malicious one, and a memory file is a prompt.
• Prune on a schedule. Irrelevant memories do not hurt retrieval, they just take space. But pages do go stale. My rule of thumb: the updated field is older than the interval at which the underlying fact changes, it gets re-verified next time it is surfaced.

And one honest gap in the format itself: there is no locking or merge story. Two agents writing to the same field over Syncthing will eventually clobber a page, because pages are replaced whole. For a solo operator like me that is a non-issue. For a team, git is the obvious transport precisely because it brings the merge semantics with it.

Why this landed for me

I have spent two years assuming that "agent memory" meant adopting somebody's platform. It never once occurred to me that the correct unit was a file format, and that the pipeline I thought I needed was the product being sold to me. Paterson's version of the Mythical Man Month quote makes the argument better than I can: show me your tables and I will not need your flowcharts. My agents do not need a memory pipeline. They need a folder of Markdown that travels with them, and a standing instruction to keep it honest.

The 8 KB page limit turned out to be my favorite part. It is a constraint that forces memories to be written the way good notes are written: one topic, dense, self-contained. When a page wants to grow past that, the answer is another page, and the search index handles finding it.

I write about AI infrastructure, agents, and the unglamorous engineering that makes them reliable every week. Subscribe, it is free.

Now you: is your agent's memory a file format, a vendor feature, or just vibes? Have you tried memoryfields or a Karpathy-style wiki with your agents? Tell me what worked, I am genuinely deciding how much of this to adopt.

Sources

• Agent memory as a file format - Cal Paterson's original post
• The Hacker News discussion - 174 points, 89 comments at time of writing
• The memoryfield spec (draft v0.1)
• memoryfield-tool - the official CLI, also on PyPI as memoryfield-tool
• Karpathy's wiki gist - the graph-walking prior art the post argues against