Our autonomous agent has been running a small publishing business for three months: it posts, replies, follows, publishes articles, and tracks every decision it makes. The state layer behind all of that is not Postgres, not SQLite, not Redis. It is a directory of JSONL files committed to git.
This choice gets us laughed at occasionally, so this post is the honest case for it — the patterns that make append-only text files survive crashes, retries, concurrent writers, and an LLM's enthusiasm for re-running things it already ran.
Why files-in-git at all
Three properties turned out to matter more than query power:
-
Every state change is a diff. When the agent follows someone, replies to a thread, or publishes an article, the evidence lands in
git logwith a timestamp and an author. Auditing an autonomous system is the hard part of running one; with ledgers in git, the audit trail is the storage engine. - Scheduled jobs and interactive sessions share state with no server. Our GitHub Actions jobs check out the repo, read the ledgers, act, commit. The interactive session pulls before deciding anything. The merge boundary is git's problem, which is a well-understood problem.
-
The LLM can read its own state natively. An agent that can
grepits full decision history is meaningfully smarter than one that needs a query layer written for it.
Pattern 1: append-only, with one exception
Almost every ledger is append-only: one JSON object per line, new facts go at the end. Append-only means a crashed write corrupts at most the final line, and recovery is "drop the broken tail," not "restore from backup."
The exception: consumption ledgers (a stock of pre-written posts, a queue of follow candidates) need a consumedAt stamp on existing rows. For those we load-modify-rewrite the whole file — acceptable because the files are small — with one hard rule: a consumed mark is never overwritten. The update function refuses to touch a row whose consumedAt is already set. Retry-safety comes from that refusal, not from hoping the caller behaves.
Pattern 2: idempotency keys from the outside world
Every ledger row that mirrors an external event carries the external system's own identifier — the post URI, the article ID, the comment permalink. Ingestion dedupes on that key, so fetching the same feedback twice records it once. This is what makes "the cron fired twice" and "the agent re-ran the command after a timeout" non-events.
The corollary: never let the LLM hand-type an identifier. Every DID, URI, and ID in an input file is copied mechanically from a previous command's output. We learned this after one hand-typed identifier — a single wrong character in a DID — created a follow record pointing at an account that does not exist. The API accepted it because the string was syntactically valid, there is no unfollow in our pipeline, and the row is in the history forever, because ledgers don't forget.
Pattern 3: two-phase validation, side effects last
Commands that act on the world validate the entire batch before performing any of it. If one entry in a reply batch is malformed, the whole batch throws before the first reply is sent. A half-executed batch is the worst state an autonomous system can be in — the ledger says one thing, the world says another — so we simply never create it.
Pattern 4: the ledger is the gate
The best part of state-in-repo: your test suite can read production state. Our commit gate includes tests that load the real ledgers and assert invariants — every stocked post is under the platform's length limit, no stocked article's title collides with a published one, no open TODO item is older than its grace period. Corrupt or contradictory state cannot be committed, because the tests that guard it run on every commit. State bugs get caught at write time by CI, not at 3 a.m. by the scheduled job that tried to consume the bad row.
Where it genuinely hurts
Fairness section. You give up: cross-file transactions (we scope every command to one ledger write where possible), concurrent writers on the same file (git rebase handles cross-job races; two writers in the same working tree need coordination — we've hit this and had to serialize by agreement), and any query fancier than a linear scan (fine at our scale: our largest ledger is under a thousand lines, and all of them together are under four thousand).
If your agent handles thousands of events an hour, use a database. Ours handles dozens of decisions a day that we need to trust and audit years later. For that shape of problem, a pile of JSONL files under git has been the most boring — and therefore best — infrastructure decision we made.
The agent described here runs Rulestack, and its config patterns are what we package and sell.
Day-to-day operational notes: @ai-shop.bsky.social on Bluesky.
Top comments (1)
This is a good fit for low-volume, review-heavy work, but the hardest gap is the one between an external side effect and the ledger append. “Side effects last” still leaves: publish succeeds, process crashes before recording the URI, retry publishes again. I’d give every logical operation a stable idempotency key, write a durable
preparedrecord before dispatch, pass that key to the provider when possible, then reconcile indeterminate operations against provider state before retrying. Also, git rebase detects text conflicts; it does not prove two non-conflicting appends preserve a cross-record invariant. A single-writer lease plus generation/hash preconditions and a replayable reducer would make the concurrency contract explicit.