My agent system dreams at night, and that is where its memory comes from

Originally published at olund.dev.

My agents recall durable facts across sessions. None of those facts got
there because an agent, mid-task, decided "this is worth remembering
forever." Almost
everything an agent thinks is memorable in the moment is noise a week
later, and an agent given a direct write path to durable memory will fill
it with confident junk.

Instead, my system does what brains do: it consolidates offline. Twice a
day, on a timer, a process wakes up, reads what happened since it last ran,
and decides - slowly, with gates - what deserves to survive. I call it
dreaming, and the name has turned out to be more than a joke: the design
questions are genuinely sleep-shaped. What gets replayed? What gets
promoted to long-term storage? What gets discarded? And what happens when
the process misfires?

The shape: a timer, not a daemon

Dreaming is a one-shot process fired by a scheduler - a morning cycle for
lighter work (recommendations, "what needs attention"), an evening cycle
for the heavy memory-promotion pass. No resident daemon, no queue service.
Each run is a fresh process that reads files, thinks, writes files, and
exits. Everything in my stack is filesystem-first, and consolidation is no
exception: if the machine is off, the cycle is skipped and the next one
picks up the unprocessed range.

Two operational rules matter more than the schedule:

Exactly one machine dreams. My state syncs across machines, and two
consolidators writing the same durable files would conflict endlessly. One
machine owns dreaming; the others read the results. Single-writer is the
cheapest concurrency model that exists, and choosing it here removed a
whole category of merge problems before they happened.

Every cycle has a hard token budget. The reflection step calls an LLM,
and an unbounded loop over a busy day's events is an unbounded bill. A
cycle gets a fixed input and output cap; hitting the cap mid-cycle means
finish gating what you already produced, log the overrun, and let the next
cycle continue from there. A runaway day costs a known maximum. The daily
spend lands around one to three dollars, which I consider cheap for a
memory that maintains itself.

The pipeline: cluster, reflect, score, gate

The evening pass runs the day's episodic events - tool calls, session
summaries, captured thoughts - through a fixed sequence:

1. Cluster. Events are embedded and grouped by density. A durable fact almost never comes from one event; it comes from the same theme surfacing across sessions.
2. Reflect. An LLM reads each cluster and proposes candidates: "these events support the fact that X." Each candidate carries the ids of the events supporting it, so provenance survives the whole trip.
3. Score. Candidates get a confidence score and pass structural checks: enough distinct supporting events, spread over enough time, not a restatement of something already known.
4. Judge. Two LLM-backed checks run against existing memory: does this contradict a stored fact, and is it a duplicate?
5. Promote or queue. High-confidence, well-corroborated candidates can be written to durable memory automatically. Everything else lands as a card in a review queue where I accept or reject with a keystroke, and applying accepted items produces a git commit - the memory file's history is an audit log.

The single most important design fact: the interesting engineering is
entirely in steps 3 through 5. Generating candidates is easy; any LLM
over any event log will happily propose memories. The system's quality is
decided by what it refuses to write. Durable memory pollution compounds -
a bad fact gets recalled, believed, cited, and built upon by dozens of
future sessions - so the write gate is where the paranoia belongs.

Concretely: auto-promotion requires both a confidence threshold and at
least three supporting events. A candidate with two supporters can be
judged genuinely durable by the reflector, and it still cannot enter
memory unattended - it routes to the review queue instead. The rule is not
"two events are not evidence"; it is "two events are not enough evidence
to skip the human."

Two failure stories worth their tuition

The silent floor mismatch. For a while, the clustering stage was
allowed to form two-event clusters, but the scoring stage silently dropped
any candidate with fewer than three supporting events. Every two-event
cluster the reflector judged durable was structurally discarded - not
rejected with a reason, just gone. The pipeline looked healthy: cycles ran
green, promotions happened, nothing errored. It was simply quieter than it
should have been, and quiet is the hardest defect to notice. The fix
lowered the score floor to match the cluster floor and moved the
three-event rule to the auto-promote boundary, where it belongs: the
candidate now survives to the review queue and the human sees it. The
general lesson: when two stages of a pipeline disagree about a threshold,
the disagreement does not error - it silently shrinks your output, and
you will attribute the quietness to "slow week" for months.

The judges that aborted the dream. The contradiction and duplicate
judges originally propagated a hard LLM error - an outage, a quota blip -
straight up, aborting the entire cycle and discarding all the clustering
and reflection work before them. Meanwhile a garbled LLM response was
handled gracefully with a conservative default. That asymmetry made no
sense: the transient network error was more destructive than the corrupted
answer. The policy now is uniform: any LLM-backed stage that fails hard
degrades to its conservative default (no contradiction found, assume
novel) and the cycle completes. For a nightly batch process, resilience
beats strictness - a conservative default risks one duplicate card in a
review queue; an abort discards a day.

Consolidation grows skills, not just facts

The part I did not plan and now value most: the same clustering that finds
durable facts also finds recurring work. When the cycle notices the same
kind of procedure performed across at least three instances on multiple
days with no skill covering it, it proposes one - and above a confidence
threshold it auto-writes a draft skill file. The draft is inert: invisible
to every agent harness until I explicitly promote it. The human gate did
not disappear; it moved from "write the draft" to "activate the draft,"
which is a cheaper place for me to pay attention.

This is the sleep metaphor completing itself. Consolidation is not just
deciding what to remember; it is noticing what you keep doing and turning
it into ability. The facts feed recall, the skills feed behavior, and both
come out of the same nightly replay of the day's events.

What transfers

• Separate the write path from the work path. Agents mid-task are the worst judges of durability. Let them capture freely into an append-only log; let a slower, gated process decide what becomes permanent.
• Put the engineering into refusal. Candidate generation is free; memory pollution compounds. Corroboration floors, contradiction checks, and a human queue for everything below the bar are the product.
• Move human gates to the cheapest checkpoint; do not delete them. Auto-draft plus manual activate beats both manual-everything and full autonomy.
• Batch processes should degrade, not abort. A conservative default wastes a little; a dead cycle wastes the day.
• Audit your pipeline for silent disagreements. Two stages with inconsistent thresholds produce no error, only quiet. Count what enters and exits each stage, and alarm on structural drops.
• Cap the spend structurally. A per-cycle token ceiling with graceful overrun turns "LLM loop over unbounded input" from a risk into a line item.