DEV Community: Sovantica

What our coverage number did not protect

Przemek Marzec — Mon, 24 Aug 2026 20:20:02 +0000

Engrava's test suite had 3,845 tests and 94.22% line coverage. There was more test code in the repository than production code. By every number we had, the thing was well tested.

Then we ran a mutation audit over it and found that the suite did not protect a broken concurrency guard, several deletion paths, five configuration sections, two of its own safety tests, or a field that would silently discard a legitimate value, in the version people were running. That is nine findings in three groups, and the groups are not alike. The accounting is below.

This post is about how that happens, because the mechanism is more interesting than the individual bugs, and because we would have told you the suite was solid the day before.

Coverage answers a question nobody asked

Line coverage tells you a line executed while the tests ran. It does not tell you that anything would have noticed if the line were wrong.

Those sound close. They are not. A test can execute a guard, assert something true about the result, and stay green after you delete the guard entirely, because the outcome it asserts is produced by something else in the path. The line was covered the whole time. It was never checked.

The instrument for finding this is mutation testing: change the production code deliberately, run the suite, and see whether anything goes red. What survives the mutation is the finding. Not the code you broke, the tests that let you.

We ran that against Engrava before the 0.6.0 release. Here is what it returned, grouped honestly, because the three groups are not the same kind of thing and lumping them together would flatter us in one direction and alarm you in another.

Group one: six guards we hardened, none reachable from outside the process

Six findings were guards that validated a value and then used the caller's original object rather than the validated one. Validate, then discard the result of validating.

If you have ever written a validator, you know this shape immediately. You check an argument, the check passes, and then the code below reaches for the argument again instead of for what the check produced. If the value can answer differently between those two reads, the check and the use are no longer talking about the same thing.

Six of those were reachable (and here is the part that matters more than the finding) only by a caller already executing code inside the same process. Not through a config file, not through a query string, not through data arriving from anywhere. Every one needed code running next to ours, deliberately handing the library something built to behave inconsistently.

We checked that against the published release itself rather than against our own descriptions of what we had fixed, which is a different claim. The answer was the same for all six: none of them crossed a trust boundary in any version that was ever on PyPI. No advisory is owed, no CVE, no patch to the 0.5 line.

The framing we ended up with, and it is the honest one: this is the absence of a boundary, not a hole in one. Someone executing arbitrary code in your process can replace the function these guards protect, or the standard library call underneath it. Defending against that specific caller is not a thing a library can do, and a test asserting we had would be theatre.

So why harden them at all? Because defence in depth is worth having when it is cheap, because the mechanism generalizes to places where the caller is less trusted, and because a guard that can be talked out of its own conclusion is wrong on its own terms even when nothing can currently exploit it. We fixed all six. We are not going to describe them in language that suggests you were exposed, because you were not.

Group two: two safety tests that proved nothing

This is the group that unsettled us most.

Two tests in the hygiene suite (the code path that decides what memory gets archived and eventually deleted) passed over guards that were working correctly. They were not detecting the guards. They were detecting something else in the path that happened to produce the same outcome.

One of them had a docstring stating, in plain English, that a specific pin was what caused the row to be skipped. That was not true. A different condition entirely was doing the skipping, and you could remove the pin protection at all three layers and watch the test stay green.

Nobody wrote those tests carelessly. They were written by people who understood the feature, and read by people who understood it too. Reading a test tells you what its author believed. It does not tell you what the test can detect. Only mutating the code it claims to protect tells you that, and neither of us thought to do that until a sweep did it mechanically.

There is a related finding that makes the point sharper. Elsewhere in the codebase, deleting a guarantee outright left 4,316 tests green: the whole suite, not some narrow slice of it. A green suite is not evidence that a guarantee holds. It is evidence that nothing in the suite noticed it was gone.

Group three: the one that could actually lose your data

Nothing above changed what a normal user's data did: the first group needs code already running inside your process, and the second is about our tests rather than about shipped behaviour. This one is different, and it is the reason this post exists in the form it does.

Engrava's edges carry a decay_multiplier, a float that is documented as valid from 0.0 upward. Setting it to 0.0 is a legitimate thing to do: it means this edge does not decay by that mechanism.

In released 0.5.x, the code that read that value back from the database tested it for truthiness. 0.0 is falsy. So a stored 0.0 read back as 1.0, the default.

That alone would be a bad-but-recoverable read bug. It got worse on the way out: the update path rebuilds the whole edge record from the values it just read and writes every column back. So the next time anything updates that edge (for an unrelated reason: changing the weight, touching metadata), the misread 1.0 is written over your 0.0. Permanently. No error, no warning, no log line.

No attacker. No unusual configuration. Ordinary correct use of the public, documented API, silently discarding a value the documentation says is valid.

It is fixed in 0.6.0. If you set decay_multiplier=0.0 on any 0.5.x release, check those edges, and note that upgrading does not restore a value that was already overwritten. The 0.5 database has no record of what the number used to be.

We are stating this plainly because a post about our own testing gap that quietly omitted the one defect that could cost someone data would be worth less than not writing it.

What the count actually is

Six defence-in-depth hardenings, all of which require code already running in the process. Two vacuous tests over working guards. One real data-loss defect.

Those nine are the same nine the opening lists: the concurrency guard, the deletion paths and the configuration sections are among the six; the hygiene safety tests are the two; the field that could lose data is the one. We are spelling the arithmetic out because the easy version of this post describes all nine in security language, and that would be a worse error than any of the individual findings: six of them cannot be reached by anyone who is not already running code in your process, and two of them are about our tests rather than about anything that shipped.

What we changed, and what we did not

The suite went from 3,845 tests to 4,316; coverage went from 94.22% to about 95%. The coverage move is the least interesting number on this page, which is the argument closing on itself.

The changes that matter are structural. Guards now use the value the validation produced. Several hand-maintained lists that were supposed to enumerate protected operations are now derived from the code rather than typed out and hoped over. The two vacuous tests are able to fail. And we now require, for a claim about a test, that someone demonstrate the signal going red on the broken version, not that the suite is green on the fixed one.

What we did not do: replace the concurrency mechanism. The specific guard the audit broke is fixed, but the broader design question underneath it (how two processes writing to one file coordinate) needs a schema change, and landing that in a release days from publishing was the wrong trade. It is deferred and written down as deferred, rather than rushed in so this post could have a tidier ending.

If you want to try this on your own suite

You do not need a mutation-testing framework to start. Take a guard you believe is tested: an authorization check, a validation, a deletion safety condition. Delete it. Run your tests.

If they stay green, you have learned something about your suite that no coverage report was ever going to tell you. That is the whole technique. The tooling only makes it systematic.

We found six hardenings, two tests that proved nothing, and one real bug this way, in a codebase we would have described the day before as well tested. We think that says less about Engrava than about what a coverage number is capable of promising.

pip install engrava

Repo: github.com/sovantica/engrava
Upgrade notes: github.com/sovantica/engrava/blob/v0.6.0/docs/upgrade.md

Engrava 0.5.0: a first-class MCP server

Przemek Marzec — Sat, 22 Aug 2026 20:36:42 +0000

Engrava 0.5.0 pulls the MCP server out of the library.

Install engrava now and you get the memory library and nothing else. The server ships separately, as engrava-mcp. The two were never really the same kind of thing — one is a dependency you import, the other is a process your MCP client spawns — and keeping them in one package meant everyone who installed the library also pulled in the server's dependencies, whether or not they ran a server. Splitting them gives each a clean install surface.

If you build directly on the Python API, nothing about your install changes:

pip install engrava

If you want Engrava as a memory server for an MCP client, reach for the standalone package instead:

uvx engrava-mcp
# or
pip install engrava-mcp

engrava-mcp is a native stdio Model Context Protocol server that sits in front of an Engrava store. It exposes read tools, optional write tools, engrava:// resources, and guided prompts to any MCP client that speaks stdio.

It's also listed in the official MCP Registry as ai.sovantica/engrava — PyPI package engrava-mcp, stdio transport — so a client that reads the registry can find it without you wiring anything by hand.

One detail that looks like a mismatch and isn't: the library is engrava 0.5.0, but the server you install is engrava-mcp 0.5.1. The two version independently. engrava-mcp follows Engrava's minor line rather than its patch releases — every 0.5.x build targets engrava>=0.5,<0.6 — so it moves on its own patch schedule underneath. Today that lands at 0.5.1.

What changed for MCP users

In 0.4 the server rode along inside engrava as the engrava[mcp] extra. In 0.5 it graduates to a package of its own:

Before	After
`pip install "engrava[mcp]"`	`pip install engrava-mcp` or `uvx engrava-mcp`
`engrava-mcp` installed by `engrava`	`engrava-mcp` installed by the standalone package
MCP client command: `engrava-mcp`	MCP client command: `uvx`, args: `["engrava-mcp"]`

This only breaks you if you were running Engrava as an MCP server. A plain pip install engrava library upgrade is untouched, and the server still reads the same engrava.yaml or database path it always did — what moves is the install command and the launch command your client uses.

What else shipped in 0.5.0

The packaging is the visible change, but the library moved too, mostly around retrieval and the audit path:

Scoped ranked retrieval — metadata and visibility filters now apply inside the ranked retrieval path, not only the raw scan.
Audit verification — check the hash-chain journal from code with store.verify_journal(), or from a shell with engrava --db engrava.db verify.
Typed provenance capture — callers can attach bounded write-time context such as session_id and actor_id.
Reliability fixes — the sqlite-vec backend and several hybrid-fusion edge cases behave correctly now. If you were hitting either, this is the release that fixes it.

There's also a set of opt-in lifecycle features you can switch on and evaluate against your own workload: consolidation activation, deterministic memory hygiene, and caller-side whole-turn assembly. They ship in 0.5.0, off by default; whether they earn a place in your setup is yours to judge.

engrava is the library, engrava-mcp is the server — install whichever you need:

pip install engrava
uvx engrava-mcp

Code and server docs:

Action Records: Memory for Things an Agent Did

Przemek Marzec — Mon, 20 Jul 2026 20:26:10 +0000

Most agent memory examples start with facts: a user preference, a summary, a document chunk, a note extracted from a conversation. Those are useful, but they leave out a large part of what an agent actually needs to reason about later: what it planned to do, what it attempted, what succeeded, and what failed.

Engrava models that surface with Action Records. An action isn't a chat message, and it isn't a replacement for a task runner. It's a durable memory record for a state change the agent considers operationally meaningful.

That distinction stays abstract until an agent has to recover context across sessions. "The deployment was discussed" is a different thing to have in memory than "the deployment was attempted and failed after the migration step." A text summary can hold the second sentence, but a first-class action record gives the agent something structured to query, connect, and verify.

Why actions belong in memory

Agents move through a loop: infer what should happen next, call a tool or ask a human for approval, observe the result, adjust. If memory only stores natural-language notes, that loop goes opaque over time — the agent can search for words, but it can't reliably ask for "the last failed attempt to update this resource" or "actions that were planned but never confirmed."

Action Records make that state explicit. They represent states like PLANNED, EXECUTING, CONFIRMED, and FAILED, and connect to thoughts and edges in the same local store. This isn't about turning the memory system into an orchestrator — it's about keeping the memory database honest about the gap between what the agent knows and what it actually did.

Structured state, still embedded

Engrava keeps this in the same embedded SQLite-backed store as the rest of the memory graph, which matters for a couple of reasons. There's no separate service to stand up just to ask operational questions — thoughts, edges, action records, timestamps, lifecycle state, and optional journal entries all live in one local database. And the action state participates in the same query and retrieval model: an agent can record a thought for a request, link it to a planned action, update that action when the tool call completes, then later use MindQL and the Python API to inspect recent actions, narrow by status, or pull back the surrounding graph.

The journal is evidence, not a spell

When journaling is enabled, Engrava records thought and edge mutations and action state transitions as hash-linked journal entries — a tamper-evident chain for the events that were journaled. The wording is deliberate. The journal is optional. It is not a database-wide integrity system, it doesn't make external side effects reversible, and it doesn't stop a privileged writer from replacing the database and journal together. What it gives you is a way to verify the continuity of the journaled chain when that journal has been enabled — which matters because action state has a different trust profile from an ordinary note. If an agent later sees an action marked confirmed, the application may want to know whether that confirmation is part of the expected local history.

Not just a task log

A task log answers "what happened?" Agent memory has to answer a wider question: not only that an action happened, but what the agent believed when it chose it, which earlier facts led there, whether it moved through a known state sequence, and whether a failure should change how it retrieves or plans next time. Action Records are built to live with the surrounding graph, not beside it — so execution memory stays close to semantic memory instead of collapsing into unstructured prose.

A concrete example

Say an agent is rolling out a config change. It stores a thought for the request, records a PLANNED action linked to that thought, and moves the action to EXECUTING when it calls the deploy tool. The migration step fails, so the action lands in FAILED — not as a line buried in a summary, but as a queryable record tied to the resource and to the thoughts around it.

Two sessions later the agent picks the task back up. Instead of re-reading a wall of notes hoping the failure is mentioned, it asks the store directly: the last FAILED action on that resource, and the thoughts connected to it. It retrieves the failed migration and the reasoning linked to it — and can choose not to repeat it blindly.

A plain text log could hold "deploy failed." What it can't do is let the agent ask that question precisely, connect the answer to the decision that led there, and treat a failed action differently from a fact it merely read. That's the point of giving actions a first-class record instead of leaving them in prose.

Where it sits

Action Records are one piece of a compositional store: graph memory for relationships, hybrid search for recall, MindQL for structured reads, and an optional tamper-evident journal. None of it claims the memory "thinks" for the agent — the application still owns policy, approvals, and side effects. Engrava just gives it a local, queryable substrate that can remember more than facts alone.

How we built Engrava: from cognitive-architecture research to a production library

Przemek Marzec — Tue, 14 Jul 2026 20:23:39 +0000

Deterministic consolidation, a typed graph in SQLite, and an honest look at what agent-memory benchmarks can and can't measure.

You're building an agent. It answers questions across many sessions, and by session three it has forgotten what it learned in session one.

The reflex is to put a vector database in front of it - and now it remembers a blurry average of everything, ranked by cosine distance, contradicting itself and unable to tell a three-week-old preference from a stale throwaway. A graph database gives you structure, at the cost of a second persistence model with its own query language and deployment. A managed memory service starts you in a few lines, and moves the decision of what your agent remembers onto someone else's infrastructure.

Underneath all of them is one problem: if every plausible fact is written the moment it appears, memory turns into an accumulation layer instead of a judgement layer. Engrava is our answer to that - local, structured, and deliberate about what it keeps in reach. Here is how it is built, and why.

We started with a question, not a schema

Before the storage design there was a long stretch of reading cognitive-architecture research, circling one question: what does a long-running agent actually need from memory if that memory has to stay inspectable? That framing is the reason Engrava is a typed graph and not a bag of embeddings, the reason consolidation is deterministic instead of an LLM rewrite pass, and the reason extraction stays above the database instead of hiding inside it.

What the memory actually is

Engrava is a typed knowledge graph with hybrid search, in a single SQLite file. Thoughts are nodes; typed edges carry the relationships a flat vector can't - that A caused B, that C specializes D. Retrieval fuses vector similarity, keyword match, recency, and priority rather than leaning on cosine distance alone. Turn on the journal and every thought and edge mutation is recorded in a tamper-evident SHA-256 chain. It is all in the free package.

How consolidation works

Every thought has a priority, recomputed each time the consolidation cycle (the dreaming: block) runs - one deterministic pass over the store weighing five signals: recency, staleness, confirmation, confidence, frequency. They combine into a score, checked against gates before promotion:

dreaming:
  enabled: true
  signals: [recency, staleness, confirmation, confidence, frequency]
  promote_threshold: 0.75
  gates:
    min_confirmations: 2
    max_promoted_per_run: 20

The default is conservative on purpose: with min_confirmations: 2, a fact the agent has seen once is not promoted - it stays fully retrievable, just not lifted into the active set until the agent has re-confirmed it. That is the point of the cycle: keep memory a judgement layer, so a single noisy pass can't reshape what the agent treats as settled. No language-model call, no network call, no embedding recomputation - arithmetic over SQLite rows, same inputs and same outputs every run, and the policy is a YAML file you can review in a pull request.

Why it is shaped like the brain's memory - and where it isn't

Three findings shaped three choices. Diekelmann & Born (2010) describe memory stabilization during sleep as selective - the brain keeps a subset of traces, not all of them, which is what the priority score and the promotion gates do. Yassa & Stark (2011) describe how the hippocampus keeps similar experiences from collapsing into one average, so in Engrava thoughts stay distinct nodes and similarity doesn't auto-merge them. Rao & Ballard (1999), and Clark's synthesis (2013), frame the brain as stabilizing what repeated experience confirms, which is where the confirmation and confidence signals come from. We did not build a brain - we took the pressures biological memory evolved under and applied them to a file on disk.

The part of benchmarks nobody wants to say

We didn't tune Engrava to top a leaderboard. And dreaming - the consolidation cycle that gives Engrava its character - doesn't move our retrieval benchmark. We turned it on, we turned it off, and the accuracy barely shifts.

It isn't a bug we're hiding; it's a sign the benchmark is measuring something dreaming isn't for. These benchmarks are short-horizon question-answering over a fixed transcript. Dreaming is lifecycle management - over the weeks an agent stays alive, it decides what stays in reach and what settles out, so the store doesn't rot into an accumulation layer. Whether that discipline pays off over long horizons is genuinely hard to measure, and the benchmarks that exist can't see it.

Comparable evidence across consolidation systems is thin, too: implementations, datasets, reader models, and evaluation setups vary too much for clean comparisons, and the underlying sleep-consolidation research is itself contested. So we won't attach an improvement claim to dreaming that our own retrieval test doesn't support. Dreaming is deterministic, inspectable, and does a specific job; whether that job matters is yours to decide.

The wider point holds for the field: half of these benchmarks aren't measuring the same mechanism. Some run a language model inside the memory pipeline; some don't. And the reader model can move the headline number more than the memory architecture does - hold the memory fixed, swap in a stronger reader, and the same system posts a very different score. When we do publish a number, we publish the whole run - reader, judge, dataset, setup - so anyone can reproduce it.

Stealing SQLite's posture

SQLite runs inside the host process, writes to one file, ships as one library, and is one of the most-tested databases in existence. We wanted that posture. Engrava is a Python library - pip install engrava, and the store is a file on your disk. No server to run, no port to open, no separate auth; your agent imports it like any dependency, nothing leaves the host unless you wire up a remote embedding provider yourself, and it is MIT-licensed. It is not a multi-tenant fleet service - if your agent is a horizontally-scaled cluster needing shared memory across machines, embedded is the wrong shape. For a single agent, it is usually the simpler fit.

Where the category is now

When we started, this was fairly open space. By the time we shipped it wasn't - several teams, Anthropic among them, had shipped consolidation under the same "dreaming" name, borrowing the same sleep metaphor, around the same time. The category converged fast. What separates these tools now isn't the vocabulary; it's whether the consolidation is something you can open up, configure, and run yourself, or something that happens elsewhere on your behalf.

What's next

Engrava is live on PyPI - the graph, deterministic consolidation, hybrid search, the audit journal, and MindQL are all in the free package.
If you're building an agent, try it:

pip install engrava

The repo is on GitHub; issues and discussions are open.