The seed is just the oldest record, and other things I got wrong about a YantrikDB merge bug

On 2026-10-16, antonius888 opened issue #253 on YantrikDB, the Rust memory engine for AI agents I maintain. In one think() consolidation pass, five memories from four unrelated topics (two delegated-task prompts, a fact about DPI community tools, two Bitwarden facts) were folded into one consolidated record whose text began "Hermes dashboard autostart: TWO mechanisms...". None of the five had anything to do with dashboard autostart.

yantrikdb-hermes-plugin runs the engine in-process as a memory backend for the Hermes agent framework, and its on_pre_compress hook writes a "compression gist" into the store before a conversation chunk gets trimmed: importance 0.75, metadata {"source": "compression_summary", "pre_compression": true, ...}. antonius888's gists ran over 1,300 characters, up to about 1,400.

What they posted:

• Consolidation multiplies an absorbed record's importance by 0.3 (consolidate.rs:1191), so a 0.6 record comes out at 0.18. Their absorbed records landed between 0.18 and 0.24.
• Over many think() passes (the engine only commits a handful of consolidations per pass by default), the store had built up 84 distinct consolidation targets: {2: 31, 10: 22, 3: 8, 1: 5, 6: 5, 4: 4, 5: 3, 8: 3, 7: 2, 9: 1}. The 22 at exactly 10 are sitting on the max_cluster_size hardcoded at the call site, consolidate.rs:774.
• Ten pending "redundancy" triggers, urgency 0.988 to 0.997. Of the three quoted with scores, the 0.9973 one paired a gist with an unrelated 1C ERP fact written in Russian. The 0.9915 one was two different "Strata install" gists, which might honestly be near-duplicates, and the 0.9890 one was also gist against gist.

They proposed four fixes (exclude summaries from clustering, cap a gist's similarity influence, promote the pre_compression tag to a column, cap gist length) and had already added a local env var, YANTRIKDB_COMPRESSION_GIST, to turn the write off.

Two things I had wrong

I replied the same day, on engine 0.23.2, with a theory for why the entity-overlap guard in find_clusters didn't stop this. That guard has been in the engine since April (commit a27ab9f, the v0.4.2 to v0.4.4 core sync). It skips a candidate when both records have extracted entities and the sets share nothing, which keeps "Alice is CEO" from merging with "Sarah is CTO". My theory was that a gist's entity set covers everything it summarized, so it's never disjoint from anything and the guard can't fire.

That describes a record the plugin doesn't write. Here's the end of _compose_compression_gist on the plugin's main:

intent = (first_user or "").strip().replace("\n", " ")[:200]
outcome = (last_assistant or "").strip().replace("\n", " ")[:200]
n = len(middle)
if outcome:
    return (
        f"[compression-summary, {n} turns] "
        f"intent: {intent} | outcome: {outcome}"
    )

First 200 characters of the first user turn, first 200 of the last assistant turn, inside a fixed template. Nothing from the middle of the conversation gets read. That tops out around 450 characters on main, and v0.27.0, the release antonius888 was running, has the same cap. So 1,300-character gists match neither. I asked in the thread whether their copy of the hook had been modified. I still don't know.

The second mistake was one I'd been carrying around in my head, and I only caught it re-reading the Rust for this post. I'd been picturing the engine choosing the gist as a cluster's seed because it sits near so many other vectors. Seed choice has nothing to do with similarity:

// Sort by creation time (return indices)
let mut indices: Vec<usize> = (0..memories.len()).collect();
indices.sort_by(|&a, &b| memories[a].created_at.total_cmp(&memories[b].created_at));

let mut used = HashSet::new();
for (seed_pos, &i) in indices.iter().enumerate() {
    if used.contains(&i) { continue; }
    let mut cluster = vec![i];
    used.insert(i);
    // ... candidates are later records in time order, checked against
    // time window, entity-overlap guard, then cosine similarity

The earliest unused record becomes the next seed. A gist seeds a cluster only when it happens to be the oldest unused record in its neighborhood.

Where the merged text comes from matters more. The consolidation target is a brand-new record built by extractive_summary, which sorts members by importance, highest first, and leads with the top one's text. Ordinary facts in a store like this sit around 0.5 to 0.6. A gist is 0.75, so whenever one lands in a cluster, as seed or member, its text will very likely open the merged record.

The "Hermes dashboard autostart" record from the top isn't a gist (gists always start with the literal [compression-summary,), so I'm not pinning that one on gists. Same mechanism, though: the highest-importance member named a cluster that was mostly about other things.

The embedder

On 2026-10-17, a day after my first reply, I posted a partial correction: the gist is a real but secondary factor, and I think the main cause is the embedder.

I'd installed the published yantrikdb==0.23.2 wheel into a clean venv and, on a fresh store, scored text modeled on the report by calling the engine directly, once with the default embedder and once with potion-multilingual-128M:

pair	default embedder	potion-multilingual-128M
unrelated Russian↔Russian (1C/Bitwarden/gist/FoodTorg)	0.83–0.93	0.14–0.47
unrelated English↔English	0.04–0.23	0.21–0.33
Russian gist ↔ English gist	0.17	0.46

The default, potion-base, is the static, English-only model bundled in the wheel. Unrelated Cyrillic text scores 0.83 to 0.93 against other unrelated Cyrillic text, well over the 0.6 clustering threshold. That alone is enough to explain the gist-to-1C trigger, clusters filling to the size-10 cap, and the importance loss.

The last row is where the gist does contribute. Under the multilingual model, a Russian gist and an English gist still score 0.46, and the only thing they share is the [compression-summary, N turns] intent: ... | outcome: ... scaffolding. Every gist carries it, which gives gists a similarity floor with each other that ordinary facts don't have, and is a likelier story for the two gist-to-gist triggers than anything in their content.

Switching embedders has its own wall. On a store that already holds data, set_embedder_named() refuses and points you at db.reembed(), which isn't callable from Python (#241). My test also bypassed the plugin's config path, so I can't vouch for it end to end. Through the plugin, YANTRIKDB_EMBEDDER also needs YANTRIKDB_EMBEDDING_DIM set or it raises, and the plugin README documents a second route, YANTRIKDB_EMBEDDER_MODEL2VEC=minishlab/potion-multilingual-128M, which needs the model2vec package. Either way it means a fresh store and a re-import; consolidated originals keep their full text, so nothing is lost.

I committed to three follow-ups: make reembed() callable from Python; warn when stored text is mostly outside the embedder's language instead of degrading silently; and on the plugin side, a config switch for the gist write plus keeping gists out of the seed role.

The question I owed wolf-crazy

wolf-crazy, who already has one merged fix in this repo, picked up the seed-role piece today. Gists stay eligible as members and never become the seed. Their useful find: MemoryWithEmbedding, the struct find_clusters already works on, carries a metadata field filled from decrypted JSON at two construction sites, so there's no schema change and no extra decrypt. The proposed shape is an Option<&dyn Fn(&MemoryWithEmbedding) -> bool> parameter, None by default, so every other caller keeps today's behavior. That includes patterns.rs:327 and the Python binding at py_consolidate.rs:45, the _find_clusters antonius888 named in the report. (Checking that one, I noticed it passes None for the entity map, so the guard my first theory was about doesn't even run on that path.)

Their claims checked out against origin/main at 1d48d09 earlier today — the files involved hadn't changed since before the issue was even filed.

They asked one direct question: key the predicate off the pre_compression tag already in the metadata blob, or promote it to a real column first?

The reply that verified their work, posted under my account, was written as triage and pushed that data-model call to me instead of making it in a comment. When I sat down to write this, I still hadn't answered wolf-crazy.

So here's my answer: use the tag.

The reason sits a few dozen lines above where find_clusters gets called, at consolidate.rs:727-731:

let memories: Vec<MemoryWithEmbedding> = memories
    .into_iter()
    .filter(|m| m.metadata.get("consolidated_from").is_none())
    .collect();

Records tagged consolidated_from are dropped before clustering starts, so the next pass doesn't re-merge a summary with its own sources. A metadata key already decides clustering eligibility in the same file, and reading pre_compression for seed eligibility is the same kind of decision. A column earns its cost once there are enough opt-out knobs to want to query or index them. Right now there are two, both in a blob we already decrypt. If more show up I'll be migrating a live table with encrypted metadata on it, which won't be fun, and I'd still rather pay that later than pay for it now. Someone with more production migrations behind them could fairly call that backwards.

What I don't have an answer for is the other half. Keeping gists out of the seed role changes which record a cluster is measured against. It doesn't touch extractive_summary, which will still put a 0.75 gist at the front of any cluster it joins as a member, and with that template floor, gists join clusters easily. The plugin's source calls 0.75 "higher than baseline; this is a deliberate save." Lowering it, skipping gists when picking the lead, or dropping them from clustering the way consolidated_from records are dropped would each undo something chosen on purpose. I haven't picked one, and wolf-crazy's PR shouldn't wait on it.

---

Pranab Sarkar, Independent Researcher