DEV Community: Dawncoral

We grouped 31 agent memory gates into 9. No checks were deleted.

Dawncoral — Wed, 04 Nov 2026 07:54:12 +0000

We grouped 31 agent memory control points into 9 groups, kept all 31 detailed checks, and put the new map through a two-week usage review instead of treating the smaller index as proof of equivalent coverage.

Agent memory is often discussed as a retrieval problem. Our recent work was about the less visible control plane, the checks that decide how a memory system reads and changes project state. We borrowed the outcome-level interface idea from an AWS MCP rebuild and applied it to that control plane.

The system had accumulated session-loading checks, write validators, evidence rules, delivery checks, periodic scans, and human approvals. Each one had a job. The flat list was the problem.

Fifteen entries merged into five groups, four became conditional checks inside a group, one moved back to ordinary reading discipline, and eleven kept their own identity. All 31 detailed rows remain available. No safety check was deleted.

Sarvar Nadaf's recent AWS MCP rebuild offered a useful comparison. He cut 15 tools to 4 outcome-level tools, and the same question took 2 tool calls instead of 7 in his test. The lesson is larger than MCP: an interface should be organized around the decision someone is trying to make, with detail available one level down.

Two mechanisms made that possible without pretending that grouping is the same as safety. The first is an outcome-level control map: start with the failure in front of you, find its group, then open the detailed check. The second is a shadow projection: derive a state view from the append-only event log, compare it with the handwritten view, and change the write path only after the differences have mechanical explanations.

The scoreboard: 31 entries, 9 groups, 0 deleted checks

The analogy with Nadaf's MCP rebuild has a limit. His main costs were tool schemas, response size, and tool-selection confusion. Our control points also consume attention, but a missing memory check can leave two versions of the truth behind. We did not treat a smaller count as evidence of equivalent coverage.

The coverage argument has to come from the mapping itself:

What changed	Count	What it means
Flat control entries before	31	Every check competed for attention at the same level
Groups after	9	Each group answers one kind of operational question
Entries merged into five groups	15	Their actions still run inside those groups
Entries made conditional	4	They now run at a named trigger inside their host group
Entries reclassified	1	A source-priority rule moved back to reading discipline
Entries kept independent	11	Their timing or decision authority requires a separate identity
Detailed rows retained	31	The group is an index, not a replacement for the detail
Safety checks deleted	0	Consolidation changed navigation, not the required work

That last row matters. A control map can hide a check behind a friendly group name just as easily as an oversized AGENTS.md can hide it behind three hundred lines. We kept the detailed rows beside the group index so a reviewer can trace every consolidated entry to its new home.

The groups are addresses, not nine new rituals

We grouped controls by the moment at which they act and the failure they catch. We did not group them by which script or document happened to own them.

Group	The question it answers
Intake	Did the session read current long-term facts, active state, and the relevant domain record?
Admission	Should this action exist, and does it have an owner or task identity?
Write	Did the change pass through the approved writer, contract checks, evidence rules, and location checks?
Change conduction	Did this change reach every dependent document, with contradictions recorded?
Closeout	Is there an explicit account of unfinished artifacts and uncommitted work?
Periodic checks	What do the daily scan, freshness checks, projections, and weekly reconciliation say?
Human review	Does this action cross a red line, require authorization, touch money or credentials, or expose private data?
Delivery self-check	Does every status claim have evidence, and does every surprise have a classification?
Feedback	Is the failure registered for recurrence checks, and is the capability still earning its place?

The first five groups roughly follow a task. The other four cut across tasks. A reviewer can say “this is a write-group failure” or “this belongs in change conduction,” then open the exact validator or receipt underneath.

This is the same reason find_waste is more useful than seven near-identical AWS wrappers in the reference article. The caller starts with an outcome. The workflow lives behind the interface.

Some controls do not get merged

Grouping works only when it preserves decision boundaries. Five kinds of human review stayed distinct inside the human-review checklist: red lines, pre-authorization, irreversible actions, data classification, and monthly audit. Sharing a checklist does not turn five decisions into one.

The periodic checks also kept separate schedules. A daily diff cannot prove that a weekly semantic review happened. A freshness light cannot stand in for either one. Closeout kept its own group because no other control accounts for unfinished files. Failure registration stayed visible because a repeated problem needs an identity before anyone can count its recurrence.

Our rule is simple: merge the lookup, not the judgment.

A shadow projection tests the source before the switch

The control-map change reduced navigation overhead. The shadow system addressed a different question: if a state view can be derived from events, why maintain that view by hand?

We started with one high-frequency view. It had collected 111 rows in 11 days, which made manual drift easy to observe and gave the experiment enough material to be useful. The existing handwritten view stayed in place. A deterministic projector read the append-only event log, matched events to rows using shared anchors, and wrote a separate shadow view.

The pipeline has three stages:

extract: read timestamped events from the append-only log
project: match events to view rows with slugs, task IDs, or text anchors
diff:    report coverage, lag, and unexplained drift

No model calls sit in this path. The projector checks that the source hash has not changed before writing, rotates five backups, and skips rows it cannot parse instead of deleting them. A person reads the diff.

The first run produced a useful, unglamorous table:

First shadow run	Result
Handwritten rows in scope	144
Rows matched to an event	129
Coverage	89.6%
Strong-signal lag	0
Rows without an event anchor	15

The 15 unmatched rows had a mechanical explanation: the activity existed in the handwritten view but had never entered the event log. Increasing the matcher's vocabulary would only hide that structural gap. The better fix was to make the event log the single write entrance before asking the projection to replace the handwritten view.

Promotion changes who writes the view

A publishing view moved first. The projector produced 41 entries that matched the existing handwritten publication list. A separate activity view followed with 30 generated rows, each carrying a pointer back to its source event. Its reconciliation run reported 100% coverage and zero drift.

Those numbers describe agreement between the event log and the derived views. They do not measure whether an agent remembered the right fact, made a better decision, or saved money. A projection can be perfectly consistent with an incomplete event log. That is why the write entrance and the external audit remain separate controls.

After promotion, people stop editing the generated view. They write the event; the projector refreshes the view. The append-only log stays intact, and the generated rows retain their evidence pointers. Rollback means restoring the handwritten view, not reconstructing history from the projection.

Three kill rules stayed armed after promotion:

Two or more unexplained drift cases trigger rollback.
One dispute over whether the log or the projection is authoritative triggers rollback.
Maintenance time exceeding the old handwritten workflow triggers rollback.

A shadow system needs a way to lose. Otherwise it is just a second production path with a reassuring name.

The 9-group map is still under review

The shadow results above are completed reconciliation checks. The control-map consolidation is earlier in its life. It is currently in a two-week review with three tests:

five sample failures must be traceable to the right group within 30 seconds;
the four newly conditional checks must have zero missed executions;
the nine groups must be used in at least three real deliveries or reviews.

If any test fails, the detailed 31-row map becomes the primary reference again. We have not redrawn the architecture diagram around the new groups yet. A diagram would make the consolidation look settled before the usage evidence exists.

We also have not measured a token reduction for the 31-to-9 change, and we are not claiming one. The review measures lookup speed, missed conditional checks, and actual use. That is a narrower claim than “71% less governance,” which would confuse a smaller index with less work.

What to copy before you consolidate anything

Start with a control inventory that names each trigger, owner, and evidence artifact. Group entries only when they share a decision moment or failure mode. Keep the detailed inventory one level below the group index, and leave human decisions individually accountable.

Then pick one derived view, not the whole memory system. Keep its source append-only. Run the projector beside the existing view and count coverage, lag, and unexplained drift. Change the write path only when every remaining difference has a mechanical explanation. Keep the rollback criteria active after the switch.

Our earlier posts cover the adjacent layers: why memory needs an audit between its log and current state, how change-conduction receipts verify dependent documents, and why a saved status is not evidence of a durable write. The public wallaby-agent-rules repository contains the file-based memory and governance layers. This post describes our internal control map and shadow-projection work; it is not a release announcement for that projector.

Disclosure: Wallaby Token builds open-weight inference infrastructure and develops wallaby-agent-rules. We use the memory controls described here on our own projects.

Originally published on the Wallaby Dispatch.

Kimi K3.1 Pro: 5,315 multilingual test results

Dawncoral — Tue, 03 Nov 2026 02:19:31 +0000

Kimi K3.1 Pro scored 90.9% on Japanese idioms and 99.5% on Turkish paraphrase selection. We ran 5,315 source-dataset entries across Japanese, German and Turkish to see where those strengths hold, what they do not prove, and why the difference matters for business AI.

German conversational meaning reached 87.7%. Turkish figurative-versus-literal usage reached 99.0%. The complete run returned 1,551,208 tokens and cost an estimated $6.169655 at list price, with every response, prediction and score preserved for review.

Disclosure: Wallaby builds open-weight inference infrastructure. We ran this evaluation on Moonshot's official API on 2 November 2026. It is not a Wallaby production benchmark or a customer result.

The numbers worth talking about

Japanese OpenMWE asks whether a marked phrase is an idiom or literal language. Kimi K3.1 Pro answered 270 of 297 cases correctly.

German MultiPragEval tests conversational meaning. The model answered 263 of 300 cases correctly.

The two Turkish idiom tasks were stronger still: 199 of 200 on paraphrase selection and 195 of 197 on figurative-versus-literal usage.

Those are useful results for teams building terminology checks, document classification and local-language workflows. They are also deliberately specific. Each percentage belongs to a named dataset, not to an entire language.

What 5,315 entries actually include

The full set covers public conversational, idiom and workplace tasks, plus nine small custom tasks for structured extraction and context reversal.

Dataset	Entries	What it measures
German MultiPragEval	300	Conversational meaning
German InQA+	384	Indirect answers
Japanese OpenMWE	297	Idiom versus literal meaning
Japanese JaSocial	10	Public speech-act samples
Japanese JWCAS	30	Open workplace scenarios
Turkish ImplicaTR	3,888	Conversational implication
Turkish Idioms	397	Paraphrase and figurative usage
Custom tasks in three languages	9	Extraction and context reversal
Total	5,315	—

Turkish makes up 80.7% of the entries because ImplicaTR alone contributes 3,888. That is why we do not average the six task scores into a single multilingual number. A large test set should not hide which tasks dominate it.

How we counted responses and prompts

The 5,315 source-dataset entries contain 5,311 unique prompts. Four repeated prompts remain separate source-dataset rows. The 60 InQA+ supplement samples are part of the 384-entry InQA+ set, not an extra 60.

We recorded 5,368 successful API responses in total. The difference comes from earlier duplicate sampling. Duplicate responses count toward execution cost, not toward the scored total. Every recorded response returned HTTP 200 and passed a content-hash check.

The more useful finding sits between the high scores

Turkish idiom selection reached 99.5% and 99.0%. Turkish ImplicaTR scored 2,179 of 3,888, or 56.0%.

German conversational meaning reached 87.7%. German InQA+ scored 256 of 384, or 66.7%. The label breakdown is more informative than the average:

Label 4, the non-answer category, reached 85.5%. Label 6, insufficient context, reached 0 of 22. A team using the model to decide when a conversation lacks enough information should keep an explicit review step for that decision.

The same caution applies to Japanese workplace writing. We generated 30 JWCAS open-ended responses, but they still need native-speaker review. JaSocial covered 10 public samples with keyword-based function checks, not a naturalness score. We did not turn either one into a made-up accuracy number.

What this means for local-language products

Japanese: idiom and literal-meaning classification is a promising fit for structured document workflows. Open-ended business writing needs a separate native review process.

German: conversational-meaning tasks are useful, but indirect answers and missing-context decisions need task-specific tests and escalation rules.

Turkish: terminology and idiom workflows show strong results. Open-ended implication should be evaluated on representative customer text rather than inferred from idiom accuracy.

For a product team, the practical question is not “Is this model multilingual?” It is “Does it work on this document, in this language, under these rules?” That is the evaluation we would run before choosing a model for a customer workflow.

What the evaluation cost

The run returned 1,551,208 tokens. Our list-price estimate is $6.169655, including duplicate sampling and different execution modes.

Execution stage	Estimated cost
Initial 4,207 entries	$4.701157
Supplementary 90 entries	$0.348890
Professional-set direct calls	$1.119608
Total	$6.169655

The first 1,000 Batch calls were estimated at 60% of standard price; the other stages used standard list rates. This is an evaluation-cost estimate, not an invoice, and it excludes taxes and any separate cache-write charges.

Evidence and sources

Download the evaluation artifacts:

Public datasets: MultiPragEval, Multilingual-IQA, OpenMWE, ImplicaTR, Turkish Idioms, JaSocial, JWCAS.

We used kimi-k3.1-pro with low reasoning effort and the same output-format requirements within each task. Objective answers were scored against their labels; open-ended generations were left unscored. We did not run a same-prompt, same-condition Claude comparison, so this article makes no Kimi–Claude superiority claim.

Try your own local-language workflow

Wallaby provides OpenAI-compatible access to curated open-weight models, itemized usage billing and automatic provider failover. Kimi K3 is live on Wallaby today; the K3.1 Pro evaluation above does not imply that K3.1 Pro is available on Wallaby.

New accounts receive $0.50 in free credit, with no card required until top-up. You can start your first API call, compare it with your own documents and see every token itemized. Self-serve top-ups start at $20.

For the commercial details, see the current Kimi K3 pricing and our broader open-weight API pricing comparison. If you are estimating a longer evaluation, our agent context cost audit shows how repeated context changes the bill.

We stopped grading agent prose and counted 240 tool calls

Dawncoral — Sun, 01 Nov 2026 13:04:18 +0000

Our old experiment gave a footer instruction 0/20; the new one gave the same position 20/20 with three different models.

The sentence did not suddenly become persuasive. We changed the experiment.

In the old runs, the named feedback tool did not exist, and a judge scored the output prose. In the new runs, the agent could call a real skill_feedback function. Every valid call wrote a row that we could retrieve from a ledger after the run. Across 240 executions, we counted tool calls instead of grading 240 explanations.

A dev.to reader's criticism of our earlier statistics shaped the other half of the design. We pre-registered the step-versus-footer comparison, kept the per-task pairs and reported discordant cases rather than asking readers to interpret overlapping intervals.

The new result contradicts our old story, but it also shows why the old number was weaker than we realized.

Zero was not twenty refusals

The original instruction sat at the end of the skill file. It named skill_feedback, specified the arguments, and told the model to skip silently if the tool was unavailable.

There was no such tool in that environment.

A judge read each transcript and scored whether the output recorded a failure. Pooled across two forge-built Flash arms and one Pro arm, the score was 0/20. We discussed that result as if twenty agents had declined to act.

Later, we found eight zero-byte files among the twenty outputs. Only twelve were non-empty. A model with no tool, an empty response and a text-only judge cannot support the behavioral claim we attached to that zero.

The old dataset remains what it is. We cannot turn it into a tool-call experiment after the fact.

For the next run, we made the action exist and gave it a record. skill_feedback became a real harness function. A call counted only when it had valid arguments, matched the execution identity and appeared in the ledger afterwards.

No judge read the prose to decide whether a call had happened.

A reader had already challenged our statistics

Before this follow-up, we published a different experiment comparing one-shot prompting with Skill Forge. One-shot recalled 118/120 fire-question judgments and Skill Forge recalled 110/120. Our first write-up leaned on overlapping Wilson intervals when discussing the gap.

In the comments, arhancanli explained why that was the wrong comparison. Both arms had answered the same questions, so the informative cases were the discordant pairs. The comment also challenged the independence assumption behind counting two trials of one question as separate draws.

We recomputed the results. By question and trial, one-shot won 8 discordant pairs and Skill Forge won 0, giving p = 0.0078 in an exact two-sided sign test. Collapsed to unique questions, the result was 6-0 with p = 0.031.

That correction was public. Its more useful effect showed up in this experiment's design.

We registered step versus footer as the primary comparison before execution. We retained the pairing by task. We also decided how to handle engineering failures before looking at the result table.

The reader did not supply the new hypothesis. The comment changed how we were willing to test it.

The 240-run matrix

The models were DeepSeek V4.1 Flash 1017, Kimi K3.1 Pro and DeepSeek V4 Pro. Our local Flash arm name was ds-v41-flash-1017-unverified, while the API returned deepseek-flash; the build identifier is therefore not confirmed by the API.

Each model received the same twenty anomaly tasks under four conditions. The feedback sentence was absent from the skill, placed in the footer, placed in its own section, or inserted as step 3 of 5 immediately after base step 2.

Every condition shared a system prompt saying to call skill_feedback when feedback was needed. Even the no-sentence arm therefore had a mandatory feedback instruction outside the skill.

The new sentence was mandatory. It did not repeat the old silent-skip fallback, which is one more reason this was not a controlled rerun of the original 0/20 experiment.

Here are the real-call counts:

Instruction inside the skill	V4.1 Flash 1017	K3.1 Pro	V4 Pro
No sentence	1/20	0/20	1/20
Footer	20/20	20/20	20/20
Own section	20/20	20/20	20/20
Step 3 of 5	20/20	20/20	17/20

All three footer arms reached 20/20. Both Flash positioned arms reached 20/20 as well, and K3.1 Pro matched them. V4 Pro reached 20/20 in the footer and own-section arms, then 17/20 in the step arm.

The old draft said the numbered step was the contract and the footer was a comment. The real-call matrix did not reproduce that advantage.

Read the pairs before the percentages

The pre-registered comparison was step against footer. For Flash, all 20 pairs had calls in both arms, giving p = 1. K3.1 Pro had the same 20 double-call pairs and p = 1. Excluding its footer API-error pair for the sensitivity analysis left 19 pairs, again with calls in both arms and p = 1.

V4 Pro had 17 double-call pairs and three footer-only pairs, giving p = 0.25. The three step-arm failures were SSL and transport errors with zero completion. They were not cases where a completed run declined to call the tool.

We kept the primary step count at 17/20 because that was the pre-registered rule. The sensitivity analysis removed the engineering failures and produced 17 double-call pairs, p = 1.

Presented this way, the result does not depend on whether two confidence intervals happen to overlap. The reader can see the pairs, the exclusions and the exact test.

That still buys a narrow conclusion: this experiment did not detect an advantage for step 3 of 5 over the footer. It does not prove equivalence, and it does not show that instruction placement is irrelevant elsewhere.

Ceilings are bad places to look for small differences.

What could have produced the ceiling

The shared system prompt may have done much of the work. It named the tool in every arm, including the one without a sentence in the skill. Once the model knew about skill_feedback, adding a mandatory mention in the footer may have been enough to make the call reliable.

The contrast between absence and presence is descriptive. No-sentence arms scored 1/20, 0/20 and 1/20, while all footer arms scored 20/20. This matrix did not isolate why that happened, and it did not score whether the ending prose mentioned the instruction.

There are practical limits too. The business actions in the tasks were simulated; only the feedback call was a real tool. Fixed arm order, unshuffled tasks and one execution per cell leave room for time and server-side drift.

The 240 executions produced 240 unique terminal states, zero ledger mismatches and zero empty final outputs. Eight API errors remained in the record: three in V4 Pro's step arm, four in K3.1 Pro's no-sentence arm and one in its footer arm. The footer error followed a valid ledger row, which is why its call remained in the primary count.

The new measurement is more concrete. It is not a universal theory of agent instructions.

The rule that survived the reversal

We would still rather see an important action in a task plan than hope a closing note survives the run. Step 3 of 5 remains a reasonable implementation choice. This experiment simply does not give us evidence that it outperforms the footer.

The stronger rule sits below placement. Build the tool before grading the sentence, then count the side effect where it lands. In this harness, that side effect was a ledger row with a skill name, scenario and failure line, tied back to a specific execution.

That is also why the new 20/20 and the old 0/20 can coexist without choosing whichever number makes the better headline. One came from judged prose without the named tool. The other came from real calls in a harness whose shared prompt may have pushed positioned arms toward a ceiling.

The harness, pre-registration and raw results are open for inspection. The previous reader improved this run by recomputing our comparison; the same kind of review is welcome here.

Disclosure: Wallaby Token sells API access to Kimi K3, one of the models in this comparison.

A dev.to comment changed how we test agent instructions

Dawncoral — Sun, 01 Nov 2026 13:03:48 +0000

A reader named arhancanli caught a problem in our last experiment: we had used overlapping Wilson intervals to compare two arms that answered the same questions. The intervals said less than we thought. The paired recount said more: 8-0 against our tool, p = 0.0078.

That comment changed the next experiment.

For the follow-up, we registered step versus footer as the primary comparison before running it. We kept the per-task pairs. We also replaced judged prose with a real tool and a ledger. After 240 executions, the result was inconvenient in a new way: the footer scored 20/20 with all three models, and moving the instruction into the numbered steps did not improve it.

This is the chain from a comment to a different experimental design, and then to a result we did not expect.

The comment that forced the recount

The earlier post compared one-shot prompting with Skill Forge, our pipeline for building skill files. One-shot recalled 118 of 120 fire-question judgments. Skill Forge recalled 110. We reported overlapping Wilson intervals and initially treated the overlap as evidence that the gap was unclear.

In the dev.to discussion, arhancanli pointed out two problems. First, two trials of the same question were not independent draws. Second, overlapping intervals were the wrong comparison for arms that had answered identical questions.

The useful unit was the discordant pair: cases where one arm passed and the other failed. We went back to the per-question results and counted them. By question and trial, one-shot won 8 pairs and Skill Forge won 0. The exact two-sided sign test gave p = 0.0078. Collapsing to unique questions gave 6-0 and p = 0.031.

The correction made our result worse. It also made the analysis better.

We updated the public write-up with the paired counts. More importantly, we carried the criticism into the next experiment instead of leaving it in a correction paragraph.

Three decisions before the next run

The next question concerned instruction placement. If a skill tells an agent to report a failure, does it matter whether that instruction sits in a closing footer or inside the numbered task steps?

Our first design decision came straight from the comment: compare paired outcomes, not two separate success rates with overlapping intervals. We pre-registered step versus footer as the primary comparison before executing the matrix.

The second decision was to count an action rather than ask a judge whether the transcript sounded compliant. The harness exposed a real skill_feedback function. A valid call appended a row containing the skill name, scenario and failure line. After each run, we could re-read the ledger and match the row to the execution.

The third decision was to register the exclusions before seeing the results. Transport failures could still occur, but the primary analysis would follow the pre-registration rather than a rule chosen after the table looked interesting.

These changes did not make the experiment perfect. They did make three criticisms easy to answer: which comparison was primary, what counted as a pair, and which failures could be removed.

The older zero had a measurement problem too

There was another reason to change the measurement. In an older round, a footer instruction had scored 0/20. We had treated that number as twenty examples of the model failing to act.

The environment told a different story. No skill_feedback tool existed in those runs. The instruction itself said to skip silently if the tool was missing. A judge then read the output text and scored whether it recorded a failure.

Eight of the twenty output files were also zero bytes. The non-empty denominator was twelve, a detail nobody had counted when we first quoted the zero.

So the old 0/20 was a text score collected without the named tool. It was not twenty observed refusals to call that tool.

The new run could not repair that old dataset. It could stop us from repeating the same measurement.

Three models, four positions, 240 executions

We tested DeepSeek V4.1 Flash 1017, Kimi K3.1 Pro and DeepSeek V4 Pro. The Flash arm carried the local label ds-v41-flash-1017-unverified; the API returned deepseek-flash, so the API did not confirm the 1017 build identifier.

Each model ran twenty anomaly tasks in four arms:

No feedback sentence inside the skill.
The sentence in the footer.
The sentence in its own section.
The sentence as step 3 of 5, immediately after base step 2.

The mandatory sentence was otherwise unchanged across the positioned arms. Every arm also shared a system prompt: if you need feedback, you must call skill_feedback.

That last detail matters. The no-sentence arm had no feedback sentence in the skill, but the model still had an instruction in the system prompt.

Three models, four arms and twenty tasks produced 240 executions. The ledger, not a judge reading the final prose, supplied the counts.

Arm	V4.1 Flash 1017	K3.1 Pro	V4 Pro
No sentence in the skill	1/20	0/20	1/20
Footer	20/20	20/20	20/20
Own section	20/20	20/20	20/20
Step 3 of 5	20/20	20/20	17/20

The footer worked everywhere.

The numbered step did not beat it.

The paired result, reported first

Flash produced 20 step-footer pairs with calls in both arms, so p = 1. K3.1 Pro also produced 20 double-call pairs and p = 1. Its sensitivity analysis excluded one footer API-error pair, leaving 19 double-call pairs and p = 1.

V4 Pro produced 17 double-call pairs and three footer-only pairs. The primary paired result was p = 0.25. Those three missing step calls were SSL and transport errors with zero completion, not model refusals. The pre-registered primary count nevertheless stayed at 17/20 for the step arm. Removing the engineering failures in the sensitivity analysis left 17 pairs with calls in both arms and p = 1.

This is where the earlier comment changed our behavior. Instead of printing two percentages and inviting the reader to guess whether they agreed, we could show the pair structure and the exact test first.

The result still needs a narrow reading. It did not detect a step advantage in this environment. It does not establish that footers and steps are equivalent in general, or that instruction position never matters.

A null result at a ceiling has limited reach.

Why the ceiling may be there

Every arm carried the shared system instruction about skill_feedback. That may have made the action so salient that any mention inside the skill was enough to push the run toward 20/20. The no-sentence arm remained near zero, while footer, own section and step reached the ceiling or approached it.

The tool also changed what a skip looked like. In the old environment, a missing tool and a declined call could both end as prose with no verifiable action. In the new harness, a missing ledger row was visible. Valid arguments and a matching execution identity could be checked after the run.

We changed more than the measurement. The old instruction allowed a silent skip when the tool was missing; the new sentence was mandatory. The new run was therefore not a controlled rerun of the old experiment, and it cannot isolate sentence wording, placement and tool availability as separate causes.

The business actions in the anomaly tasks were simulated. Only skill_feedback was a real tool. This shows whether the models called a feedback tool they were told about, not whether they completed a real business workflow.

The arm order was fixed and the tasks were not shuffled. Each cell ran once, so time and server-side drift remain possible confounds. Across the 240 executions there were eight API errors: three in the V4 Pro step arm, four in the K3.1 Pro no-sentence arm and one in its footer arm. The footer error occurred after a valid ledger row, so the primary count retained that cell.

Those constraints belong beside the 20/20, not below a triumphant conclusion.

What survived from both critiques

The statistical critique left us with a concrete rule: when two arms answer the same tasks, preserve the pairs and report the discordant cases. In this run, the primary comparison and its sensitivity analyses were visible before anyone had to ask for them in the comments.

The measurement critique left us with another: grade the side effect you can retrieve. A skill_feedback row either matched the execution or it did not. There were 240 unique terminal states, zero ledger mismatches and zero empty final outputs in the new matrix.

Together, those rules explain why we now trust this null result more than the old zero. The old number required a judge to infer behavior from prose in an environment missing the tool. The new number came from a pre-registered comparison over readable tool records.

This time, all 60 footer executions left a matching ledger row.

The harness, pre-registration and raw results are public. If you find another design flaw, the useful response is the one arhancanli gave us: name the unit, show the comparison and, if you can, recompute it.

Disclosure: Wallaby Token sells API access to Kimi K3, one of the models in this comparison.

A reported $2.3 API bill doesn't tell you what the agent costs.

Dawncoral — Sat, 31 Oct 2026 03:14:33 +0000

The spend reported to us for this line of work was ¥16.8, about $2.3 at roughly 7.2 RMB to the dollar. In a separate client-side reading for 28 October, the median request carried about 158k cache-read tokens. Neither reading answers the question people actually ask, which is what the agent cost per useful output.

This piece is about the gap between them, and why we are not going to close it with arithmetic.

Three numbers that look like the same number

There are three denominators in any agent cost story.

The invoice: money, in the currency you were charged, for a period of time. It is the number a finance person needs and the number every cost post quotes.

Wire volume: tokens the client sends and receives. Cache reads, fresh input, output, retries. It is the number an engineer can measure locally, per request, without asking anyone.

The unit cost: money per accepted output. It is the only one of the three that tells you whether a change to the agent was worth making.

You cannot convert one into another without three extra facts: the rate charged for each token class, which stages of work the invoice covers, and a quality gate that decides what counts as accepted. We have one of the three numbers cleanly, one partially, and the third not at all.

What the reported figure covers, and what it cannot

The figure is reported, not verified. It came to us from the person who paid it, and our repository holds no independent receipt or stage-level breakdown.

That is the first limit. The second is coverage. This program ran a pre-registered feedback experiment with 240 executions, a structure experiment with 180 executions and 180 judge calls, a calibration pass, smoke tests, and retries caused by transport errors. Nothing in our logs assigns a share of the spend to any of those stages, and we cannot verify which of them the figure covers.

So we do not use about $2.3 as the cost of either experiment, and we do not treat it as a verified total for both. It is a reported amount attached to a line of work.

What the client carries on every request

Our metering script reads the client session logs and reports, per request, how much of the prompt was already-seen context.

For 28 October it counted 795 requests with a median of 158k cache-read tokens per request and a 90th percentile of 421k. Cache reads summed to 151,017,550 tokens. Fresh input summed to 9,146,736. In this client log, re-read context is the largest input-token class: 16.5 times the fresh-input volume. That is a volume ratio, not evidence about which token class dominated the bill.

Two cautions before anyone quotes that number.

It is not fresh input, not total context, and not a bill. That number is one class of token, from one meter, on one machine.

An earlier saved note records 768 requests and a 159k median for the same date; the later rescan returned 795 and 158k. We cannot reconstruct the cause of that change from the saved evidence, so we treat the readings as diagnostics, not accounting figures.

What the two experiments record

P1 has 240 terminal executions. All 411 recorded completions include usage, but seven API-error cells have no completion or usage record, so usage coverage is not 240 out of 240 executions. Where usage exists, output totals were 99,654 tokens for the arm labelled DeepSeek V4.1 Flash 1017, 110,401 for DeepSeek V4 Pro, and 98,266 for Kimi K3.1 Pro. The API returned only deepseek-flash for the first of those, so the 1017 build is unverified.

That is a token total, not money. We never stored a rate or a currency on those runs. The token column is partial and the money column does not exist.

P2 recorded 180 terminal judge runs across the 90 ambiguous cells: 179 parsed successfully and one ended in a judge-parse error. One third of the executions lack execution-usage records. Judge usage exists, but the missing execution usage prevents a complete P2 token total.

There is a smaller distortion on top. Eight P1 cells ended with API or transport errors. Seven have no completion record, so their token use is unknown; the eighth had already written a valid feedback ledger before the final error. The three V4 Pro step errors had no completion, which is why that arm reads 17/20 in the primary count.

Two numbers that look alike

P2 records 159,215 execution output tokens for V4 Pro in aggregate. The client meter reports about 158k cache-read tokens per request for 28 October. The digits are similar and the quantities have nothing to do with each other: one is generated text, the other is re-read input.

The same discipline applies to the older figures. Our skill furnace program cost ¥10.6 in API fees. A separate Claude review cost $10.04. Neither is comparable to the reported $2.3 figure here: different programs, different models, different currencies, different coverage.

Cost per accepted output

If you want one number for design decisions, it is money per accepted output. There is a longer argument for it on the blog; this program is a case where the denominator is unusually well defined.

P2 has a quality gate: two judges for each of the 90 ambiguous cells, each scoring against a pre-registered rubric, for 180 terminal judge runs. That is the kind of gate an accepted-output count needs.

It also has a problem. In our last piece, we did not detect a full-versus-shell quality difference. If two configurations produce indistinguishable quality, cost per accepted output becomes the thing that separates them. We cannot compute it: the numerator is a reported figure with no stage split, and one third of the executions have no usage record.

The number we do not have

We are not going to reverse-engineer it.

Multiplying the token totals by our public rate card would produce a number with the shape of a cost and none of the substance: the wrong currency, the wrong rate class, and no stage attribution. We would rather say the number is missing.

What it would take is three small changes, and we are starting on them.

Price each stage in the run log, at the rate actually charged, not at the list rate we publish.

Keep execution usage and judge usage in the same record, so a token total can be checked against a completion count.

Keep a ledger that says which outputs passed, so the denominator is a query and not a memory.

What changed in our pipeline

We are adopting two rules after this program.

A cell without usage is marked incomplete, never zero. P2 taught us that the hard way: sixty executions carried no usage, and a zero would have looked like a free run.

Every cost claim carries its stage scope in the same line. A reported total says reported total. A one-day meter reading says one day and one machine.

Neither rule makes the analysis cheaper. They make it impossible to quote a number we cannot defend.

The reported figure tells you something about what the work cost to run. It does not tell you what the agent costs. For that you need the wire reading and the accepted-output denominator. We have one and a half of the three, and publishing the half we have would be the easy mistake.

The metering script, the experiment harness and the raw results live in the open repository: Dawncoral/wallaby-agent-rules.

Disclosure: Wallaby Token sells access to Kimi K3, and K3.1 Pro spend is the reported figure discussed here.

The footer scored zero. The real tool log gave it 20/20.

Dawncoral — Sat, 31 Oct 2026 03:13:49 +0000

In our old scoring, the closing footer fired zero times out of twenty. We moved the same sentence into the numbered steps, the score moved, and we wrote a post about addresses: the footer is a comment, the step is a contract.

Then we built a real tool with a ledger you can re-read, and the footer scored twenty out of twenty.

Before you file that under contradiction: the old number was not an action. It was a text score, produced in an environment where no feedback tool existed. And the sentence we were testing ended with an instruction to skip silently if the tool was missing. The model may have been obeying us.

This was not a controlled rerun of the old experiment. The tool and the scoring rule changed, and P1 also used a different feedback sentence. The old silent-skip fallback was not retested; P1 used a mandatory sentence without that clause.

What actually changed: tool, scoring rule and sentence

The old run had no skill_feedback tool. The skill files ended with a closing line naming the tool, its arguments, and the fallback. A judge then read each transcript and scored the text: did the output record a failure.

Pooled across the two forge-built Flash arms and one Pro arm, that score was 0/20. Of the twenty output files behind it, eight were zero bytes. The non-empty denominator was twelve. Nobody counted that at the time. We reported 0/20 as if it were twenty observations of a refusal.

The new run has a tool. skill_feedback is a real function in the harness. Every call appends a row to a ledger with the skill name, the scenario and the failure line. A call counts only if it appears as a real tool call, with valid arguments and a matching identity, and can be re-read from the ledger afterwards. No judge reads the prose. 240 executions, zero judgment calls.

The measurement now counts a real side effect where it lands; the comparison with the old run does not isolate sentence quality or position as the cause.

Three models, four positions, 240 executions

Three models, four arms, twenty anomaly tasks per model. The models were DeepSeek V4.1 Flash 1017, Kimi K3.1 Pro and DeepSeek V4 Pro. The first is labelled 1017 only in our local arm name, ds-v41-flash-1017-unverified; the API returned deepseek-flash, so the build is not confirmed by the API. The arms differ only in where the same mandatory sentence sits: nowhere inside the skill, in the footer, in its own section, or as step 3 of 5, inserted right after base step 2. Every arm, including the one with no sentence, carries a shared system prompt: if you need feedback, call skill_feedback.

Arm	V4 Pro	V4.1 Flash 1017	K3.1 Pro
No sentence in the skill	1/20	1/20	0/20
Footer	20/20	20/20	20/20
Own section	20/20	20/20	20/20
Step 3 of 5	17/20	20/20	20/20

The pre-registered comparison was step versus footer. Flash: 20 pairs, all double-called, p = 1. K3.1 Pro: 20 pairs, all double-called, p = 1; the sensitivity analysis excludes the footer API-error pair and gives 19 pairs, all double-called, p = 1. V4 Pro: 17 double calls and 3 footer-only, p = 0.25, and the sensitivity analysis that drops engineering failures gives 17 pairs, p = 1.

The three V4 Pro gaps deserve their own line. All three are SSL and transport errors with zero completion. They are not the model declining to call the tool. We keep the primary number at 85% because the pre-registration says so, and we report the 17-of-17 sensitivity beside it.

Footer reached 20/20 in all three models, and the pre-registered comparison did not detect a step advantage in this environment. That null result does not establish equivalence or show that position is generally irrelevant.

Why the footer works now

The obvious answer is that the tool exists, so a sentence that names it has something to point at. A skip is also visible now: no ledger row. That lets us separate declined from could not call, and it is one reason the old number and this one are not comparable. P1 did not retest the old fallback clause itself.

Position may still do something. It decides whether the action sits inside the plan the model executes, or outside it as advice about how to work. The tool alone did not account for the ceiling: none stayed at 1/20, 0/20 and 1/20 while the footer reached 20/20 in every model. That presence-versus-absence contrast is descriptive, and P1 did not score whether the ending mentioned the instruction.

The part of the old lesson that survives

One claim from the old draft still holds, and the new data sharpens it instead of erasing it. A rule that lives only in prose is not verifiable. You cannot tell compliance from narration by reading the transcript, because a model that narrates well produces the same surface as a model that acted.

We learned this the expensive way in our memory work: writes that reported success and never landed, discovered only when someone re-read the store instead of the summary. The report is not the action. The footer is not the call.

What the new run adds is the fix, and the fix is boring. Make the action write something you can re-read, then count that. Once the ledger exists, the argument about footers and steps becomes a secondary question.

What this corrects

Two statements from the old draft need to go.

The first is the headline number read as behavior. "The instruction fired zero times" was true of a text score in a tool-less environment. It was not twenty observations of a model refusing. Eight of the twenty outputs were empty, and the instruction we wrote told the model to skip when the tool was missing.

Statement two is the final line, "We have not run a follow-up." We have. It is this piece.

Published posts are not being rewritten. Our rule is that live posts stay as published, and this article carries the correction instead.

Limits

Every arm, including the no-sentence arm, carried the same line in the system prompt: if you need feedback, call skill_feedback. That line is not a variable in this experiment. It plausibly pushes every arm toward the ceiling, and a ceiling is what a null result looks like. The no-sentence arm means the skill contains no such sentence. It does not mean the model had no feedback instruction.

Only skill_feedback is a real tool. The business actions inside the tasks are simulated, so this measures whether the model calls a tool it was told about. It does not measure whether it can complete a real workflow.

The arm order was fixed, the tasks were not shuffled, and each cell ran once, so time and server-side drift cannot be ruled out. The Flash build label is our own, as noted above; the API never returned a build identifier.

240 executions produced 240 unique terminal states, with zero ledger mismatches and zero empty finals. There were eight API errors: three in the V4 Pro step arm, four in the K3.1 Pro no-sentence arm, one in its footer arm. The footer error landed after a valid ledger row, which is why the primary count keeps that cell.

The rule we now run

Build the tool before you grade the sentence. If the action cannot be observed, you are grading prose, and prose is the one thing a good model always produces.

Then count the side effect instead of the self-report. One readable row per action, no judge.

Then put the instruction where the model's plan is, and expect the tool to do the work. Step 3 of 5 is a reasonable place for it. This run does not show that place matters more than the footer.

The footer did not become persuasive. It became auditable.

The harness, the pre-registration and the raw results are open: Dawncoral/wallaby-agent-rules.

Related from the same series: the three-regimes map and writes that report success.

Disclosure: Wallaby Token sells API access to Kimi K3, one of the models in this comparison.

Skill contracts vs. ceremony: a narrow retest

Dawncoral — Sat, 31 Oct 2026 03:09:22 +0000

Kimi K3.1 Pro failed six of ten clear tasks with no skill file. Give it a skill and it failed one. That is the number we expected. The number we did not expect: a thin shell scored the same nine out of ten as the full scaffold.

We ran 180 executions and 180 judgments across three models and three arms to check a claim we had published ourselves. The claim was that full structure starts to drag on strong models once the task gets fuzzy. Over 28 valid pairs on the main metric, we did not reproduce it.

If you maintain a SKILL.md or an AGENTS.md, the useful question is about which parts of the file a single run can see, and which parts only a maintainer can. The retest below is narrow on purpose.

The retest, and what it was built to kill

Two weeks ago we published a map with three regions. A weak model on routine work: structure pays. A strong model on routine work: a ceiling. A strong model on hard work: the freeze, where one-shot files beat the forge-built ones and the skill's own boundary clauses were quoted as the reason to stop.

The K3 postscript that failed to reproduce the map's third region used one judge, three to ten executions per cell, and no pre-registration. We treated that smoke test as a warning and built P2 to retest it.

Three models, three arms, twenty tasks. The models were DeepSeek V4.1 Flash 1017, Kimi K3.1 Pro and DeepSeek V4 Pro. The Flash build is named only in our local arm label, ds-v41-flash-1017-unverified; the API returned deepseek-flash, so treat 1017 as declared rather than confirmed. The arms were no skill, a full skill file, and a shell of the same file. Ten tasks were fuzzy, across five families. Ten were clear, with a mechanical pass condition. Each cell ran once. Clear tasks were checked mechanically; each fuzzy execution was judged twice.

The shell is the only new object here. We removed the description, the closing skill_feedback logging step, and the empty pitfall placeholder. The numbered task steps, the specific red lines, the safety constraints and the completion criteria stayed.

On the hard tasks, we did not detect a difference

The main metric was unjustified stop: the run quits while a safe partial result was still available. We pre-registered it, which is why the result is allowed to be boring.

Arm	Stops	Rate
Full scaffold	2	7.14%
Thin shell	3	10.71%

Our primary comparison keeps only cells where both arms returned normally and both judges agreed on this field. That left 28 pairs. Full stopped twice, shell three times. The difference is 3.57 percentage points in the shell's favor, exact McNemar p = 1. For unjustified_stop only, the preregistered full-favorable and shell-favorable sensitivity analyses gave p = 0.25 and p = 1 respectively; neither detected a significant structural difference.

Read that as narrowly as it is written. We did not detect the drag. We did not show the two arms are equal, and nothing here says a strong model needs a skill file in general.

The clear tasks are where the split shows

A clear task has a mechanical pass condition: required fields present, banned fields absent. Literal compliance, not a semantic quality score.

Model	No skill	Full	Shell
DeepSeek V4 Pro	5/10	8/10	8/10
DeepSeek V4.1 Flash 1017	7/10	7/10	8/10
Kimi K3.1 Pro	4/10	9/10	9/10
Pooled	16/30	24/30	25/30

The observed gains are in literal task-following scores, not general capability, and they are concentrated in V4 Pro and K3.1 Pro; both skill arms yield the same numerical gains for those models in this set. The Flash build barely moves. The skill is not making the models smarter. It is telling them what this job's output has to contain, which they cannot infer from a one-line task statement.

Read the order in that table as a flat list, not a ranking; these single-run literal scores do not establish a general capability ordering.

A shell gets the same numerical gain. That is the part worth keeping. The design does not isolate which sections produced it, so it does not show that the deleted material never matters.

The line that moved: contract versus ceremony

Our old axes were model strength and task fuzziness. Those still describe something. They do not describe what a skill file is doing at the moment it helps.

The axis that survived is what the file is for. A contract is what one run must satisfy: the steps, the red lines, the definition of done. Ceremony is what a maintainer does between runs: tuning the description, writing the failure ledger, accumulating pitfall notes, keeping the ritual alive. Contract changes the run. Ceremony changes the next file.

A strong model already knows how to work. What it cannot know is your banned field, your definition of finished, and which shortcut is a red line in your domain. Those sentences are the cheap part of the file. The ceremony is the expensive part, and a single run cannot see it. This run cannot separate which sentences did the work.

That reframes the freeze we published. The clauses that stopped the model in that run were boundary clauses written in a stop-unless-everything-matches mood. That is a contract problem. The model was obeying the file. We had written the wrong instruction.

Why this is not "delete your skill file"

The shell is still a substantial document. It keeps the numbered task steps minus the closing logging step, plus the red lines, the safety constraints and the completion criteria. A shell with the safety section deleted would be a different experiment, and probably a worse one. We did not run it.

All three arms carried a shared instruction telling the model to deliver any part it could complete safely and not to stall the whole task without cause. That instruction targets the main metric directly. It plausibly created a floor, and a floor is exactly what a null result looks like. The low stop rate may belong to the shared prompt rather than to any arm.

The sample is small in the ways that matter. Five task families, ten fuzzy tasks, one round per cell. The 28 pairs are model-by-task cells, not 28 different questions. The business tools in the tasks are simulated. The judges disagreed on 29 field-level items across 23 cells, and we kept those disagreements rather than resolving them upward. One no-skill judgment failed to parse and was excluded under the pre-registration; it did not affect the full/shell primary comparison. The Flash build label is our own, as noted above; the API never returned a build identifier.

What we now ask before shipping a skill

Three questions, in this order.

What must one run do to be finished? Write that down, including the banned outputs and the one shortcut that would embarrass you. This is the contract, and it is the part a strong model cannot invent.

Which parts does only a maintainer care about? The description wording, the ledger format, the note about a pitfall from three weeks ago. Those belong in a changelog, a lint, or a review checklist. Inside the skill file they are read once and then forgotten.

Where is the check that fails the run? A contract without an observable artifact is still a comment.

The size rule from our own measurements points the same way. Files grow because ceremony is easy to add and hard to delete. If the contract is four sections, the file should be close to four sections.

What this changed in our pipeline

In our own pipeline, we now keep the maintenance layer out of the file that one run loads. The forge pipeline still writes the ledger and the pitfall notes, because a maintainer reads them. It no longer pastes them into the file that one run loads, unless a task family has shown that it needs them.

That is a routing decision inside the skill, not a smaller skill for its own sake.

The honest summary is narrower than the headline. On these three models, with this file and this harness, we did not detect a full-versus-shell difference in unjustified stops, and the clear-task scores were close. This does not establish general equivalence or show that every strong model needs a skill file. What it does support is narrower: a contract-shaped file is worth keeping, and the maintenance layer is the part to question.

The harness, the pre-registration and the raw results are in the open repository: Dawncoral/wallaby-agent-rules.

Two related pieces from the same run of work: the map this retests and the file size law.

Disclosure: Wallaby Token sells API access to Kimi K3, one of the models in this comparison.

AI agent memory needs a verification step

Dawncoral — Fri, 30 Oct 2026 13:55:59 +0000

An AI can update MEMORY.md and still leave five other files describing the old project. We added a small CLI that makes those missed updates visible. It maps nine change types to the documents that need review, checks explicit premises, tracks contradictions, and only then marks a change verified.

The new extras/memory-governance/ pack in wallaby-agent-rules does this without a model, a database, or a service. One Python file. JSON for state. A fresh install takes about a minute.

The first run

From the repository:

python3 extras/memory-governance/memory_governance.py \
  --target /path/to/your-project init
cd /path/to/your-project
python3 scripts/memory_governance.py doctor

A fresh install ends with:

CONTRADICTIONS open=0
RECEIPTS pending=0
DOCTOR OK

init never overwrites an existing config or script. The core memory files stay independent, so you can add the pack to a project that already has MEMORY.md, NOW.md, and INDEX.md.

A rename is not done when one file changes

Tell the CLI which old term should disappear:

python3 scripts/memory_governance.py impact \
  --event file_move \
  --terms "old-component" \
  --record

In our clean-project demo, it found one reference in README.md and created receipt R-000001. The first verification failed:

NOT VERIFIED R-000001: 1 unresolved and 0 unread file(s)
  README.md (1)

That is the useful part. A log entry saying “renamed” would not have caught the second reference.

After updating the reference, verify runs the scan again. If an old mention is intentional history, you allow that path explicitly:

python3 scripts/memory_governance.py verify \
  --receipt R-000001 \
  --allow "archive/**"

The impact and verification stay as separate records in .memory/receipts.jsonl.

Premises have failure signals, not just good intentions

Some rules depend on a relationship staying true. A required file must exist. A JSON field must keep a specific value. A forbidden artifact must stay absent.

The pack turns those into four deterministic checks: file_missing, file_exists, text_contains, and json_equals. It does not run shell commands from the config.

There are three levels:

P0: stop the action, exit 2.
P1: require review, exit 1.
P2: show a warning without blocking routine work.

A manual premise cannot silently pass at P0 or P1. It needs evidence.

Contradictions keep their history

When a new statement replaces an old one, you record both and the evidence:

python3 scripts/memory_governance.py contradict add \
  --old "INDEX.md is optional" \
  --new "INDEX.md changes with files" \
  --evidence "DECISION-42"

Open contradictions make doctor return nonzero. Resolving one appends a new history entry instead of erasing the original claim.

What the scanner will not do

It does not edit your documents. It reports paths and match counts, not file contents. It skips .git, .credentials, .memory, dependency directories, and build output.

Unreadable or oversized files block verification. A file the scanner could not read must not become evidence that no old references exist.

The pack has 14 regression tests covering installation, all nine event types, premise exit codes, contradiction history, explicit scan limits, and false-green prevention. Those tests show that the checks behave as specified. They do not prove a memory-accuracy improvement across other projects.

Try it

The repository has the full README, the CLI, example JSON files, and the tests:

wallaby-agent-rules on GitHub

Start with doctor. Then give it one real rename from your project and see whether the first verification passes.

Wallaby Token builds open-weight inference infrastructure. We use file-based memory and governance on our own projects; the CLI is the part that keeps neighboring documents from being left behind.

Building a Letta Code custom provider: static first, live second

Dawncoral — Wed, 28 Oct 2026 16:13:07 +0000

Fresh sessions in Letta Code failed on their first message with Unknown model "wallaby/kimi-k3". Opening /model once fixed it. That symptom sent us down the wrong path for two releases, and the real cause had been public in the issue tracker for ten days.

Letta Code is the CLI for stateful agents from the Letta project (formerly MemGPT). A provider mod teaches its local runtime about a new model endpoint.

This is the build log behind wallaby-letta-provider 0.1.2, a small npm package that puts our gateway inside Letta Code's /connect flow. It is also the checklist we would hand to anyone shipping a provider mod: three lessons, one wrong turn, and a live catalog that now updates without a release.

What the package gives a Letta Code user today:

kimi-k3 with a 1M context window, image input, and effort levels from off to max
A live catalog: a new model id appears in /model once it is listed and priced, without another npm release
Costs read from the same public price file the gateway bills against

Why a mod instead of a config file

Letta Code's local backend wants provider metadata: a display name, an API style, a base URL, an auth variable, a model list with context limits and prices. A provider mod hands that metadata to the runtime at registration time, so the provider shows up in /connect and /model like a built-in one.

Letta's documented recipe is small. You call letta.providers.register("wallaby", {...}) with api: "openai-completions", a base URL, an env var name, and a list of models. Model ids must not contain a slash, which matters later. Provider mods are local-backend only; they do not add providers for agents managed through the Letta API.

Our first release did the obvious thing and also implemented listModels, so the catalog would track whatever the gateway serves. That decision produced the bug.

The bug we misdiagnosed

Version 0.1.0 shipped with listModels. A fresh session then failed on its first message with Unknown model "wallaby/kimi-k3", until you opened /model once and picked the model. After that, the session worked.

We read the symptom as a catalog timing problem: the dynamic catalog is empty at session start, so the first lookup misses. Version 0.1.1 removed listModels and shipped a static catalog. The symptom disappeared.

That was a workaround built on a wrong diagnosis. We filed issue #5147, and the project's integration bot answered the next day: dynamic catalogs do start empty in each session, but resolving a turn already refreshes the provider catalog and retries when the model is missing. The error we saw came from a different bug, where a saved model handle for a mod provider could pick up an extra provider/ prefix, producing handles like wallaby/wallaby/kimi-k3. Reselecting the model rewrote the handle, which is exactly why the /model re-select worked. The prefix bug was fixed in PR #5079, which closes the public root-cause report #4995, and shipped in v0.34.17.

So we retested v0.34.17 with listModels turned back on, using two brand-new agents and one headless prompt each. Both answered on the first turn, no /model visit needed.

One edge case survives. Resuming an agent whose model handle was saved before v0.34.17 still fails, because the lookup runs against the openai provider even though the saved handle is correct; re-selecting the model clears it. We asked whether a migration is planned and have no answer yet. A fix does not always reach saved state.

The diagnosis was theirs, the fix was theirs, and our static catalog had been hiding a bug that was already gone.

The published 0.1.2 tarball's header comment still records the original catalog-timing diagnosis. The repository note is corrected; the next release will carry it.

Two lessons came out of that. Search the issue tracker before shipping a workaround: #4995 had been public since October 8 with the full root cause, ten days before our 0.1.0. And keep the workaround after the fix, because a provider that fails closed is worse than one that carries a static fallback.

What 0.1.2 does instead

The current release keeps both paths and orders them:

The static catalog registers kimi-k3 immediately, so the first turn of a fresh session always resolves.
listModels then merges whatever the gateway's /models endpoint returns. Ids containing a slash are dropped, because Letta model ids cannot contain one.
A model without a matching entry in the public pricing.json is skipped rather than shown at $0. A wrong price is worse than a missing row.
If /models returns 401 or 403, the mod raises Wallaby rejected this API key (HTTP 401) when the model list refreshes. A headless run that never refreshes the list can still fall back to the host's default model instead of surfacing it.

Three metadata fields that decide the experience

Most of a provider's quality is metadata, not the HTTP call. Three fields mattered most.

Thinking levels. The model schema accepts a thinkingLevelMap that maps each level to a provider-specific value; it comes from pi-ai's Model type rather than the provider recipe docs. We map off to none, minimal to minimal, low to low, medium to medium, high to high, and max to max, with xhigh folded into high. We verified on the endpoint that none, minimal, and max are all accepted and that reasoning_tokens responds.

Image input. A model declares input: ["text", "image"]. We checked that the gateway accepts an image_url data URL before declaring it; a 1x1 PNG came back with the pixel described correctly. Declaring image support without that check is how you get a 400 on the first paste.

Cost. The model entry carries per-million input, output, and cached-input prices. We read them from the public price file and keep the static values in sync with it. If the number in the picker disagrees with the bill, users stop trusting the picker.

Living with a two-day release cadence

letta-code shipped six tagged releases in ten npm publishes between October 18 and October 28, from v0.34.17 to v0.34.26. That cadence is the real maintenance cost of a mod, more than the code itself.

Our routine is small: run the unit tests, run one headless turn against a fresh agent on the latest release, and read the provider API surface when a release lands. The mod is one TypeScript file with no dependencies, so a break surfaces in minutes. The npm package had about 490 downloads in its first nine days, which is enough to make a silent break embarrassing.

If you write a mod, pin your repro to a version, keep a static path that cannot fail, and assume the API will grow. Ours did: listModels and connect fields arrived as documented surface, and the model schema already carried a thinking-level map.

Try it

letta install npm:wallaby-letta-provider

Then /reload, /connect wallaby, /model, and pick wallaby/kimi-k3. Kimi K3 runs $2.70 per million input tokens, $13.50 output, and $0.27 for cached input, 10% below the official list; new accounts start with $0.50 of trial credit.

What comes next

The next model we add needs no code change: an entry in /models and a price in pricing.json, and the provider picks it up. That is why the live path exists, and why the static path stays.

Disclosure: Wallaby Token sells API access to Kimi K3, and wallaby-letta-provider is our package. The Letta issue and the release notes cited here are public.

If you want the install steps rather than the build log, the Letta Code walkthrough covers them in four commands. Running a different terminal agent? The pi setup is the same pattern with a different host. To try the endpoint behind the mod, wallabytoken.com gives new accounts $0.50 in trial credit, no card required.

The Model Is Not the Bottleneck. The Routing Is.

Dawncoral — Fri, 23 Oct 2026 17:38:13 +0000

I still route work by hand: three models, two agent sessions, and a text file of handoffs. A task arrives, I read it, decide whether it needs the fast model or the slow one, write the prompt, and pass it along. That decision is the most valuable thing I do all day, and it is the least systematized.

The obvious response is to wait for a bigger model. I don't think that is the bottleneck. The bottleneck is the routing decision: which kind of thinking a task deserves, and what evidence would make me change my mind.

Daniel Kahneman's fast and slow thinking is a useful metaphor here, but it is not a model-size chart. Fast thinking is not a small model. Slow thinking is not a large one. Fast is compiled: a trigger, a rule, a template, a cached playbook. Slow is deliberate: decomposition, multiple candidates, verification, a human checkpoint. A frontier model can be fast and wrong. A mid-tier model can be slow and right if the process around it is good.

Fast and slow are paths, not model sizes

Path	Use it when	Mechanism	Cost
Fast	Known, reversible, low stakes	Reflex, rule, template, cache	Low
Slow	Novel, irreversible, ambiguous, high stakes	Decompose, generate, verify, decide	High

The fast path should not call a model

The first surprise is that the fast path should usually not call a model at all. If a task has a known trigger and a known action, the fastest path is a function: when X, do Y, because Z. The model only enters when the trigger is ambiguous. That is why a cheap model with a big memory is still slower and less reliable than a good reflex. You are paying for inference to rediscover something you already knew.

Slow is a process

The second surprise is that slow is not "ask the biggest model." Slow is a sequence. Decompose the task. Retrieve the relevant context. Generate independent candidates. Verify them. Decide. A bigger model helps at every step, but it helps less than a tight harness. I have watched a well-scoped refactor go better on a mid-tier coding model with a checklist than on a frontier model with a vague prompt. That says more about the harness than about the frontier model.

Multiple models are useful, but not as a free-form debate. Independent candidates plus a verifier beat a round table. Debate tends to converge on the most confident voice, which is not the same as the best answer. Use debate for values and ambiguity. Use verification for facts and code.

Choosing a model is the easy part. The harder problem is deciding when the fast path has stopped being true.

A reflex is a bet

A reflex is a bet that the world repeats. Most of the time it does. The job is to notice the times it doesn't. Every reflex needs a life cycle, not just a definition:

trigger:
action:
scope:
confidence:
evidence:
last_validated:
failure_count:
owner:

A strong reflex can be promoted to a higher layer: personal, team, department, company. Promotion should require evidence across contexts, a low failure rate, and an owner who will answer for it. Repetition alone is not evidence. A reflex that fires a thousand times and is wrong ten percent of the time is not a strong reflex; it is a superstition with a good memory.

Weak reflexes need a way out. Some should decay. Some should be challenged. Some should be retired. The immune system metaphor is useful: white blood cells scan for contradictions, lymphocytes go after a specific bad reflex. Passive decay alone is not enough, because a stale reflex can sit in the fast path for months, quietly costing you the one case where it no longer applies.

The same applies to the slow path. A slow path that never becomes a reflex is just a permanent tax. The point of deliberate work is to compile some of it: a decision that was hard once becomes a checklist, then a rule, then a reflex. The system gets faster without getting dumber.

The human is the sensor

Then there is the part that is harder to engineer: the human.

The model has read more than I have. It has not sat in a customer call at 2 a.m., or watched a launch go sideways because one sentence on a pricing page was ambiguous. The gap is experience, not knowledge. The only way to give the system that experience is to record the decisions, not just the outcomes.

A preference is cheap. A decision trace is expensive, and worth it:

options:
evidence:
choice:
rationale:
outcome:
post_use_review:

I built an architecture assistant around this idea. It reads our system definition, our failure cases, and a set of positive and negative examples. It is good at producing options; judgment is still the human's job. What helped was not more documentation; it was the trace: what I saw, what I considered, why I chose, what happened, and what I would do differently. Positive and negative cases helped, but they were sparse. The trace was denser, and it was the only thing that let the assistant see the shape of a decision rather than the shape of an answer.

This is also why "human in the loop" is not a checkbox. A thumbs-up is an opinion. A two-week outcome is evidence. If you only collect preferences, you train the system to sound like you. If you collect decisions and outcomes, you train it to think like you in the situations you have actually survived.

A portfolio of local optima

The worry is local optima. It is a real worry. A system that learns from its own decisions can get very good at the cases it has seen, and very confident in the wrong places. The answer is a portfolio of local optima, plus a way to retire one when the world moves.

That means a few unglamorous things: a champion and a challenger, an exploration budget, a red team, a scheduled retirement review. Not a single global optimum. A set of local optima that can be recombined. Success, in a changing environment, is the best set of answers you can still revise, not the single best answer.

Four components

If I were building the routing layer today, I would start with four components:

A router. Score each task on novelty, ambiguity, stakes, reversibility, and dependency. Route to fast, slow, or escalate. The router should be allowed to say "I don't know."
A reflex registry. Every reflex has a trigger, an action, a scope, a confidence, evidence, a last-validated date, a failure count, and an owner. Promotion is a decision, not an accumulation.
A model portfolio. Model cards, not a leaderboard: what each model is good at, what it costs, how much context it needs, how it behaves with tools. Route by fit, not by rank.
A decision trace. Options, evidence, choice, rationale, outcome, post-use review. This is the data that teaches the system how you judge, not just what you say.

Start with one domain. Measure four numbers: reflex hit rate, escalation rate, cost per successful task, and rework rate. If the fast path never escalates, it is not fast; it is reckless. If the slow path never compiles a reflex, it is not learning; it is just expensive.

The schema is the easy part. The discipline is writing the decision down before you know the outcome.

The model is the CPU. The routing layer is the operating system. We have spent a lot of energy making the CPU faster. The OS is still a person with a clipboard.

That is the layer I want to build next.

Disclosure: I work on Wallaby Token, an inference platform for open-weight models. This essay is about the routing layer in my own workflow; it is not a product announcement.

Memory Is Not Written. It's Audited.

Dawncoral — Thu, 22 Oct 2026 16:09:57 +0000

Here is a failure mode nobody screenshots. Your project log says the checkout redesign shipped on Tuesday. Your current-state file still lists it as in flight, waiting on assets. Both files are "your AI's memory." One of them is lying, and nothing in the system is going to tell you which.

The files are plain text: a dated log of what happened, a current-state file, and a few rule files an agent reads at startup. No database, no service. That is the point, and also the problem.

We found our own version of this the unglamorous way: not by noticing, but by writing a script that notices. It reads the dated log, reads the state file, and diffs what the one claims against what the other shows. The first run flagged entries we would have sworn were accurate. That is the entire argument of this post: write discipline always loosens, so memory needs an audit layer, not more discipline.

Disclosure: Wallaby Token operates an OpenAI-compatible inference API. The memory system described here is open source, MIT licensed, and is the exact setup our own agents run on.

The two lies memory files tell

Once we started looking, the drift came in exactly two shapes:

The contradiction. The log records a closure (shipped, fixed, merged, published) and the state file still carries the same topic as open work. Usually nobody updated the state file, because updating it was part of the closeout, and the closeout got skipped on a Friday. The log is append-only and honest; the state file is edited and forgetful. Left alone, the gap compounds until the state file is a museum of things you finished months ago.

The rumor. The log claims done, but points at nothing: no file, no ticket, no link. A closure you cannot verify is indistinguishable from one that never happened. This one is subtler, because the entry feels complete when you write it. Six weeks later, "fixed the retry bug" with no pointer is a rumor you told yourself.

Both are boring. That is what makes them dangerous: there is no dramatic failure, just a slow divergence between what your records claim and what is true.

Why mechanical, not motivational

The obvious fix is a sterner resolution to keep the files in sync. We tried that. It works for about two weeks. We know, because our audit script is how we found out it had stopped working.

So the rule we now run is: the human ritual writes, the machine audits. The closeout ritual (from our last post) is still the entry point; it sorts each session's output into memory, log, state, or the bin. But rituals have bad weeks, so a scheduled script checks the results. And it checks the one thing the ritual cannot check about itself: whether the story the log tells and the story the state file tells are still the same story.

The script is deliberately dumb. It does keyword matching, not semantic judgment, and it reports candidates rather than verdicts; a human decides which file is stale.

CLOSURE_WORDS = re.compile(
    r"\b(shipped|ships|done|closed|closes|fixed|fixes|merged|published|"
    r"released|resolved|launched|completed|landed|deployed|closed out)\b", re.I
)

That regex is the whole semantic layer. No model, no embeddings, no configuration file. Dumb is the feature: a smarter audit would need tuning, and tuning would grow its own maintenance ritual. A dumb one just runs.

What's in v3.1

The audit ships today in wallaby-agent-rules as templates/reconcile.py: zero dependencies, same exit-code contract as the health check (0 clean, 1 findings), two scans:

Closure contradictions: log entries that claim something is done while the same topic still sits in the state file's "in flight" section.
Evidence-free closures: "done" entries with no path, ticket, link, or backtick reference to verify against.

New installs get it by default: the L0 and L1 prompts in PROMPT.md now build six files instead of five. Existing installs need exactly one step: copy templates/reconcile.py into your project as scripts/reconcile.py and run it weekly next to the health check. The L2 upgrade check detects v3.1 by the file's presence, and like health_check.py, the script is scaffolding: it holds none of your content, so upgrades can offer to swap it outright.

The honest limit

A keyword diff will flag some false positives: a closure whose topic overlaps unrelated open work, a pointer written in a format the script doesn't recognize. We accept that trade knowingly. The audit's job is to make drift visible, not to be right about it. A candidate you dismiss in three seconds costs three seconds. A contradiction nobody surfaces costs you the week you spent trusting a file that had quietly become fiction.

The harder limit is the one from the last post, unchanged: none of this survives if you never run it. Hang the audit on the same schedule as the health check, and let the machine be the part of the system that never gets tired.

What the audit led to

An audit makes drift visible. It does not stop it, and after enough weekly findings you stop treating them as incidents and start reading them as a design signal. Two files can disagree because nothing forces them to agree at the moment of writing.

So we moved every memory change behind one entrance. Twelve write surfaces go through a single writer with a diff preview and a test suite of 37 cases. The source of truth is an append-only event stream; the SQLite index is derived, and we rebuilt 1,263 of 1,263 records from it in one command to prove it. Corrections are appended too: when 403 timestamps turned out to be missing seconds, we added correction events instead of editing history.

The validators run in shadow first. They report for two weeks before they are allowed to reject a write. A check that can block on day one is a check nobody trusts.

What we run, and what we publish

The reconcile audit is the third layer of the open stack: the file conventions and health check from v2, the closeout ritual from v3, and now the consistency audit, all in wallaby-agent-rules, MIT licensed, all running daily on our own project. The design principles are in AI remembers, you find; the maintenance layer is in Your AGENTS.md is rotting. And in 1.0.0 the stack grew a write-side counterpart to this audit: a probation period for new memories, where the agent proposes and you approve.

Our internal version of this script checks more (size budgets, decision ledgers, registration drift), because our workspace grew those needs. What we publish is the part every project needs on day one: the diff between what you said you did and what your files say is true. We also outsourced one audit on purpose: we paid Claude to review our own blog — a second pair of eyes with no stake in our conclusions.

Get started

New here? The fastest path is the L0 prompt in PROMPT.md: paste two lines, and the audit ships with the install. Already running the system? Copy templates/reconcile.py into scripts/ and run it once today. If it comes back clean, congratulations; your log and your state file agree. If it doesn't, you just found out what the audit is for.

Related: Your rule file says "later." That's why it never happens. — three incidents where one deferral word was the whole bug.

Why your AI agent's memory says it saved but didn't

Dawncoral — Mon, 19 Oct 2026 15:32:37 +0000

On October 13, a user of agentmemory, an open-source memory system for coding agents, filed an issue with a code block that should not exist. Two memory_save calls, two returned IDs, and about six days later, no memories. The engine had hung on October 7. It held its listening sockets open, never accepted a connection, and the client kept confirming saves that never reached disk. Call it the state-assertion bug: a system's report about its own state diverges from the state.

It is not a retrieval failure. We think it is the most expensive failure class in agent memory, because it corrupts the one asset you need to recover from every other failure, the record of what happened, and it does so one green check at a time.

This post takes three threads of that class from agentmemory and pairs them with the four escalating fixes we ran on the same class inside our own memory discipline, over two days in October. The short version: a status claim is not evidence. Make the claim name the check that would prove it, make the check read the live system rather than the record of it, and make the wrong thing impossible before you make it visible.

A receipt from a till the store never reads

The issue is titled "hung agentmemory was not saving but giving ids," and the body is nearly boring, which is what makes it useful. The reporter ran two saves and pasted the responses:

{"saved":"mem_mv6tghgk_02a5a2f023b8"}
{"saved":"mem_mv6tghzw_4aba68f9c6f7"}

A later session could not find either one. The reporter's diagnosis: the engine had been hung for about six days, holding its listeners but never accepting, so the UI showed zero memories while both saves returned IDs for rows that never touched disk.

The follow-up pull request explains the mechanism better than the report does. When the health check or a proxied call fails, the MCP shim writes the memory to a local fallback file, ~/.agentmemory/standalone.json, and replies {"saved": id}. Nothing copies that file to the server. The fallback is announced only on stderr.

That last sentence is the whole story. The save did happen. The receipt was accurate about the ID and silent about the address, and the address is the only part that matters, because a memory in a file the server never reads is a memory you will never recall.

The open patch does not prevent the loss. It prevents the lie: fallback saves and deletes would carry storage: "local" and a warning that the server did not confirm the call. A patch that makes a failure legible is worth more than a patch that makes it rare, because the rare one still happens on a day you are not looking.

Not saving, or saving somewhere else?

That pull request also shows the second form of the same bug: a labeling problem wearing a storage costume. The user's mental state is "it is not saving." The system's state is "it is saving somewhere the server does not read." Both sentences are true, and the interface offers one word for three different outcomes: saved, queued, and gone.

This shape is everywhere: an offline write queue, a mock returning fixtures, a cache serving stale values. The write path is not wrong. The vocabulary is.

The test we now apply to any status field is whether it can be false. A field that always reads saved is decoration, not a status. If the server did not confirm, the response should say the server did not confirm, and it should name the store that actually holds the bytes. The patch here does exactly that, and the interesting part is how small it is. One field and one warning, in the place where the user is already looking.

Contrast that with the original signal. The fallback was announced on stderr, which is where truthful information goes to be ignored. A warning that requires a human to tail a log during an agent loop is a note in a drawer.

`success: true` after the data is gone

On October 16, a second report took the same shape through a different door. POST /agentmemory/import with strategy: "replace" deletes every existing scope before it checks the optional sections. Send lessons as an empty object, and the response is success: true with every lesson gone. Send it as a string, and the call throws after the deletes. Either way the store is empty by the time anyone finds out.

The proposed fix validates before deleting. That is the strongest shape in this post, because it does not depend on anyone reading a warning at the right moment. A malformed section is rejected before anything is lost.

Why do tests miss this? Because each step is correct alone. Replace does delete scopes. The validator does reject malformed sections. The handler does return success. The bug is the order, and order rarely gets its own test until someone has been burned by it.

Why is this class so expensive?

Retrieval errors announce themselves. You ask for a fact, get the wrong one, and notice. A state-report error waits. You build the next three steps on it, and when something finally breaks, you debug using a record that is already false.

Then comes the calibration cost. Every time the interface says fine and it isn't, you pay twice: once for the lost write, and once for the habit of not checking. That habit is the expensive part, and nothing in a dashboard measures it.

The benchmark layer does not cover this. agentmemory publishes 95.2% R@5 on LongMemEval-S from its own run, reproducible from the harness in the repo. That number is real, and it says nothing about whether a write reached the store. Retrieval quality and write truthfulness are different contracts. A system can be excellent at the first and still fail the second on a Tuesday, which is roughly what happened here.

None of this is an argument against the project. The tracker is public, the fix proposals are public, and the reports are specific enough to reproduce. The interesting fact is that a team this careful about evaluation still shipped the failure. Carelessness is the wrong explanation. The cause is a status message that promises more than the system can know.

The four rounds we ran on the same class

We recognized the shape because we have been losing to it for months. Our own memory discipline makes claims about a live project: what shipped, what is still in flight, what was decided. Four times in two days in October, one of those claims was wrong, and each fix was a stronger shape than the last.

The first fix added a question to our instructions. Before stating a status, ask whether this session actually observed it. Within a day we produced two wrong status sentences, one built on a snapshot that was twenty-five days old and one that called an item pending after an exception had already closed it. A question in a document is a reminder, and reminders are read by the same mind that is about to be wrong.

A script replaced the question the next day. It reads five sources in a fixed priority order, verdicts first and stale plans last, and prints which source says what. It also added a write-back rule: when an item closes, the plan line gets a pointer. The script was correct. What failed was the installation. We recorded that the check ran before status sentences, and the file that would have told anyone to run it never contained that line. The gate existed in the report about the gate.

Then came a machine light. A daily scan flags log entries that contain status words without an evidence anchor. That scan found real problems, and it still missed the failure that mattered, because it scanned the log while the failure happened in a sentence about to be spoken in a conversation. The sensor was mounted in the archive while the leak was in the room.

The fourth fix moved the check into the generator. A live mode reads the draft's front matter, the built output, and the live page before a status claim can be written, and the drafting template has a field for the evidence that cannot be left blank. The class recurred afterward in a sentence rather than a document. We count that as evidence for the ranking below, not against the check, because a check only covers the surface it is mounted on.

Fix shapes, ranked by when they act

Shape	When it acts	What it gives you	Example
Make it impossible	Before the action	The bad state cannot occur	Validate before delete (#1576)
Wire it to the moment	At the action	The reader sees the truth at the moment of choice	Response says `storage: "local"` (#1563)
Detect after the fact	After the action	You can find it later, if you look	Daily scan for unevidenced claims
Remember to remember	Depends on the moment	Nothing you can rely on	Our self-check fixes: 0 for 4

Rank the shapes by where the failure can enter, not by how much effort the fix takes. The weakest shape is also the one that feels most like progress, because writing a rule feels like doing something, and the day you write it is usually the day the last failure is still fresh.

Across our four rounds, the first three rules were all true. The question was right, the script was right, the scan was right. They failed because they lived in a channel the decision maker does not visit at the moment of decision. That is the same defect as a fallback announced on stderr, and it is why we now rank fixes by when they act.

The checks that hold

Our rule now has two halves. A positive claim names the check that would prove it, and that check reads the live system, not the record of the live system. A negative claim, the kind that says not done, does not exist, or cannot be done, has to answer three questions before it leaves the desk:

Which source I have not checked is most likely to hold the counterexample?
What evidence would overturn this conclusion, and did I look for it?
Which related fact did I already see, and then ignore?

These questions are mechanical on purpose. They are the cheapest substitute we have for a machine that cannot be lied to, and they work best when the answer has to name a command and its output.

Four things to copy on Monday:

Read the write back. memory_save returns an ID. Recall that ID in a fresh session and assert the content, not the status code. One extra call buys you the only evidence that counts.
Label the fallback. If any client path can store locally, its response must say so in the result, not in stderr. If the only signal is a log line, you have no signal.
Order destructive operations. Validate, then delete, and test the malformed input path, because that is the branch where the store disappears.
Count your fixes. If the same class fails twice, change the shape. On the third failure, move the check into the tool that produces the claim, not into the notes around it.

What we have not solved

Our failure registry is still event driven and written by a human, which means the judgment stays human and the coverage depends on someone noticing. The live check reads the post source, the build output, and the live URL. It does not read the rest of what we claim about our own work. The patches here are mostly reactive in the same way: the fallback still writes to a file the server never reads, and the open patch makes it say so.

There is a specification question underneath, and we would like to see it taken seriously. When a memory API cannot confirm that a write reached durable storage, should the response say saved at all? Accepted, queued, and confirmed are three different promises, and most memory APIs collapse them into one. The first vendor to publish that distinction in its contract will make this entire bug class cheaper to avoid, for everyone, including the systems that already handle it well.

Until then, the discipline is the same one we apply to our own records: treat every status message as a claim, and check the claim against the live system before you build on it. The memory files we use are open source in wallaby-agent-rules, and the failure registry behind this post is internal, which is the next asymmetry worth closing.

Sources and further reading

agentmemory, the project discussed here (Apache-2.0, about 29,700 stars in October 2026)
Issue #1552: hung agentmemory was not saving but giving ids
PR #1563: say when a write only reached the local fallback store
Issue #1573: replace import deletes the store before it validates and PR #1576
agentmemory benchmarks, including the note on which numbers are the project's own run
Memory Is Not Written. It's Audited., the audit layer this post extends
We Ran Our Memory System Inside Hermes, the same discipline inside another engine
Muse Remembers. Who Governs the Memory?, on memory that lives inside the vendor

Disclosure: Wallaby Token sells inference for open-weight models, and we keep our own project records in a file-based memory discipline. We have a stake in this layer, so check the receipts and the links.