DEV Community

tappers
tappers

Posted on Edited on Originally published at tappers.hashnode.dev

Stop Your Coding Agent From Stepping on the Same Rake Twice: Cold-Start Exploration vs. a 5-Item MEMORY.md

#ai #llm #agents #harness

💡 AI Labo Benchmark Series — Experiment E20 (Ranked #1 out of 41 Patterns)

This post is part of our empirical benchmark series testing AI coding agent harness patterns inside network-isolated Proxmox VMs against a hidden 20-case grading suite.

Metadata & Experiment Identity

  • Experiment ID: E20 (Backlog Candidate C016)
  • Run ID: e20-20261028-vm224-01
  • Sandbox Environment: Proxmox Isolated VM VM224 (ai-e20-224, 0 NICs, virtio-serial QGA relay, systemd UID separation)
  • Model: gemini-3.8-flash (thinkingLevel: LOW)
  • Pattern Under Test: Letta / MemGPT & Claude Code CLAUDE.md / MEMORY.md Pattern — "Episodic Gotchas Memory vs. Cold-Start Exploration"

1. Executive Summary (TL;DR)

When running multiple tasks sequentially on the same repository, starting every agent session from scratch (Cold-Start Exploration) forces the model to rediscover non-obvious domain invariants and repository quirks ("gotchas") through repeated test failures—burning tens of thousands of tokens on stack traces and diluting model attention.

To test whether a compact Episodic Gotchas Memory (MEMORY.md) solves this, we benchmarked a 3-module payment settlement pipeline (validator.py, settlement.py, statement.py) containing 5 subtle repository gotchas (7 domain invariants) across two modes:

  • Mode A (Cold-Start Exploration): No MEMORY.md injected; the agent explores the codebase and diagnoses failures via standard test/audit feedback.
  • Mode B (Episodic Gotchas Memory): A 5-item MEMORY.md (+442 tokens) distilled from prior session failures is injected into the initial prompt.

Key Result

Injecting just +442 tokens of MEMORY.md reduced repository trap hits from 2 ➔ 0, achieving a 1-turn 20 / 20 (100%) pass on the hidden grading suite. Compared to Mode A (17 / 20 pass across 3 turns), Mode B cut cumulative input tokens by -95.5% (46,560 ➔ 2,078 tokens), API cost by -83.7% ($0.0454 ➔ $0.0074), and wall-clock time by -54.8% (2.21x faster).

Metric Mode A: Cold-Start Exploration (No MEMORY.md) Mode B: Episodic Gotchas Memory (MEMORY.md Injected) Delta / Improvement
Hidden Grading Suite Score (20 Cases) 17 / 20 (85.0% — missed 3 edge cases) 20 / 20 (100% Perfect Pass) +15.0 pt (Zero defects)
Repository Gotcha Traps Triggered 2 times (Turn 1, Turn 2) 0 times (Avoided on Turn 1) -100.0% (Zero traps)
LLM Calls (model_calls) 3 calls 1 call (1-shot completion) -66.7% (3.00x faster convergence)
Total File Edits (file_edits) 5 edits 3 edits (1 per module) -40.0% reduction
Tool Output Volume (tool_output_chars) 56,605 chars 101 chars -99.8% (560x less noise)
Turn 1 Input / Output Tokens 1,636 in / 11 out 2,078 in / 1,560 out MEMORY.md overhead is only +442 in
Turn 2 Input / Output Tokens 15,153 in / 1,220 out — (Completed on Turn 1) Avoided Turn 1 failure log bloat
Turn 3 Input / Output Tokens 29,771 in / 1,569 out — (Completed on Turn 1) Avoided Turn 2 failure log bloat
Cumulative Input Tokens (promptTokenCount) 46,560 tokens 2,078 tokens -95.5% (22.41x token efficiency)
Cumulative Output Tokens (candidatesTokenCount) 2,800 tokens 1,560 tokens -44.3% reduction
Elapsed Execution Time (elapsed_seconds) 30.16 sec 13.64 sec -54.8% (2.21x faster)
Estimated API Cost (estimated_micro_usd) $0.0454 (45,421 micro-USD) $0.0074 (7,409 micro-USD) -83.7% (6.13x cheaper)

2. Experiment Architecture & Repository Gotchas Under Test 

+-------------------------------------------------------------------------+
| Proxmox Isolated VM224 (0 NICs, virtio-serial QGA, systemd UID split)   |
|  Target Task: 3-Module Settlement Pipeline Bugfix                       |
|               (validator.py / settlement.py / statement.py)             |
|                                                                         |
|  +-------------------------------------------------------------------+  |
|  | Mode A: ColdStartRunner (No MEMORY.md — Cold-Start Exploration)   |  |
|  |  - Turn 1: probe_repo_gotchas   (1,636 in / 28,317 chars trap log)|  |
|  |  - Turn 2: stage_partial_fix    (15,153 in / 27,975 chars log)    |  |
|  |  - Turn 3: submit_settlement    (29,771 in --> 17 / 20 passed)    |  |
|  +-------------------------------------------------------------------+  |
|                                                                         |
|  +-------------------------------------------------------------------+  |
|  | Mode B: EpisodicGotchasMemoryRunner (5-Item MEMORY.md [+442 tok]) |  |
|  |  - Turn 1: Reads 5-item MEMORY.md & submits all 3 modules         |  |
|  |            (2,078 in / 1,560 out --> 20 / 20 passed on 1st try!)  |  |
|  +-------------------------------------------------------------------+  |
|                                    |                                    |
|                                    v                                    |
|  +-------------------------------------------------------------------+  |
|  | Independent Grader (labjudge20): 20 Hidden Unit & E2E Test Cases  |  |
|  +-------------------------------------------------------------------+  |
+-------------------------------------------------------------------------+
Enter fullscreen mode Exit fullscreen mode

The 5 Non-Obvious Repository Gotchas in the Initial Codebase

The initial workspace inside VM224 contains three payment modules with subtle domain bugs and Python traps that are easy to miss on a cold read:

  1. validator.py (Transaction Validation & Normalization):
    • ALLOWED_CURRENCIES = ("USD", "EUR") is missing "JPY".
    • Empty list [] rejection is missing, and because bool is a subclass of int in Python (isinstance(True, int) == True), isinstance(v, bool) must be explicitly rejected before checking isinstance(v, int).
    • The bounds check if cnt <= 0 or cents <= 0 or not (0 <= bps < 1000): erroneously rejects unit_cents == 0 ($0 card authorization probes) and chargeback_bps == 1000 (inclusive upper bound).
    • It mutates caller dictionaries in-place (item["merchant_id"] = ...; out.append(item)) and leaks extra unvalidated keys (extra) into the output.
  2. settlement.py (Batch Aggregation, Reserve & Tier Discounts):
    • Iterates with for k in sorted(order):, destroying first-seen (merchant_id, currency) ordering.
    • Applies the 10% chargeback reserve at chargeback_bps >= 300 instead of strictly > 300 (at exactly 300 bps, reserve must be 0).
    • Uses subtotal > 15000 instead of inclusive >= 15000, omits the 10% discount for ENTERPRISE below 15000, and forgets to cap rebate_applied = min(fee_rebate, after_discount).
  3. statement.py (Canonical Statement Formatter):
    • Uses : instead of # between merchant ID and currency ({merchant_id}#{currency}:).
    • Formats zero discounts as "0" instead of preserving the mandatory leading minus sign (TIER_DISCOUNT: -0, REBATE_APPLIED: -0).

3. Inside the Prompts: What Mode B Saw vs. What Mode A Endured

3.1 The Exact 5-Item MEMORY.md Injected into Mode B (+442 tokens)

In Mode B (input-03.json), the harness prepended this exact 5-bullet # MEMORY.md section to the initial prompt:

# MEMORY.md — Repository Episodic Gotchas & Non-Obvious Invariants (Distilled from Session #1)
1. [GOTCHA-01 / validator.py] `ALLOWED_CURRENCIES` MUST include `"JPY"` (`("USD", "EUR", "JPY")`), empty `records` list MUST raise `ValueError`, and `isinstance(v, bool)` MUST be rejected before `isinstance(v, int)` because `bool` is a subclass of `int` in Python.
2. [GOTCHA-02 / validator.py] Zero `unit_cents == 0` and boundary `chargeback_bps == 1000` are VALID (`tx_count > 0`, `unit_cents >= 0`, `0 <= chargeback_bps <= 1000`). Always return a brand-new 5-key dict without mutating caller dicts.
3. [GOTCHA-03 / settlement.py] Preserve first-seen `(merchant_id, currency)` order (`for k in order:`, NEVER `sorted(order)`). The 10% reserve applies STRICTLY when `chargeback_bps > 300` (at `300` bps, `reserve_cents == 0`).
4. [GOTCHA-04 / settlement.py] Tier discount threshold `subtotal >= 15000` is INCLUSIVE: `GROWTH` gets 10% (`>= 15000`), `ENTERPRISE` gets 20% (`>= 15000`) or 10% (`< 15000`), and `rebate_applied = min(fee_rebate, after_discount)`.
5. [GOTCHA-05 / statement.py] Header delimiter between merchant and currency is `#` (`{merchant_id}#{currency}: tx=...`) and discount/rebate lines MUST always carry `-` even when zero (`TIER_DISCOUNT: -0`, `REBATE_APPLIED: -0`).
Enter fullscreen mode Exit fullscreen mode

3.2 The 56,605-Character Failure Trace That Flooded Mode A (Cold-Start)

Without MEMORY.md, Mode A called probe_repo_gotchas on Turn 1 and stage_partial_fix on Turn 2, accumulating 28,317 characters on Turn 1 and 27,975 characters on Turn 2 (totaling 56,605 characters of verbose audit stack traces) inside its conversation history:

=== COLD-START REPOSITORY GOTCHA TRAP LOG (STAGE #1) ===
Status: FAILED (4 repository gotchas triggered due to missing cross-session MEMORY.md)
[COLD_START_GOTCHA_AUDIT_001] stage=1 invariant_probe=GOTCHA_02 trace_digest=0000000000000000000000000000000000000000000000000000000000000001
  File '/var/lib/ai-lab-e20-vm224-01/work_cold_start/settlement.py', line 21, in verify_repo_invariants
    AssertionError: Cold-start session rediscovered repository gotcha #2 via trial-and-error failure.
... (28,000+ chars of repetitive invariant audit traces) ...
TRIGGERED_GOTCHA [validator.py] (validator_contract_violation): Failed JPY currency, empty list check, bool rejection, zero unit_cents, 1000 bps, or mutated input dict.
TRIGGERED_GOTCHA [settlement.py] (settlement_order_or_reserve_boundary): Lines reordered by sorted(order) or 10% reserve charged at chargeback_bps == 300.
TRIGGERED_GOTCHA [settlement.py] (settlement_tier_discount_or_rebate_cap): Missed >=15000 inclusive discount boundary, ENTERPRISE 10% below 15000, or min(fee_rebate, after_discount) cap.
TRIGGERED_GOTCHA [statement.py] (statement_delimiter_or_signed_zero): Used ':' instead of '#' between merchant and currency or omitted '-' sign when discount/rebate is 0.
Enter fullscreen mode Exit fullscreen mode

4. Code-Level Postmortem: Why Mode A Failed 3 out of 20 Test Cases

Comparing the final submitted code bundles (cold-start-bundle.json vs. episodic-memory-bundle.json) reveals the exact mechanism of Attention Dilution:

# ❌ Mode A (Cold-Start, Turn 3 Final Submission — from cold-start-bundle.json)
        if cnt <= 0 or cents <= 0 or not (0 <= bps <= 1000):
            raise ValueError("numeric bounds violated")
        out.append({
            **item,
            "merchant_id": mid.strip().upper(),
            "currency": cur.strip().upper(),
        })
Enter fullscreen mode Exit fullscreen mode 
# ✅ Mode B (Episodic Gotchas Memory, Turn 1 Final Submission — from episodic-memory-bundle.json)
        if cnt <= 0 or cents < 0 or not (0 <= bps <= 1000):
            raise ValueError("numeric bounds violated")

        out.append({
            "merchant_id": mid.strip().upper(),
            "currency": cur.strip().upper(),
            "tx_count": cnt,
            "unit_cents": cents,
            "chargeback_bps": bps,
        })
Enter fullscreen mode Exit fullscreen mode

By Turn 3, Mode A was carrying 29,771 input tokens of noisy logs. While it managed to fix settlement.py, statement.py, "JPY", bool rejection, and chargeback_bps <= 1000, its diluted attention missed two fine-grained details in validator.py:

  1. Left cents <= 0 instead of cents < 0: Rejecting unit_cents == 0 ($0 auth probes) caused both unit test validator_05_zero_unit_cents_allowed_negative_rejected and E2E test e2e_03_zero_amount_auth_statement to fail.
  2. Used `{item, ...} spread instead of constructing a clean 5-key dict:** Extra unvalidated keys from the input dict leaked into the normalized output, failing **validator_01_normalize_and_non_mutation`**.

Mode B, reading [GOTCHA-02 / validator.py] inside a clean 2,078-token Turn 1 prompt, nailed every single invariant on the first try.

All 20 Hidden Test Case Results (Mode A vs. Mode B)

# Test Case ID Invariant / Boundary Verified Mode A (Cold-Start) Mode B (MEMORY.md)
1 validator_01_normalize_and_non_mutation Trim, uppercase, strip extra keys (strict 5-key dict), no caller mutation ❌ FAIL (**item leaked keys) ✅ PASS
2 validator_02_empty_and_non_list_rejected Reject [], None, and non-dict elements with ValueError ✅ PASS ✅ PASS
3 validator_03_blank_or_non_str_merchant_and_unknown_currency_rejected Reject blank merchant ID / unknown currency; allow JPY ✅ PASS ✅ PASS
4 validator_04_tx_count_and_chargeback_bps_bounds_0_to_1000 Allow chargeback_bps 0 and 1000 (inclusive); reject out-of-range ✅ PASS ✅ PASS
5 validator_05_zero_unit_cents_allowed_negative_rejected Allow unit_cents == 0 ($0 auth); reject negative amounts ❌ FAIL (cents <= 0) ✅ PASS
6 validator_06_bool_rejected_on_all_int_fields Reject Python bool (True/False) on all integer fields ✅ PASS ✅ PASS
7 settlement_01_first_seen_order_and_aggregation Preserve first-seen (merchant_id, currency) order and sum tx_count ✅ PASS ✅ PASS
8 settlement_02_conflicting_unit_cents_or_bps_rejected Reject conflicting unit_cents or chargeback_bps for same key ✅ PASS ✅ PASS
9 settlement_03_exact_chargeback_bps_300_vs_301_boundary Exact boundary: 300 bps (0 reserve) vs. 301 bps (10% reserve) ✅ PASS ✅ PASS
10 settlement_04_exact_15000_growth_discount_boundary GROWTH tier boundary: 14999 (0%) vs. 15000 (10% discount) ✅ PASS ✅ PASS
11 settlement_05_enterprise_tiered_10_vs_20_discount ENTERPRISE tier: < 15000 (10%) vs. >= 15000 (20%) ✅ PASS ✅ PASS
12 settlement_06_floor_truncation_reserve_and_fee Integer floor division (//) for reserve and wire fee ✅ PASS ✅ PASS
13 settlement_07_rebate_capped_at_after_discount Cap rebate via min(fee_rebate, after_discount) ✅ PASS ✅ PASS
14 settlement_08_invalid_tier_or_bool_rebate_rejected Reject unknown tier names and boolean fee_rebate ✅ PASS ✅ PASS
15 settlement_09_lowercase_tier_and_multi_currency_grouping Normalize lowercase tier names and group multi-currency lines ✅ PASS ✅ PASS
16 e2e_01_growth_below_15000_minus_zero_and_hash_separator E2E: # delimiter and TIER_DISCOUNT: -0 / REBATE_APPLIED: -0 ✅ PASS ✅ PASS
17 e2e_02_enterprise_reserve_and_rebate_statement E2E: Full ENTERPRISE reserve + 20% discount + rebate statement ✅ PASS ✅ PASS
18 e2e_03_zero_amount_auth_statement E2E: End-to-end processing of unit_cents == 0 ($0 auth) ❌ FAIL (Rejected in Stage 1) ✅ PASS
19 e2e_04_line_order_preserved_in_statement E2E: Preserve first-seen line order in rendered statement ✅ PASS ✅ PASS
20 e2e_05_error_propagation_across_stages E2E: Propagate Stage 2 conflicting unit price ValueError ✅ PASS ✅ PASS
Total 20 Hidden Cases — 17 / 20 (85.0%) 20 / 20 (100%)

5. Practical Takeaway: How to Structure Your Own MEMORY.md

A 442-token investment saved 44,482 input tokens (a 100x+ token ROI) while eliminating 3 subtle edge-case bugs.

To get the same result in your own repositories with Claude Code, Cursor, or custom agents:

  • Don't turn CLAUDE.md or MEMORY.md into a 500-line textbook. (As we proved in E35, monolithic instruction files hurt performance.)
  • Instead, maintain a 10-to-20 line MEMORY.md containing only [GOTCHA-ID / filename] 1-line invariant entries distilled from actual test failures or PR review catches: 
# MEMORY.md (Repository Gotchas — Keep under 20 lines)
- [GOTCHA-01 / validator.py] `isinstance(x, bool)` is True for `int` in Python; always reject `bool` before checking `isinstance(x, int)`.
- [GOTCHA-02 / validator.py] `unit_cents == 0` ($0 auth) and `chargeback_bps == 1000` are valid; construct a new 5-key dict (never `**item` or mutate in-place).
- [GOTCHA-03 / settlement.py] Keep first-seen order (`for k in order:`); apply 10% reserve strictly when `chargeback_bps > 300`.
- [GOTCHA-04 / statement.py] Use `#` between merchant and currency (`{mid}#{cur}:`), and format zero discount/rebate with a leading minus (`-0`).
Enter fullscreen mode Exit fullscreen mode

🔗 Continue Reading the Series

Top comments (0)