Context Drift on a Free Server: A Reproducible Memory Stress Test

A developer asks a free coding model to refactor a service in twelve steps. Steps one through three pass cleanly. By step five, the model renames a function that was already renamed. By step seven, it reintroduces a deleted dependency. The model did not get dumber. It lost context. This is context drift. It is the silent killer of long AI-assisted refactors.

Reasoning ledgers are a popular pattern for AI agents. The pattern gained traction in recent agent discussions. Write decisions to a file between steps. Send only the ledger and the current code on the next call. The idea sounds good. This post turns it into a measurable test. MonkeyCode is an open-source project with a free tier. That tier includes model access, a token budget, and a free server. Disclosure: This article was prepared as part of MonkeyCode's product outreach. The harness below runs against that setup. A full run costs nothing.

Why context drift happens

Free coding models run stateless calls. Each request starts fresh. The model only knows what the prompt contains. Long tasks force a choice. Paste the full history and burn tokens. Send a summary and lose details. A ledger offers a third path. It keeps decisions without replaying every token.

A previous post measured the 10 million token budget on this setup. This post measures something different. It measures memory. A model with a huge budget can still forget a rename from three steps ago.

The experiment

Use a twelve-step refactoring task. Each step depends on the previous one. Each step has a passing test and a hidden constraint. The constraint is the memory check. A model that forgets a rename fails the test.

Step	Change	Memory constraint
1	Add Todo type	None
2	Rename done to completed	No done anywhere
3	Add in-memory store	API: add, list, setCompleted
4	Rename toggle to setCompleted	No toggle anywhere
5	Add title validation	Empty titles rejected
6	Extract validation to a module	Behavior unchanged
7	Add an HTTP handler	Uses Request/Response
8	Rename db to store	No db identifiers
9	Handle unknown IDs	Returns 404
10	Split the handler into two files	Same routes
11	Add a /stats endpoint	Counts come from the store
12	Remove dead exports	No unused exports

Score one point per step. A step passes only when tests pass and constraints hold. Maximum score is twelve. Run the task twice. 1. Full conversation history. 2. Reasoning ledger only. 3. Compare the scores.

The seed files

Start with a tiny service. The seed file is small on purpose.

// src/todo.ts
export interface Todo {
  id: string;
  title: string;
  done: boolean;
}

// src/todo.test.ts
import { test } from "node:test";
import assert from "node:assert/strict";
import { Todo } from "./todo.js";

test("todo has id, title, done", () => {
  const t: Todo = { id: "1", title: "write test", done: false };
  assert.equal(t.done, false);
});

Each step adds its own test file. The pattern stays the same. Assert the behavior. Grep for the constraint. The full suite is twelve small files.

The harness

The harness runs the loop. It sends a prompt, applies the response, verifies the step, and appends to the ledger.

#!/usr/bin/env bash
set -euo pipefail

for STEP in $(seq 1 12); do
  PROMPT=$(cat "prompts/step-$STEP.md")
  CURRENT=$(cat src/todo.ts)
  LEDGER_TEXT=$(cat LEDGER.md 2>/dev/null || true)

  RESPONSE=$(call_model "$PROMPT" "$CURRENT" "$LEDGER_TEXT")
  apply_diff "$RESPONSE"

  if verify_step "$STEP"; then
    echo "step $STEP: PASS" >> results.csv
    append_ledger "$STEP"
  else
    echo "step $STEP: FAIL" >> results.csv
    exit 1
  fi
done

The model call is an adapter. Replace it with your provider's client.

call_model() {
  # Pseudocode. Swap in the SDK for your provider.
  curl -s "$ENDPOINT" \
    -H "Authorization: Bearer $API_KEY" \
    -d "$(jq -n --arg p "$1" '{prompt: $p}')" \
    | jq -r '.choices[0].message.content'
}

verify_step checks the constraint. Step eight, for example, greps for db.

verify_step() {
  case "$1" in
    8) ! grep -rn "db" src/ ;;
    *) npm test -- --test-name-pattern="step $1" ;;
  esac
}

The ledger format

Keep the ledger short. One decision per line. The next prompt sends the ledger instead of the full history.

## Step 3
Decision: store is an in-memory Map keyed by id.
Constraint: `toggle` renamed to `setCompleted`.
Files: src/store.ts, src/todo.ts

What to record

Record four numbers per run. Steps passed. Constraint violations. Tokens per step. Wall-clock time. The ledger mode should use fewer tokens than full history. That is the trade-off this test quantifies.

mode,steps_passed,constraint_violations,tokens_per_step,notes
ledger,12,0,1840,example row - replace with your run
full_history,9,3,4120,example row - replace with your run

The rows above are placeholders. They show the format, not a result. Run the harness to get real numbers.

Reading the score

Use the score to decide. The table below is a rubric, not a benchmark.

Score	Verdict	Action
10-12	Reliable for staged refactors	Use the ledger, review diffs
6-9	Usable with guardrails	Keep per-step tests, split prompts
0-5	Not for multi-step work	One change per call, human review

Where it breaks

Expect failures in specific places. Renames break first. The model forgets a constraint once the file grows. Deleted imports reappear. Error handling drifts across files. The harness is built to catch these exact failures.

Limitations

The harness tests one task type. It says nothing about greenfield code, debugging, or architecture. Free-tier models can change between runs. A passing score is not a permanent guarantee. Token budgets cap long sessions. The ledger trades tokens for reliability.

Skip this approach if you need deterministic output. Skip it if you cannot review generated diffs. Anyone running untrusted code should still use a disposable server.

Conclusion

The test is small. It answers one question: whether a free model holds context across a long refactor. The ledger pattern helps, but only measurement proves it. Run the harness against MonkeyCode's free tier and free server. Post your score. The data helps everyone.