Keep Memory Off the Remote Generator

Keep all durable memory off the remote generator. You own the cache, the identity, and apply rights. The remote model is a compiler, not a colleague.

A free remote server will not keep your session. It can retry, stall, or drop mid-stream. Design for that failure before you prompt.

Think about a CI worker you do not own. You ship a job ticket and then wait. You never leave the deploy key there.

Constraints

This remote generation path gives you three hard constraints. You still do not control end-to-end latency. You also do not control the server-side retries.

You do not control the remote process lifetime. The output remains untrusted text even after success. It is not a merged commit by itself.

Secrets must never ride inside the prompt body. Apply rights must never live beside the model. Mix those two and you built a trap.

Availability on a free path is not an SLA. You should treat silence as a normal outcome. Your local plane remains the system of record.

Network partitions are not rare events here. The compiler host can vanish between poll calls. Your ticket must still mean one apply.

Give every compile job a local time budget. The budget is a constraint, not a guess. When it expires, the key stays unapplied.

Data flow

Start from a frozen worktree, not chat history. Hash the tree, the intent file, and a nonce. That tuple becomes your only idempotency key.

HEAD is not enough for a dirty tree. Uncommitted edits still sit outside that commit hash. Freeze the worktree or refuse to generate.

A fresh nonce stops accidental key reuse tomorrow. A wall clock would make identical jobs diverge. Keep time out of the identity tuple.

You send only the envelope the compiler needs. The remote side returns bytes or a failure. You write those bytes under the key.

Data still moves one way during the generate step. The envelope leaves your host as a job. The artifact returns as bytes under the key.

Apply must read the local artifact file only. It should never read the live HTTP stream. A dead stream can still leave a file.

If the server retries, the key already exists. The second payload collapses into the first file. That is how you survive duplicate 200s.

Treat this like a package registry, not chat. Chat is a poor store for patches. A content-addressed file is a real store.

Here is a proposed orchestrator for that flow. Label it as example code, not production. You should adapt paths before you run it.

# proposed_orchestrator.py — example only, not production telemetry
import hashlib, json, os, time, urllib.request
from pathlib import Path

CACHE = Path(".gen-cache")
APPLIED = CACHE / "applied.json"

def idempotency_key(tree_hash: str, intent: str, nonce: str) -> str:
    raw = f"{tree_hash}\n{intent}\n{nonce}".encode()
    return hashlib.sha256(raw).hexdigest()[:16]

def store_artifact(key: str, body: bytes) -> Path:
    CACHE.mkdir(exist_ok=True)
    path = CACHE / f"{key}.diff"
    if not path.exists():
        path.write_bytes(body)
    return path

def already_applied(key: str) -> bool:
    if not APPLIED.exists():
        return False
    return key in json.loads(APPLIED.read_text())

def mark_applied(key: str) -> None:
    data = json.loads(APPLIED.read_text()) if APPLIED.exists() else {}
    data[key] = int(time.time())
    APPLIED.write_text(json.dumps(data, indent=2))

def post_job(url: str, envelope: dict, timeout: int = 30) -> bytes:
    req = urllib.request.Request(
        url,
        data=json.dumps(envelope).encode(),
        headers={"Content-Type": "application/json"},
        method="POST",
    )
    with urllib.request.urlopen(req, timeout=timeout) as resp:
        return resp.read()

You pin the tree before the request leaves. A dirty worktree makes the key a lie. Freeze the index first, then compile remotely.

# freeze the review surface before you generate
git update-index -q --refresh
git diff --quiet && git diff --cached --quiet || {
  echo "worktree dirty; refuse generate" >&2
  exit 1
}
TREE=$(git rev-parse HEAD)
INTENT=$(sha256sum intent.md | awk '{print $1}')
NONCE=$(openssl rand -hex 8)
echo "$TREE $INTENT $NONCE"

Poll with the same key until bytes land. Do not start apply from the live stream. The cache file is the only apply input.

# proposed poll loop — example only
def wait_for_artifact(url, key, envelope, cancel_path, budget=120):
    deadline = time.time() + budget
    while time.time() < deadline:
        if Path(cancel_path).exists():
            raise RuntimeError("local cancel flag set")
        cached = CACHE / f"{key}.diff"
        if cached.exists():
            return cached
        try:
            body = post_job(url, envelope, timeout=20)
            return store_artifact(key, body)
        except TimeoutError:
            time.sleep(2)
            continue
    raise TimeoutError("generation budget exhausted")

Cancellation should live on your local disk. A flag file is a boring honest signal. The remote job may keep running anyway.

You still refuse apply after that flag. Spend on the wire may continue briefly. Your repo does not have to follow it.

Put the flag next to the cache, not in chat. Chat delete does not cancel a running job. The apply gate should watch the flag path.

Failure domains

Draw three boxes and keep them hostile. The compiler box can lie, stall, or duplicate. The cache box can fill or go stale.

The apply box can still break the tree. Do not share credentials across those three boxes. The compiler token must not write git objects.

The apply token must not call the model. A duplicate HTTP 200 is not another patch. It is the same key arriving twice.

Your cache makes that collapse into one file. A partial diff is a failed compile. Truncation belongs in the compiler failure domain.

Apply should reject a truncated envelope right away. If the cache disk fills, you stop generation. Filling the disk is a local failure.

That failure still beats a silent double apply. Keep those domains on separate credentials at all times. One leaked token should not do both jobs.

Apply never talks to the compiler host itself. Git is a local tool in this design. That split is how rollback stays yours.

You can aim that local plane at a remote generator. MonkeyCode offers free model access and a free server option. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Point the compiler URL at that option when useful. Keep cache and apply steps on your machine. The free server is the untrusted compiler host.

What you should change next

What you should change next is explicit signing. This section is a proposal, not a field report. The apply plane should refuse an unsigned diff.

You should add per-path leases inside the worktree. Two keys must not touch the same file. A lease file would make that conflict visible.

You should add a dead-letter directory for rejects. Failed apply is its own failure domain. Do not retry it through the model by default.

You should add an explicit cancel protocol later. Today the flag only stops local apply. A real protocol would also stop wire spend.

Do not use this if chat is your record. People who live in threads will hate it. The design throws the thread away on purpose.

Do not use this for secrets that cannot leave. A remote compiler is still an egress path. A local model belongs in that case instead.

Do not use this when you need a contractual SLA. A free server is a courtesy, not a promise. On-call generation needs a named service contract.

The pattern also fails on huge binary blobs. Hashing and caching diffs assumes text patches only. Generated binary assets still need a different store.

A real compiler is deterministic in normal practice. A remote model is not deterministic at all. The idempotency key covers transport, not patch semantics.

Two runs with one key should store one file. They will not guarantee the same future bytes. Freeze the artifact after the first success.

You still need a human or a verifier. This architecture does not grade the patch quality. It only stops double apply and lost bytes.

Latency will stay ugly on a shared free path. Backpressure belongs in your local poll budget. Do not hide it behind a spinner UI.

Wire a tiny apply gate around the cache. The gate checks the key, the lease, and truncation. Then it runs a dry-apply before git write.

# proposed_apply_gate.py — example only
import subprocess, sys
from pathlib import Path

def looks_truncated(diff: str) -> bool:
    if not diff.endswith("\n"):
        return True
    if "diff --git " not in diff:
        return True
    return False

def dry_apply(diff_path: Path) -> None:
    subprocess.run(
        ["git", "apply", "--check", str(diff_path)],
        check=True,
    )

def apply_once(key: str, diff_path: Path) -> None:
    if already_applied(key):
        raise SystemExit(f"key {key} already applied")
    text = diff_path.read_text(errors="replace")
    if looks_truncated(text):
        raise SystemExit("truncated artifact; stay in compiler domain")
    dry_apply(diff_path)
    subprocess.run(["git", "apply", str(diff_path)], check=True)
    mark_applied(key)

if __name__ == "__main__":
    key, path = sys.argv[1], Path(sys.argv[2])
    apply_once(key, path)

Run it only against a cached artifact path. Never pipe the HTTP body into git apply. The file is the contract you can re-read.

python proposed_apply_gate.py "$KEY" ".gen-cache/${KEY}.diff"

If you try the local cache pattern, keep apply local. The generator can stay cheap and remote. Your repo should remain the only memory that matters.