DEV Community

Sam Chen
Sam Chen

Posted on

Debug the Context Pack Before You Debug the Model

The model did not fail your last coding task. Your context pack failed that task instead. I keep seeing the same six packing mistakes. They still look like ordinary model quality problems. They are packing errors and not weight errors.

Want the real failure point right now? Watch the files you actually send. Then watch the files you quietly skip. The miss is rarely sitting in the weights. The miss is sitting in the pack.

What I mean by a context pack

A context pack is the snapshot the model sees. It holds files, diffs, commands, and constraints. It is not your entire laptop state. It is not last week's chat transcript. It is only that frozen snapshot.

Did you record that snapshot at all? Most debugging sessions never do. Then the postmortem blames the model again. That blame is usually the first leak.

How to read this catalog

Each anti-pattern has three tight parts.

  • Symptom: what you observe during the live session
  • Root cause: the packing mistake hiding underneath
  • Replacement: the pattern I actually run instead

I am not ranking hosted models in this piece. I am ranking the packs we feed them.

Anti-pattern 1: The Whole-Repo Dump

You archive the repository and paste the tree. You hope the model finally has everything. That completeness still feels like real engineering discipline.

Symptom: The answer cites dead folders without shame. Latency climbs while the useful patch misses. The edit lands in the wrong package.

Root cause: You treated complete as helpful input. Complete is mostly noise in this loop. Noise drowns the one sibling file that matters.

Replacement: Pack by blast radius around the failure. Start from the failing test file. Add the implementation file it imports. Add one neighbor, then stop packing.

Can you name every file in the pack? If you cannot, the pack is already too fat.

Anti-pattern 2: The Lucky File

You send only app.py because it compiles today. The regression actually lives in app_utils.py. You never packed the file that scared you.

Symptom: The model rewrites a function that is innocent. The original failing test still fails hard. You call the model weak and stop.

Root cause: You packed the file you already understand. You skipped the file you did not want. Comfort is not a serious retrieval strategy here.

Replacement: Pack the failing assertion before any implementation. Then pack every import that assertion touches. Then stop packing again on purpose.

Which file are you avoiding right now? Send that feared file next.

Anti-pattern 3: Chat as Source of Truth

The model proposed a patch ten turns ago. You kept talking without freezing the tree. Nobody wrote down the bytes that were sent.

Symptom: Fresh advice targets symbols you already deleted. Imports point at names from an old branch. You feel gaslit by a helpful tone.

Root cause: The chat drifted while the pack stayed stale. You never wrote a pack manifest. Debugging then becomes folklore instead of honest engineering.

Replacement: Write a manifest before every model call. Hash each file and record the command. Refuse to debug a session without that file.

Anti-pattern 4: The Orphan Diff

The model returns a clean looking unified diff. You never apply it to a fresh worktree. You eyeball the patch inside the chat bubble.

Symptom: The patch looks right beside the explanation. git apply --check fails on first contact. Hidden context never made the pack.

Root cause: Review happened on prose, not on git. Chat remains a terrible merge tool here. The orphan diff never paid rent in git.

Replacement: Apply the patch on a throwaway branch. Re-run the exact failing command. Review git diff and nothing else.

Did you apply the diff before arguing quality? If not, you reviewed a story.

Anti-pattern 5: Token Theatre

You pad the pack with README, LICENSE, and lockfiles. The prompt looks thorough and strangely official. That padding is theatre and not signal.

Symptom: The model recites slogans from your README. It never touches the flaky test line. You burned a turn on branding copy.

Root cause: You optimized for looking complete and serious. You did not optimize for the failing line. Ceremony is not a substitute for context.

Replacement: Keep an explicit deny list of junk paths. Fail the packer when those paths appear. Make that failure loud in local CI.

Why did the model quote your license header? Because you packed that header.

Anti-pattern 6: Review in the Bubble

You ask the same model if the patch is safe. It agrees with its own earlier story. You ship on that warm agreement.

Symptom: Production hits a case the chat never saw. The postmortem blames "AI generated code" again. The bubble never contained the real users.

Root cause: Author and reviewer shared one context window. That window cannot surprise itself honestly. Agreement inside one window is not evidence.

Replacement: Split generation and review into two packs. Keep the review pack smaller and meaner. Use a checklist the model did not write.

Who reviews the reviewer in your loop? If nobody, the bubble ships.

The replacement workflow I actually run

I run one boring loop on purpose. That boredom is the entire point of the loop.

  1. Capture the exact failing command output.
  2. Build a pack from that failure only.
  3. Write pack_manifest.json with file hashes.
  4. Call the model with only that pack.
  5. Apply the patch on a clean branch.
  6. Re-run the same failing command.
  7. Keep the patch or revert immediately.

Need a remote model for step four? A free model endpoint belongs there when you do not want a local GPU. MonkeyCode fits that narrow slot because it offers free model access and a free server option.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

I only need those two things from that stack. I need free model access for the generation call. I also need a free server option for the endpoint. I do not need a story about permanence or quotas. I need a repeatable call against a frozen pack.

Artifact: a packer that can fail

The script below is a worked example. I am not claiming production metrics. Run it in a throwaway clone only.

#!/usr/bin/env python3
"""Blast-radius context packer. Worked example, not a benchmark."""

from __future__ import annotations

import argparse
import hashlib
import json
from pathlib import Path

DENY_NAMES = {
    "license",
    "license.md",
    "copying",
    "readme.md",
    "package-lock.json",
    "yarn.lock",
    "pnpm-lock.yaml",
    "poetry.lock",
    "cargo.lock",
}
SKIP_DIRS = {
    ".git",
    ".hg",
    ".svn",
    "node_modules",
    "__pycache__",
    ".venv",
    "venv",
}


def sha256_bytes(data: bytes) -> str:
    return hashlib.sha256(data).hexdigest()


def is_denied(path: Path) -> bool:
    return path.name.lower() in DENY_NAMES


def iter_files(root: Path):
    for path in root.rglob("*"):
        if not path.is_file():
            continue
        if any(part in SKIP_DIRS for part in path.parts):
            continue
        if is_denied(path):
            continue
        yield path


def score_file(path: Path, failing: Path) -> int:
    score = 0
    if path == failing:
        score += 100
    if path.parent == failing.parent:
        score += 20
    if path.suffix == failing.suffix:
        score += 5
    left = path.stem.replace("test_", "")
    right = failing.stem.replace("test_", "")
    if left == right:
        score += 40
    return score


def build_pack(root: Path, failing: Path, limit: int = 6) -> dict:
    ranked = sorted(
        iter_files(root),
        key=lambda p: score_file(p, failing),
        reverse=True,
    )
    chosen = []
    for path in ranked:
        if len(chosen) >= limit:
            break
        if score_file(path, failing) <= 0:
            continue
        chosen.append(
            {
                "path": str(path.relative_to(root)),
                "sha256": sha256_bytes(path.read_bytes()),
                "bytes": path.stat().st_size,
            }
        )
    return {
        "root": str(root.resolve()),
        "failing": str(failing.relative_to(root)),
        "files": chosen,
    }


def main() -> None:
    parser = argparse.ArgumentParser()
    parser.add_argument("--root", type=Path, required=True)
    parser.add_argument("--failing", type=Path, required=True)
    parser.add_argument("--out", type=Path, required=True)
    parser.add_argument("--limit", type=int, default=6)
    args = parser.parse_args()
    root = args.root.resolve()
    failing = (root / args.failing).resolve()
    pack = build_pack(root, failing, limit=args.limit)
    args.out.write_text(json.dumps(pack, indent=2), encoding="utf-8")
    print(f"wrote {args.out} with {len(pack['files'])} files")


if __name__ == "__main__":
    main()
Enter fullscreen mode Exit fullscreen mode

A tiny test belongs beside the packer. The test should fail closed.

from pathlib import Path
from pack_context import build_pack


def test_deny_list_blocks_license(tmp_path: Path):
    (tmp_path / "LICENSE").write_text("MIT", encoding="utf-8")
    test_file = tmp_path / "test_app.py"
    test_file.write_text("def test_ok():\n    assert True\n", encoding="utf-8")
    (tmp_path / "app.py").write_text("x = 1\n", encoding="utf-8")
    pack = build_pack(tmp_path, test_file, limit=6)
    paths = {item["path"].lower() for item in pack["files"]}
    assert "license" not in paths


def test_failing_test_is_included(tmp_path: Path):
    test_file = tmp_path / "test_app.py"
    test_file.write_text("def test_ok():\n    assert True\n", encoding="utf-8")
    (tmp_path / "app.py").write_text("x = 1\n", encoding="utf-8")
    pack = build_pack(tmp_path, test_file, limit=6)
    paths = {item["path"] for item in pack["files"]}
    assert "test_app.py" in paths
Enter fullscreen mode Exit fullscreen mode

Run the example like this.

python pack_context.py --root . --failing tests/test_app.py --out pack_manifest.json
python -m pytest test_pack_context.py -q
git checkout -b pack-scratch
# apply the model patch here, then:
git apply --check proposed.patch
python -m pytest tests/test_app.py -q
Enter fullscreen mode Exit fullscreen mode

If LICENSE lands in the manifest, your packer is lying. Fix the packer before you touch the model. The manifest is the test you forgot to write.

Decision table I keep on the desk

Observation Likely anti-pattern First move
Whole tree cited in the answer Whole-Repo Dump Cut the pack to blast radius
Innocent function rewritten first Lucky File Pack the feared sibling file
Advice uses deleted symbol names Chat as Source of Truth Rewrite the pack manifest
git apply --check fails immediately Orphan Diff Apply on a throwaway branch
README slogans quoted back to you Token Theatre Enforce the deny list in CI
Model blesses its own earlier patch Review in the Bubble Split generation and review packs

Print that table and tape it near the terminal. Use it before you change model providers. Provider hopping will not fix a lying pack.

Limitations

This loop will not catch security issues for you. It will not design your architecture either. A deny list is not a retrieval system. Hashes prove what you sent. They do not prove the patch is right.

Free model access is not an availability contract. A free server option is not an SLO. Do not paste secrets into any remote pack. Do not treat this catalog as a legal audit trail.

Large monorepos need path indexes I did not ship here. This packer is a seatbelt around the call. It is not an agent, and it will not steer.

Who should not use this

Skip this if policy forbids third-party model hosts. Skip this if your tree is mostly secrets. Skip this if you need guaranteed latency numbers.

Skip this if you want the model to pick architecture. This catalog assumes you already have a failing test. No failing test means no honest pack. No honest pack means you are back to folklore.

What I want you to try tomorrow

Pick one failing test and nothing else. Build a pack you can name out loud. Write the manifest before the model call. Apply the diff on a clean branch.

Keep the manifest even if the patch is ugly. The manifest is the actual product of the session. Blame the pack first. Only then blame the model.

Top comments (0)