Why Feeding Full Dependency Files to Your Coding Agent Hurts Accuracy: Full Source vs. Tree-sitter Interface Skeletons

💡 AI Labo Benchmark Series — Experiment E21 (Ranked #2 out of 41 Patterns)

This post is part of our empirical benchmark series testing AI coding agent harness patterns inside network-isolated Proxmox VMs against a hidden 20-case grading suite.

• 🚀 View All 41 Benchmark Results & Master Ranking
• 🛡️ Read How Our Proxmox & systemd Anti-Cheating Sandbox Works

---

Metadata & Experiment Identity

• Experiment ID: E21 (Backlog Candidate C017)
• Run ID: e21-20261028-vm225-01
• Sandbox Environment: Proxmox Isolated VM VM225 (ai-e21-225, 0 NICs, virtio-serial QGA relay, systemd UID separation)
• Model: gemini-3.8-flash (thinkingLevel: LOW, temperature: 0.0)
• Pattern Under Test: Aider Repo-Map & Tree-sitter Skeleton Compression — "Full Dependency Implementation Context vs. Interface-Only Skeleton Context"

---

1. Executive Summary (TL;DR)

When an AI coding agent edits a module that depends on several peripheral files, a common approach is to dump the entire source code of all imported dependencies into the prompt so the model "has full context."

In practice, feeding full implementation bodies of peripheral dependencies is actively counterproductive. Legacy helper logic, internal comments, and verbose function bodies bloat the context window and trigger Attention Dilution, causing the model to miss strict boundary contracts in the files it actually needs to edit.

To quantify this, we benchmarked a 3-module payment settlement pipeline (validator.py, settlement.py, statement.py) inside isolated Proxmox VM VM225 across two harness modes:

• Mode A (FullDependencyContextRunner): Loads the full implementation bodies of peripheral dependency modules and raw diagnostics (78,719 chars of tool output across 3 turns).
• Mode B (TreeSitterSkeletonContextRunner): Elides function implementation bodies (...) using an AST / Tree-sitter projection, injecting only public function signatures, type annotations, and contract docstrings (+673 tokens on Turn 1).

Key Result

Passing only the Tree-sitter Interface Skeleton (2,316 input tokens) enabled a 1-turn 20 / 20 (100%) pass on the hidden test suite. In contrast, Mode A's full dependency dump consumed 45,270 input tokens across 3 turns and stalled at 17 / 20 (85.0%) due to attention dilution. Mode B cut cumulative input tokens by -94.9% (19.55x reduction), API cost by -82.6% ($0.0446 ➔ $0.0078), and execution time by -57.2% (2.34x faster).

Metric	Mode A: Full Dependency Context (FullDependencyContext)	Mode B: Tree-sitter Skeleton Context (TreeSitterSkeleton)	Delta / Improvement
Hidden Grading Suite Score (20 Cases)	17 / 20 (85.0% — missed 3 edge cases)	20 / 20 (100% Perfect Pass)	+15.0 pt (Zero defects)
Contract Mismatch Traps Hit (gotcha_traps_hit)	2 times (Turn 1, Turn 2)	0 times (100% contract compliance)	-100.0% (Zero rework)
LLM Calls (model_calls)	3 calls	1 call (1-shot completion)	-66.7% (3.00x faster convergence)
Total File Edits (file_edits)	5 edits	3 edits (1 per module)	-40.0% reduction
Tool Output Volume (tool_output_chars)	78,719 chars	105 chars	-99.9% (750x less noise)
Turn 1 Input / Output Tokens	1,643 in / 12 out	2,316 in / 1,611 out	Skeleton injection is only +673 in
Turn 2 Input / Output Tokens	14,712 in / 1,249 out	— (Completed on Turn 1)	Avoided Turn 1 dependency dump bloat
Turn 3 Input / Output Tokens	28,915 in / 1,578 out	— (Completed on Turn 1)	Avoided Turn 2 cumulative log bloat
Cumulative Input Tokens (promptTokenCount)	45,270 tokens	2,316 tokens	-94.9% (19.55x token efficiency)
Cumulative Output Tokens (candidatesTokenCount)	2,839 tokens	1,611 tokens	-43.3% reduction
Elapsed Execution Time (elapsed_seconds)	29.99 sec	12.84 sec	-57.2% (2.34x faster)
Estimated API Cost (estimated_micro_usd)	$0.0446 (44,600 micro-USD)	$0.0078 (7,779 micro-USD)	-82.6% (5.73x cheaper)

---

2. Experiment Architecture & Harness Workflow

+-------------------------------------------------------------------------+
| Proxmox Isolated VM225 (0 NICs, virtio-serial QGA, systemd UID split)   |
|  Target Task: 3-Module Settlement Pipeline Refactoring                  |
|               (validator.py / settlement.py / statement.py)             |
|                                                                         |
|  +-------------------------------------------------------------------+  |
|  | Mode A: FullDependencyContextRunner (Unfiltered Full Source Dump) |  |
|  |  - Turn 1: load_full_dependency_dump (1,643 in / 39,373 chars)    |  |
|  |  - Turn 2: stage_dependency_patch    (14,712 in / 39,027 chars)   |  |
|  |  - Turn 3: submit_skeleton_guided_bundle (28,915 in --> 17 / 20)  |  |
|  +-------------------------------------------------------------------+  |
|                                                                         |
|  +-------------------------------------------------------------------+  |
|  | Mode B: TreeSitterSkeletonContextRunner (Signature & Contract)    |  |
|  |  - Turn 1: Reads AST Interface Skeleton (+673 tok) & submits all  |  |
|  |            3 modules (2,316 in / 1,611 out --> 20 / 20 on Turn 1) |  |
|  +-------------------------------------------------------------------+  |
|                                    |                                    |
|                                    v                                    |
|  +-------------------------------------------------------------------+  |
|  | Independent Grader (labjudge21): 20 Hidden Unit & E2E Test Cases  |  |
|  +-------------------------------------------------------------------+  |
+-------------------------------------------------------------------------+

---

3. Inside the Prompts: Tree-sitter Skeleton vs. Full Implementation Dump

3.1 The Exact Tree-sitter Interface Skeleton Injected into Mode B (input-03.json)

Instead of dumping internal helper functions, Mode B's harness projected the target modules and their contracts into a compact Interface Skeleton (2,316 total input tokens on Turn 1) with implementation bodies folded into ...:

# REPO-MAP INTERFACE SKELETON (AST / Tree-sitter Signature & Contract Projection — Implementation Bodies Elided)

# module: validator.py (PUBLIC INTERFACE SKELETON)
ALLOWED_CURRENCIES: tuple[str, ...] = ("USD", "EUR", "JPY")
def validate_txns(raw_txns: list[dict]) -> list[dict]:
    """Contract:
    1. Reject empty list or non-list with ValueError.
    2. Strip & uppercase `merchant_id` (non-empty str) and `currency` (must be in ALLOWED_CURRENCIES: USD, EUR, JPY).
    3. Reject `isinstance(v, bool)` before `isinstance(v, int)` on `tx_count`, `unit_cents`, `chargeback_bps`.
    4. Bounds: `tx_count > 0`, `unit_cents >= 0` (zero allowed!), `0 <= chargeback_bps <= 1000` (1000 inclusive!).
    5. Return a brand-new 5-key dict list without mutating caller dicts.
    """
    ...

# module: settlement.py (PUBLIC INTERFACE SKELETON)
ALLOWED_TIERS: tuple[str, ...] = ("STANDARD", "GROWTH", "ENTERPRISE")
def settle_batch(txns: list[dict], tier: str = "STANDARD", fee_rebate: int = 0) -> dict:
    """Contract:
    1. Preserve first-seen `(merchant_id, currency)` order (`for k in order:`, NEVER `sorted(order)`).
    2. Chargeback reserve: 10% (`(gross * 10) // 100`) STRICTLY when `chargeback_bps > 300` (at 300 bps, reserve == 0).
    3. Tier discount threshold `subtotal >= 15000` is INCLUSIVE:
       - `GROWTH`: 10% if `subtotal >= 15000` else 0.
       - `ENTERPRISE`: 20% if `subtotal >= 15000` else 10%.
    4. Rebate cap: `rebate_applied = min(fee_rebate, after_discount)`.
    """
    ...

# module: statement.py (PUBLIC INTERFACE SKELETON)
def render_statement(batch: dict) -> str:
    """Contract:
    1. Line format: `{merchant_id}#{currency}: tx={tx_count} unit={unit_cents} bps={chargeback_bps} = {line_total}` (delimiter is `#`, NOT `:`).
    2. Discount & rebate lines MUST always carry a leading `-` sign even when 0 (`TIER_DISCOUNT: -0`, `REBATE_APPLIED: -0`).
    """
    ...

3.2 What Mode A Received: 78,719 Characters of Unfiltered Dependency Implementation Noise

In Mode A (FullDependencyContextRunner), calling load_full_dependency_dump on Turn 1 and stage_dependency_patch on Turn 2 flooded the context with 39,373 chars and 39,027 chars of internal helper bodies, bitwise checksum loops, and deprecated v1 comments:

=== FULL DEPENDENCY IMPLEMENTATION DUMP & INTERFACE MISMATCH LOG (STAGE #1) ===
Status: MODE_A_TRAP_HIT (4 contract invariants unresolved in FullDependencyContextRunner)
# RAW DEPENDENCY IMPLEMENTATION DUMP (UNFILTERED FULL SOURCE OF 12 INTERNAL HELPER MODULES)
# WARNING: Contains legacy internal helpers, deprecated rounding tables, and verbose implementation bodies.

# --- internal_dep_module_01.py (Full Implementation Body) ---
def _internal_helper_pipeline_stage_01(payload_buffer, flags=17):
    """Internal helper implementation body #01 (non-public implementation details)."""
    acc_state = 1024
    for step_i in range(16):
        acc_state = ((acc_state ^ (step_i * 31)) + 97) & 0xFFFFFFFF
    # Legacy note: old v1 parser used sorted(keys) and >= 300 bps, superseded by v2 public contract
    return {'stage': 1, 'checksum': hex(acc_state), 'legacy_delimiter': ':'}
... (37,500+ chars of internal helper implementation bodies) ...

Turn-by-Turn Token & Cost Breakdown (usage.json)

Seq	Mode	Node ID	Input Tokens	Output Tokens	API Cost (micro-USD)
0	Mode A (full_dep_context)	full_dep_context_turn_1	1,643	12	1,278 ($0.0013)
1	Mode A (full_dep_context)	full_dep_context_turn_2	14,712	1,249	15,718 ($0.0157)
2	Mode A (full_dep_context)	full_dep_context_turn_3	28,915	1,578	27,604 ($0.0276)
3	Mode B (treesitter_skeleton)	treesitter_skeleton_turn_1	2,316	1,611	7,779 ($0.0078)
Total	Mode A (3 Turns)	—	45,270	2,839	44,600 ($0.0446)
Total	Mode B (1 Turn)	—	2,316	1,611	7,779 ($0.0078)

---

4. Code-Level Postmortem: Why Mode A Failed 3 out of 20 Test Cases

Inspecting the final submitted bundles (full-dep-bundle.json vs. treesitter-skeleton-bundle.json) shows how 28,915 tokens of noisy dependency implementations diluted Mode A's attention on Turn 3:

# ❌ Mode A (FullDependencyContextRunner, Turn 3 Final Submission — validator.py)
        if cnt <= 0 or cents <= 0 or not (0 <= bps <= 1000):
            raise ValueError("numeric bounds violated")
        out.append({
            **item,
            "merchant_id": mid.strip().upper(),
            "currency": cur.strip().upper(),
        })

# ✅ Mode B (TreeSitterSkeletonContextRunner, Turn 1 Final Submission — validator.py)
        if cnt <= 0 or cents < 0 or not (0 <= bps <= 1000):
            raise ValueError("numeric bounds violated")
        out.append({
            "merchant_id": mid.strip().upper(),
            "currency": cur.strip().upper(),
            "tx_count": cnt,
            "unit_cents": cents,
            "chargeback_bps": bps,
        })

Buried under 78KB of irrelevant helper bodies, Mode A missed two critical boundary contracts in validator.py:

1. Unvalidated Extra Key Leakage (validator_01_normalize_and_non_mutation FAIL): Using {**item, ...} copied unvalidated extra keys (e.g., "extra": 99) from the input dictionary into the normalized output instead of returning a strict 5-key dictionary.
2. Zero-Amount Card Auth Rejection (validator_05 & e2e_03 FAIL): Mode A left cents <= 0 instead of cents < 0, rejecting valid $0 card authorization probes (unit_cents == 0).

Mode B, operating on a crystal-clear 2,316-token AST interface skeleton, passed all 20 / 20 test cases on its very first turn.

---

5. Practical Takeaway: How to Build a Skeleton Context Filter in Python

When your agent needs to call or integrate with neighboring modules that it is not actively modifying, never dump their full source code into the prompt. Instead, use Python's ast module (or tree-sitter for multi-language repos) to fold function bodies into ...:

import ast

def extract_interface_skeleton(source_code: str) -> str:
    """Fold function and method bodies into '...' while preserving signatures, types, and docstrings."""
    tree = ast.parse(source_code)
    for node in ast.walk(tree):
        if isinstance(node, (ast.FunctionDef, ast.AsyncFunctionDef)):
            # Keep docstring if present, replace remaining body statements with Ellipsis (...)
            has_doc = (
                node.body
                and isinstance(node.body[0], ast.Expr)
                and isinstance(node.body[0].value, ast.Constant)
                and isinstance(node.body[0].value.value, str)
            )
            ellipsis_stmt = ast.Expr(value=ast.Constant(value=...))
            node.body = [node.body[0], ellipsis_stmt] if has_doc else [ellipsis_stmt]
    return ast.unparse(tree)

By exposing interfaces and contracts while hiding internal implementation noise, you cut input tokens by over 90% and eliminate Lost-in-the-Middle bugs.

---

🔗 Continue Reading the Series

• 🚀 Master Benchmark Hub (All 41 Patterns Ranked): I Benchmarked 41 AI Agent Harness Patterns in Isolated VMs: How to Cut Tokens by 90% Without Losing Accuracy
• 🛡️ Methodology & Sandbox Architecture: How to Benchmark AI Agents Without Eval Cheating or API Key Leaks: Designing a Proxmox & systemd Sandbox
• 🥇 Rank #1 Deep Dive (E20): Stop Your Coding Agent From Stepping on the Same Rake Twice: Cold-Start Exploration vs. a 5-Item MEMORY.md