I have an MCP server that digests a GitHub repository into markdown so a model can read it without cloning anything. Directory tree, plus the contents of the files that matter.
I pointed it at a real repository and got this back:
Error: result (109,668 characters) exceeds maximum allowed tokens
Not a truncated answer. Not a partial digest with a warning. Nothing. The tool call failed and the caller was left with an error message where a repository summary should have been.
The cause was a constant I had written months earlier without thinking about it:
MAX_FILE_BYTES = 32_000
TOP_N_FILES = 10
Ten files, each capped at 32,000 characters. I had reasoned that ten files is a reasonable amount of a repository to show. Which is true, and also not a size limit at all. Ten files is anywhere from a few hundred characters to 320,000 depending on whose repository you point it at.
The obvious fix, and why I did not just do it
The obvious fix is to add a total character cap. Pick a number, fill it, stop.
The problem is picking the number. I could have reasoned my way to one. 50,000 sounds fine. 32,000 sounds fine too. So does 64,000. Any of them sounds fine, which is a good sign that reasoning is not the tool for the job, and is exactly how TOP_N_FILES = 10 got there in the first place.
So before changing anything I measured. Six repositories, spanning the range I would realistically encounter: two small ones of my own, a mid-sized MCP server, a docs-heavy list repo, and two large real codebases.
| repository | files | tree | file content | total |
|---|---|---|---|---|
| GopherMCP/GopherCache | 5 | 89 | 4,505 | 4,594 |
| pyarchana/gopher | 22 | 591 | 23,640 | 24,231 |
| sktime/sktime-mcp | 104 | 3,296 | 100,490 | 103,786 |
| astral-sh/uv | 1,578 | 65,530 | 142,337 | 207,867 |
| modelcontextprotocol/python-sdk | 1,643 | 57,306 | 160,182 | 217,488 |
| punkpeye/awesome-mcp-servers | 10 | 173 | 228,137 | 228,310 |
A fifty-fold range between the smallest and largest output, from the same tool with the same settings.
Three things fell out of that table that I would not have guessed.
The tree alone can eat everything
Look at the tree column for uv and the python SDK. 65,530 and 57,306 characters, just to list filenames.
I had been thinking of the budget as a cap on file contents. But on a repository with 1,600 files, the directory listing on its own is larger than any sensible total. If I had capped only the file content, uv would have blown past any budget I set before fetching a single file.
The tree needed its own cap, at a fraction of the total, and a note saying how many entries it left out.
The ranking was picking the wrong files
This is the one that actually mattered, and I only saw it because I printed what was being selected rather than just how big it was.
Here is what my tool chose to show about uv, a Rust build tool:
27,194 test/ecosystem/airflow/pyproject.toml
32,048 test/ecosystem/home-assistant-core/pyproject.toml
27,517 test/ecosystem/pandas/pyproject.toml
9,156 test/ecosystem/jupyterlab/pyproject.toml
8,757 test/ecosystem/black/pyproject.toml
Eighty-nine thousand characters of other projects' dependency lists, scraped out of uv's test fixtures. The only uv code that made it in was three main.rs files, all of them thin entry points, and Cargo.toml ranked below every one of those fixtures.
And here is awesome-mcp-servers:
32,048 README-fa-ir.md
32,048 README-ja.md
32,048 README-ko.md
32,048 README-pt_BR.md
32,048 README-th.md
32,048 README-zh.md
32,048 README-zh_TW.md
The same README in seven languages, 224,000 characters of it.
The cause was one line:
if name in PRIORITY_NAMES:
score += 1000
Any file called pyproject.toml got a thousand points, wherever it sat. A vendored copy six directories deep in a test fixture scored exactly the same as the one at the root describing the actual project.
This is why the measurement mattered. If I had shipped the budget on its own, the digest would have gone from 207,867 characters of the wrong files to 40,000 characters of the wrong files. Smaller, still useless, and now looking deliberate.
The file it wanted most came back empty
One more, which I would never have found by reading code.
In the awesome-mcp-servers run, the top-ranked file scored 1200 and returned zero characters:
0 score=1200 README.md
32,048 score=200 README-fa-ir.md
GitHub's contents API will not serve a file over 1MB. It does not return an error. It returns 200 OK with an empty content field and encoding: "none". So the base64 decode succeeds, produces an empty string, and the file silently disappears.
That README is 1,616,144 bytes. The single most important file in the repository was being dropped without a word, and seven translations of it were filling the space instead. The fix is to notice encoding: "none" and refetch through the blobs endpoint, which serves up to 100MB.
What I changed
A total budget, not a file count. 40,000 characters, configurable. Spent on the tree first, then files in priority order, until it runs out. The number of files now falls out of what fits instead of being pinned in advance.
A cap on the tree, a quarter of the total, with a line saying how many entries it omitted.
A per-file ceiling of 40% of what remains, so one large file cannot crowd out everything else.
Ranking by position, not just name. The priority bonus now decays with directory depth, so the root manifest beats a vendored one. Test and fixture directories lose points, vendored ones lose more, examples lose only a little because sometimes an examples directory is the best documentation a project has. Files in the repository's primary language, which the API already tells you, gain some.
Here is the same table after:
| repository | before | after |
|---|---|---|
| GopherMCP/GopherCache | 4,594 | 5,005 |
| pyarchana/gopher | 24,231 | 39,263 |
| sktime/sktime-mcp | 103,786 | 39,231 |
| astral-sh/uv | 207,867 | 39,321 |
| modelcontextprotocol/python-sdk | 217,488 | 39,224 |
| punkpeye/awesome-mcp-servers | 228,310 | 39,168 |
uv's digest now leads with Cargo.toml and README.md. awesome-mcp-servers leads with its actual README. My own small repo got bigger, because it was only ever showing 10 of its 22 files and now shows all of them.
The thing I would tell past me
I nearly shipped a one-line change. Add a constant, clamp the output, close the issue. It would have passed review, passed tests, and produced a tool that was confidently wrong in a smaller font.
What stopped it was spending an hour printing what the tool actually produced against real inputs. Not unit tests, which only check what I already thought to assert. Not reading the code, which is where the bug had been sitting unnoticed for months. Just running it against six real repositories and looking at the output.
If you are building anything that assembles context for a model, you are making size and selection decisions whether you notice them or not. Print what your tool actually sends, against real inputs, at real sizes. The numbers are frequently not what you expect, and the interesting failure is rarely the one you set out to fix.
The tool is pyarchana/gopher, an MCP server that fetches, caches and digests context for Claude. The measurements above live in #5 and #14 if you want the full before and after.
pyarchana
/
gopher
One MCP server for Claude: digest any GitHub repo, keep memory across conversations, and extract facts locally with Ollama
Gopher
One MCP server that fetches, caches, and digests context for Claude.
Gopher is three things that used to be three separate servers:
-
fetch: point it at a GitHub repo and get back a clean Markdown digest: full directory tree, plus the contents of the files that actually matter. Filters out binaries, lock files,
node_modules,venv, and the rest of the noise, then ranks what's left. No README? It builds one for you. -
cache: persistent memory across conversations, stored as two plain files you can read yourself: a structured
context.jsonand an append-onlydiary.md. - digest: reads a conversation transcript, extracts the facts with a local Ollama model, and merges them into the cache.
Everything runs locally over stdio. Nothing leaves your machine except GitHub API calls.
Tools
Tool
What it does
fetch_github_repo(repo_url)
Markdown digest of a public repo, tree plus top files
read_context()
Return
Top comments (1)
The 109k-character wall is the number-one enemy of any repo-digestion tool — I hit the identical error building a similar digest. What saved me: a budget per layer instead of per file (tree, then symbols and signatures, then bodies), so the call always returns something degraded rather than nothing at all.
How are you handling the fallback now — does the server degrade gracefully to signatures when the full digest won't fit, or do callers have to retry with narrower paths?