I asked my coding agent a simple question about a Python package. It fetched the package's PyPI page and read 92,348 tokens.
The part of that page I actually cared about, the project description, is about 2,000 tokens. The rest? Navigation, a sidebar, a footer, and a hidden "Download files" tab listing every file of every NumPy release with its security hash. You never see that list in a browser. My agent read all of it anyway, and I paid for it.
That's the problem I built leantext to solve.
What leantext does
leantext turns web pages, PDFs, screenshots and Office files into lean Markdown before your agent reads them. It keeps the content, drops the noise, checks that nothing important went missing, and tells you exactly what it saved.
On that NumPy page: 92,348 → 755 tokens. Every word of the project description kept.
It works in two ways:
1. Automatically in Claude Code. leantext installs a hook that sits under the agent. When Claude Code runs curl on a docs page, or reads an HTML file, a scanned PDF or a Word document, the hook converts it first. You don't change your prompts, and you don't rely on the agent remembering to call a tool.
2. From the command line, for any agent or any pipeline:
leantext https://docs.stripe.com/api/customers/create --report
Markdown comes out on stdout. --report prints the tokens before and after.
The rule that matters most: fail open
A tool that shrinks context can do real damage if it quietly drops the one table your agent needed. So leantext checks every conversion against its source:
- Are all the code blocks, tables and headings still there?
- Is the page's own title at the top, or did navigation sneak in instead?
- Is the output garbled, near-empty, or a "please enable JavaScript" page?
- Does it save at least 15%?
If anything fails, the agent gets the original, untouched. The worst case is "no savings", never "wrong answer".
The results, including where it loses
Here's every real input I measured, sorted by how much it saved:
| Input | Before | With leantext | Saved |
|---|---|---|---|
| PyPI: NumPy | 92,348 | 755 | 99% |
| PyPI: uv | 43,177 | 2,501 | 94% |
| Next.js README | 7,394 | 452 | 94% |
| Screenshot: rich table | 1,999 | 127 | 94% |
| Stripe API docs | 3,958 | 1,194 | 70% |
| MDN: Promise | 11,901 | 7,503 | 37% |
| Python tutorial | 11,950 | 10,746 | 10% |
| Attention Is All You Need (PDF) | 10,168 | 9,838 | 3% |
Two things I want to be upfront about.
"Before" is the honest baseline. It's the whole page converted to Markdown, roughly what an agent's built-in fetch gives it. Compared with raw HTML, every number looks bigger (the uv page is 158,141 tokens raw), but nobody sensible feeds raw HTML to an agent.
The range is wide on purpose. leantext can only remove noise. PyPI pages are mostly hidden download lists, so they shrink by 94 to 99%. A research paper is almost all content, so it barely shrinks, and that's the correct result.
Fewer tokens, same answers?
Token counts alone prove nothing. So I tested two things.
Is the content still there? On both PyPI pages, I compared leantext's output word by word against the project description. uv: everything kept, including all 85 lines of code examples. NumPy: 375 of 375 words. What got dropped was the hidden download lists, the menus, and PyPI's sidebar (release date, maintainers, license). That sidebar loss is real, so it's listed in the README's known limits.
Does the agent still get it right? I ran 7 tasks in fresh Claude Code sessions, hook off and hook on, three times each. 21 of 21 answers stayed correct.
And here's the result that surprised me most: screenshots lose inside Claude Code. Converting a screenshot to text saves 90%+ of the image tokens, but an image only costs about 2k tokens to begin with. When the agent reads OCR text instead, it sometimes takes one extra turn to double-check it, and that turn costs around 15k. Net result: more expensive.
So the hook's defaults follow the data. It converts web pages, scanned PDFs and Office files, and leaves screenshots and text PDFs alone unless you opt in. I'd rather ship a tool that admits where it loses than one that claims 90% everywhere.
Things I learned building it
Your dependencies are your first impression. My first install pulled in 133 MB on a Mac, mostly a backup OCR engine (OpenCV and ONNX Runtime) that a Mac never uses, since Apple Vision is built in. On a slow connection, that meant a 17-minute install. Moving the backup OCR and Office support into optional add-ons cut it to 39 MB, with zero change to the results. On my slowest connection, a fresh install now takes 11 minutes instead of an estimated 30.
"My internet is fine" isn't the whole story. That slow install taught me something else: my Wi-Fi did 18 MB/s to Cloudflare but only 58 KB/s to PyPI at the same moment. Speed depends on the route to each server, not just your connection. The README now has a "slow install?" tip because of that night.
Every bug becomes a test. Glued-together parameter names on Stripe's docs, dropped headings on GitHub READMEs, hidden tab panels on PyPI: each one became a fixture from a real page. The suite is at 74 tests now.
Try it
git clone https://github.com/shouvik12/leantext && cd leantext && ./install.sh
The installer brings everything it needs (no Homebrew packages required) and asks two questions: whether you want Word/Excel/PowerPoint support, and whether to set up the Claude Code hook.
Then:
leantext https://pypi.org/project/numpy/ --report > numpy.md
Using another agent? Any agent that can run shell commands can use it today. Add a line to its rules file telling it to run leantext <url-or-file> instead of curl. An MCP server is on the roadmap.
What I'd love from you
The most valuable thing you can send me is a page where leantext gets it wrong: the URL, the output, and what your agent missed. Every one becomes a test.
The repo is here: github.com/shouvik12/leantext (Apache-2.0). Stars help other people find it, but a bad conversion report helps more.
Top comments (0)