DEV Community

Shouvik Palit
Shouvik Palit

Posted on

leantext: up to 99% fewer tokens when your coding agent reads the web

#ai #python #opensource #productivity

Your coding agent opens a docs page to answer one question. It gets the navigation bar, the sidebar, the footer, hidden menus, scripts and markup. Somewhere in there is the answer, and you pay for all of it, in tokens, in context window and in latency.

On PyPI's NumPy page, the project description is about 2,000 tokens. The whole page, converted to Markdown the usual way, is 92,348. Most of it is a hidden list of every file in every release, with hashes, that you never see in a browser.

leantext removes that noise before it reaches your agent, checks that nothing important went with it, and tells you what it saved.

https://github.com/shouvik12/leantext

What it does

Point it at a URL or a file. Markdown comes out on stdout.

leantext https://docs.stripe.com/api/customers/create --report
Enter fullscreen mode Exit fullscreen mode 
# Create a customer

## Request

```curl
curl https://api.stripe.com/v1/customers \
  -u "<<YOUR_SECRET_KEY>>" \
  -d "name=Jenny Rosen" \
  --data-urlencode "email=jennyrosen@example.com"
```

## Parameters

- `email` (string, optional)
  Customer's email address. ...
...

leantext report  (markdown served by the site)
  raw HTML                        521,378  ->    1,194   100% saved
  naive Markdown (whole page)       3,958  ->    1,194    70% saved
Enter fullscreen mode Exit fullscreen mode

The agent gets the request, the response, every parameter with its type, and the return value. Nothing else.

It handles web pages, PDFs (text and scanned), screenshots (OCR first, a vision model only as a fallback) and Office files.

The numbers

Savings are measured against the page converted to Markdown the usual way, not against raw HTML, because that's what an agent would otherwise get. Token counts use tiktoken.

Input Before With leantext Saved
PyPI: NumPy 92,348 755 99%
PyPI: uv 43,177 2,501 94%
Next.js README 7,394 452 94%
Screenshot: rich table 1,999 127 94%
Stripe API docs 3,958 1,194 70%
MDN: Promise 11,901 7,503 37%
Python tutorial 11,950 10,746 10%
Attention Is All You Need (PDF) 10,168 9,838 3%

On the docs you read every day

I also ran it against reference pages developers open daily:

Page Before With leantext Saved
MDN: Array.prototype.map() 8,265 2,754 67%
FastAPI: first steps 7,671 4,104 46%
git: git rebase 24,115 17,225 29%
Node.js: fs 129,576 102,898 21%
Python: pathlib 23,339 22,863 2%

Every row is lossless: all code blocks and tables from the source are in the output. Pages that are almost pure content, like Python's pathlib reference or a dense research paper, have little to cut, and leantext correctly leaves them alone (more on that below).

In end-to-end runs inside Claude Code, the agent answered 21 of 21 tasks correctly with leantext switched on.

It's automatic in Claude Code

Say yes to the hook during install, restart Claude Code, and keep working. When the agent reads a web page, a scanned PDF or an Office file, leantext converts it first:

When the agent does The hook does
Read page.html, Read scan.pdf, Read spec.docx points Read at a cached Markdown conversion
curl -s URL, `curl -s URL \ grep …, wget -qO- URL`
anything else (source code, curl -o, `curl \ jq`, POSTs)

Every decision is logged with a reason:

tail -3 ~/.cache/leantext/hook.jsonl
Enter fullscreen mode Exit fullscreen mode

Turn it off any time with LEANTEXT_HOOK=off.

It fails open

This is the part I care about most. Cutting tokens is easy if you don't mind cutting the answer too.

After every conversion, leantext compares the output's structure with the source: code blocks, tables, headings, how much text survived, whether the page title is there. It also recognises bot walls and pages that only render with JavaScript. If anything looks off, or the saving is under 15%, the hook steps aside and the agent reads the original.

leantext: extraction suspicious: code blocks: 83 in source, 14 in output
Enter fullscreen mode Exit fullscreen mode

So the worst case is the same as not having leantext installed.

That check paid for itself before launch. When I first ran the table above, Python's pathlib page kept only 14 of its 83 code examples intact. Sphinx puts examples inside <dl><dd> API blocks, and the extractor flattened them into the surrounding prose. Node's docs lost 116 of 123 version-history tables, which sit inside collapsed <details> boxes. Both were flagged, so no agent would have read the broken version.

The fix: before extraction, every <pre> and data table is swapped for a placeholder. The extractor still decides what's main content (so sidebar code is still dropped), and afterwards the blocks go back in verbatim, with their language tags. Python's page is now there with all 83 examples, as a regression test.

How it works

  1. Fetch. If the site serves its own Markdown (Stripe, GitHub Docs, Docker docs do), use that.
  2. Strip what a reader can't see. Hidden elements, unselected tab panels, closed dialogs.
  3. Extract the main content with trafilatura, with code and tables shielded.
  4. Check the result against the source.
  5. Measure tokens before and after.

Install

About 20 seconds on a normal connection, 39 MB on a Mac, no system packages needed:

git clone https://github.com/shouvik12/leantext
cd leantext && ./install.sh
Enter fullscreen mode Exit fullscreen mode

The installer asks whether to add the Claude Code hook.

Try it on your docs

The best test is the docs you read most. Run it with --report, and if something comes out wrong or gets flagged, open an issue with the URL. The flag tells you exactly what went missing, which makes it quick to fix.

leantext is Apache-2.0: github.com/shouvik12/leantext

Top comments (0)