Six minutes into testing a PDF tool, Claude found its Windows download held only a README.
PaperPilot , a Rust PDF tool, showed up on Hacker News on 26 October. Its command-line interface looked useful for the arXiv papers I use in my research workflow, and it also came with an MCP server for agents.
That same day, Simon Willison posted about having Claude Opus 5.5 try a new tool and write up what it found. I wanted to try that approach, so instead of installing PaperPilot myself, I asked Claude to investigate it first, on a small job I could check: turning two papers into a four-page reading pack.
As it turned out, the empty download was the smaller surprise. A bigger one came once Claude Code started checking the results, and I had to ask how I’d do this job without PaperPilot at all.
Subscribe for more hands-on tests of tools you can use with your agents, including the attempts that go nowhere.
In This Edition
Claude found that PaperPilot’s Windows download held only a README, before I installed anything.
PaperPilot built a pixel-perfect four-page pack inside a locked-down Docker container, but its plain text came back as Markdown.
A short pypdf and PyMuPDF script tied on pages and gave me cleaner text, so that’s what I’m keeping.
You get the trial prompt I used, rewritten so you can point it at a tool you’ve been meaning to install.
Nothing to Install
I started in the Claude desktop app and asked Claude to find out what PaperPilot could do before I installed anything. I gave it one rule: no PaperPilot code runs without a real sandbox. It’s the same idea as the approval gates I add to Hermes: nothing risky runs until a check passes.
It came back with a problem. PaperPilot’s Windows downloads were tiny zips with a README inside and no program. A packaging script hides that error, so nobody gets warned when the program goes missing.
Claude also spotted a wrong flag in the README and confirmed that the Linux version worked. Then it found cloud credentials in its own workspace and stopped, exactly as I’d asked.
It even corrected me. My notes said people had reported security problems on Hacker News. Claude reread the thread and found questions about licensing and where the code came from, with no security reports at all. Reading past the docs paid off, a habit I wrote about in How to Search Beyond the Docs When Your Code Won’t Work.
Before I’d installed a thing, I knew to skip Windows and run the test on Linux. I learned to check first the hard way in The Twenty Minute Audit That Found Where Hermes Was Getting Lost, after adding three tools before looking at what they’d read.
Three Tries to Run It
I moved to my home server and handed the job to Claude Code. It ran PaperPilot inside a Docker container with no network access and only one folder it could see:
docker run --rm --network none --cap-drop ALL \
--security-opt no-new-privileges --user 1000:1000 \
-v ~/pp-trial:/work -w /work ubuntu:26.04 <command>
You could do the same with Docker Desktop on a laptop.
Getting a clean run took three tries, and none of that was PaperPilot’s fault. On the first, I’d pasted only half my instructions and named no papers, so Claude Code grabbed the repo’s own test files and ran the tool on those. On the second, it stopped and asked which papers I meant. Seems fair.
Five minutes of planning would have saved me a try, as I argued in How To Architect A Feature In 5 Minutes Before Talking To AI.
On the third try, it had both papers, Attention Is All You Need and the LoRA paper. About two minutes later, I had my four-page reading pack and its text. Claude Code then checked both with pypdf and PyMuPDF , two Python PDF libraries that have nothing to do with PaperPilot.
Remember those two.
Perfect Pages, Messy Text
PaperPilot nailed the pages. Every page in the pack matched the original exactly, down to the arXiv stamps running along the margin.
But it sometimes failed too. When Claude Code asked for page 16 of a 15-page paper, PaperPilot returned an error and wrote nothing. You won’t end up with a half-finished file that looks complete.
Text extraction was the catch. Here’s how the Attention paper’s authors came out:
**Ashish** **Vaswani**
**Noam** **Shazeer**
**Niki** **Parmar**
**Jakob** **Uszkoreit** Google Brain
Google Brain
Google Research Google Research [avaswani@google.com](mailto:avaswani@google.com)
Every word survived, but PaperPilot wrapped bold words in Markdown, turned emails into links and split authors from their affiliations. It broke the LoRA title into four pieces and mangled the inline maths too.
You could search that text. I wouldn’t want to read it, or hand it to another tool, without cleaning it first.
Send this to someone with a pile of starred GitHub repos they never get around to trying.
Then Claude Wrote Its Own
Before writing this up, I asked myself how I’d do this job without PaperPilot. I didn’t have an answer. My best guess was that I’d ask an agent, and it would write a Python script.
That’s when it clicked. Claude Code had already used pypdf and PyMuPDF to check PaperPilot’s work, and those same two libraries could do the whole job. It’s the question from The Build vs Buy Scorecard, with an agent doing the building.
I sent Claude Code back with the same papers, no PaperPilot, and free rein over the method. It wrote a short script.
On pages, it was a tie. Both packs came out identical.
On text, the script won. LoRA’s title stayed in one piece, every author kept their affiliation and email, and the maths stayed readable.
It wasn’t perfect, and Claude Code said so. One setting glued a few hyphenated words together, turning “English-to-German” into “Englishto-German”, and some letter pairs came through as special characters that trip up search. Both are one-line fixes, and Claude Code left them in so the comparison stayed fair. Honest comparisons are rarer than they should be, which is why I test models on real Hermes work instead of trusting benchmarks.
It also refused to call the script faster, because the second run had everything the first one taught it. Fair enough.
Try It on Your Next Tool
I’m keeping the script. Its text needs less cleanup, and Claude Code couldn’t find a PaperPilot setting that fixes PaperPilot’s text. Tweaking one line in my own script is easier. It’s also code I own, which matters to me since I built my stack to survive vendor lock-in.
If you only need to cut and merge pages, PaperPilot did that well, as long as you’re not on Windows. For OCR, big documents or two-column papers, this trial can’t tell you much, so run your own.
In When to Use MCPs, CLIs, or Your Own Tool, I wrote that small custom tools are often enough for the job. I’ve had agents build those for me before, like the Substack API I built with Hermes and Codex. With this trial, I got to test the idea on a real tool, with files I could check myself.
Here’s the prompt to start from. Give it your real files, or your agent will grab demo files like mine did.
I want to know whether <TOOL URL> is worth installing for one job: <JOB>.
Start by checking what the current release supports for this job, using
the official documentation and downloads as evidence. Record the commit
you inspect so we can distinguish that version from later changes.
Keep planned features separate from anything available in the release.
Check whether a binary exists for my OS before suggesting a source build.
Before executing any project code, establish which isolation boundaries
are enforced in the environment where it will run. Stop if those
boundaries aren't available, since an empty folder doesn't provide them.
Use the tool inside that environment to do the job on <INPUTS>.
If another tool is needed to produce an output, explain that rather than
crediting the result to the tool we're testing.
Verify the result using an independent method and tell me how you checked
it. Include a failure case that shows what happens when an input is invalid.
Try the same job your own way on the same inputs, without using the tool
under evaluation. Compare the outputs closely enough that I can decide
which approach I'd want to keep using.
Save enough evidence in run-notes.md for someone else to follow the run.
Include the exact versions and file hashes needed to identify what was
tested. Preserve the commands with their output and any errors encountered.
Record actual timestamps without turning them into estimated savings.
Finish with your recommendation for this particular job, making clear
where the evidence is still too limited for a decision.
If I’d stopped after the PaperPilot run, I’d have a working PDF and some text to clean up. Asking Claude Code to do the job without it gave me a script I’d rather keep, all from an experiment that started with an empty Windows download.
Leave a link to a tool you’ve been meaning to try in the comments, and I’ll pick one to put through the same comparison.
Subscribe if you’d like to see what comes back when I try the next tool this way.
Top comments (0)