vapmail16

Posted on Sep 20

A complete engineering log of building `findmypylibrary` with an AI pair-programmer (Claude Code)

#ai #coding #python #softwaredevelopment

From a placeholder to a release: every bug, wrong turn and decision behind a small PyPI search tool

A complete engineering log of building findmypylibrary with an AI pair-programmer (Claude Code): what broke, how we found out, what we tried, what we measured, and why each decision went the way it did. Nothing is left out, including the mistakes that were ours.

What the tool is

findmypylibrary answers one question: "I need to do X in Python. Which package?"

pip install findmypylibrary
findmypylibrary refresh
findmypylibrary "fuzzy string matching"

1. RapidFuzz  (score 0.90)
   rapid fuzzy string matching
   downloads/30d: 163,835,611  last release: 2026-08-30
2. pfzy  (score 0.81)
   Python port of the fzy fuzzy string matching algorithm
   downloads/30d: 27,024,226  last release: 2022-01-28
3. fuzzywuzzy  (score 0.73)
   Fuzzy string matching in python
   downloads/30d: 14,789,165  last release: 2020-02-13

Three constraints were fixed on day one and never changed, and they explain most of the decisions below:

Grounded in real data, not a language model's memory. Every result is a real package with real download counts and a real release date.
Offline after the first download. Queries never leave your machine.
No API key, no account, no heavy dependencies. Twice during the project the "obvious" fix was embeddings and a model download. Twice we said no.

The project started as a 0.0.1 placeholder on PyPI: a name reservation with a README that described two commands that did not exist yet. This is the story of getting from there to something we were willing to release.

Step 1. Where does the data come from?

The plan needed, for the most-used packages on PyPI: name, description, download count and last release date.

The first obstacle: PyPI has no free bulk endpoint for "all packages with downloads". Download statistics live in a public BigQuery dataset, and BigQuery needs a Google Cloud account and key. That breaks constraint 3.

What we did instead: two public sources, neither needing a key.

Source 1: hugovk/top-pypi-packages, a JSON file of the most-downloaded packages over 30 days, itself rebuilt periodically from BigQuery by its maintainer.
Source 2: the PyPI JSON API (https://pypi.org/pypi/<name>/json), one request per package, for the summary and release date.

A small first bug: the dataset's documented URL returned an HTML page, not JSON. It was a 301 redirect to a new domain that our first curl did not follow. We switched to the raw GitHub URL. Lesson filed for later: never assume a 200, and follow redirects (this came back to bite us in Step 12).

Decision: how many packages? The question was 200, 2,000 or 8,000. The answer from the product owner was "15,000, or whatever covers all actively downloaded libraries". When we opened the dataset it contained exactly 15,000 rows, down to packages with about 68,000 downloads a month. So "top 15,000 by 30-day downloads" became the definition of "active", for free.

The cost of that decision: 15,000 packages means 15,000 HTTP requests per snapshot. Sequentially that is over an hour. With httpx.AsyncClient and a semaphore of 25 concurrent requests it takes a few minutes. Hold that thought; it becomes the scaling problem in Step 5.

Step 2. The first working version

Architecture, deliberately boring:

fetch.py   two sources -> list of package dicts (async, bounded concurrency, retries)
cache.py   sqlite file in ~/.cache/findmypylibrary/
rank.py    query -> ranked packages
cli.py     click commands: refresh, search

Three details worth recording.

1. A bare query had to work. The README promised findmypylibrary "parse messy pdfs", with no search subcommand. Click groups reject unknown command names, so we subclassed the group:

class DefaultGroup(click.Group):
    def resolve_command(self, ctx, args):
        try:
            return super().resolve_command(ctx, args)
        except click.UsageError:
            return super().resolve_command(ctx, ["search", *args])

This looked finished. It was not (Steps 12 and 15 found three more cases).

2. An ordering trap in the async crawl. We used asyncio.as_completed to drive a progress bar. It yields results in completion order, so you cannot zip them back onto the input list to attach download counts. We caught this at design time and made each task return its own complete record, download count included, instead of re-joining afterwards.

3. We did not hammer PyPI while developing. Every test of the pipeline during development used refresh --limit 40. The full 15,000-package crawl was run only when we needed real data. Result of the first full run: 14,999 of 15,000 packages (the missing one was a genuine 404, a delisted package).

The first ranking was textbook: a pure-Python BM25 over name + summary + keywords, then a weighted blend.

score = 0.60 * relevance + 0.25 * popularity + 0.15 * recency     (each min-max normalised)

It produced sensible results for the first few queries we tried. That was the problem: we only tried a few.

Step 3. The first ranking bug: `openpyxl` loses to a package nobody has heard of

Query: read and write excel spreadsheets.

1. numbers-parser   Read and write Apple Numbers spreadsheets     924,605 downloads
2. tifffile         Read and write TIFF files                  27,744,478 downloads
3. openpyxl         A Python library to read/write Excel ...  339,316,525 downloads

openpyxl is the right answer and it has 370 times the downloads of the winner.

Diagnosis (we looked at the numbers instead of guessing). openpyxl matched three of the four query words, so it was not a recall problem. Two things combined:

BM25's short-document bias. numbers-parser's whole summary is six words, three of which are in the query. BM25's length normalisation rewards that density.
Min-max normalisation of each factor separately. A large relevance gap, weighted 0.6, can never be recovered by a popularity factor weighted 0.25, no matter how big the download gap is.

The fix: stop blending, start gating. Relevance became a gate (keep anything within 50% of the best match), and the survivors were ranked mainly by popularity. This is how real package search engines work: text match for recall, authority for ordering. We also weighted matches in the package name above matches in the summary, and added a small synonym list (excel/xlsx/spreadsheet and a dozen more).

This was done test-first. The regression test asserts the outcome a user sees, not an internal number:

def test_popular_relevant_package_beats_short_keyword_dense_niche_one():
    result = names("read and write excel spreadsheets", [openpyxl, numbers_parser, tifffile])
    assert result[0] == "openpyxl"

We rejected one idea here on principle: a local embedding model. It would have pulled in a deep-learning runtime and a model download for a tool whose selling point is being light.

Step 4. Quality gates, borrowed from a completely different project

The product owner uses a strict audit methodology on a large TypeScript / Postgres / Playwright application, and asked for the same gates here:

Test-driven development
Over 90% coverage on new code
100% pass rate
Zero type and lint errors, plus Playwright end-to-end tests and a blast-radius check

Applied literally, half of that is meaningless for a Python CLI with no browser, no database server and no TypeScript. Copying the letter of a process into a different stack is how you end up with ceremonies nobody can explain. So we translated the intent:

Their gate                      What it became here
-----------------------------   ---------------------------------------------------
TDD                             pytest, regression test written before each fix
>= 90% coverage                 pytest-cov, enforced in CI
0 TypeScript / lint errors      ruff (lint + format) and mypy, zero findings
Playwright end-to-end           subprocess tests that run the real installed binary
Leftover / twin host inventory  grep every importer of anything we change or remove
"Ledger-green, product-red"     assert the outcome users see, never a proxy for it

That last line is the core lesson of their methodology: a test that checks "the file exists" or "the score changed" can be green while the product is broken. It turned out to apply to us more than once.

Mistake 1. We deleted the real data to make the tests clean

To isolate test runs, a cleanup command removed ~/.cache/findmypylibrary before running pytest. That directory was not a test artefact. It was the real 15,000-package snapshot that had taken a full crawl to build.

Fix: rebuild it, then make the mistake impossible rather than merely avoided. Every test now runs with XDG_CACHE_HOME pointed at a temporary directory, through one autouse fixture:

@pytest.fixture(autouse=True)
def isolated_cache_dir(tmp_path, monkeypatch):
    monkeypatch.setenv("XDG_CACHE_HOME", str(tmp_path / "xdg-cache"))

Lesson: "be careful" is not a control. A fixture that every test gets automatically is.

(We broke this rule again, through a different door, in Step 15.)

Step 5. The scaling problem: every user would crawl PyPI

With a working tool, the obvious flaw surfaced: refresh makes 15,000 requests to PyPI. One developer doing that monthly is fine. A thousand users doing it is not, and it would get everyone rate-limited.

Decision: build the snapshot once, centrally, and let users download it. A scheduled GitHub Actions workflow builds the snapshot monthly and publishes it as a GitHub Release asset. findmypylibrary refresh downloads that file (one request, a couple of seconds). refresh --build-locally keeps the full crawl for anyone who wants a snapshot as of right now.

This is the same pattern the upstream dataset uses for itself, which is a good sign it is the right shape.

A guardrail worth mentioning: the AI assistant's permission system refused to create a public GitHub repository on its own. It stopped, explained, and asked. Creating a public surface is a human decision. We approved it explicitly, and only then did it create the repo and push.

We verified the loop end to end, not just the workflow: trigger it manually, confirm the release exists, then run refresh on a clean machine profile and confirm it downloads and answers queries.

Step 6. "Is it really ready?"

Asked directly, the honest answer was a table of what had not been verified:

Gap                                 Status after this step
----------------------------------  ---------------------------------------------
No CI; gates only ran by hand       CI on every push and pull request
Only tested on macOS, Python 3.12   Matrix: Linux, macOS, Windows x 3 Pythons
Reader during a refresh?            Concurrency test added
Real HTTP 429 from PyPI             Still mock-only. We will not hammer a public
                                    service to force a rate limit. Said so plainly.

Saying "this one we cannot test ethically" is part of being ready. Hiding it is not.

Step 7. The full review, and the uncomfortable discovery

We ran 15 everyday queries against the real snapshot. About half were good. Five failed in a way that would embarrass the tool on day one:

Query                           Missing from the top results   Why (confirmed in the data)
------------------------------  -----------------------------  ---------------------------------------
dataframes                      pandas                         its metadata never says "dataframe"
resize images                   Pillow                         summary says "Imaging", not "images"
plot charts                     matplotlib                     summary says "plotting", not "plot"
connect to postgres database    psycopg2 (pymssql came first)  "postgres" is not "PostgreSQL"
unit testing                    pytest                         matched one word; the gate dropped it

Two root causes. Matching was exact-token, with no stemming. And the text was tiny: summaries average 8 words, and 259 packages have none at all. A 13-entry synonym list cannot paper over that.

The same review found seven robustness gaps. Each is a story of "it works on the happy path":

A failed crawl overwrote a good snapshot. refresh saved whatever it got. A network drop at 10% would replace 15,000 packages with 1,500, and in the workflow that would be published to everyone.
The download URL was a trap. The CLI downloaded from releases/latest/.... The first time we published a code release on GitHub, "latest" would stop being a snapshot, the URL would 404, and every user would silently fall back to the 15,000-request crawl.
Re-running the workflow in the same month failed, because the dated release tag already existed.
Raw tracebacks reached users for a corrupt cache file and for no network.
The silent fallback crawl was too aggressive. GitHub unreachable should not mean "hit PyPI 15,000 times without asking".
No schema version. A future table change would break every old install that downloaded the new file.
No staleness signal. A snapshot could be a year old and the tool would never say.

We decided not to publish until the ranking and gaps 1 to 5 were fixed. A tool that cannot find pandas is not a release.

Step 8. Rebuilding ranking on SQLite FTS5

The design decision. We needed stemming and more text per package. More text kills the pure-Python BM25: building an index over 15,000 documents of ~250 tokens on every query would take seconds. The index had to be built once and shipped.

SQLite already has this. FTS5 is a full-text engine with a Porter stemmer and a BM25 ranking function, it is compiled into standard CPython on all three platforms (the CI matrix later proved it), and it adds zero dependencies:

CREATE VIRTUAL TABLE packages_fts USING fts5(
    name, summary, keywords, topics, description,
    content='', tokenize='porter unicode61'
);

content='' makes it contentless: README text is searchable but not stored, which keeps the file small. Result: 19.8 MB on disk, 10.9 MB gzipped, and queries in tens of milliseconds.

More text, carefully. We added each package's topic classifiers and a cleaned excerpt of its README. "Cleaned" matters: in pandas' README the word "DataFrame" first appears about 3,000 characters in, behind badges, links and HTML. We strip images, links, URLs, tags and reST directives before indexing.

The golden queries came first. Before touching a ranking constant we wrote 40 everyday queries, each with a list of acceptable right answers ("resize images" -> pillow, opencv-python, ...). Writing the expectations before seeing results is the only way to keep yourself honest. Baseline on the new index: 37 of 40.

The discovery that shaped the design: README noise. For unit testing, the results were boto3 and tqdm. Why? boto3's README has a section that says "run the unit tests". tqdm's talks about iterations-per-second "units". We had given the description column a weight of 1 against 8 for the name, and it made almost no difference. The reason is a property of BM25: term frequency saturates, so once a term appears at all, the column weight barely moves the score. boto3's README-only match scored 0.73 of the best real match.

Fix: score the core fields (name, summary, keywords, topics) and the description in two separate queries, and add the description score at a heavy discount. A parameter sweep showed the pass rate was flat at 39/40 across a wide band of settings, which is what you want to see: a plateau, not a knife-edge.

A better metric. Pass/fail at top 5 hides ordering. We added mean reciprocal rank (1 for a right answer in first place, 1/2 for second, and so on). It showed that scaling popularity by "log-downloads divided by the maximum" barely separated a package with a million downloads from one with a billion (the logs are 6 and 9). Min-max scaling over the surviving candidates lifted MRR from 0.69 to 0.88.

A test whose premise was wrong. We asserted that tqdm must not appear for "unit testing". It kept appearing. Instead of tuning until it vanished, we looked: tqdm's own PyPI classifiers include Education :: Testing. That is the package describing itself, not README noise. The test was wrong, so we fixed the test, and kept the assertion for boto3, which was the genuine case.

Step 9. A real bug hiding behind a green test

The concurrency test (searches running while a refresh rewrites the snapshot) was passing. Its output contained a warning: the writer thread had died with sqlite3.OperationalError: disk I/O error.

The test passed because it only collected errors from the reader thread. It was asserting a proxy.

We reproduced it in isolation and asked SQLite for the extended error: SQLITE_IOERR_LOCK. The cause was opening the snapshot through a read-only URI (file:...?mode=ro). On macOS, those connections made a concurrent writer's lock acquisition fail, three to five times per run. Plain connections never failed.

Fix: open normally and make the connection read-only with PRAGMA query_only = ON. And fix the test so it records writer failures too. A bug fix without the test fix would have left the same blind spot for the next bug.

Step 10. Closing the robustness gaps

Each with the reason, because the reason is what transfers to other projects.

Fixed release tag. Downloads now come from releases/download/snapshot-latest/..., a tag that only the snapshot workflow writes to. Code releases can never change what that URL points at.
Schema version in the file name (snapshot-v2.sqlite.gz) and in a meta table. Old installs keep downloading a layout they understand. A mismatch produces a clear "run refresh" message.
The 95% rule. A crawl that fetched under 95% of the list is discarded, and the existing snapshot is kept.
No silent fallback. If the download fails, refresh fails with a message that names --build-locally. The user decides whether to make 15,000 requests.
Validate, then swap. A download is decompressed to a temporary file, validated, and only then moved over the real snapshot. A bad download can never damage a good one.
A quality gate before publishing. The workflow runs the golden queries against the freshly built snapshot and refuses to publish if the pass rate falls below 85%.
Staleness. A status command, and a warning after 45 days.
Trusted Publishing. Releases to PyPI happen from a GitHub Actions workflow triggered by a version tag, using OpenID Connect. There is no PyPI token stored anywhere, and nobody pastes one into a terminal.

Step 11. Measuring an impossibility: pandas versus boto3

pandas still did not appear for dataframes (polars did, so the golden query passed, but the review had named pandas). Could we fix it?

pandas matches dataframes only through its README. boto3 matches unit testing only through its README. We swept the two relevant parameters and printed both positions side by side:

gate   readme weight   pandas for "dataframes"   boto3 for "unit testing"
-----  -------------   -----------------------   ------------------------
0.05   0.5             7th                       6th
0.10   0.5             7th                       6th
0.15   0.5             7th                       6th
0.20   0.5             not shown                 not shown
0.25   0.5             not shown                 not shown

They move together, every time. To a lexical ranker the two cases are the same case. You cannot have one without the other.

Decision: precision wins. No boto3 for "unit testing", and pandas is found by the words its own metadata uses (data analysis puts it first). We wrote this limit into the README with those exact examples, instead of tuning until a demo looked good. The same sweep gave a free improvement (a README weight of 0.5 took the golden set to 40/40 with boto3 still excluded), so we took it.

We also tried weighting the coverage gate by term rarity. It removed one junk result and dropped Pillow and pytest. Rejected, with the numbers in the commit message.

Mistakes 2 and 3. Pushing on red, and being lied to by bytecode

Mistake 2. A commit was gated like this:

pytest -q | tail -3 && git commit ... && git push

pytest failed. The commit and push happened anyway, because the exit status of a pipeline is the exit status of its last command, and tail always succeeds. Red code reached the main branch.

Fix: never put a pipe between a gate and the thing it guards. Capture output to a file and test the real exit code:

fail=0
pytest -q > out.txt 2>&1 || fail=1
ruff check -q . || fail=1
mypy src > /dev/null || fail=1
if [ "$fail" -ne 0 ]; then echo "GATES RED - NOT COMMITTING"; exit 1; fi

Mistake 3, found while investigating mistake 2. The "failures" were not real. To prove a regression test had teeth, we had temporarily changed a constant in rank.py from 0.5 to 1.0, confirmed the test failed, and restored it. Both versions of the file had the same size and were written within the same second. Python validates cached bytecode by modification time and size, so it considered the stale .pyc valid and kept loading the 1.0 version. The source said one thing and the interpreter ran another.

Fix: after any mutation check, delete __pycache__. CI, which starts from a clean checkout, confirmed the pushed commit was green all along.

Two lessons: check exit codes, not output; and when results contradict the source code, suspect the cache.

Step 12. An independent, adversarial review

Asked "is anything left?", re-running our own checklist would only confirm what we already believed. So we started a second AI reviewer with no edit rights and a specific brief: here is what the author believes is handled; find what they missed. In parallel we attacked the real CLI with odd inputs. Between the two:

Search crashed. A three-word query where no package matched half the words (say, two typos) raised ValueError: min() iterable argument is empty. This sat behind 98% coverage.
refresh --build-locally --limit 0 wiped the snapshot. The 95% guard computed 0 < 0.95 * 0, which is false, and saved an empty list. 14,999 packages became 0, exit code 0.
-n -1 printed 389 results (a negative slice).
One bad response killed a 15,000-package crawl. A single 403 with an HTML body raised JSONDecodeError out of the whole run. It now costs one package. We also follow redirects now (remember Step 1).
Corruption past the first page of the file crashed queries, and would have passed download validation. We now run PRAGMA quick_check before swapping a download in.
résumé parser was tokenised as r, sum, parser. The tokeniser was ASCII-only while the index folds accents.
findmypylibrary status bar widget was a usage error because status is a command. So was -n 2 parse pdf.

And two findings about the tests, which mattered more than any single bug:

Vacuous assertions. Several CLI tests asserted "Traceback" not in output. Click's test runner keeps uncaught exceptions in result.exception and never writes them to output. Those assertions could not fail. That is exactly why the search crash survived. The test helper now fails on any uncaught exception:

  crashed = result.exception is not None and not isinstance(result.exception, SystemExit)
  assert not crashed, f"CLI crashed with {result.exception!r}"

A circular fixture. The offline golden test ran against a slice of the corpus built from "the ranker's current top 25 per query". It contained only packages the ranker already liked, so a new bad competitor could never appear. We rebuilt it to include the most-downloaded packages matching each query's terms regardless of score, plus a seeded random sample of the corpus.

Performance, measured rather than assumed. Every search took 0.30 seconds, and 0.14 of that was importing httpx, which search never uses. Importing the fetch module lazily halved the start-up time to 0.15 seconds. We also measured FTS5 optimize plus VACUUM: 5% smaller, no speed-up. Not worth a line of code, so we did not add it.

Step 13. The bug only Windows could show us

Piping results to head closed the pipe early. A new, broad except OSError handler caught the resulting error and told the user to "check permissions and free disk space". We fixed it by letting BrokenPipeError through.

Local tests passed. CI failed on all four Windows jobs. On Windows a closed pipe does not raise BrokenPipeError (EPIPE). It raises OSError with EINVAL.

The real problem was the design, not the missing case: a catch-all around the whole command could not tell "I cannot write the cache" from "nobody is reading my output". We removed it and did two precise things instead:

File errors are converted to a clear message where they happen (the cache directory, the snapshot write, the download).
The console entry point handles "my reader went away" once, for both error codes:

  def run() -> None:
      try:
          main()
      except OSError as exc:
          if exc.errno not in (errno.EPIPE, errno.EINVAL):
              raise
          os.dup2(os.open(os.devnull, os.O_WRONLY), sys.stdout.fileno())
          sys.exit(1)

No amount of local care would have found this. A cross-platform CI matrix is not a formality.

Step 14. The honest number: held-out evaluation

The golden set said 100%. But we had tuned on it. So we wrote 30 new queries, fixed their expected answers before running anything, and ran them once.

Tuned set (40 queries):      40/40 in the top 5   MRR 0.89
Held-out set (30 queries):   25/30 in the top 5   MRR 0.72

83%, not 100%. That gap is the price of tuning on your test set, and it is the number users actually experience.

The failures had one pattern. A package with a billion downloads that matched only the commonest word of the query floated to the top:

gui application   ->  idna, platformdirs, filelock      (all match only "application")
geospatial data   ->  pandas, platformdirs, tzdata      (all match only "data")
discord bot       ->  python-telegram-bot first         (matches only "bot")

Experiment 1: multiply the whole score by how much of the query's rare vocabulary a package covers. The junk vanished, but so did the good partial matches (Pillow, pytest). Total passes fell from 65 to 61. Rejected.

Experiment 2: apply that factor to the popularity term only. A package keeps its relevance score, but it only earns popularity credit in proportion to the informative words it matches.

exponent   tuned set     held-out set    junk entries in four probe queries
--------   ----------    ------------    ----------------------------------
0.0        40  (0.892)   25  (0.715)     10
0.25       39  (0.869)   25  (0.737)      5
0.5        39  (0.840)   25  (0.764)      2
1.0        38  (0.843)   25  (0.759)      1

Better on the held-out set. But the held-out set had now influenced a decision, so it was no longer held out. We wrote a third set of 25 fresh queries to validate:

Validation set (25 queries):   MRR 0.755 -> 0.858
ldap authentication:    google-auth, google-auth-oauthlib, ...  ->  django-auth-ldap, python-ldap, ldap3
currency conversion:    pillow, pymupdf, forex-python           ->  forex-python, currency-symbols, ...
bloom filter:           soupsieve, eth-bloom, Markdown          ->  eth-bloom, bloom-filter2, pybloom-live

Adopted. The one query it lost, graph algorithms, exposed a flaw in our own synonym list: "graph" had been grouped with "chart" and "plot", so matplotlib beat networkx. We removed it.

One more rejected idea, because rejections are results too. pytest dropped out of unit testing (its keywords say "unittest", one word). Adding unit <-> unittest as a synonym fixed it and immediately polluted unit conversion with testing tools. The "general" fix, treating every adjacent pair of query words as a possible compound, made things much worse: 84 of 95 instead of 89, because rare accidental compounds like "machinelearning" inflate the top relevance score and push good packages below the gate. What worked was a four-entry list of real compounds (unit test, time zone, data frame, web socket) that applies only when both words are in the query.

Final numbers. All 95 queries are now the permanent regression suite. 90 pass. Of the 55 that were never used for tuning, 49 passed when first run (89%). The README reports the 89% and explains why the 95% is flattering. The rebuilt offline fixture (2,743 real packages) reproduces the full-snapshot result exactly: same 90, same five misses.

Step 15. The release-manager review, and the reviewer's own mistake

A second independent review used a different lens: check every claim in the docs against the code, be a first-time user, think about supply chain. It found:

refresh --limit 5 (without --build-locally) silently ignored --limit and downloaded the full snapshot. Now a usage error.
status --json ran a search for the word "status". Now a usage error.
The Python API returned internal fields (id, relevance), used download_count where the JSON output said downloads_30d, did not validate top_n, and raised an exception it did not export. All settled before a first release froze the contract.
Package summaries were printed to the terminal unsanitised. We now strip control characters at crawl time so an escape sequence in a PyPI summary cannot reach anyone's terminal.
Downloads are size-capped, and the downloaded SQLite file is opened with trusted_schema = OFF.
The workflow's write token is now scoped to the single publishing step, not the step that crawls untrusted data.
Licence metadata updated for PEP 639; Python 3.14 added to CI.

And the reviewer made a mistake of its own. Told in plain words never to touch the real cache, it ran refresh --limit 5 expecting a usage error, without isolating the cache directory. The command ran and re-downloaded the snapshot over the real one. The content was identical and nothing was lost, and it reported the deviation itself, first, at the top of its report. But it is Mistake 1 again, through a different door. It also found a real bug in the process (--limit being ignored), which is a nice illustration that accidents are data.

Lesson: an instruction is not a sandbox. If something must not be touched, make it unreachable (an environment variable set for the whole process, a read-only mount), do not just ask nicely.

Verifying the automation for real

A workflow you edited and did not run is an untested program. Each time we changed the refresh workflow we triggered it and checked the result, four times in total, including the path that only runs the second time (release already exists, asset replaced in place, notes updated after the upload). The publish workflow refuses a tag that is not on the main branch, refuses a tag that does not match the package version, and refuses to publish if the snapshot that version downloads is not already on the release page, so a new install can never hit a 404.

Finally, the whole first-run journey from a wheel installed into a clean environment: search with no snapshot (clear message), contradictory options (usage error), refresh (real download), status, searches, JSON piped to head, the Python API, the typing marker, and proof that a search does not import the HTTP stack.

Where it ended up

Tests                135, coverage 97%, ruff and mypy clean
CI                   15 jobs: Linux, macOS, Windows x Python 3.10 to 3.14
Ranking              90 of 95 golden queries; 89% on queries never used for tuning
Snapshot             14,999 packages, 10.8 MB download, rebuilt monthly behind two gates
Search               ~0.15 s per invocation, offline
Source distribution  18 KB (was 780 KB before we excluded the test corpus)

Known limits, stated in the README rather than discovered by users: matching is lexical, so numpy does not appear for "linear algebra"; about one search in ten will not show a package you would call right; a real PyPI rate limit has only ever been simulated; GitHub pauses scheduled workflows after 60 days without commits, and the 45-day staleness warning is the safety net for that.

The lessons, collected

On testing

Assert what the user sees. "No traceback in the output" could never fail; "the command did not raise" can.
A test that only records half the failures (the reader thread, not the writer) is a proxy. Fix the test when you fix the bug.
When a test fails, first ask whether its premise is true. tqdm really does declare a Testing topic.
A two-document test corpus has degenerate statistics. Make fixtures realistic, then prove the test still fails when the feature is broken.
Do not build an evaluation set from the output of the thing you are evaluating.

On evaluation

Write expected answers before you look at results.
Whatever you tune on stops being a test. Keep a held-out set, and when it influences a decision, write another.
Report the unflattering number. We publish 89%, not 95%.
Look for a plateau, not a peak. If a parameter only works at one value, you have fitted noise.
Rejected experiments are results. Write down what you tried, the number it produced, and why you dropped it.

On robustness

Every "it can't be zero" will be zero: --limit 0, an empty crawl, an empty survivor list.
One bad item must cost one item, never the batch.
Validate, then swap. Never write a download over good data.
Do not fall back silently to the expensive path. Fail and say how to opt in.
Catch errors where you know what they mean. A catch-all at the top turns "reader closed the pipe" into "check your disk".
Version anything you download, in the file name, from the first release.
Never build a permanent URL on "latest".

On process

Translate a methodology's intent to your stack; do not copy its checklist.
Never put a pipe between a gate and what it guards. Check exit codes.
When behaviour contradicts the source, suspect the cache (__pycache__, in our case).
Instructions are not sandboxes. Make the dangerous thing unreachable.
Run the real path: the installed wheel, the real workflow, every operating system you claim to support.
A second reviewer is only worth having if it looks for something different. Tell it what you already believe, and ask it to prove you wrong.
Measure before optimising. One of our two "obvious" optimisations halved start-up time; the other did nothing.
Put the limits in the README. Users forgive a documented limit far more easily than a surprise.

Appendix: the decision log

Decision                                  Alternatives considered            Why
----------------------------------------  ---------------------------------  ------------------------------------------
Two keyless public data sources           BigQuery                           no account or key for users or maintainers
Top 15,000 packages                       200 / 2,000 / 8,000                matches the upstream dataset; covers the
                                                                             long tail down to ~68k downloads a month
Central monthly snapshot + download       every user crawls                  15,000 requests per user does not scale
Fixed release tag "snapshot-latest"       releases/latest, dated tags        code releases must not move the URL;
                                                                             re-runs in a month must not fail
SQLite FTS5 (porter, contentless)         pure-Python BM25; embeddings       stemming and a shipped index with zero new
                                                                             dependencies; embeddings break "lightweight"
Core fields and README scored apart       one weighted index                 BM25 term-frequency saturation makes column
                                                                             weights nearly useless against README noise
Gate, then rank                           linear blend                       short-document bias swamped popularity
Popularity credit x informative coverage  hard rarity gate; whole-score      only variant that removed junk without
                                          penalty                            losing the good partial matches
Curated compounds (4 entries)             synonym; compound every pair       synonym leaked; general rule scored 84/95
Precision over pandas-for-"dataframes"    lower the gate                     inseparable from boto3-for-"unit testing"
No silent crawl fallback                  automatic fallback                 the user must opt in to 15,000 requests
Trusted Publishing on a version tag       manual twine upload with a token   no stored or pasted secrets
Lazy import of the HTTP stack             eager import                       measured: 0.30 s -> 0.15 s per search
No FTS optimize / VACUUM                  add it                             measured: 5% smaller, no speed-up

If you take one thing from this log: most of the bugs above were invisible to the checks we already had. They were found by asking a different kind of question each time ("what happens on Windows?", "what if this is zero?", "what does it score on queries it has never seen?", "what would someone who wants me to be wrong find?"). Asking the same question twice, more carefully, would have found none of them.

Top comments (1)

mthburnsbarber-web • Sep 20

The held-out evaluation section is the most important part and it's buried near the end. Publishing 89% (the held-out number) instead of 95% (the tuned number) is exactly the right call — "report the unflattering number" should be a principle in every engineering log.

The circular fixture problem is subtle and easy to miss: building the offline test corpus from "the ranker's current top 25 per query" means a new bad competitor can never appear. The corpus is self-validating the wrong thing. The fix — seeded random sample of the corpus plus most-downloaded packages matching query terms regardless of score — is the correct structure.

The pipe-between-gate-and-what-it-guards mistake (pytest -q | tail -3 && git commit) is one of those failures that looks embarrassingly obvious in hindsight but is genuinely easy to do in the moment. Capturing to a file and testing the real exit code is the fix that transfers everywhere.

"Instructions are not sandboxes. Make the dangerous thing unreachable." — this applies far beyond this project. The reviewer's mistake in Step 15 (same as Mistake 1, through a different door) demonstrates that you can't reliably tell humans or AI assistants "don't touch the real cache." The autouse fixture that redirects XDG_CACHE_HOME is the structural solution, not the careful warning.

The decision log at the end is worth its own post.