Search your own past writing — journals, notes, drafts — and get passages back with receipts. Runs entirely on your machine: no API key, no network, dependencies = [].
That part is a weekend build. This is about the part underneath it.
Project 1 of 5 · Agent Lab Vol 4 · a new volume
Page: https://dev48.infy.uk/agentlab/vol4-01-past-you-searcher.html
Repo: https://github.com/dev48v/past-you-searcher
The archive is invented, and that is stated before the first number
There is no public corpus of one person's journals, and no honest way to make one out of somebody else's. So the 96 documents, 17,221 words, 9 years and 28 subjects were written by hand from a fixed seed.
What it costs: every point estimate would move on a real archive. Only the shapes transfer, because they are arithmetic on the retrieval.
What it buys: the fixture knows which document is about what, so relevance is declared rather than judged. Every number below is a count over all 5,348 query–chunk pairs, not somebody's opinion of what looked relevant.
1. Five citations is not five passages
The standard recipe cuts documents into fixed windows with a fixed overlap — 700 characters and 150 are the numbers that appear most often. The overlap is there for a good reason: a sentence straddling a boundary would otherwise be split across two chunks and findable in neither.
It also means text in the overlap region is indexed twice, both copies are searchable, and both can come back in the same result set.
| indexed characters | 104,960 |
| of which a second copy of text already indexed | 14,250 — 13.58% |
| distinct documents behind 5 citations | 4.75 |
A footer that says "5 sources" is counting chunks. The reader is counting evidence.
2. A similarity threshold is a number about the scorer
The recipe ships a fixed 0.20, below which a result is a "weak match". Three real, local scorers on the same archive, over every query–chunk pair:
| scorer | share admitted by 0.20 | highest score anywhere |
|---|---|---|
| hashed bag-of-words | 43.64% | 0.6888 |
| tf-idf | 1.53% | 0.3831 |
| BM25 | 80.16% | 12.2821 |
Same cutoff, same corpus, same questions — 28× more pairs for one scorer than another. And BM25 is not on the [0,1] scale at all, so a "cosine threshold" applied to it is a category error rather than a tuning mistake.
The constructive half. A cutoff is not portable, but it is derivable: score everything once and read off the quantile. For the same 5% admitted everywhere the cutoff must be 0.4031, 0.1524 and 4.1029. Three numbers, no two alike. There is no constant to copy out of a README — including mine.
3. The obvious fix costs accuracy
Keep at most one chunk per document and the duplicate receipts vanish completely. It also makes the results worse:
| scorer | precision@5 as shipped | one per document | Δ |
|---|---|---|---|
| hashed | 0.1857 | 0.1929 | +0.0071 |
| tf-idf | 0.5429 | 0.5286 | −0.0143 |
| BM25 | 0.5214 | 0.5071 | −0.0143 |
The rule cannot tell "the same sentence twice" from "this document has two separate passages that both answer the question" — both look like two chunks with one doc_id. It helps the weakest scorer and hurts the two good ones. That is a trade, not a fix.
4. The one claim I could check, checked
The recipe says the hashing fallback is weaker than the model it recommends. Half of that is unverifiable without the model; the other half holds and by a wide margin — precision@5 0.1857 against 0.5429.
Worth noticing that the two good scorers disagree about which is better depending on which number you read: tf-idf wins on precision@5, BM25 wins at rank 1 (20 of 28 against 19).
5. A weak temporal effect, reported as measured
The expectation was that questions with no date in them would be answered disproportionately out of the years you wrote most in, crowding out the past you came for.
The older half is 28.80% of the archive and 21.43% of the answers — under-returned, but not in order. The least-returned years are 2022 (lift 0.45) and 2019 (0.50), and 2017, the oldest of all, comes out slightly over-returned. Part of the reason is top-k itself: a year holding a third of the archive can still win at most 5 slots per question, so the fixed budget caps exactly the years that would otherwise dominate.
Two porting hazards worth stating
The page runs the whole engine in your browser — a port of the Python, including SHA-256 — so two JavaScript traps had to be handled rather than hoped over:
// wrong: bitwise operands are coerced to 32 bits, and nextInt() * n
// overflows that for every n above 1
return (this.nextInt() * n) >>> 31;
// right
return Math.floor((this.nextInt() * n) / 2147483648);
And the control demonstrating the low-bit LCG defect needs BigInt: 1103515245 * state exceeds 2^53, so a plain multiply silently rounds and the control "disproves" the defect by computing a different generator. It reported 18 flips instead of 1999 until that was fixed.
What checks what
- 51 pytest, several of them controls that show the failure happens without the guard
- a differential verifier, 3,579 asserts, 0 failures: it lifts the page's engine out of the HTML, runs it under Node, and diffs it against reference JSON from the real Python package — every document by SHA-256, every chunk span as an exact integer, and all 840 ranked hits across 28 questions × 3 scorers × both dedupe modes
- the page's own self-check, 33 of 33, running in your browser on load
Python standard library only. MIT.
Top comments (0)