DEV Community: Artur Daschevici

What the rewrite deleted

Artur Daschevici — Wed, 09 Sep 2026 10:34:00 +0000

Part three of a series on chops-search, a hybrid search engine for static sites. This one is less about search than about code review, and it stands alone. Part two is here.

Part two ended with a persistence batch: three calibrated scoring values (min_gap, rrf_alpha, an optional min_cos override) becoming config keys, riding into index.bin next to the field weights under one version bump, and getting read out by the engine at construction. The point was to close an honesty gap, since CI had been certifying a configuration that existed only as CLI flags while every visitor's browser ran the defaults. After the batch: the config file states the scoring, the build bakes it, the browser runs it, and a bare eval measures it. One configuration in four places, none of them shell history.

The batch came with a boundary rule worth restating because everything below leans on it: persist what you calibrated, default what you did not. Only the three knobs that earned calibrated values got config keys. rrf_k stays a compiled constant because its sweep was flat at every value. strong_cos stays off because no fixture case has ever needed the hatch. Every knob now has one of three provenances, corpus-calibrated in the artifact, compiled constant, or per-run flag override, and the scoring: line that eval prints marks flag overrides with an asterisk, because a sweep transcript that cannot distinguish "the index shipped 0.08" from "a flag injected 0.08" is the exact debt the batch exists to close, in a new denomination.

The batch landed. Tests green, artifacts building, search working. This post is about the review that happened next, because the review found seven problems, and the pattern across all seven is the thing worth writing down: a rewrite deletes what it does not understand, and deletions are invisible in the green path. Tests pass, everything works, and the file is quietly more fragile than the one it replaced. The review question that paid was not "is the new code right?" It was "what did the old code do that the new code no longer does?"

First, the one genuinely interesting design decision

Before the review findings, the wire-format decision the batch forced, because two of the tests it produced show up later.

min_cos is an override, not a default, and "absent" and "0.0" are different sentences. Absent means "derive the floor from embedding dimensionality at construction," which is right for almost every corpus and tracks a dims change automatically. An explicit 0.0 means "floor off," per the house convention that floors disable at zero. Conflate them and one of two legitimate engines becomes unspellable.

So the wire keeps them distinct with a fixed-width presence flag: one byte, then a four-byte f32 value field, written as zero when absent and ignored on read. Five bytes where four would have blurred the distinction. Fixed width rather than conditional, so the layout stays trivially seekable and the absent case has exactly one byte representation, which matters here because the artifact filenames are content hashes, and a nondeterministic byte forces every visitor on the planet to re-download the index.

The distinction had to be pinned at every layer that could lose it, because every layer had a habit that would merge them: a defaulted TOML key, a bare float field in the struct, an unwrap_or in the constructor. Three tests: the wire test asserts None and Some(0.0) produce different bytes and both round-trip, the config test asserts min_cos = 0.0 parses to an armed override while an absent key stays None, and the engine test asserts all three states produce the three intended engines. Keeping one distinction alive across three serialization boundaries takes a presence flag, three tests, and the willingness to spend the extra byte on it.

The review: seven absences

The batch landed as large rewrites of format.rs and config.rs rather than surgical diffs, which is what made the review question necessary. Here is what the old code did that the new code no longer did, ordered by how much it would have cost.

The dropped bounds checks. The old Index::read rejected chunk entries and posting doc ids pointing past the document count. The rewrite dropped the checks, and Engine::new indexes per-doc arrays with those values unguarded. In native code that is a caught panic with a stack trace. In wasm it is an aborted module. A truncated or corrupted artifact went from "loud FormatError with a rebuild instruction" to "the search box dies silently on every page of the site." The same rewrite dropped the NaN rejection on field weights, whose failure mode is quieter still: NaN does not crash anything, it just NaNs every score it touches, and search returns garbage with complete confidence.

The thematically perfect one. The config parser's unknown-key rejection disappeared in the same batch that made config keys load-bearing for scoring. The old code had the check, and a comment defending it: "a misspelled chunk_size that silently does nothing is worse than a failed build." Without it, a typo'd min_gp = 0.08 parses cleanly and ships the gate disarmed. Which is the honesty gap this entire batch was built to close, reintroduced through a spelling error. The check went back with the three new keys in its list, and the regression test's fixture is literally the string min_gp. A typo should cost a build, not a calibration, and the stricter the meaning of a config file, the stricter its parser has to be about words it does not know.

The parser's answer to the typo. min_gap sits right there in the known-keys list, which is the point.

Two defaults changed silently inside the refactor. The compiled dims default moved from None (native size) to Some(128), making "native" unspellable and contradicting the field's own doc comment three lines above it. The model path changed too. Both reverted, with the 128 claim moved to where it belongs: the two deployed sites' config files, where the size-versus-recall decision is visible in a diff. Here is the uncomfortable part: 128 was probably even the right value, and that is exactly what made it dangerous, because nothing looked wrong. Defaults changed inside a refactor are claims smuggled past review. A behavior change carries its own commit, its own doc update, and its own measurement, or it carries a future confusion.

The unreachable error message. The batch renamed the artifact magic strings in the same commit that bumped the format version. That combination made the carefully worded version-mismatch error unreachable by its only intended audience: a developer with a stale pre-batch out/ directory now fails the magic check before the version check ever runs, and gets "not an index.bin," which is actively confusing because it is one, just old. The fix recognizes the legacy magics specifically and emits the rebuild message from the magic branch. Writing a good error is half the work. The other half is checking who can actually arrive at it, because an error message is only as good as the path that reaches it.

A pre-batch out/ meeting the new binary. The meta file trips the check first, and the message names the fix instead of denying the file.

And the housekeeping tier, listed because absences come in all sizes: value rails lost (dims = 0 and chunk_chars = 10 were accepted without complaint), a doc comment on with_overrides stating the exact opposite of the function below it, and bytes.rs orphaned because the format rewrite grew its own private byte-cursor, leaving two implementations for the next format change to drift between. Reunified rather than deleted.

Every one of the seven was an absence: a bounds check, a key list, a rail, a reachable error path. The green path cannot show you a deleted guard, because guards only exist for inputs the happy tests never send. The question that finds them is mechanical and worth ritualizing: diff the old file against the new one and account for every rejection the old code could produce.

Rails, mirrored

The hardening that came out of the review has one organizing idea: the artifact must not be able to carry a value the config parser would have refused.

The same range checks, cosine-space quantities finite in 0..=1, alpha finite in 0..=100, now run in three places with one shape: on the TOML keys, on the build flags (through a shared validator, so a flag cannot bake what a key could not), and in the binary reader. The reader check is the one that earns its keep in the dark. A NaN min_gap read from a corrupted artifact would never gate anything, because gap < NaN is false, and the engine would run ungated forever while every test stayed green, every build succeeded, and every visitor got the junk results the gate exists to suppress. Nobody would ever know.

The payoff of checking at the door is what it licenses inside. Engine::new gets to index unguarded and the gate gets to compare unchecked precisely because read refused everything else. Validation is not paranoia layered on trust. Validation is the license for trust.

The scaffold teaches without deciding

chops-search init generates a starter config, and the batch posed a small design question with a sharp edge: the generated file is where a user learns what keys exist, and the three most consequential keys were invisible in it. But scaffolding min_gap = 0.08 would ship one corpus's calibration as if it were a universal default, and even an innocent-looking min_cos = 0.0 would arm the override on a site that should be deriving its floor.

The resolution: commented-out examples, with the calibration loop spelled out beside them. Sweep with eval, verify the mechanism with explain, pin the winner in config, rebuild. The scaffold teaches the keys exist and how to earn values for them, without asserting values it has no basis for.

Then the part that makes it durable: the template went under test. One test parses the generated file and asserts every scoring knob comes out inert, every live key is known to the parser, and no value is smuggled inside a comment block. A second strips the comment markers off the example lines and parses that, so the exact text a user will uncomment is guaranteed to be valid keys with in-range values, and a future key rename cannot leave the scaffold documenting a key the parser rejects. The template is now under test the same way the engine is, because it is an interface the same way the engine is.

Help text is a claim about defaults

A side effect of index-carried defaults that was easy to miss: four flag doc comments became lies the moment the batch landed.

"--min-gap, default 0 (disabled)" was true when the engine could only ever get 0. On a calibrated corpus, the default is now whatever index.bin shipped, and a user reading the help and omitting the flag would believe they were running ungated while the artifact gated at 0.08. Same for --rrf-alpha. Same for --min-cos, whose stated 0.20 was doubly wrong, being the 256-dim constant on top of the stale assumption. Same for build's --dims after the native-default restoration. Each now says "default: whatever index.bin was built with" or names the derivation instead of asserting a number.

The post-sweep "lock it in" hint got the same treatment. It used to prescribe carrying the winning flags around in your shell. Now it prescribes a config key and a rebuild, because locking in is precisely the thing the batch redefined. Help text is documentation that executes in the user's head at the worst possible moment, mid-debugging, and it deserves the same review as the code it describes.

Definition of done

The captures in this post are from 534e1dd, one batch after the one described here, by which point chunk_penalty had earned a key of its own through the same sweep-and-pin loop and the min_cos override had been dropped in favor of the derived floor. min_gap and rrf_alpha are as this batch shipped them.

The acceptance sequence for the batch, in order: workspace tests green. Two consecutive builds byte-identical on the real corpus, because the unit test asserts byte-stability on a toy index and only the real corpus proves it at scale. A stale pre-batch out/ producing the rebuild message rather than a parse failure. And then the line the whole arc was for: bare chops-search eval, no flags, scoring header reading the calibrated values with no asterisks on them.

Bare eval, no flags. Every value came out of index.bin; the 0.28 floor is derived from dims = 128, not set anywhere. 37 cases, 86% recall@1.

The four negative controls from the same run. Three return nothing: rows are warm in eval, so [kw] means the gate suppressed the semantic list and keyword matching had no evidence to offer. The fourth gets through in hybrid mode: no keyword evidence, but a top-median gap of 0.161 against min_gap 0.08, so the gate never fired. That is a real failure at the shipped configuration, and it belongs in the picture.

The last verification happens after deploy, and it is the one this whole series has been pointed at: type "toddler bedtime routine" into the live search box and watch it return nothing.

One open item surfaced while planning the next round of measurement, and it makes a fitting place to stop. No fixture case exercises the strong_cos hatch shape: keyword-empty, flat field, genuinely relevant top document. Which means a sweep of that knob is blind by construction until the case is written. The gap was found not by a failure but by asking what a sweep would even see, and the principle generalizes past search engines: a knob you cannot measure is a knob you cannot calibrate, and the time to notice is before the sweep, not after it returns a column of identical numbers you cannot interpret.

The engine, the config parser with its min_gp fixture, the scaffold tests, and the format reader are all in the repo: github.com/gitbadger-clan/chops-search. The series starts with a launch post about range-fetching an embedding model and runs through the eval that scored 65% and the knobs running out.

The knobs run out

Artur Daschevici — Wed, 02 Sep 2026 09:42:00 +0000

Part one ended at a plateau: 34 of 46 cases passing at the honest configuration, recall@3 thirteen points above recall@1, and a hypothesis. The right page was almost always retrieved and simply was not first, so the problem had to live wherever "first" gets decided, and in a hybrid engine that place is rank fusion. RRF's conventional k = 60 is calibrated for TREC-scale runs fusing thousand-deep lists. On a 19-document corpus the reciprocal-rank curve is nearly flat (rank 1 versus rank 2 is 1/61 versus 1/62, a gap of 0.0003), so fusion degenerates toward "best average rank wins" and a decisive first place in one engine cannot survive mediocrity in the other. Nobody calibrates k. It looked like the constant nobody had checked.

I built a sweep to test it. rrf_k threaded through ScoreOpts, seeded from rrf::K so there is exactly one copy of the constant. --sweep-rrf-k and --sweep-rrf-alpha on eval, running all 46 cases per grid point. And a self-validation requirement: the k=60 / alpha=0 cell must reproduce the baseline byte for byte, or the harness itself is suspect. It did.

Here is what the grid did to the hypothesis:

recall@1	alpha 0	alpha 0.5	alpha 1	alpha 2
k = 2	74%	74%	72%	67%
k = 4	74%	72%	72%	67%
k = 8	74%	74%	72%	72%
k = 16	74%	74%	74%	72%
k = 32	74%	74%	76%	74%
k = 60	74%	74%	76%	74%

(This table is re-measured on the 46-case union at 42593bf: all twenty-four cells reproduced exactly, including the flat first column.)

Read the first column. 74% at every k from 2 to 60. Fusion resolution is not where recall@1 lives on this corpus, and the hypothesis died in one table. The k that nobody calibrates turns out not to need calibrating here, and the sweep built to confirm the obvious instead falsified it in a single run.

Two smaller findings came along. The alpha=2 corner at low k drops to 67%: confidence-weighted fusion on a sharp curve actively amplifies keyword mistakes, which is the knob's danger zone documented empirically rather than reasoned about. And the grid's best cell, k=32 / alpha=1 at 76%, is exactly one case above baseline, which on a 46-case set I dismissed as argmax noise. Argmax over a 24-cell grid will always find a cell one case up by chance. That dismissal gets revisited below, and the revisit is the most instructive mistake of the arc.

A negative result recorded with its grid is a deliverable. This sweep produced no tuning change and was still the most valuable run of its week, because of what it printed next.

What the sweep actually bought

The per-kind table redrew the map of where the problem was:

kind	n	recall@1	recall@3
exact	14	86%	100%
navigational	9	100%	100%
negative	4	50%	50%
paraphrase	19	58%	79%

(Measured at 42593bf, no flags, summed over the gate and known-failures files: 12/14, 9/9, 2/4, 11/19, total 34/46, matching the baseline exactly. One instrument trap worth recording: the sweep re-runs its best cell verbosely after the grid, so a per-kind table trailing sweep output belongs to the best cell, not the baseline. An earlier draft of this table was the k=32/alpha=1 cell for exactly that reason.)

Navigational is solved outright and exact is close behind. That is BM25F doing the job it was built for in part one: the "getting started"-class failures are gone, and both kinds sit at 100% by recall@3. The negatives are leaking at exactly 50%, meaning the relevance floor lets junk through on half the off-topic queries. And paraphrase at 58% is the ceiling, with four of its nineteen misses absent from the top 3 entirely, which is retrieval, not ordering, and no fusion or gating knob reaches it.

The deficit had names, and none of them was fusion.

The floor versus the gate

The two leaking negatives ("toddler bedtime routine", "bicycle chain lubricant") were semantic cosines sneaking over the derived relevance floor, 0.28 at 128 dims. The obvious fix: raise the floor. min_cos 0.34 cleared both.

The per-case diff is why that fix was rejected. It read +2/-1, and the casualty was "make the download smaller for visitors", a legitimate paraphrase whose expected winner's best chunk sat at cosine 0.313, inside the same 0.28-to-0.34 band as the noise. The raised floor deleted the right answer's evidence and the fused order collapsed onto the hub page. A second fingerprint confirmed the mechanism: "elasticlunr" flipped from hybrid to keyword-only, a legitimate exact query whose entire semantic corroboration fell below the raised floor.

On this corpus at this dimensionality, the 0.28-to-0.34 band contains noise and signal simultaneously. No absolute per-document threshold separates two populations occupying the same interval. That sentence is the whole reason the next mechanism exists.

The corroboration gate, built in part one and shipped disarmed, is the distributional version of the same judgment. Instead of asking "is this cosine high," it asks "does anything stand out from this query's own field," measured as top-minus-median over raw cosines, applied only when the keyword engine contributed nothing. Off-topic queries on a homogeneous corpus produce a flat pack; real paraphrases produce one document above the field.

The calibration is the part worth stealing. Part one shipped the gate with diagnostics: --explain prints the gap and top values per query. Toddler's measured gap was 0.059. The sweep missed it at min_gap 0.04 and caught it at 0.06, and the per-case table was identical to baseline at every value from 0.06 through 0.12 except for the one intended flip. Instrument and knob agreeing to the third decimal. When explain and eval move together like that, calibration is a lookup, not a search, and that is the entire argument for shipping a disarmed mechanism with armed diagnostics. The value chosen was 0.08, roughly 35% above the measured noise gap. The strong_cos escape hatch stayed off: nothing on this corpus needed exempting, and a hatch that ships armed for a problem the corpus does not present is a knob without a constituency.

The gate is casualty-free by construction on the floor's failure mode: the min_cos casualty had keyword confidence 1.00, and the gate never fires on corroborated queries. The two mechanisms judge disjoint populations, which is why one of them could be armed and the other could not.

And the second negative survived every gap value, which is the honest counterexample the section needs. "Bicycle chain lubricant" is a structurally different noise mode: one document ringing at a nonsense query, standing out from the pack, which a flatness statistic reads as signal by definition. Any gap value that caught it would also suppress every legitimate semantic-only query whose winner stands out. A statistic designed to detect flat noise cannot detect pointed noise. Disposition: a known-failures entry under a new cause label, spurious resonance, with the blindness documented as inherent rather than pending. Not every failure gets fixed. Some get confessions.

Alpha's redemption

One explain output changed a verdict I had already published to myself.

The data-chops-open failure was the case rrf_alpha was designed against, in its purest observed form. The keyword engine had the compound term right: the correct page at kw#1, score 7.135, on df=2 evidence, about as unambiguous as keyword evidence gets. The semantic engine crowned an honestly adjacent page. And unweighted RRF's arithmetic preferred kw#3 + sem#1 over kw#1 + sem#4, because on a flat curve mediocrity in both engines beats excellence in one. Doing the weighted arithmetic by hand from the explain columns showed alpha=1 flipping it:

2/61 + 1/64 = 0.04841   >   2/63 + 1/61 = 0.04814

That hand calculation forced the revision. The grid's k=60 / alpha=1 cell had shown one case above baseline and been dismissed as argmax noise. It was almost certainly this case. The corrected principle, and both directions of it fired inside one arc: one flipped case without a mechanism is noise; one flipped case with an explain-verified mechanism is a finding. The k=32 cell stays dismissed. The alpha=1 cell gets rehabilitated the moment its case has arithmetic behind it. I applied the noise heuristic one round too early, and the only reason the mistake got caught is that the diagnostic loop kept running after the verdict.

The combined run, min_gap 0.08 plus alpha 1, was predicted at two up, zero down. It came back three up, one down. The bonus was "what should i commit and what should i ignore" flipping to PASS on the expected page, and its known-failures comment, written 2026-08-08 in the same commit that re-baselined the gate (2205069), read: "Verify with chops-search query when weighted RRF lands: if the keyword list already had manage-the-model first, this comes along for free." It did, and it did. A dated prediction in a fixture comment paying off is the strongest single argument in the file for writing diagnoses down instead of just fixing things.

The casualty was "cli", a navigational regression test, and its explain was the best single artifact of the arc. The page titled CLI held kw#3. Ahead of it at kw#1, scoring 4.324, sat a page containing zero occurrences of the word "cli", carried entirely by summed damped prefix expansions: "client", "client-side". And the fused-order arithmetic showed the case had been passing at baseline by 0.032266 to 0.032258.

A margin of eight millionths. Alpha did not break a solid case. It revealed a case that had been passing by luck, balanced on a coin edge that any keyword up-weight would tip. A regression test passing by 8e-6 is not a passing test; it is a coin that has not been flipped yet. The fix recorded the margin so nobody trusts that case as load-bearing again, and the per-case diff discipline is the only reason the coin was ever noticed before it landed wrong in production.

Two keyword fixes, both defensible with the fixture deleted

The "cli" explain exposed a principle-level bug, not a tuning problem. Completions of a half-typed word are competing hypotheses about what the user means, not corroborating evidence. A document matching several completions of "cli" has not multiplied its evidence for anything. The fix scores the trailing word's hypothesis set, the exact match plus its expansions, as a per-document maximum instead of a sum, while distinct typed words still sum as independent evidence. Confidence deliberately kept summing: it answers "does the keyword side have any evidence at all" rather than "how much does one document have", and its floor was calibrated against that arithmetic.

Counterfactual analysis before building showed best-of alone would not rescue "cli": with the phantom kw#1 demoted, the install page's genuine description-plus-body evidence still edged the CLI page's lone title hit. The enabler was the second fix, a title-cover tier, an idea from the first brainstorm of the whole arc that finally found its case. Documents whose titles contain every typed query word rank ahead of documents whose titles do not, BM25F order within each tier. Ordinal on purpose, so it composes with rank fusion untouched instead of fighting the saturation curve, and the trailing word can be covered through an expansion, so the tier works mid-keystroke.

The tier carries one deliberate semantic inversion, pinned by a named test rather than slipped in: for covered queries, title coverage outranks the tag weight. w_tag encodes topical authority; coverage encodes navigational intent; the navigational claim wins across tiers and the weights decide within. The old "tag outranks title" test was split into its two claims, score arithmetic (unchanged, still pinned) and rank order (now inverted for covered queries, pinned as deliberate). And the safety property got its own test: an all-false cover reproduces the plain ranking byte for byte, so queries the tier does not touch score exactly as they did before.

Predictions and outcomes, because pre-registering them is the house style: the tier and best-of were predicted to reach 35 with no outcome changes from the tier itself, and 37 with alpha armed. Actual: 36 and 37. The overshoot was best-of alone flipping "rebuild automatically when i publish a post" to PASS, unpredicted, and alpha then taking it back, weighted fusion re-amplifying the sense-collision evidence ("content rebuild" in the cache-invalidation sense) that best-of had demoted. Alpha's honest ledger on the merged set: +2 / -1, net +1, with the casualty being a case whose own diagnosis already prescribes a content fix no fusion weight can substitute for. Armed anyway, with the taketh-away recorded in the case's notes.

The fixture debt comes due

Running the gate file at the honest configuration for the first time showed two cases down from the recorded 37/37, and the immediate read was "two regressions". The audit said otherwise.

The 37/37 had been measured at --min-cos 0.34, the hand-raised floor. The file's own header had warned that "any --fail-under here gates a configuration nobody actually runs", and the negatives section had predicted that these exact two cases were the most likely to flip when the gate started measuring what ships. Both had been failing at every honest configuration all along. Their green was measurement debt, and the honest floor collected it on the predicted schedule. The gate file's true history, restated without the debt: 33/37 at this chapter's start, +2 from the calibration and keyword work, two never-real passes now visible as the failures they always were.

A fixture file that names, in advance, which of its own passes are fake is doing science. The predictions coming due is what the apparatus was for.

The litigation round that preceded this settled the open fixture verdicts by reading the actual corpus, and two pre-reading guesses reversed on contact with the pages, which is the argument for the procedure over judgment-from-the-diff. "model2vec-rs" was not widened: the crate name appears in exactly one place in the corpus, the returned winner never mentions it, and blessing it would be blessing a noise winner. "data-chops-open" turned out not to be a fixture problem at all but the fusion case above. "which files land in the deployed site" was widened to include the configuration page, weakly held and recorded as such. The verdict rule that did the work: would you add this URL to expect if the case were currently passing? Corpus edits made solely to flip a test stayed off the table, with the one carve-out the known-failures file itself blesses: docs fixes that make the docs genuinely better count as documentation work that happens to be measured.

The ledger, and the cell I refused

The configuration ladder on the merged 46-case set, all at the derived floor: baseline 34, gate armed 35, keyword fixes 36, alpha armed 37. Baseline and gate pin to 42593bf, the keyword fixes and alpha to c9905d8, and all four rungs re-measure exactly. 80% recall@1, up from 74% at the chapter's start, with every point traceable to a mechanism: a gate, a tier, a scoring correction, or the measurement getting honest. Zero points came from resweeping weights. The tuning surface was flat; the recall was hiding in the fusion arithmetic, the expansion semantics, the fixtures, and the prose.

On the gate file: 35/37 at the shipping candidate configuration, and its two failures were exactly the two standing verdicts, which landed: bicycle demoted to the known-failures file on 2026-08-19 (0731290), where its confession lives, and "which files" widened to include the configuration page on 2026-08-21 (46c27c5), weakly held and recorded as such. At the chapter's shipping configuration against the settled fixture, the gate reads 36/36: the c9905d8 engine, --min-gap 0.08 --rrf-alpha 1, fixture as of 46c27c5, 100% at recall@1 and recall@3.

The engine that started this series at 65% now passes its whole gate, and every case it cannot win lives in the known-failures file with a signed confession.

And one number I turned down. The configuration matrix contained a tempting 97% cell: everything above plus --min-cos 0.34, 36/37 at c9905d8, and it re-measures exactly, casualty included: at that floor, "make the download smaller for visitors" slips to third place in the file the gate cannot read. Rejected, because its casualty lives in the known-failures file, so the gate file cannot see the collateral. Shipping a configuration whose damage is hidden by the file split would be optimizing the scoreboard against the honesty the two-file design exists to protect. The split earns its keep in both directions: it keeps known failures from blocking CI, and it keeps CI from laundering them.

The asterisk, and what closed it

Everything above was measured through CLI flags. The browser constructs its scoring from index.bin, which at that point carried the field weights and nothing else, so live search was still running the baseline column: ungated, unweighted, junk on toddler-class queries. The CI gate was about to certify a configuration nobody's visitor ran, which is the measured-at-0.34 debt in a new denomination.

The persistence batch closed it. min_gap, rrf_alpha, and the min_cos override became chops-search.toml keys, ride into index.bin next to the field weights under one version bump, and get read out by the engine at construction. The config file states the scoring, the build bakes it, the browser runs it, and a bare eval measures it: one configuration in four places, none of them shell history. The boundary rule that came with it: persist what you calibrated, default what you did not. rrf_k stays compiled because its sweep was flat; strong_cos stays off because no case has ever needed the hatch. A knob earns a config key the day a sweep produces a value worth shipping, and on that day it also earns an artifact field, because a config key that does not reach the browser recreates the exact gap the batch closed.

The final verification is the one this whole series was pointed at: type "toddler bedtime routine" into the live search box and watch it return nothing.

Landing the batch meant rewriting two files, and the review of that rewrite found seven things the new code had quietly stopped doing, none of which any passing test could see. That is its own story, and it is the next post.

One prediction, pre-registered here so it can fail in public: a comprehensive multi-knob sweep is planned, and the expected result is flat, everywhere, except possibly chunk_penalty, which has never been swept systematically and is the one knob a future chunk_chars change could wake up. The coupled grids (alpha crossed with k, min_gap crossed with strong_cos) should reproduce the flat-column result. If that prediction fails, the failure is the finding, and it gets the same write-up a win would.

The prediction has since been tested, and recording the outcome here keeps the chronology honest, because the paragraph above was written first. A calibration round on 2026-08-14 swept every knob against this gate and woke exactly the knob this post's closing prediction named: chunk_penalty, pinned at 0.12 the next day (534e1dd), the same commit that retired the hand-raised floor for good. The sweep regenerates from the repo alone, corpus, fixture, and harness all being committed, and its verdict re-litigates in one command: at the post-verdict fixture, the pin's per-case ledger against the old 0.02 is +1/-0. It wins Content-Range and costs nothing. And two weeks of documentation growth then did what documentation growth does on a dogfooded corpus: today's bare eval at HEAD reads 33/36, three failures whose winners are the freshly written measurement pages out-competing the fixture's expectations, none of them the penalty's fault by the same per-case diff. The gate caught the corpus moving. That is what it is for.

The engine, the eval harness, both fixture files, and every sweep in this post are in the repo: github.com/gitbadger-clan/chops-search. Part one, the evidence-changing work that got the corpus to the plateau this post started from, is here.

My search engine scored 65% on its own docs site

Artur Daschevici — Wed, 26 Aug 2026 12:49:00 +0000

A couple of weeks ago I launched chops-search with a blog post claiming 92% recall@1. That number is real and it still stands: 24 labelled queries, my blog corpus, methodology in the post. Then I wired the engine into its own docs site, added a CI gate to block deploys on eval failures, and ran the eval honestly for the first time.

65%.

This is part one of three on what happened next. It ends at a plateau, on purpose: recall@1 stuck in the mid-70s on a fixture that kept growing, every ranking knob swept flat or regressive, and a hypothesis about rank fusion waiting to be tested. Part two is what the test did to the hypothesis.

The claim this part earns: every change that moved recall changed what evidence existed for the ranker to see. Every knob that merely reweighted existing evidence went flat or backfired. I built four ranking levers during this arc, measured all four, and shipped all four inert. That sentence would have read as an indictment to me a month ago. It is the most useful thing this corpus taught me. (Part two complicates it, with receipts. Evidence changes still win, but reweighting knobs eventually earn their values, one inside part two's arc and one at its very close, and the rule for telling those cases apart from noise is part two's story.)

The scoreboard

The query set changed twice during the arc (23 cases grown to 46, plus a gate/known-failures split), so no single percentage pair is an honest before/after. Every checkpoint with its own denominator:

Checkpoint	Fixture	recall@1	Passing	What changed
First honest run	23 cases	65%	15/23	Nothing yet. Baseline.
Labels, content, compound tokens, keyword floor	23 cases	65%	15/23	Diagnosis pass: engine net -1, fixture repair +1
BM25F	23 cases	83%	19/23	Field-normalized weighting lands
Fixture grown for coverage	46 cases	76%	35/46	Same engine, harder test
Descriptions as a keyword field	46 cases	80%	37/46	The description experiment's winner
Honest configuration	46 cases	74%	34/46	Where this part ends

Read the passing column, not the percentage column, and read the last two rows together, because the number goes down on a better engine. What happened there is the floor: rows three through five were measured at a hand-raised min_cos 0.34, and the last row put it back to its derived default. Some green under that raised floor was measurement debt, passes that only existed at a configuration nobody ships, and the honest configuration collected it. The full audit of that debt is part two. The rule it taught fits in one line here: a gate that measures a configuration nobody runs is not a gate, it is a scoreboard.

Every number was measured at dims = 128. The rows pin to commits: 047ebd8 (baseline), 588e4e3 (diagnosis pass), 7e3e280 (the BM25F row on its own fixtures, and the grown-fixture row against the fixture from 2205069), fc1e235 (descriptions), and 42593bf (honest configuration; the fusion sweep at that commit reproduces this row as its self-check cell). Rows one, two, and six are one git checkout from reproducible; rows three through five additionally pass --min-cos 0.34, the floor they were measured under, and rows four and five point eval at the fixture from 2205069, since the grown fixture landed after the engine work in commit order.

Why a docs site is the hostile corpus

The setup: 19 pages, Diátaxis structure, 132 chunks, 1,592 keyword terms at the BM25F commit (1,607 by the descriptions commit and 1,629 at the plateau; the count moved with docs edits during the arc). CI builds the CLI from workspace HEAD, builds the index, and gates deploys on the eval. Dogfooding with teeth.

A docs site about one tool is topically homogeneous, and homogeneity hurts both halves of a hybrid engine at once:

Keyword side: vocabulary collapse. Words like "search", "index", "query", "build" appear on every one of the 19 pages. Their IDF is near zero. BM25 has nothing to discriminate with, because the vocabulary that describes the tool is the vocabulary of every page about the tool. The extreme case: "chops" has document frequency 19 of 19. On this corpus, the project's own name is a stopword.

Semantic side: elevated noise floor. Every page is "about chops-search", so the cosine between a query and the wrong pages is uniformly higher than on a diverse corpus. Off-topic queries were clearing the relevance floor with confident nonsense.

The engine was calibrated on my blog: mixed topics, distinctive vocabulary per post, exactly the corpus where both halves work as designed. The docs corpus is the opposite in every dimension that matters. Not an excuse for 65%, but the variable the whole post turns on.

Diagnose before touching any knob

The first move was explicitly not tuning. The ladder, in order:

1. Read the per-kind recall table. The red row tells you which subsystem is failing. Exact and navigational failing points at the keyword side. Paraphrase failing points at the semantic side. Negative controls failing means the relevance floor is too permissive for this corpus.

2. Compare recall@3 to recall@1. If @3 is high while @1 is low, the right answers are on the podium and you have a tie-breaking problem. If @3 is also low, retrieval itself is failing, and no threshold sweep will save you. Those are fixed in completely different places.

3. Audit every failure as a possible labeling question first. The docs-specific trap: many queries have two legitimate answers, a flag documented in both a how-to and the CLI reference. expect is a list and the check is contains, so both URLs belong in it. I had a handful of cases where the engine returned the reference page and I had labelled the guide page. The engine was right. My labels were wrong.

4. Only then touch the engine.

One eval UX gotcha that fed a false sense of failure: the URL column in per-case rows shows the top-1 result, not the expectation. On PASS rows they coincide, so you never notice. On failures it shows what beat you. Several of my "failures" were the engine returning a page that legitimately answered.

The audit also surfaced two real bugs. Hyphenated identifiers like data-chops-open were split into parts, and with "chops" a corpus-wide stopword, compound-identifier queries were ranked on noise. Fix: emit the whole token alongside its parts. And stopword-only queries were entering rank fusion with junk keyword lists that still vote. Fix: a keyword confidence floor (kw_floor) that suppresses the list when matched IDF mass falls below a threshold fraction of potential mass, with the trailing word exempt from the denominator when it produced prefix expansions, so mid-typing stays live while completed junk gets gated.

I remembered that pass as a seventeen-point jump. The committed history says otherwise, and the correction is the most on-theme finding in this post: at any held-constant configuration, the engine side of the diagnosis pass was net minus one case. It won exactly the case it targeted, data-chops-open flipping top3 to PASS in the per-case diff, the compound-token fix visibly doing its job, and it paid two collateral cases. Everything else I had attributed to the pass lived in fixture repair (the label widening is worth one case in every configuration), in content fixes that had not been committed yet, and in a relevance floor that would not exist for another two days. The scoreboard above is the committed version, and the diagnosis ladder below is still the right procedure; what it produced first was better labels and better docs, not a better ranker, which is precisely this post's thesis wearing its own clothes.

The fixture split. I'll be straight about the motive: I wanted CI green, and a set of diagnosed engine issues stood between me and green. Quarantining failures to pass a gate is the classic way to lie to yourself, so the split only works with discipline attached. fixtures/queries.toml is the contract the engine meets today, gating CI. fixtures/known-failures.toml holds annotated cases reproducing open issues, each with a diagnosis and candidate fix in a comment, runnable manually, not gating. The rule is written into the file header: when a fix lands, promote its case back in the same PR, and never "fix" a case by blessing whatever the current noise winner is in expect. A gate you tuned to green on a small set is a gate that lies. This one has a paper trail instead. (Part two shows the paper trail doing something I did not expect: predicting its own future, twice.)

The work that moved the number

Three changes account for most of the movement in this part, and all three share a property: they put evidence in front of the ranker that it previously could not see.

Field weights were doing the wrong thing, confidently

The keyword engine had supported field weights from early on. A title mention counted like two body mentions, a tag like four, and the builder implemented that literally: it multiplied the term frequency before writing the posting.

That is wrong in a way that is easy to miss, because the direction is right. BM25 saturates term frequency against k1 (1.2 by default). The saturation curve is what stops a document that says "caching" forty times from scoring forty times higher than one that says it once. Pre-multiplying tf pushes a term straight past the interesting part of that curve: a single tag occurrence entered scoring as tf 4, already deep into saturation, so the distinction between "tagged with this" and "tagged with this and about it" disappeared. Worse, the inflated tf also inflated the document's computed length, so weighting one field quietly penalized every other term on the page.

BM25F is the standard answer and it inverts the order of operations. Each field carries its own term frequency and its own average length. Each field's tf is normalized by that field's length, the normalized values combine under the weights, and saturation applies once to the result:

ntf_f  = tf_f / (1 - b + b * dl_f / avg_f)
tfw    = w_title * ntf_title + w_tag * ntf_tag + w_desc * ntf_desc + ntf_body
score += idf * tfw * (k1 + 1) / (tfw + k1)

The practical difference: a term in a five-word title is now scored against the average title, not against two thousand words of body. That is the entire reason a query like "artifacts" can pick the page titled Artifacts out of five pages that use the word in prose. By the next checkpoint, navigational recall measured 100%, and it stayed there through the plateau.

Two implementation notes that turned into rules. The posting record widened twice on the way here: per-field term frequencies took it from four bytes (doc, tf) to eight, and the description field below takes it to ten. Each widening forces a format version bump, and the committed history reads v1 to v3 to v4; the mid-session v2 never landed, a small sibling of the phantom 44. The bump is not for tidiness, but because a reader one version behind would parse the newer file into plausible garbage rather than failing, since every field is a u16 and nothing about the byte stream announces its own shape. And the weights moved from build time to query time, persisted in index.bin, so they can be swept against an existing index with eval --w-title instead of a rebuild per value, and the browser scores with whatever the site configured. That introduced a new way to be wrong: any caller that builds a ScoreOpts::default() instead of starting from engine.score_opts() silently discards the index's own weights. That is now a test.

The description experiment

Nineteen pages, every one with a hand-written description in its front matter, none of them indexed. The descriptions read exactly like the questions a searcher asks: "rebuilds the search index whenever content changes", "why chops-search degrades to keyword-only". Three of four failing paraphrase cases had their answer sitting in an unindexed field. The obvious move was to index it, and the obvious place was the semantic side, since descriptions are natural language rather than rare identifiers.

The obvious place was wrong. Five configurations, same 46 cases, all at --min-cos 0.34:

Where the description goes	recall@1	exact
Nowhere (baseline)	74%	86%
Synthetic chunk 0, with title and tags	72%	79%
Chunk 0 and keyword field	76%	86%
Keyword field only	80%	93%
Its own semantic chunk, plus keyword	78%	86%
Keyword field, tf counted twice	80%	93%

Every row is measured on the 46-case fixture. The keyword rows are --w-desc 0, 1, and 2 at fc1e235 (descriptions reach the keyword engine as a BM25F field there, which is this table's "keyword side"); the two placement rows are one-commit experiment branches atop fc1e235, pinned at bea4a0b (chunk 0) and 6e93675 (own chunk). The "tf counted twice" run reproduced the "keyword field only" per-case lines byte for byte. One case separates this table's baseline (34) from scoreboard row four (35), and the commits explain it: row four is 7e3e280, before the description field existed, while this baseline is fc1e235 with the field present but disarmed, and the format and chunking changes between the two commits carry the case.

Every configuration that put description text into an embedding lost exact cases: 13 of 14 with descriptions on the keyword side alone, 11 or 12 in every cell with an embedding placement. And the recall@1 column says something sharper than "keyword won": pure embedding placement lands below doing nothing (33 cases against the baseline's 34), and stacking an embedding placement on top of the keyword field only gives back cases the keyword field had won (35 and 36 against keyword-only's 37). The embedding contribution is negative in every cell it appears in.

The mechanism is short synthetic chunks. An embedding is the mean of its token vectors, so a twenty-word chunk is dominated by those twenty words in a way a six-hundred-word chunk never is. Title plus tags is fine there: eight words that name the page produce a vector pointed at the page. A description is summary prose, and across a documentation corpus every summary is written in the same register. "The headers file for Cloudflare and Netlify, the CSP directives, and the three gotchas." "Fetch, lock, and verify the model2vec model." Those sentences are more similar to each other than the pages they describe are. Giving every document a compact vector pointed at its own summary clusters all nineteen documents together and hands topical pages a strong best-chunk score on queries they merely share vocabulary with. The keyword engine has the opposite property: it cares which specific rare words appeared, and a description contributes a handful of high-idf terms the prose never used.

The finding, stated the way it is worth remembering: the field that reads most like a search query is the field the vector engine handles worst. Summary prose is low-variance across a corpus. Low variance is poison for a similarity measure and harmless for an inverted index.

Descriptions now live as their own BM25F field at w_desc = 1.0. Worth exactly three cases: 34 to 37 on the 46-case set, re-measured, six and a half points. And 1 versus 2 is byte-identical per case, re-measured too, which is the saturation math saying the signal is presence, not weight. The two-line experiment that established weight-insensitivity is also what avoided building a heavier mechanism first and justifying it afterward.

The knobs that could not reach

Now the other half of the ledger. Four ranking levers built, measured, and shipped inert or at defaults in this part of the arc, each with a recorded reason.

chunk_penalty: structurally inert on this corpus. Three remaining failures were won by the same page, a getting-started tutorial that touches install, model, build, and serve, and therefore has a chunk near almost any query. That is precisely the bias this knob exists to correct: max-pooling over n chunks is a biased estimator, and the correction subtracts coeff * sqrt(2 ln n). Sweeping 0.02 and 0.04 produced byte-identical per-case runs. At 0.06 exactly one case moved: "which files land in the deployed site" slid from the caching how-to to the configuration reference, two pages that both answer it, and part two records the fixture widening to say so. The arithmetic explains both halves: what reorders two documents is the difference between their penalties, and for a twelve-chunk document against a five-chunk one that difference is sqrt(2 ln 12) - sqrt(2 ln 5) = 0.44, which at coeff = 0.02 is 0.009 of cosine against rank-gaps of 0.02 to 0.05 on this corpus, and at 0.06 is 0.026, just inside the band, where it found the one pair sitting at the bottom of it. Below roughly 0.05 the knob cannot reach. The reason is corpus shape: 132 chunks over 19 documents is seven per document, and a correction that scales with the log of a count needs the count to vary. It was designed against a blog corpus where a 23-chunk essay competed with a one-page note. Here every page is a screen of prose, so at the default and twice the default it is structurally inert, and the first thing it reaches when pushed is a label, not a ranking. Not mistuned, inert here. Different diagnoses: a mistuned knob wants a sweep, an inert one wants to be left at default and understood. And the same arithmetic that convicts the range names the escape: the differential scales linearly with the coefficient, so at roughly six times the default it clears the top of the observed rank-gap band. "Inert" is a claim about a range, not a knob, and part two's closing records what happened when a later sweep went where this one did not.

min_gap / strong_cos: built for a real problem, shipped disarmed. An earlier experiment at higher dimensionality broke both negative controls, and the failure taught me the shape of the problem: on a homogeneous corpus, the signature of noise is flatness, not low absolute cosine. An off-topic query produces a flat pack where every doc scores 0.15 to 0.35 with no standout. A real paraphrase produces one document clearly above the field. So I built a corroboration gate on the relative statistic top_median_gap, suppressing an uncorroborated semantic list only when the field is flat and nothing is strongly relevant in absolute terms. In this part of the arc it ships with both knobs disarmed (min_gap 0, strong_cos off, an asymmetry documented on the fields because it is exactly what someone "cleans up" later). What ships armed is the diagnostics: --explain prints gap and top, so that if this gate is ever needed, calibrating it is a lookup rather than a sweep. It turned out to be needed sooner than expected. Part two.

rrf_alpha: built, first sweep regressive, ships at 0. Confidence-weighted rank fusion cost points at every armed value on the first pass. Its story is not over either, and the way it comes back is the best argument in the whole arc for keeping measured dead ends written down. Part two.

w_tag: flat above zero. Every one of the 19 pages carries hand-curated tags under [taxonomies], and the field is live: at 42593bf, w_tag 0 costs one case, "shrink vectors to save bandwidth", a page that is found by its tags when the weight is armed and by nothing else when it is not. 2, 4, and 8 are byte-identical per case. Presence is the signal and weight is not, the same shape the description field showed, and the same reason a sweep across the armed range reads flat. The default of 4 is inherited; the sweep says any value above zero would do, and the config comment now says that instead of implying the number was earned.

Four levers shipped inert. Not waste: each has a recorded reason, and three of them ruled out a hypothesis about one stubborn failure that I would otherwise have re-tried in a month. A measured dead end you wrote down is cheaper than the same dead end explored twice.

Two methodology lessons, learned the hard way

Summary-level A/B testing hid a cancellation. The description experiment initially ran two changes in one commit: text into synthetic chunk 0, and terms into the keyword body. The summary read 77%, then 75%, then 77% again, which looked like a change that did nothing. It was two changes pulling in opposite directions, each worth one to two cases, netting to zero, and I read "descriptions do not help this corpus" off the summary line twice. What exposed it was diffing per-case lines instead:

chops-search eval | grep -E '^(PASS|top3|FAIL)' > /tmp/a.txt
# change exactly one thing, rebuild
chops-search eval | grep -E '^(PASS|top3|FAIL)' > /tmp/b.txt
diff /tmp/a.txt /tmp/b.txt

A one-line diff is a real result. An empty diff is a real result. An unchanged summary is not, because it cannot distinguish "nothing happened" from "two things happened and cancelled".

Growing the fixture lowered the score, which was the point. The set went from 23 cases to 46, chosen for coverage rather than for passing: every indexed page became the primary expectation of at least one case, which closed three pages nothing had been asking about. Recall@1 read 83% before and 76% after (19/23 to 35/46, same engine, same floor, both re-measured), on the same engine facing a fixture that finally asked about everything. The denominator changed and the honest response is to say so, which is what the scoreboard at the top of this post is for. The growth also flushed out fixture bugs wearing ranking-bug clothes: one case handed a distinctive word to a wrong-but-plausible page (fix: reword the case), another asked a question whose only answer lives in a section landing page the indexer skips by design (fix: delete the case, or improve the docs for their own sake). The distinction matters. Editing content so a test passes is only legitimate when the content is better afterward. Otherwise it is fitting the corpus to the fixture, which is the same failure as fitting thresholds to the fixture, wearing a different hat.

The plateau

Where part one ends: 46 cases, honest configuration, 34 passing. 74% recall@1, 87% recall@3, both re-measured at 42593bf with no flags (34 passing, 40 within the top 3).

That thirteen-point gap between @1 and @3 is the tell. The right page is almost always retrieved. It is just not first. Which means the remaining problem lives wherever "first" gets decided, and in a hybrid engine that place is rank fusion. So I had a hypothesis, and it was attractive: RRF's conventional k = 60 is calibrated for TREC-scale runs fusing thousand-deep lists. On a 19-document corpus the reciprocal-rank curve is nearly flat (rank 1 versus rank 2 is 1/61 versus 1/62, a gap of 0.0003), so fusion degenerates toward "best average rank wins" and a decisive first place in one engine cannot survive mediocrity in the other. That is exactly the @1-fails, @3-passes signature. Nobody calibrates k. It looked like the constant nobody had checked.

I built a sweep to test it: a full grid over rrf_k and rrf_alpha, every cell running all 46 cases, with the k=60 / alpha=0 cell required to reproduce the baseline exactly as a self-check on the harness.

What the grid did to the hypothesis, what the per-kind table said instead, a regression test that turned out to be passing by a margin of eight millionths, and the fixture file that predicted its own future: part two.

The engine, the eval harness, and both fixture files with their header rules are in the repo: github.com/gitbadger-clan/chops-search.

Semantic search on a static site with no server: range-fetching the embedding model

Artur Daschevici — Mon, 10 Aug 2026 15:31:09 +0000

The search box on my site downloads about 0.1 KB per query. That number sounded wrong to me too, so I put the devtools network tab in the demo GIF. Watch the range requests. That's the whole trick.

There's no search server behind it. No SaaS, no API key, no request leaving the page except range fetches against static files sitting next to the HTML. The engine is called chops-search, it's on crates.io, and the search on the docs site is a live deployment you can poke at. This post is about the one architectural decision that made it work: treating the embedding model as a file you read byte offsets out of, instead of a blob you download.

You can check the claim from your terminal without trusting my recording:

curl -s -o /dev/null -w "%{http_code} %{size_download} bytes\n" \
  -H "Range: bytes=0-127" \
  https://chops-search.gitbadger.com/search/model.rows.2b9eb020575d2857.i8

That hash in the filename is a content hash, and every rebuild mints a new one. If this URL 404s by the time you read it, grab the current name from manifest.json in the same directory. The hash is what lets the artifacts ship under immutable cache headers, so it is doing real work, not just breaking my curl example.

The problem, stated plainly

Static sites and search have a long, awkward history. Keyword search is a solved problem: Pagefind figured out how to fragment an index so the browser only pulls what a query needs, and it works beautifully. But keyword search fails on paraphrase. Search my site for "packing a repo into a prompt" and a pure keyword engine shrugs, because the post says "context packing" and never uses your words.

Semantic search fixes that, and every existing answer to "semantic search on a static site" is some flavor of the same compromise. Run a server. Pay a SaaS. Or ship the entire embedding model to the browser and ask your visitors to download tens of megabytes before they can type.

I didn't want any of those. I wanted Pagefind's loading discipline applied to the model itself.

Why that's even possible

The reason this works is model2vec. A potion-base-8M model isn't a transformer at inference time. It's a lookup table: one static vector per vocabulary token, and a sentence embedding is just the mean of its token rows. No attention, no layers, no runtime beyond "look up rows, average them."

A lookup table has a property transformers don't: you can read one row without the others. If a query tokenizes to six tokens, you need six rows. At int8 quantization, a row is dim bytes, and row i lives at byte i × dim. That's an HTTP range request. The model stops being a download and becomes an address space.

Four files, four loading rules

The build tool emits four artifacts, and each one has a different answer to "when does the browser fetch this."

model.meta.bin holds the complete vocabulary plus per-row quantization scales. Around 500 KB, and it gzips hard. This one is never partial. A truncated vocab doesn't fail loudly, it tokenizes wrong and gives you confidently bad embeddings, so completeness is the rule here.

model.prefix.i8 is the top ~2048 rows by token frequency, loaded eagerly. Common tokens cover a lot of real queries, so most lookups never leave this block.

model.rows.i8 is the full matrix as headerless raw i8. No framing, no metadata, just bytes, because the offset arithmetic is the format. This is the file the browser range-fetches per query.

index.bin carries the chunk vectors, document table, and keyword postings for the site content itself.

At query time a Web Worker asks the wasm engine which byte ranges it's missing, fetches them, feeds them back, and renders ranked results. Fetched rows go into a Cache API row cache, and the artifacts ship content-hashed under immutable cache headers. So the second time anyone searches for anything vaguely similar, the network doesn't get involved at all.

The numbers, with their caveats attached

On my 24-query labelled eval set: 92% recall@1 and 100% recall@3. The set covers exact matches, paraphrases, navigational queries, and a negative control that must return nothing. Exact and navigational sit at 100% recall@1. The two misses are both paraphrase queries, and both still put the right document in the top 3.

Caveats, next to the claim where they belong: 24 queries is small, I wrote and labelled them myself, and they run against my own site's content. This is a regression gate, not a benchmark against anyone else. It exists so a ranking change can't quietly make things worse, and it runs in CI against a real demo Zola site whose worst page (one post covering thirty unrelated topics) is deliberately kept in as the chunker's stress test.

The traffic numbers are the part I actually care about. After initial load, 16 of the 24 eval queries need no network at all, either a prefix hit or a warm row cache. The remaining eight average 0.1 KB range-fetched, 0.8 KB worst case. Compare that to shipping the matrix eagerly and the whole design justifies itself.

Ranking, briefly

Retrieval fuses BM25 (with length normalisation) and cosine similarity over embedded chunks via reciprocal rank fusion. A relevance floor suppresses junk when neither engine is confident, and the trailing query term gets prefix matching so results show up mid-word while you type. Snippets come from the best-scoring chunk with query terms highlighted.

There's a chops-search query --explain command that prints the evidence behind a ranking: keyword scores, best-chunk cosine, each engine's RRF contribution per document. It calls the same scoring code as the ranker, so the explanation cannot drift from the behavior. I built it to debug my own fusion weights and kept it because a search engine that can't show its work is a search engine you can't tune.

One tokenizer, enforced structurally

The core crate, chops-search-core, is pure Rust with no I/O: WordPiece tokenizer, int8 row store, scoring, RRF, artifact formats. It compiles unchanged to native for the build CLI and to wasm for the browser. The tokenizer that indexed your content is bit-for-bit the tokenizer that handles queries, because it's the same compiled code, not two implementations someone promised to keep in sync.

A parity test drives fixture sentences through my implementation and through MinishLab's official model2vec-rs, asserting cosine > 0.9999 per input. If my tokenizer or quantization drifts from the reference, CI fails. The guarantee is structural, not aspirational. That distinction deserves its own post, and it'll get one.

When it breaks, it says so

Partial loading creates a failure mode most search libraries don't have: what if a row you need isn't loaded? Offline, a strict CSP, a host that ignores range requests.

The wrong answer is to average the rows you have. That produces a shrunken, wrong embedding that returns plausible-looking garbage, which is worse than returning nothing. So embed() returns nothing. Search degrades to keyword-only and reports that it did, so the UI can tell the user semantic matching is off rather than silently getting dumber. A range-hostile server degrades to eager loading of the full matrix. Slower, not broken.

I spend a fair amount of my time deliberately breaking data pipelines to see what they do. Building the degradation paths on purpose, instead of discovering them in an issue tracker, was the part of this project that felt most like that work.

One gotcha worth flagging

There's a --dims flag that reduces dimensionality at build time, and it re-runs PCA on the token matrix rather than truncating columns. Potion models are trained after model2vec's distillation-time PCA, so the columns aren't ordered by variance anymore and naive truncation is silently wrong. Cost me an evening. Short follow-up post coming.

Try it

The live demo is the search box on my blog, so the fastest way to evaluate it is to search there for something I've written about, phrased in words I didn't use.

cargo install chops-search
chops-search init    # initializes the required static files in your Zola site
chops-search build   # walks a Zola content tree, emits the artifacts
chops-search eval    # runs your labelled queries as a regression gate

It's a Rust workspace (chops-search-core, chops-search-cli, chops-search-wasm), dual-licensed MIT/Apache-2.0. Zola is the only site generator with first-class support right now because it's what I run. If you want it for Hugo or Eleventy, open an issue and tell me what your front matter looks like.

gitbadger-clan / chops-search

Hybrid semantic + keyword search for static sites. Runs entirely in the browser via WASM — no server, no API keys.

chops-search

Hybrid keyword + semantic search for static sites, running entirely in the browser. No API keys and no server.

The "model" is a model2vec/potion int8 lookup table streamed over HTTP range requests, so a query fetches a few 128-byte rows rather than the 23 MB a transformer would: on the project's own docs site the mean is 71 bytes per query and most queries fetch nothing at all (measured with chops-search plan, see below). The engine is one Rust core compiled twice: natively for the build tool, to wasm for the browser. That means the tokenizer indexing your content is the same code that tokenizes queries.

On the demo corpus (9 posts, 24 labelled queries):

query kind	recall@1	recall@3
exact (`chromedp iframes`)	100%	100%
paraphrase (`how long will this project take`)	82%	100%
navigational (`about`)	100%	100%
negative (`sourdough starter hydration`

…

View on GitHub

Choosing what the model sees: a repo-to-prompt tool for 2026

Artur Daschevici — Wed, 29 Jul 2026 13:17:00 +0000

For about a week I told people my tool packed the Next.js monorepo in 7 to 10 seconds. The real number was 1.6.

I was reading the wrong line of the benchmark output. time gives you user time and wall time, and I'd been quoting user time, which is CPU-seconds summed across cores. My tool runs on five cores. So I was underselling it by 5x, confidently, to anyone who asked. If you take one thing from this post: wall clock is the number humans experience. Quote that one.

Anyway. This is a post about a repo-to-prompt tool. Yes, in 2026. Stay with me.

The awkward timing question first

Repo-to-prompt tools had their moment two years ago, when the workflow was "assemble context by hand, paste it into a chat window, pray." Then agents learned to grep. Claude Code and Cursor pick their own context now, and most people stopped running a packer before every session.

So why build one?

Because sometimes the agent's retrieval is exactly what I don't want. Code review across a whole subsystem. A migration plan. A raw API call where nobody is doing retrieval for me. In those moments I want to pick the files myself, see what it costs in tokens, and know that .env.production didn't tag along.

Three requirements. Fast. Interactive with visible token costs. Secret-scanned.

No tool had all three. The fast ones (yek, repomix-rs) are fire-and-forget: point, dump, done. The interactive one (code2prompt) doesn't show token costs. So I forked code2prompt and built gnaw.

The benchmark

Vendor benchmarks are junk by default. Different machines, cherry-picked flags, and my favorite trick: comparing against output nobody checked. A tool that emits half the files in half the time isn't fast. It's incomplete.

So here's the setup, built to survive you rerunning it:

Docker, --cpus 8 --memory 8g. One command, same environment as mine.
Corpus: vercel/next.js at 70e3934.
hyperfine, 10 runs, warmup, mean ± σ.
File-count check: every tool must emit within ~1% of the others or the comparison is void. All landed between 26.6k and 26.9k files.
repomix-rs built from source at 5798dc0, not the npm package. The npm build is roughly 2x slower. I benchmarked their engine, not their packaging. If anything this makes my numbers look worse. Good.
Scan settings, stated exactly because it matters: three of the five tools secret-scan (gnaw, repomix, repomix-rs). Run 1 turns scanning off where it can be turned off. repomix-rs scans by default and I found no way to disable it, so its run-1 number includes its scan. Which means run 1 is not a clean gnaw-vs-repomix-rs comparison, and I won't quote it as one. Run 2 is the clean one: everything that can scan, scanning.
I ran this three times across three days while working on the tool. Absolute times wobbled about ±6% with the host's mood. The ratios barely moved. Trust the ratios; treat the absolutes as "this hardware, that day."

Script and Dockerfile are in the repo. Rerun it. I mean it. My least favorite genre of reply is "works differently on my machine" and I'd rather get it as a GitHub issue with numbers attached.

Run 1: extraction, scanning off where possible

tool	mean ± σ	peak RSS	cpu×	files	version
gnaw	1.55 ± 0.10 s	378 MB	5.2	26,706	local
repomix-rs*	1.73 ± 0.11 s	311 MB	4.8	26,635	git:5798dc0
code2prompt	2.11 ± 0.22 s	463 MB	4.1	26,706	4.3.0
yek	8.05 ± 0.20 s	228 MB	1.0	26,790	git:0.25.5
repomix	35.44 ± 1.87 s	769 MB	1.1	26,928	repomix@1.16.0, incl. Node startup

*includes its default regex scan, which can't be disabled. See run 2 for the equal-footing comparison.

Run 2: scanning on, for the three tools that can

tool	mean ± σ	peak RSS	cpu×	scanner
gnaw	1.76 ± 0.13 s	477 MB	5.2	gitleaks ruleset
repomix-rs	1.78 ± 0.03 s	311 MB	5.2	custom regex set
repomix	35.97 ± 1.24 s	878 MB	1.1	Secretlint, incl. Node startup

What I'll claim, and what I won't

Against the tools people actually have installed, the gap is not subtle: 5x faster than yek. 20x faster than repomix, with both tools scanning — and that repomix number already includes Node startup in both time and memory, and it's running Secretlint, a different ruleset than gnaw's gitleaks. All three caveats, same sentence, because a 20x headline with the fine print three scrolls away is how vendor benchmarks lie. And no, repomix is not the 22-minute dinosaur from the old yek benchmark everyone still quotes. They shipped serious parallelization work. It's 35 seconds now, current version. Still 20x.

Against repomix-rs, with both tools scanning: 1.01 ± 0.08. A dead heat. I'm not going to torture that into a win. Here's the full trade in one breath: gnaw ties repomix-rs on wall-clock, carries 166 MB more peak RSS, and is running the full gitleaks ruleset where repomix-rs runs a custom regex set. The memory is the price of the bigger ruleset. Whether thoroughness is worth 166 MB is your call; I've made mine.

One more honest one. Look at yek's user time: 8.0 CPU-seconds. Now look at gnaw's: 7.4. Nearly the same total work. gnaw spreads it across roughly five of the eight cores while yek sits on one — the cpu× column shows 5.2, and that's what the wall-clock gap is made of. On a 2-core CI runner, expect much less of it. (Why 5.2 and not 8? Traversal is partly I/O-bound. Profiling where the other cores went is on the list.)

Now the number I actually care about. gnaw with scanning off: 1.55s. With a full gitleaks pass over 26,706 files: 1.76s.

The scan costs about 200 milliseconds.

I picked gitleaks knowing it was the heavier, slower option, and braced for a real cost. The bill came to a blink. Whatever era "secret scanning is too slow to leave on by default" belonged to, it's over.

The fork

code2prompt got the big thing right: packing a repo is something you should be able to steer, not just trigger. It has a TUI. Its extraction speed hangs with the native tools. That's why I forked it instead of starting from zero.

What I changed:

Live token counts. The reason gnaw exists. You browse the tree, toggle files and directories in and out, and the token total updates as you go. Deselect test/, watch tokens fall off the bill. A context window is a budget. The tool should show you spending it.

Search that behaves the way I kept expecting it to. Not a bug report, a taste difference. code2prompt's search does what it does; my fingers disagreed with it several times a day, so gnaw's does what my fingers expect. I'll write up the specific UX splits in a follow-up, because they're small, concrete, and the fun kind of arguable.

gitleaks as the default scanner. The thorough option. I assumed I was trading speed for it. See above: 200ms. Easiest trade I've made this year. I'm building a planted-secrets corpus to measure recall and false positives against the lighter scanners properly. Until that's published, "more thorough" is my working belief, not a measured verdict.

A parallel extraction engine. cpu× 5.2. The table already made this argument.

The pure taste changes stay in the fork. Anything that turns out to be an objective fix goes upstream, where it belongs.

What gnaw is not

An agent. If Claude Code is already picking good context for your task, close this tab, you're fine. gnaw is for when you want to be the retrieval step: deterministic, inspectable, scanned, and done in under two seconds instead of twenty minutes of clicking through GitHub copying files like it's 2023.

It's also not done. Next on the roadmap: an MCP server, so agents can call gnaw as a token-budgeted, secret-scanned context extractor instead of this being a manual-only tool. That's the version that makes sense in 2026, and it's the part I'm most looking forward to building.

Check my work

Repo: gnaw. Quickstart in the README, benchmark script and Dockerfile next to it. docker run it, and if your numbers disagree with mine, open an issue. I once misread my own benchmark by 5x; I have no standing to be precious about corrections.

If it's useful, a star helps the next person find it.

Rusty puppets, Websockets and Voyeurism (part II): Driving Chromium in Docker with a Window

Artur Daschevici — Wed, 19 Nov 2025 09:05:00 +0000

TL;DR

You swapped Chrome → Chromium for better arm64 support, strapped on VNC + noVNC to watch the chaos, made Alpine optional to chase size gains, and pimped a Makefile so everything feels like a dead man's switch.

The Backstory

In some of my previous experiments I have used browser automation with Chrome to extract info from pages, usually via a wrapper around CDP. CDP has a pretty neat and quite huge API for driving Chrome and Chromium based browsers. Instead of using an existing wrapper, I built my own in Rust with tungstenite to communicate over WebSocket.

Why?

Ideally what I want is a server style primitive component that is able to speak CDP over a websocket without having to babysit a local browser instance. Normally the way you run most of the chrome automations is by running the process and if you want some transparency into what is happening you flip --headless flag to false, but this is not what you want, the automation is outside the docker container, the container only holds the browser with the remote debugging port exposed. Something else that tripped me up was that there were no Chrome repos with arm64 builds that i could run on my M1 mac, so I switched to Chromium that supports arm64 better.

Yes, "headless" is efficient; no, I don't trust it until I can see it wiggle, that is why an extra feature that felt right was having VNC enabled on one of the container variants.

What?

Components (lego brick style)

Chromium (not Chrome) – better support for arm64 builds, CDP at 9222.
Socat - port forwarding for CDP, since binding to 0.0.0.0 did not work well with Chromium.
Xvfb + lightweight WM (fluxbox) – fake display for the VNC stream.
VNC server (x11vnc) – the eyeballs.
noVNC + websockify – view it in the browser at :6080.
supervisord – herd cats (multiple daemons).
Alpine (optional) – minimal base; trade-offs: fonts, glibc shims, weird edges.
Rust client – talks to http://container:9222/json

Architecture

+------------------ Docker Container --------------------+
|                                                        |
|  [Xvfb] --- [WM] --- [VNC Server] --- [noVNC]          |
|                         ^             (HTTP 8080)      |
|                         |                              |
|                    screen:0                            |
|                                                        |
|  [Chromium --headless=new --remote-debugging-port=9222]|
|                                         (WS:9222)      |
|                                ^                       |
|                                |                       |
|                            [Socat proxy]               |
+--------------------------------------------------------+

 Outside:
   Rust (chromiumoxide/others) --> http://host:<exposed-socat-port>
   Human -> http://host:6080 (noVNC)

How?

I. Image strategy (two by two):

Essentially what I wanted was efficiency at the container level and at the browser level. I applied that in practice by using alpine for smaller image size and running headless for better runtime efficiency. However running blind is kinda hard to introspect so I added two additional images with VNC support and those came with both alpine and debian flavors.

Debian/Ubuntu base: fewer papercuts, bigger image, fastest to "it works."
Alpine base: smallest, but bring your own fonts, codecs, and glibc cuddles.

Pro tip: start Debian for DX, ship Alpine once you tame fonts/codecs.

II. Dockerfile (there is a common base between debian and alpine)

#....
# Default ports
ENV CHROME_PORT=9222
ENV SOCAT_PORT=9224
ENV VNC_PORT=5900
ENV NOVNC_PORT=6080

#....packages common to both debian and alpine
  vim \ # want to edit stuff?
  chromium \ # duh
  socat \ # port forwarding for the 9222 remote debugging port
  curl \ # healthcheck and manual testing
  net-tools \ # netstat useful for listing open ports
  iproute2 \ # ss useful for listing open ports
  ca-certificates \
  procps \


#.... Install VNC and GUI components
  supervisor \
  xvfb \
  x11vnc \
  websockify \
  fluxbox \
  git \

#....

# Install noVNC from source (most reliable method)
RUN cd /opt && \
  git clone --depth 1 https://github.com/novnc/noVNC.git && \
  cd noVNC && \
  ln -s vnc.html index.html

#....

# Create necessary directories with proper permissions
RUN mkdir -p /home/chrome/data && \
  mkdir -p /var/log/supervisor && \
  chown -R chrome:chrome /home/chrome && \
  chown -R chrome:chrome /var/log/supervisor


# Copy startup script
ARG STEALTH=basic
COPY ${STEALTH}.sh /usr/local/bin/start-chrome.sh
RUN chmod +x /usr/local/bin/start-chrome.sh && \
  chown chrome:chrome /usr/local/bin/start-chrome.sh

#....


EXPOSE ${SOCAT_PORT} ${VNC_PORT} ${NOVNC_PORT}

CMD ["supervisord", "-n", "-c", "/etc/supervisor/conf.d/supervisord.conf"]

III. Dockerfile (lightweight Debian to get a more frictionless experience):

FROM debian:bookworm-slim
ENV DEBIAN_FRONTEND=noninteractive

# Install base dependencies
RUN apt-get update && apt-get install -y \
  #....
  fonts-liberation \
  fonts-noto-color-emoji \
  fonts-roboto \
  fonts-noto \
  libasound2 \
  libatk-bridge2.0-0 \
  libatk1.0-0 \
  libatspi2.0-0 \
  libcups2 \
  libdbus-1-3 \
  libdrm2 \
  libgbm1 \
  libgtk-3-0 \
  libnspr4 \
  libnss3 \
  libwayland-client0 \
  libxcomposite1 \
  libxdamage1 \
  libxfixes3 \
  libxkbcommon0 \
  libxrandr2 \
  xdg-utils \
  && rm -rf /var/lib/apt/lists/*

# Install VNC and GUI components
RUN apt-get update && apt-get install -y --no-install-recommends \
  gnupg \
  #....
  python3 \
  python3-numpy \
  && rm -rf /var/lib/apt/lists/*

#....

# Create a non-root user
RUN useradd -m -s /bin/bash chrome

#....

# Copy supervisord config
COPY supervisord.conf.debian /etc/supervisor/conf.d/supervisord.conf

#....

IV. The alpine version, similar but not quite the same, some commands are different and some of the package names and dependencies differ too:

FROM alpine:3.19

#....

# Install base dependencies and Chromium
RUN apk add --no-cache \
  # Core utilities
  bash \ # using some custom scripts to launch the browser
  # ....
  xdpyinfo \ # x11 utilities that are not included by default
  xauth \
  xprop \
  xwininfo \
  # Fonts
  font-liberation \
  font-noto \
  font-noto-emoji \
  font-noto-cjk \
  # Chromium dependencies
  libstdc++ \
  harfbuzz \
  nss \
  freetype \
  ttf-freefont \
  wqy-zenhei \
  # Audio/Video libraries
  alsa-lib \
  at-spi2-core \
  cups-libs \
  dbus-libs \
  libdrm \
  mesa-gbm \
  libxcomposite \
  libxdamage \
  libxfixes \
  libxkbcommon \
  libxrandr \
  wayland-libs-client \
  # X11 libraries
  libx11 \
  libxext \
  libxrender \
  libxtst \
  libxi

# Install VNC and GUI components
RUN apk add --no-cache \
  #....
  python3 \
  py3-numpy \
  py3-pip

#....

# Create a non-root user (Alpine uses adduser instead of useradd)
RUN adduser -D -s /bin/sh chrome

#....

# Copy supervisord config
COPY supervisord.conf.alpine /etc/supervisor/conf.d/supervisord.conf

#....

V. In order to manage the multiple containers k8s would be overkill so I used docker-compose instead. I like it for simple experiments when I have to do quick and incremental iteration for testing my containers. There is less of a need for cleaning up things after and less of a chance to make mistakes with docker run commands.

name: chrome-cdp

x-chrome-common: &chrome-common
  image: chrome-cdp:${DISTRO:-debian}-${MODE:-headless}-stealth-${STEALTH:-basic}
  container_name: chrome-${DISTRO:-debian}-${MODE:-headless}-stealth-${STEALTH:-basic}
  networks: [cdpnet]
  shm_size: "${HEADLESS_SHM_SIZE}"
  restart: unless-stopped
  healthcheck:
    test: ["CMD", "curl", "-fsS", "http://0.0.0.0:${SOCAT_PORT}/json/version"]
    interval: 10s
    timeout: 3s
    retries: 5
    start_period: 15s

services:
  chrome-headless:
    <<: *chrome-common
    build:
      context: ./headless
      dockerfile: Dockerfile.${DISTRO:-debian}
      args:
        STEALTH: ${STEALTH:-basic}
    ports:
      - "${CDP_HOST_PORT}:${SOCAT_PORT}"

  chrome-gui:
    <<: *chrome-common
    build:
      context: ./gui
      dockerfile: Dockerfile.${DISTRO:-debian}
      args:
        STEALTH: ${STEALTH:-basic}
    ports:
      - "${CDP_HOST_PORT}:${SOCAT_PORT}"
      - "${VNC_PORT}:5900"
      - "${NOVNC_PORT}:6080"

networks:
  cdpnet:
    driver: bridge

as you can see the docker-compose file is slightly streamlined by using YAML anchors and aliases to avoid repetition between the two services. But this is not the most interesting DX improvements I added.

VI. The pimped out Makefile. The big deal with this one is that I got everything working almost like an extension of make, where previously I would just be creating targets to wrap docker-compose now I added a bunch of more advanced params to allow parametrization of each target instead of doing different names for each variant.

SHELL := /bin/bash
ENV_FILE ?= .env
STEALTH ?= basic
MODE ?= headless
DISTRO ?= debian

# Container name (adjust to match your docker-compose service name)
CONTAINER_NAME ?= chromium
COMPOSE_FILE ?= docker-compose.yml

.PHONY: health list-containers ps stats stats-all stats-live top \
   ports ports-all ports-detailed logs logs-chrome logs-chrome-live \
   shell rebuild up down chrome-version chrome-tabs chrome-health verify-chrome-flags \
   restart-all stop-all wsurl

# ============================================
# Configuration
# ============================================
CHROME_IMAGE_PREFIX := chrome-cdp
CHROME_IMAGES := \
 $(CHROME_IMAGE_PREFIX):debian-headless-stealth-basic \
 $(CHROME_IMAGE_PREFIX):debian-headless-stealth-advanced \
 $(CHROME_IMAGE_PREFIX):debian-gui-stealth-basic \
 $(CHROME_IMAGE_PREFIX):debian-gui-stealth-advanced \
 $(CHROME_IMAGE_PREFIX):alpine-headless-stealth-basic \
 $(CHROME_IMAGE_PREFIX):alpine-headless-stealth-advanced \
 $(CHROME_IMAGE_PREFIX):alpine-gui-stealth-basic \
 $(CHROME_IMAGE_PREFIX):alpine-gui-stealth-advanced

# Build docker filter arguments
DOCKER_FILTERS := $(foreach img,$(CHROME_IMAGES),--filter "ancestor=$(img)")

# Command to get container
GET_CONTAINER = docker ps $(DOCKER_FILTERS) -q | head -1

# Command to get all containers
GET_ALL_CONTAINERS = docker ps $(DOCKER_FILTERS) -q

# ============================================
# Helper Functions
# ============================================
# Check if container exists and set CONTAINER variable
define require_container
 $(eval CONTAINER := $(shell $(GET_CONTAINER)))
 @if [ -z "$(CONTAINER)" ]; then \
  echo "❌ No $(CHROME_IMAGE_PREFIX) container running"; \
  echo "Available images: $(CHROME_IMAGES)"; \
  exit 1; \
 fi
 @echo "✓ Using container: $(CONTAINER)"
endef

# ============================================
# Targets
# ============================================
help:
 @echo "Chrome Container Management"
 @echo ""
 @echo "Available targets:"
 @echo "  make list-containers    - List all running chrome containers"
 @echo "  make verify-chrome-flags - Verify stealth flags in Chrome"
 @echo "  make stats              - Show container stats"
 @echo "  make stats-all          - Show stats for all chrome containers"
 @echo "  make logs               - Show container logs"
 # @echo "  make exec CMD=<cmd>     - Execute command in container"
 @echo "  make shell              - Open shell in container"

list-containers:
 @echo "Running chrome-cdp containers:"
 @docker ps $(DOCKER_FILTERS) --format "table {{.ID}}\t{{.Image}}\t{{.Status}}\t{{.Ports}}"

up:
 @echo "Building and starting chromium container in $(MODE) mode with $(DISTRO) (stealth: $(STEALTH))"
 DISTRO=$(DISTRO) MODE=$(MODE) STEALTH=$(STEALTH) docker compose --env-file $(ENV_FILE) up chrome-$(MODE) -d --build

down:
 docker compose --env-file $(ENV_FILE) down

logs:
 $(call require_container)
 @docker logs -f $(CONTAINER)

shell:
 $(call require_container)
 @docker exec -it $(CONTAINER) bash

rebuild:
 DISTRO=$(DISTRO) MODE=$(MODE) STEALTH=$(STEALTH) docker compose --env-file $(ENV_FILE) build --no-cache chrome-$(MODE)

ps:
 docker compose --env-file $(ENV_FILE) ps

health:
 @echo "Headless:" && curl -fsS http://127.0.0.1:$$(grep ^CDP_HOST_PORT $(ENV_FILE) | cut -d= -f2)/json/version | jq -r .Browser || true
 @echo "GUI:" && curl -fsS http://127.0.0.1:$$(grep ^CDP_PORT_GUI $(ENV_FILE) | cut -d= -f2)/json/version | jq -r .Browser || true

# Quick helper to print a page websocketDebuggerUrl (requires jq)
wsurl:
 @curl -s "http://127.0.0.1:$$(grep ^CDP_PORT_HEADLESS $(ENV_FILE) | cut -d= -f2)/json/new?about:blank" | jq -r .webSocketDebuggerUrl
 @curl -s "http://127.0.0.1:$$(grep ^CDP_PORT_GUI $(ENV_FILE) | cut -d= -f2)/json/new?about:blank" | jq -r .webSocketDebuggerUrl


# ============================================
# Chrome-specific commands
# ============================================
chrome-version:
 $(call require_container)
 @docker exec $(CONTAINER) chromium --version

chrome-tabs:
 $(call require_container)
 @CHROME_PORT=$$(docker port $(CONTAINER) | grep -E '9222|9223|9224' | head -1 | awk -F' -> ' '{print $$2}'); \
 if [ -z "$$CHROME_PORT" ]; then \
  echo "❌ Chrome DevTools port not found"; \
  exit 1; \
 fi; \
 echo "Fetching tabs from http://$$CHROME_PORT/json"; \
 curl -s "http://$$CHROME_PORT/json" | jq -r '.[] | "\(.id): \(.title) - \(.url)"'

chrome-health:
 $(call require_container)
 @CHROME_PORT=$$(docker port $(CONTAINER) | grep -E '9222|9223|9224' | head -1 | awk -F' -> ' '{print $$2}'); \
 if [ -z "$$CHROME_PORT" ]; then \
  echo "❌ Chrome DevTools port not found"; \
  exit 1; \
 fi; \
 echo "=== Chrome DevTools Health Check ==="; \
 echo "Endpoint: http://$$CHROME_PORT"; \
 echo ""; \
 if curl -sf "http://$$CHROME_PORT/json/version" >/dev/null 2>&1; then \
  echo "✅ Chrome DevTools is responding"; \
  echo ""; \
  curl -s "http://$$CHROME_PORT/json/version" | jq -r '"Browser: \(.Browser)\nProtocol Version: \(."Protocol-Version")\nUser Agent: \(."User-Agent")"'; \
 else \
  echo "❌ Chrome not responding"; \
  exit 1; \
 fi

verify-chrome-flags:
 $(call require_container)
 @echo "Verifying Chrome flags in container $(CONTAINER)..."
 @docker exec $(CONTAINER) sh -c \
  "cat /proc/\$$(pgrep -o chromium)/cmdline | tr '\0' '\n' | grep -E 'disable-blink-features|user-agent'" \
  && echo "✅ Stealth flags detected" \
  || echo "❌ No stealth flags"

stats:
 $(call require_container)
 @echo "Container stats:"
 @docker stats --no-stream --format "table {{.Container}}\t{{.CPUPerc}}\t{{.MemUsage}}\t{{.MemPerc}}\t{{.NetIO}}" $(CONTAINER)

stats-all:
 @CONTAINERS=$$($(GET_ALL_CONTAINERS)); \
 if [ -z "$$CONTAINERS" ]; then \
  echo "❌ No $(CHROME_IMAGE_PREFIX) containers running"; \
  exit 1; \
 fi; \
 echo "Stats for all chrome containers:"; \
 docker stats --no-stream --format "table {{.Container}}\t{{.Image}}\t{{.CPUPerc}}\t{{.MemUsage}}\t{{.MemPerc}}" $$CONTAINERS

stats-live:
 @CONTAINERS=$$($(GET_ALL_CONTAINERS)); \
 if [ -z "$$CONTAINERS" ]; then \
  echo "❌ No $(CHROME_IMAGE_PREFIX) containers running"; \
  exit 1; \
 fi; \
 docker stats $$CONTAINERS

top:
 @CONTAINER=$$($(GET_CONTAINER))
 @echo "=== Top Processes in Container ==="
 @docker top $$(docker-compose -f $(COMPOSE_FILE) ps -q $$CONTAINER)

# ============================================
# Bulk operations
# ============================================
restart-all:
 @CONTAINERS=$$($(GET_ALL_CONTAINERS)); \
 if [ -z "$$CONTAINERS" ]; then \
  echo "❌ No containers to restart"; \
  exit 1; \
 fi; \
 echo "Restarting all chrome containers..."; \
 docker restart $$CONTAINERS

stop-all:
 @CONTAINERS=$$($(GET_ALL_CONTAINERS)); \
 if [ -z "$$CONTAINERS" ]; then \
  echo "❌ No containers to stop"; \
  exit 1; \
 fi; \
 echo "Stopping all chrome containers..."; \
 docker stop $$CONTAINERS

# Show port mappings for a single container
ports:
 $(call require_container)
 @echo "=== Port Mappings for Container $(CONTAINER) ==="
 @echo ""
 @docker port $(CONTAINER) | awk -F' -> ' '{print $$1 "\t→\t" $$2}' | column -t -s $$'\t'

# Show port mappings for all chrome containers in a table
ports-all:
 @CONTAINERS=$$($(GET_ALL_CONTAINERS)); \
 if [ -z "$$CONTAINERS" ]; then \
  echo "❌ No $(CHROME_IMAGE_PREFIX) containers running"; \
  exit 1; \
 fi; \
 echo "=== Port Mappings for All Chrome Containers ==="; \
 echo ""; \
 printf "%-15s %-40s %-20s %-20s\n" "CONTAINER ID" "IMAGE" "CONTAINER PORT" "HOST BINDING"; \
 printf "%-15s %-40s %-20s %-20s\n" "---------------" "----------------------------------------" "--------------------" "--------------------"; \
 for container in $$CONTAINERS; do \
  IMAGE=$$(docker inspect --format='{{.Config.Image}}' $$container); \
  SHORT_ID=$$(echo $$container | cut -c1-12); \
  docker port $$container | while IFS= read -r line; do \
   CONTAINER_PORT=$$(echo "$$line" | awk -F' -> ' '{print $$1}'); \
   HOST_BINDING=$$(echo "$$line" | awk -F' -> ' '{print $$2}'); \
   printf "%-15s %-40s %-20s %-20s\n" "$$SHORT_ID" "$$IMAGE" "$$CONTAINER_PORT" "$$HOST_BINDING"; \
  done; \
 done

# Detailed port information with service names
ports-detailed:
 $(call require_container)
 @echo "=== Detailed Port Information for Container $(CONTAINER) ==="; \
 echo ""; \
 IMAGE=$$(docker inspect --format='{{.Config.Image}}' $(CONTAINER)); \
 echo "Container: $(CONTAINER)"; \
 echo "Image: $$IMAGE"; \
 echo ""; \
 printf "%-35s %-20s %-25s %-15s\n" "SERVICE" "CONTAINER PORT" "HOST BINDING" "PROTOCOL"; \
 printf "%-35s %-20s %-25s %-15s\n" "----------------------------------" "--------------------" "-------------------------" "---------------"; \
 docker port $(CONTAINER) | while IFS= read -r line; do \
  CONTAINER_PORT=$$(echo "$$line" | awk -F' -> ' '{print $$1}' | cut -d'/' -f1); \
  PROTOCOL=$$(echo "$$line" | awk -F' -> ' '{print $$1}' | cut -d'/' -f2); \
  HOST_BINDING=$$(echo "$$line" | awk -F' -> ' '{print $$2}'); \
  case $$CONTAINER_PORT in \
   9222|9223|9224) SERVICE="Chrome DevTools Socat Proxy" ;; \
   5900) SERVICE="VNC Server" ;; \
   6080) SERVICE="noVNC Web" ;; \
   *) SERVICE="Unknown" ;; \
  esac; \
  printf "%-35s %-20s %-25s %-15s\n" "$$SERVICE" "$$CONTAINER_PORT" "$$HOST_BINDING" "$$PROTOCOL"; \
 done

# View Chrome startup script logs
logs-chrome:
 $(call require_container)
 @echo "=== Supervisor Chrome Program Logs ==="
 @docker exec $(CONTAINER) cat /var/log/supervisor/chrome.log 2>/dev/null || \
  echo "Chrome log file not found"

# Live tail of Chrome logs
logs-chrome-live:
 $(call require_container)
 @echo "=== Live Chrome Logs (Ctrl+C to exit) ==="
 @docker logs -f $(CONTAINER) 2>&1 | grep --line-buffered "CHROMIUM\|chromium\|Chrome"

Perfomance & size notes

Alpine and Debian size are roughly the same with the current layout, so whatever I am doing wrong, needs a bit more investigation. The list of installed packages can probably be trimmed a bit more, maybe I get some extra savings but I don't expect them to be noticeable at small scale.

Security notes

Haven't added any specific security protocols so don't run this in production without:

Gating with VPN or bind to 127.0.0.1.
Add auth to noVNC/websockify or stick it behind a reverse proxy with auth.
Run as non-root (done above), and keep --no-sandbox only inside containers you control.
Pin package versions for reproducible builds.

WTF moments (and fixes)

Had a bunch a pain with the chromium remote debugging port as it would not bind to 0.0.0.0:2222 on the container so I had to add socat to forward the port. When trying to connect to the remote debugging port of chromium i was getting a connection reset by peer error. Turns out chromium only binds to localhost inside the container unless you use socat to forward the port.
When the browser was not launching inside the container, I thought I messed up, but, as it turns out, any kind of --headless flags should probably be omitted when running with VNC, otherwise the browser will not show up in the VNC session.
this is a hack and probably you can fix it in a better way, but I am lazy, so 2MB of overhead on alpine is me adding bash. I used bash to start chrome with the wrapper script, add the flags all that jazz.
in my first attempt I tried to use chrome, but there were some silly issues with arm64 builds, so I switched to chromium which has better support for arm64.

My conclusion

I will use this setup to automate via CDP. It decouples the browser from the language. I know that this is not ideal as a solo dev but hey it's fun. So I will give it a try.

How I optimized my blog images using Rust

Artur Daschevici — Wed, 12 Feb 2025 10:15:00 +0000

Why?

Ages ago I read a blog post about how images can be optimized and that gives visitors to your site a better experience as the site loads faster yadda, yadda. Why would anyone not want this?

My blog is built with zola so I am a bit of a rust fanboy and want to use rust whenever it makes sense and my novice skills can handle it. Not going to lie, I am not a rust expert and for me ChatGPT is a useful tool as it most often than not points me in the right direction.

But I digress, a few days ago I decided to resurrect my blog, I tried to write a bit more consistently last year and got a streak of a few articles going, was feeling pretty good about it, but then my daughter was born and I was thrown in the gauntlet of figuring things out as a first time dad and the blog was left to rot. Since I am a bit more web marketing savvy, I decided to add some SEO to my blog, maybe I get some more visitors to it and get a sense of how popular it is.

I am trying to be polite with the people that land on my blog and not track them so I don't use cookies. I host my stuff on cloudflare since that gives the best bang for my buck. In other words I want my blog to be performant and free to host.

My blog uses some analytics that are available through cloudflare but they are very respectful of user privacy in that they are GDPR and CCPA compliant. This saves me the hassle of having to add a cookie consent form that disrupts the user navigation experience. I both like and dislike the analytics from cloudflare as the numbers I am seeing are a bit weird as I am only seeing a constant number.

Since I learned a bit more about SEO and about Google Search Console I decided to check my blog's performance and see what I can do to improve it. Submitted my sitemap and ran a performance check and even if performance was at 100/100 I saw that the images were not optimized.

My OCD kicked in and I had to figure out a way to address it, especially since I remembered that I read an article talking about this. I dug into it a bit and noticed he is using ImageMagik, cavif and cwebp to optimize the images, I decided to go a different way, essentially almost reinventing the wheel. I built a rust cli that converts bigger png and jpeg images to webp or cavif.

How?

The step by step process looks like this:

Change the shortcode for images to try and render the optimal image if supported by the browser
write the rust tool that traverses the directory tree and convert images to webp or avif
integrate the tool into the github action pipeline
perform caching on the github workflow to avoid spending too many github minutes on the actual conversion

What's in it for me?

1. Storage Cost Savings

AVIF can be ~70% smaller than PNG while maintaining similar or better quality.
If you're storing images on AWS S3, Google Cloud Storage, DigitalOcean Spaces, or another cloud provider, reducing storage by 70% directly cuts storage costs by the same percentage. Example:
- 100GB of PNGs → ~30GB of AVIF
- If storage costs $0.023 per GB (AWS S3 Standard):
- PNG: $2.30/month
- AVIF: $0.69/month
- Savings: ~$1.61 per 100GB/month (~70%)

2. Egress Bandwidth Cost Savings

Most cloud providers charge for outbound bandwidth (data transferred to users).
Smaller AVIF files mean lower bandwidth usage, leading to significant savings.
AVIF reduces bandwidth usage by ~70% compared to PNG. Example with AWS CloudFront:
- Data transfer cost (to the internet): $0.085 per GB
- If you serve 1TB of PNGs per month:
- PNG: 1TB → $85/month
- AVIF (70% smaller): 0.3TB → $25.50/month
- Savings: ~$59.50 per TB/month (~70%)

3. CDN Caching & Requests

Many e-commerce sites use a CDN (Cloudflare, CloudFront, Fastly, etc.).
Smaller images:
- Improve cache hit ratio (more images fit in CDN cache).
- Reduce origin fetch requests, further lowering egress costs.
- Speed up load times, improving user experience.

Total Cost Savings Estimate

Cost Factor	PNG	AVIF	Savings
Storage (100GB)	$2.30	$0.69	$1.61 (70%)
Egress (1TB/month)	$85.00	$25.50	$59.50 (70%)
Total Savings per TB	$87.11/month

The Playbook

Step 1: Shortcode

I avoided using javascript for this since html already gives a mechanism to render an image with a fallback

<picture>
  <source srcset="{{id}}.avif" {% if alt %}alt="{{alt}}" {% endif %} />
  <source srcset="{{id}}.webp" {% if alt %}alt="{{alt}}" {% endif %} />
  <img src="{{id}}.png" {% if alt %}alt="{{alt}}" {% endif %} />
</picture>

Step 2: Rust tool

The way I structure my posts is that each post lies neatly inside its own folder, along with all the images and any other extra assets that add some sort of value to the content.

So, from the theme I grab all the png images

    let mut input_paths: Vec<Params> = glob("content/**/*.png")?
        .filter_map(Result::ok)
        .map(|path| Params {
            path,
            should_recreate: args.recreate,
            ..Default::default()
        })

The trouble is the theme also contains some images which need to be converted. At the moment the only image is my logo

    let theme_image_paths: Vec<Params> = glob("themes/**/*.jpg")?
        .filter_map(Result::ok)
        .map(|path| Params {
            path,
            should_recreate: args.recreate,
            should_resize: true,
        })
        .collect();

In order to save time, converting images that exist already is a bit redundant so the tool checks if the path exists already and if it does, conversion is skipped. Working with paths is surprisingly straightforward. I was previously quite afraid to write code in rust because I feared the overhead.

The actual code is stupid easy to understand and reason about, even for me

    //  webp file path
    let webp_file_path = parent_dir.join(format!("{}.webp", file_stem).as_str()); // Convert to .webp as an example
    // was it already converted?
    if !webp_file_path.exists() {

A big chunk of the code lies in the conversion code which also gave me the most brain pain.
We converted webp using the webp crate

fn convert_to_webp(img: &DynamicImage, output_path: &str) -> AnyResult<()> {
    let encoder = WebpEncoder::from_image(img).unwrap();
    let webp_data = encoder.encode(75.0); // Quality 75
    let mut file = File::create(output_path)?;
    file.write_all(&webp_data)?;
    println!("Saved WebP to {}", output_path);
    Ok(())
}

And then avif using the avif crate

fn convert_to_avif(img: &DynamicImage, output_path: &str) -> AnyResult<()> {
    let (width, height) = img.dimensions();

    let rgba = img.to_rgba8();
    let encoded_avif = Encoder::new()
        .with_quality(50.0)
        .with_alpha_quality(50.0)
        .with_speed(10)
        .with_alpha_color_mode(AlphaColorMode::UnassociatedClean)
        .with_num_threads(Some(4));
    let avif_pixels = rgba
        .pixels()
        .map(|p| Rgba {
            r: p[0],
            g: p[1],
            b: p[2],
            a: p[3],
        })
        .collect::<Vec<Rgba>>();

    let EncodedImage {
        avif_file,
        color_byte_size,
        alpha_byte_size,
        ..
    } = encoded_avif
        .encode_rgba(Img::new(
            &avif_pixels,
            width.try_into().unwrap(),
            height.try_into().unwrap(),
        ))
        .unwrap();
    let mut file = File::create(output_path)?;
    file.write_all(&avif_file)?;
    println!("Saved AVIF to {}", output_path);
    Ok(())
}

The reason converting to avif was a bit more convoluted was due to the requirement for pixels to be in rgba format. I had to convert the image to rgba and then convert the pixels to Rgba format(thank you libs with different types). This was a bit of a pain but I managed to get it working.

I'm not an image processing expert so the solution was the result of a long conversation with trail and error with ChatGPT, then again this is why I love rust and how strict it is. It forces you to write code in a way that if it runs it most likely is correct.

Step 3: Github action

The action installs and enables rust so that the cli can be used

- name: Install Rust
  run: |
    curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh -s -- -y
    rustup toolchain install nightly
    rustup default nightly
    echo "$HOME/.cargo/bin" >> $GITHUB_PATH

- name: Verify Rust Installation
  run: |
    rustup --version
    rustc --version
    cargo --version

- name: Build CLI tool
  run: |
    cargo build --manifest-path ./helpers/image-optimizer/Cargo.toml --release --verbose

- name: Perform the optimization
  run: ./helpers/image-optimizer/target/release/image-optimizer

One thing I particularly liked was the fact that it is possible to use a relative path to Cargo.toml which means no mucking about with paths.

Step 4: Caching

Now one thing about rust that is a bit of a bummer is that builds take quite some time. I guess that is the price to pay for static memory analysis. I would love to do a deep dive at some point on the optimization of rust build times but that is a story for another time.

The one thing that github tends to hold you accountable for is the number of build minutes you use when a workflow runs, so having rust install itself, download dependencies and then run a build for the tools can quickly add up.

The optimization for build times covers caching cargo dependencies but also caching the built binary.

I cached most things that I was able to but I am getting mixed results when trying to cache apt packages. It simply does not seem to work as intended in the naive approach.

- name: Install OS Dependencies (if needed)
  run: |
    # Create a file listing your required packages, one per line.
    cat > apt-packages.txt << EOF
    nasm
    EOF
    sudo apt-get update
    sudo apt-get install -y --no-install-recommends $(cat apt-packages.txt)

- name: Cache Rust toolchain
  uses: actions/cache@v3
  with:
    path: ~/.rustup
    key: ${{ runner.os }}-rustup-${{ hashFiles('rust-toolchain') }}
    restore-keys: |
      ${{ runner.os }}-rustup-

  ...
- name: Cache Cargo dependencies and target
  uses: actions/cache@v3
  with:
    path: |
      ~/.cargo/registry
      ~/.cargo/git
      ./helpers/image-optimizer/target
    key: ${{ runner.os }}-manual-cargo-${{ hashFiles('**/Cargo.lock') }}

The savings in time by using the caching is quite substantial. The first run of the workflow took 15 minutes, the run that had cached the deps was less than 1 minute. Even with a substantial amount of images this will most likely not be a bottleneck.

Conclusion

Optimizing images can decrease load on the server up to 70%
... and it also improves performance which is beneficial for SEO
Optimizing github workflows can save you a lot of wait time
... and github minutes
While this was interesting to do, I optimized for something that did not move the needle at all, it just made the evaluation in the Google Search Console a bit better.

This is what I have to blame my OCD for. I am happy with the result and I learned quite a few things about rust and image processing. I am also happy that I managed to get the avif conversion working as it is a format that is not yet widely supported but is the most efficient format out there.

Rustify some puppeteer code(part I)

Artur Daschevici — Thu, 27 Jun 2024 15:30:00 +0000

Why?

Rust is pretty amazing but there are a few things that you might be weary about. There are few war stories of companies building their entire stack on rust or and then living happily ever after. Software is an ever evolving organism so in the darwinian sense the more adaptable the better. Enough of that though, not here to advocate any particular language or framework, what I want is to share my experience with writing an equivalent scraper in rust to my previous post where I used golang and chromedp.

The experience using go with chromedp to automate chrome was pretty good, it is not as powerful as what is available in puppeteer so I figured I would have a look at what might be available in the rust landscape.

What?

In rust there are several libraries that deal with browser automation, a few I have had a look at are:

fantocini - A high-level API for programmatically interacting with web pages through WebDriver, but I want chrome devtools protocol instead.
rust-headless-chrome - chrome devtools protocol client library in rust, not as active as the crate I wound up using.
chromiumoxide - this is the one that seem to be the most active in terms of development so it looks like a good choice at time of writing.

TL;DR

As I was reading one of my older posts that focuses on quasi live coding I realized it was boring as hell, and if your attention span is that of a goldfish, like mine is, it would probably make sense to just drop in a link to the repo so that you can download the code and try it out yourself. The repo is a collection of rust prototypes that I have been building for fun and learning, haven't had yet a compelling reason to use rust in production unfortunately 😢.

How?

To my surprise the code was closer in structure to the puppeteer version than it was to the chromedp. The chromedp version uses nested context declarations to manage the browser and page runtimes, the rust version uses a more linear approach. You construct a browser instance and then you can interact with it as a user would. This points at the fact that the chromiumoxide api is higher level.

The way you can set things up to keep your use cases separate is by adding clap to your project and use command line flags to select the use case you want to run.

You will see that I have covered most cases but not everything is transferable from puppeteer or chromedp to the chromiumoxide version. I will not go through the setup of rustup, rust toolchain or cargo as this is a basic and well documented process, all you have to do is search for getting started with rust and you will find a bunch of resources.

Show me the code

1. Laying down the foundation

set up my project root

cargo new rust-crawl-pupp
cd rust-crawl-pupp
cargo install cargo-edit # this is useful for adding and upgrading dependencies

add dependencies via cargo add

[dependencies]
chromiumoxide = { version = "0.5.7", features = ["tokio", "tokio-runtime"] } # this is the main dependency
chromiumoxide_cdp = "0.5.2" # this is the devtools protocol
clap = { version = "4.5.7", features = ["derive", "cargo"] } # this is for command line parsing
futures = "0.3.30" # this is for async programming
tokio = { version = "1.38.0", features = ["full"] } # this is the async runtime
tracing = "0.1.40" # this is for logging
tracing-subscriber = { version = "0.3.18", features = ["registry", "env-filter"] } # this is for logging

add clap command line parsing to the project so that each different use case can be called via a subcommand
define your imports

use clap::{Parser, Subcommand};

define your command structs for parsing the command line arguments, this will allow for each use case to be called with its own subcommand like so cargo run -- first-project, cargo run -- second-project, and so on.

#[derive(Parser)]
#[command(
    name = "OxideCrawler",
    version = "0.1",
    author = "artur",
    about = "An example application using clap"
)]
struct Cli {
    #[command(subcommand)]
    command: Commands,
}
#[derive(Subcommand, Debug)]
enum Commands {
    FirstProject {},
    SecondProject {},
    ...
}

the way you can hook this into the main function is via a match statement that will call the appropriate function based on the subcommand that was passed in.

let args = Cli::parse();
...
match &args.command {
    Commands::FirstProject {} => {
        let user_agent = spoof_user_agent(&mut browser).await?;
        info!(user_agent, "User agent detected");
    }
    ...
}

2. Starting browser and the browser cleanup

use the launch method and its options to start the browser, if the viewport and window size are different, the browser will start in windowed mode, with the page size being smaller.

let (mut browser, mut handler) = Browser::launch(
    BrowserConfig::builder()
        .with_head() // this will start the browser in headless mode
        .no_sandbox() // this will disable the sandbox
        .viewport(None) // this will set the viewport size
        .window_size(1400, 1600) // this will set the window size
        .build()?,
)
.await?;

let handle = tokio::task::spawn(async move {
    loop {
        match handler.next().await {
            Some(h) => match h {
                Ok(_) => continue,
                Err(_) => break,
            },
            None => break,
        }
    }
});

the browser cleanup needs to be done correctly and there are two symptoms that you will see if you missed anything:
- the browser will not close - hangs at the end
- you might get a warning like the following:
```
  2024-06-26T08:40:01.418414Z  WARN chromiumoxide::browser: Browser was not closed manually, it will be killed automatically in the background
```
to correctly clean up your browser instance you will have to call these on the code paths that close the browser
```
browser.close().await?;
browser.wait().await?;
handle.await?;
```

3. Use cases

In the repo each use case lives in its own module most of the time. There are some cases where you might have two living in the same module when they are very closely related, like in Use Case c..

a. Spoof your user agent:

The only way I have found to set your user agent was from the Page module via the set_user_agent method

  let page = browser.new_page("about:blank").await?;
  page.set_user_agent(
      "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) \
       Chrome/58.0.3029.110 Safari/537.36",
  )
  .await?;
  page.goto("https://www.whatismybrowser.com/detect/what-is-my-user-agent")
      .await?;

b. Grabbing the full content of the page is pretty straightforward

      let page = browser
        .new_page("https://scrapingclub.com/exercise/list_infinite_scroll/")
        .await?;
    let content = page.content().await?;

c. Grabbing elements via css selectors,

  let elements_on_page = page.find_elements(".post").await?;
  let elements = stream::iter(elements_on_page)
      .then(|e| async move {
          let el_text = e.inner_text().await.ok();
          match el_text {
              Some(text) => text,
              None => None,
          }
      })
      .filter_map(|x| async { x })
      .collect::<Vec<_>>()
      .await;

performing relative selection from a specific node and mapping the content to rust types

  ...
  let product_name = e.find_element("h4").await?.inner_text().await?.unwrap();
  let product_price = e.find_element("h5").await?.inner_text().await?.unwrap();
  Ok(Product {
      name: product_name,
      price: product_price,
  })
  ...

d. When the page has infinite scroll you will have to scroll to the bottom of the page to be able to collect all the elements you are interested in. To achieve this you need to inject javascript into the page context and trigger a run of the function. The chromiumoxide api seems to have really decent support for this, I faced much less resistance than I did with chromedp and go.

  let js_script = r#"
      async () => {
        await new Promise((resolve, reject) => {
          var totalHeight = 0;
          var distance = 300; // should be less than or equal to window.innerHeight
          var timer = setInterval(() => {
            var scrollHeight = document.body.scrollHeight;
            window.scrollBy(0, distance);
            totalHeight += distance;

            if (totalHeight >= scrollHeight) {
              clearInterval(timer);
              resolve();
            }
          }, 500);
        });
    }
  "#;
  let page = browser
      .new_page("https://scrapingclub.com/exercise/list_infinite_scroll/")
      .await?;
  page.evaluate_function(js_script).await?;

e. When you need to wait for an element to load, this was not exactly part of the chromiumoxide api so I had to hack it together. Given my limited rust expertise there probably a better way to do this but this is what I managed to come up with. If the async block runs over the timeout then the element_result will be an error, otherwise poll the dom for the element we are looking for.

  use tokio::time::{timeout, Duration};
  ...
  let element_result = timeout(timeout_duration, async {
      loop {
          match page.find_element(selector).await {
              Ok(element) => return Ok(element),
              // Wait for a short interval before checking again
              Err(e) => tokio::time::sleep(Duration::from_millis(100)).await,
          }

      }
  })
  .await;

4. Fixtures to replicate various scenarios

Some websites, actually most websites have some sort of delay for loading different parts of the page, in order to prevent blocking the entire page. To replicate this behavior fixtures can be used to inject nodes into the dom with a delay. For the more edge case scenarios I created fixtures to emulate edge behaviors while not actually having to remember a website that is live and behaves like that.

The HTML is really basic:

  <div id="container">
      <!-- New node will be appended here -->
  </div>

  <script src="script.js"></script>

The script.js file is slightly more, but still fairly straightforward:

  document.addEventListener('DOMContentLoaded', () => {
    // Function to create and append the new node
    function createDelayedNode() {
      // Create a new div element
      const newNode = document.createElement('div');

      // Add some content to the new node
      newNode.textContent = 'This is a new node added after a delay.';

      // Add some styles to the new node
      newNode.style.padding = '10px';
      newNode.style.marginTop = '10px';
      newNode.style.backgroundColor = '#f0f0f0';
      newNode.style.border = '1px solid #ccc';
      newNode.id = 'come-find-me';

      // Append the new node to the container
      const container = document.getElementById('container');
      container.appendChild(newNode);
    }

    // Set a delay (in milliseconds)
    const delay = 3000; // 3000ms = 3 seconds

    // Use setTimeout to create and append the node after the delay
    setTimeout(createDelayedNode, delay);
  });

What it will do is create a new node with some text content and some styles, then append it to the container div after a delay of 3 seconds.

Why to be continued?

What I hate more than to be continued in a TV show where I don't have the next episode available is a blog post that has code that looks reasonable and that it might work, but doesn't. So going by the lesser of two evils principle I decided to make this a two parter which will give me the time to write and test the other use cases in order to make sure everything works as expected.

Conclusions

This is one of the few times I have stuck with rust through the pain and I have to say it was a better experience than I had with go and chromedp
writing the code was slightly faster since there was less boilerplate to write
messing around with wrappers and unwrap() was challenging but probably in time it gets easier
the code in rust looks more like puppeteer than the go version did

In Part II I will cover dealing with bot protection, handling frames, forms and more. Stay tuned!

OpenAI api RAG system with Qdrant

Artur Daschevici — Wed, 19 Jun 2024 12:35:00 +0000

Why?

OpenAI has been making it easier and easier to build out GPT agents that make use of your own data to improve the generated responses of the pretrained models.

Agents give a way to inject knowledge about your specific proprietary data into your pipeline, without actually sharing any private information about it. You can also improve the recency of your data too which makes you less dependent on the model's training cycle.

OpenAI has improved the DX, UX and APIs since version 3.5, and has made it easier to create agents and embed your data into your custom GPTs. They have lowered the barrier to entry which means that virtually anyone can build their own assistants that would be able to respond to queries about their data. This is perfect for people to experiment on building products. IMO this is a very good approach to enable product discovery for the masses.

Most big AI contenders on the market provide you with a toolbox of high level abstractions and low to no code solutions. The weird thing about my approach to learning things is that not having some understanding of the first principles of the tech I'm using makes me feel a bit helpless, this is why I figured trying to build my own RAG system would be a good way to figure out the nuts and bolts.

What?

I wanted to get a project for running my own pipeline with somewhat interchangeable parts. Models can be swapped around so that you can make the most of the latest models either available on Hugginface, OpenAI or wherever.

Because things are moving so fast in model research the top contenders are surpassing each other every day pretty much. A custom pipeline would allow us to quickly iterate and test out new models as they evolve. This allows you to try out new models and just as easily rollback your experiment.

What I wound up building is a Streamlit app that uses qdrant to index and search data extracted from a collection of pdf document. The app is a simple chat interface where you can ask questions about the data and get responses from a mixture of GPT-4 and the indexed data.

How?

1. Setting up the environment

use pyenv to manage python versions

   # update versions
   pyenv update
   # install any python version
   pyenv install 3.12.3 # as of writing this
   # create a virtualenv
   ~/.pyenv/versions/3.12.3/bin/python -m venv .venv
   # and then activate it
   source .venv/bin/activate

2. Install the dependencies

   # install poetry
   pip install poetry
   # install the dependencies
   poetry install

the dependencies section of the pyproject.toml file should look like this:

   ...
   [tool.poetry.dependencies]
    python = "^3.12"
    streamlit = "^1.32.1"
    langchain = "^0.1.12"
    python-dotenv = "^1.0.1"
    qdrant-client = "^1.8.0"
    openai = "^1.13.3"
    huggingface-hub = "^0.21.4"
    pydantic-settings = "^2.2.1"
    pydantic = "^2.6.4"
    pypdf2 = "^3.0.1"
    langchain-community = "^0.0.28"
    langchain-core = "^0.1.31"
    langchain-openai = "^0.0.8"
    instructorembedding = "^1.0.1"
    sentence-transformers = "2.2.2"
   ...

3. Set up the loading of the variables from a config file

a nice way to manage settings is to use pydantic and pydantic-settings

   from pydantic import Field, SecretStr
   from pydantic_settings import BaseSettings, SettingsConfigDict

   class Settings(BaseSettings):
       model_config = SettingsConfigDict(env_file="config.env", env_file_encoding="utf-8")
       hf_access_token: SecretStr = Field(alias="HUGGINGFACEHUB_API_TOKEN")
       openai_api_key: SecretStr = Field(alias="OPENAI_API_KEY")

this way you can load the settings from config.env but variables in the environment override the ones in the file.

a nice extra is that you also get type checking and validation from pydantic including SecretStr types for sensitive data.

4. Set up the UI elements

Streamlit makes it quite easy to strap together a layout for your app. You have a single script that can run via the streamlit binary:

   streamlit run app.py

The gallery has many examples of various integrations and components that you can use to build your app. You have smaller components like inputs and buttons but also more complex UI tables, charts, you even have ChatGPT style templates.

For our chat interface we require very few elements. Generally to create them you only need to use streamlit to initialize the UI.

   import streamlit as st
   ...
   def main():
       st.title("ChatGPT-4 Replica")
       st.write("Ask me anything about the data")
       question = st.text_input("Ask me anything")
       if st.button("Ask"):
           st.write("I'm thinking...")
           response = get_response(question)
           st.write(response)
   ...
   main()

The one thing I find a bit awkward is the fact that if you have elements that need to be conditionally displayed the conditions tend to resemble the javascript pyramid of doom if you have too many conditionals in the same block.

Below is a simple example so you can see what I mean:

   if len(pdf_docs) == 0:
       st.info("Please upload some PDFs to start chatting.")
   else:
       with st.sidebar:
           if st.button("Process"):
               with st.spinner("Processing..."):
                   # get raw content from pdf
                   raw_text = get_text_from_pdf(pdf_docs)
                   text_chunks = get_text_chunks(raw_text)

                   if "vector_store" not in st.session_state:
                       start = time.time()
                       st.session_state.vector_store = get_vector_store(text_chunks)
                       end = time.time()
                       # create vector store for each chunk
                       st.write(f"Time taken to create vector store: {end - start}")

This makes me think that it is probably not designed for complex UIs but rather for quick prototyping and simple interfaces.

5. pdf data extraction

I used the PyPDF2 library to extract the text from the pdfs. The library is quite simple to use and you can extract the text from a pdf file with a few lines of code.

   import PyPDF2

   def get_text_from_pdf(pdf_docs):
       raw_text = ""
       for pdf in pdf_docs:
           pdf_file = pdf["file"]
           pdf_reader = PyPDF2.PdfFileReader(pdf_file)
           for page_num in range(pdf_reader.numPages):
               page = pdf_reader.getPage(page_num)
               raw_text += page.extract_text()
       return raw_text

The extracted text should be chunked into smaller pieces that can be used to create embeddings for the qdrant index.

   def get_text_chunks(raw_text):
       text_chunks = []
       for i in range(0, len(raw_text), 1000):
           text_chunks.append(raw_text[i:i + 1000])
       return text_chunks

6. Setting up the `qdrant` server via `docker`

The best way to set up qdrant is to use docker and to keep track of the environment setup docker-compose is a nice approach. You can set up the qdrant server with a simple docker-compose.yml file like the one below:

   version: '3.9'

   services:
     qdrant:
       image: qdrant/qdrant:latest
       ports:
         - "6333:6333" # Expose Qdrant on port 6333 of the host
       volumes:
         - qdrant_data:/qdrant/data # Persistent storage for Qdrant data
       environment:
         RUST_LOG: "info" # Set logging level to info

   volumes:
     qdrant_data:
       name: qdrant_data

7. Indexing the data

The qdrant client can be used to index the embeddings and perform similarity search on the data. You can pick and choose the best model for embeddings for your data and swap them out if you find a better one.

   def get_vector_store(text_chunks, qdrant_url="http://localhost:6333"):
       embeddings = HuggingFaceInstructEmbeddings(model_name="avsolatorio/GIST-Embedding-v0", model_kwargs={"device": "mps"})
       vector_store = Qdrant.from_documents(
           text_chunks,
           embeddings,
           url=qdrant_url,
           collection_name="pdfs",
           force_recreate=True,
       )
       return vector_store

8. sending the query

In order to send the query to qdrant you again need to embed it to allow to do a similarity search over your collection of documents.

   def get_response(question, qdrant_url="http://localhost:6333"):
        embeddings = HuggingFaceInstructEmbeddings(model_name="avsolatorio/GIST-Embedding-v0", model_kwargs={"device": "mps"})
        query_vector = embeddings.encode(question)
        vector_store = Qdrant(url=qdrant_url, collection_name="pdfs")
        response = vector_store.search(query_vector, top_k=1)
        return response

9. Analysis

You can swap out any of the components in this project with something else. You could use Faiss instead of qdrant, you could use OpenAI models for everything(embeddings/chat completion) or you could use open models.

You can forego the UI and simply use fastapi to create an API to interact with the PDF documents. I hope this gives you some sense of the possibilities that are available to you when building your own RAG system.

Conclusions

you can build your own agent and have it respond to queries about your data quite easily
streamlit is great for prototyping and building out simple interfaces
qdrant is good for performing similarity search on your data
when building RAG systems you need to make use of embedding models to encode your data
embedding models are the most taxing parts of the pipeline
if you have pluggable parts in your pipeline you can swap them out easily to save costs
pydantic and pydantic-settings are great for adding type checking and validation to your python code

Converging project boilerplates with copier

Artur Daschevici — Tue, 11 Jun 2024 10:00:00 +0000

Why?

Technically when you start a new project the best way to approach it is by using the CLI tool of the realm, such as svelte-kit, astro, django-cli etc..., you get the idea. The huge bonus to doing this is that you get the best practices baked in and as new standards are created the CLI gets updated.

So far the frontend has been a lot luckier with the tools as far as project generation goes, every major framework having come out with their own project generation tool, some having more than one possibly due to multiple schools of thought.

There are some backend frameworks that have project generation tools too but so far it seems to be difficult to agree on the structure. The best you can do is find a way to structure it that looks like the majority and makes sense for you. I have been building spiders and crawlers for data ingestion pipelines using python at first and then node and go. Even more recently I have been looking at hacking out some tweaks in some of my neovim plugins(that is lua).

For example neovim plugins have a pretty standard setup, they will have a folder layout something like the following:

scratcher.nvim/
├── README.md
├── lua
│   └── scratcher
│       └── init.lua
└── plugin
    └── scratcher.lua

The standard way of naming things seems to be gravitating towards having some conventions as you can see so setting up a new plugin would be pretty much repetitive and automatable. And it will probably save you some time and willpower in the long run, provided you have some sense of what your final architecture needs to look like.

How?

If you are coming from python like I am then you may already be familiar with cookiecutter. I have been in the situation a few times where it might have made sense to use it, but every time it was a matter of balancing out the timeline and trying to stay away from over engineering.

Lately though the stuff I have been dealing with has been slightly on the more experimental side so churning out something new is something that happens quite often, so it makes more sense to have a prebaked architecture for specific project styles.

From the project templating libraries I was aware of cookiecutter and copier. cookie-cutter uses json for driving the generation while copier uses yaml.

In the end I used copier as I tend to favor yaml because it allows for comments in the config. It makes it easy to plop random pieces of info or even docs in there. Since the config files driving the wizard it can become quite convoluted and also difficult to read as you would normal code so having the ability to document different options is probably a plus I would think.

The library allows you to build a sort of setup wizard where you can set up your desired flow of questions, and you can use the choices supplied to drive what folders will be used in the final project boilerplate. This is pretty nifty as it gives you the ability to customize stuff all the way down to the build process.

Another neat thing is that when you have decent chunks of code that can be shared, so you can just put that in your boilerplate, so it will essentially give you things just the way you like them.

Cherry pick of features

There are a few notable features that I would kick myself if I didn't mention:

you can define choices for your options by using the choices key in your copier.yml file:

project_type:
type: str
help: What type of project are you creating?
default: neovim-plugin
choices:
    - neovim-plugin
    - golang-cli
    - python-cli

you can include different files in the root level copier.yml thus breaking down the wizard in composable parts, for example the CI/CD parts can be shared across projects
```
!include shared-conf/ci-cd.*.yml
```
defaults are very powerful and can also make use of current runtime context which is quite nice. Essentially you can think of it as a way to have your very own project wizard that is tweaked for every one of your needs.

Cool use-cases

you can define a folder/file be created conditionally depending on an option selection eg:

You define your copier.yml like this to give you a choice into the type of project:
```
project_type:
type: str
help: What type of project are you creating?
default: neovim-plugin
choices:
    - neovim-plugin
    - golang-cli
    - python-cli
```
you then create a conditionally rendered folder, the naming follows jinja templating rules, so it might look something like the following
```
{% if project_type == 'neovim-plugin' %}{{project_name}}.nvim{% endif %}
```
It looks a bit strange, it will probably not work on Windows and it might look daunting at first but hopefully you will only need to revisit the hierarchy when you update your project structure template. This is not something I would expect to happen very often.
in more advanced use cases you may need to write some custom python code for transforming/processing of entities in your jinja templates, or template strings.
To hook this in you need to enable the jinja template extensions and add a separate package copier-templates-extensions
```
_jinja_extensions:
- copier_templates_extensions.TemplateExtensionLoader
- extensions/context.py:ContextUpdater
```
This allows you to load specific extensions in your project generator runtime and it can serve different functions. The following snippet illustrates a way you can update the context:
```
from copier_templates_extensions import ContextHook

class ContextUpdater(ContextHook):
    def hook(self, context):
        new_context = {}
        new_context["onboarding"] = "first steps with " + context["project-type"]
        return new_context
```
For example you might decide to create a file called "first steps with .txt" once the project is generated. In template form the file name would be {{ onboarding }}

Conclusions:

you can build hybrid boilerplates for your project and drive generating the folder hierarchy from a single repo
the notation is a bit weird with templated folder names, will not work on Win
the templated naming is also very powerful allowing for conditional creation of folders

Gopherizing some puppeteer code

Artur Daschevici — Thu, 06 Jun 2024 22:00:00 +0000

Why?

As developers we sometimes get a bad case of the shiny new object syndrome. I hate to say it but every time I start hacking on something new, the urge to add something new is quite overwhelming. It is really tough to keep an interest in projects for a long time and it starts to become tedious the deeper you go into the weeds. I suppose this is why people list growth as one of their top motivations.

I consider that anything new is an opportunity for growth, and doing something over and over in a similar manner quickly becomes a tedious. I've been building various types of scrapers since 2011, and it all started because I wanted to automate a workflow and save myself some time. The time spent on automating this was probably more than if I had done this by hand but it was interesting interacting via http from code and crunching the data automatically.

The amount of data on the web is pretty crazy, you have various sources and multiple types of data that can be combined in very interesting ways. Back in those days dropshipping was becoming huge and people were performing arbitrage across Amazon/Ebay/local flea-markets etc.. Tools that were able to perform analytics across these shops were quite trendy, and the market was slightly less crowded, so for me building crawlers seemed like a nice idea to build out a good customer base.

Nowadays due to RAG systems, gathering data automatically, breaking it down and feeding it into embedding models and storing it in vector databases for LLM information enhancement has come back into the spotlight. In between then and now there have been a few changes in the way data is served up for consumption. Off the top of my head:

single page apps have gained huge traction, most everyone turning to building their content in a js bundle, loading everything on the fly as the page loads
websites have become fussy about having their data used by unknown parties, so they have been closing down access and have become very litigious(#TODO: maybe add some cases of court cases Linkedin vs those guys, Financial Times vs OpenAI)
bot detection and prevention - this one is funny since it is like a flywheel, it built 2 lucrative markets overnight - bot services and anti bot protection
TBH, it's difficult to predict where this might be heading, it kind of feels like people have been aiming to move all their datas into data centers but since data is becoming so guarded...will they move back to paper?

Because of SPAs and the wide adoption of js in websites it is much more convenient to use some sort of browser automation to crawl pages and extract the information. This makes it less prone to badgering the servers, and having to reverse engineer the page content loading, so you will probably want to use either a chrome developer tools protocol or webdriver flavored communications protocol with the browser. Back in the day IIRC I have also used the PyQt bindings for acessing the Qt browser component but nowadays its mostly straight-up browsers.

These days my goto is puppeteer. It's a weird tool that can be easily be used to scrape data from pages. The reason I say it is weird is mainly due to the deceiving nature of the internals, essentially using two js engines that communicate via the cdp protocol that is a a very dense beast and does not play nice with complex objects.

Recently it has become more appealing to me to use strongly typed languages. This is probably because I have started to narrow down my experiments to very small code samples that illustrate one thing at time. I would go as far as to call it experiment driven development. Duck typing is fun as you can print pretty much anything you want. I was thinking to use rust but it has a very tough learning curve. Node is pretty nice with mjs but it's confusing sometimes when it crosses over between the two event loops, also while it is good for communicating on cdp it is not really designed for sync code and python is a bit boring for me so I decided to look at go. Since it is a google language I expected it to have decent support for cdp, and the learning curve is slightly gentler than rust.

How?

Looking at the alternatives there are two that stand out chromedp and rod. Rod looks like it is the prodigal son of behave and cucumber some well established BDD frameworks. Personally I am not finding the MustYaddaYadda... very readable and combining it with other custom APIs would probably make it become inconsistent. It has a few nice things in the way it abstracts iframes but I am just unable to go past the higher level API.

In the end I wound up choosing chromedp. It works pretty well for most use cases, there are some places where it doesn't quite cut it and I wish it did, but by now I have come to terms there is no one technology to rule them all, wouldn't it be nice if that existed?

You can install it via go get -u github.com/chromedp/chromedp and then you can start using it in your code. It has quite a few submodules and related projects that you may want to use depending on your concrete use case.
Generally if your use case is only data extraction and you have no tricky actions to deal with(page is bot resistant, some elements are loaded at later times, iframe hell etc...).

import (
    "context"
    "log"
    "time"

    "github.com/chromedp/chromedp"
    "github.com/chromedp/cdproto/cdp"
    // for slightly more advanced use cases
    "github.com/chromedp/cdproto/browser"
    "github.com/chromedp/cdproto/dom"
    "github.com/chromedp/cdproto/storage"
    "github.com/chromedp/cdproto/network"
)

The pleasant surprises

Well, calling them surprises is a bit of a stretch, I have been golang over the years and I have to admit it is a pretty nice ecosystem and language.
chromedp automates chrome or any binary that you are able to communicate with via cdp. The API is somewhat intuitive, haven't found myself diving into the guts of it very often to figure out how stuff works. The good part is that once you extract the data from the nodes you are interested in you can map it to go structs and make use of the go typing system.

For example you can grab a list of elements via selector:

    var productNodes []*cdp.Node
    if err := chromedp.Run(ctx,
        // visit the target page
        chromedp.Navigate("https://scrapingclub.com/exercise/list_infinite_scroll/"),
        chromedp.Evaluate(script, nil),
        chromedp.WaitVisible(".post:nth-child(60)"),
        chromedp.Nodes(`.post`, &productNodes, chromedp.ByQueryAll),
    ); err != nil {
        log.Fatal("Error while trying to grab product items.", err)
    }

then map each element to a struct

    for _, node := range productNodes {
        if err := chromedp.Run(ctx,
            chromedp.Text(`h4`, &name, chromedp.ByQuery, chromedp.FromNode(node)),
            chromedp.Text(`h5`, &price, chromedp.ByQuery, chromedp.FromNode(node)),
        ); err != nil {
            log.Fatal("Error while trying to grab product items.", err)
        }
        products = append(products, Product{name: name, price: price})
    }

Another nice perk is that go is built with concurrency in mind so crunching the extracted data can be a lot more performant than in puppeteer.

Yet another pretty nifty thing I found is that you can deliver a binary that can be compiled for multiple platforms and can be distributed easily. This is a huge plus given that you may not really know who the user of the tool might be in the end.

The ugly parts

The way to communicate with the browser is still through the cdp protocol and sometimes you need to pass objects only objects that can be serialized.

If you need to work with objects that can't be serialized you will need to inject js into the page context and interact with it.

When you have a page that contains iframes it is problematic to trigger events on the elements inside them. You can extract data from it but triggering events gets messy as you need js for that.
An example of how you might extract data from an iframe might look something like this:


    var iframes []*cdp.Node
    if err := chromedp.Run(ctx, chromedp.Nodes(`iframe`, &iframes, chromedp.ByQuery)); err != nil {
        fmt.Println(err)
    }

    if err := chromedp.Run(ctx, chromedp.Nodes(`iframe`, &iframes, chromedp.ByQuery, chromedp.FromNode(iframes[0]))); err != nil {
        fmt.Println(err)
    }
    var text string
    if err := chromedp.Run(ctx,
        chromedp.Text("#second-nested-iframe", &text, chromedp.ByQuery, chromedp.FromNode(iframes[0])),
    ); err != nil {
        fmt.Println(err)
    }

But in order to trigger events on elements inside the iframe you can't just use the chromedp API, and since chromedp.Evaluate does not take a Node as context you will need to perform all the actions in javascript and that will make the resulting code a bit of a mishmash of go and js.

puppeteer also has some extra packages that can be used like puppeteer-stealth but chromedp does not seem to have an equivalent for that at this time. The rod package has rod stealth but I haven't tried it since the API is not to my liking.

The other slightly dissappointing missing feature is that when running in headless mode all the GPU features are disabled because it is running in a headless-chrome container which does not have a display server. Puppeteer is able to run with GPU features enabled allowing it to pass the webgl fingerprinting tests.

Conclusion

in some ways puppeteer is still better than chromedp, working with iframes falls short
rod is a nice alternative but its API looks like it was designed for testing, reminds me of cucumber
chromedp is a nice alternative to puppeteer if you are looking to build a binary that can be distributed easily
it is a bit more performant than puppeteer due to the concurrency model in go

Keep your estimates boring

Artur Daschevici — Wed, 29 May 2024 09:12:00 +0000

Why?

Most everyone I know has got shiny object syndrome. We all want to work on the latest and greatest. I myself am part of that crowd very much. Whenever I start on a project I will never pin the versions for the libraries. That adds anywhere between 0% and 50% on top of the project timeline. The most notable example is Javascript with its myriad of libraries. You would think everyone is familiar with this meme by now 😅.

In the first article in the series we talked about getting anchors right and trying to stay away from the uniqueness bias when choosing a good anchor. In How Big Things Get Done the authors also bring in a concept that was new to me: the reference class.
The phrase was originally coined in the 1970s by the psychologist Daniel Kahneman and his colleague Amos Tversky and is regularly used in the context of reference class forecasting.
Daniel and Amos refer to two types of views when estimating a project:

inside view which is the view while working on the project with your personal biases
outside view which is the view from the outside, looking at the project as a whole and comparing it to similar projects

Enough with the theory, in practice, what we actually want to achieve is to have our estimates work and be as accurate as possible.

How?

Now that we have the lingo down, let's get into the nitty-gritty. We want to figure out how to calculate the estimates, you got that right calculate. The calculations are based on being able to cut down your project from being a special and unique snowflake to a project that is similar to others. It's a combination of statistical and historical analysis of other projects as similar as possible to yours.

Going on a tangent here, wouldn't it be cool if we could have a database of software project estimates with numerical data, situational requirements and conditions, and perhaps how long the project took in the end? 🤔

You want to reduce your project to something as generic as possible then look for data about other projects like it. As a software developer you may be tempted to think that it is special and unique, but finding the commonalities will help you get a better estimate.
We could make use of both inside view and outside view:

outside view: see how long similar projects took(take the median) - I am referring to the reduced version where you cut out any product differentiators
inside view: see how long the differentiators will take (this you can break down further as well into common tasks and unique tasks)

... do you see where I am going with this? It's all turtles all the way down 🐢. Now you can already see how things can be broken down further into smaller pieces and how similarities can make estimating easier.

The numbers show a 30% increase in accuracy when using reference class forecasting, that is the outside view, with 50% not being uncommon.

The aspect that is different from plain anchor-based estimates is that you choose an anchor that is based on the reference class which makes it closer to the objective reality.

Into the future with AI

Last couple of years I have been working in the field of AI and I have been an avid reader of various papers and consumed a decent amount of tutorials and courses. Deep learning is an amazingly powerful tool that is able to draw conclusions based on the importance of a particular feature of the project and classify it.

If we had the data about projects we could train a multi-class classifier to predict the time it would take to complete a project(S/M/L/XL). This could be a great Trello plugin for example.

Linear regression can be another simpler approach to do a numeric estimate of the project timeline. Now, this feels like we are taking all the joy out of the agile SDLC, but remember this is only supposed to be used as a data focused approach, from the outside view i.e. looking objectively at the data, so no hard feelings to be had 😉.

Thinking a bit further we could have an LLM + RAG system that looks at the database of projects we have broken down, does a similarity search and gives us some kind of standard estimates.

The data would probably be a huge challenge for this one. You would have to get data from various sources and have it clean, usable and can be used to train the models.

Conclusion

Stay boring: don't get caught up in the thinking your project is special and unique
Use the reference class: look at similar projects and see how long they took
Use both views: inside view and outside view to get a better estimate
Use data: if you have it, use it to your advantage
Ask for help: if you are not sure, ask someone who has done it before, outside perspective can add a layer of objectivity