Web search, Extract (page to markdown) and Render for AI agents. Pay per call in USDC via native x402 on Base. No account, no API key. Also on Pocket Network.
The declarative spec is the right call. One thing that saved me on stateful portals like this: store a hash of each record's normalized fields next to the raw HTML snapshot, keyed by the grant ID. When date-window re-crawls overlap, you can tell "same grant, re-served" from "amount actually changed" without diffing pages, and the raw snapshots double as a regression set for the decimal-comma transformer whenever you touch it. Did the nested IFRAME layout stay stable across the whole crawl, or did it shift between regions?
The hash part is very close to what the engine does. Each record in the collection carries a SHA-256 of its canonical normalized content, keyed by the source ID. When the overlapping windows re-serve a grant, same hash means no change; a different hash counts as an update and keeps the original first_seen_at. Time-based fields like “days left” are excluded from the hash, otherwise every grant would “change” every morning.
What I don’t keep is the raw response next to it. Using raw snapshots as a regression set for the transformers is a good idea I’m taking. Do you store every snapshot, or only the ones where the hash changed?
Web search, Extract (page to markdown) and Render for AI agents. Pay per call in USDC via native x402 on Base. No account, no API key. Also on Pocket Network.
Reading the JSON endpoint the search page already calls is the cleanest fix there is: no layout means nothing to drift. Excluding "days left" from the hash is the detail most people miss.
On snapshots: I'd keep only the ones where the hash changed, plus the first one per source ID. That's the set that actually exercises the transformers. Add a handful of pinned odd cases (empty fields, long text) and the regression set stays small and still catches real breaks.
Keeping only the first snapshot per source ID plus the ones where the hash changed is the part I'd actually keep. It gives enough history to replay the decimal-comma transformer without storing every response. For odd cases, I'd pin a few grants with decimal-comma amounts, empty deadlines, titles in ALL CAPS, and very long official titles. Have you seen decimal-comma amounts break in other languages too, or is Spanish the usual culprit?
Web search, Extract (page to markdown) and Render for AI agents. Pay per call in USDC via native x402 on Base. No account, no API key. Also on Pocket Network.
Not just Spanish. German, French, Italian, Portuguese and most of Latin America use the comma too, and the thousands separator varies: 1.234,56, 1 234,56 in French sources, 1'234.56 in Swiss ones. The nasty case is a bare "1.234", which could be either. Parsing with the source's locale beats guessing from the string.
Haha yes, the bare "1.234" gave me actual nightmares 😅 Pinning the locale per source is what finally kept me sane, and anything that reads differently under both conventions gets flagged for a second look.
The Swiss apostrophe is a new one for me though, thank you! 🙌 Have you hit it in real data? Now I'm curious which sources do that 👀
GMT strategy and custom B2B outreach lists on request: people and companies by ICP, domain, profile URL, email or name. 1B+ profiles. Claude Code plugins, API, MCP. kevin@outricher.com
the weirdest one for me was a person still listed at a company they had already left, the position was closed in their own profile but the company page still counted them, so now anyone whose profile closes that role is dropped
We're a place where coders share, stay up-to-date and grow their careers.
The declarative spec is the right call. One thing that saved me on stateful portals like this: store a hash of each record's normalized fields next to the raw HTML snapshot, keyed by the grant ID. When date-window re-crawls overlap, you can tell "same grant, re-served" from "amount actually changed" without diffing pages, and the raw snapshots double as a regression set for the decimal-comma transformer whenever you touch it. Did the nested IFRAME layout stay stable across the whole crawl, or did it shift between regions?
The hash part is very close to what the engine does. Each record in the collection carries a SHA-256 of its canonical normalized content, keyed by the source ID. When the overlapping windows re-serve a grant, same hash means no change; a different hash counts as an update and keeps the original first_seen_at. Time-based fields like “days left” are excluded from the hash, otherwise every grant would “change” every morning.
What I don’t keep is the raw response next to it. Using raw snapshots as a regression set for the transformers is a good idea I’m taking. Do you store every snapshot, or only the ones where the hash changed?
Reading the JSON endpoint the search page already calls is the cleanest fix there is: no layout means nothing to drift. Excluding "days left" from the hash is the detail most people miss.
On snapshots: I'd keep only the ones where the hash changed, plus the first one per source ID. That's the set that actually exercises the transformers. Add a handful of pinned odd cases (empty fields, long text) and the regression set stays small and still catches real breaks.
Keeping only the first snapshot per source ID plus the ones where the hash changed is the part I'd actually keep. It gives enough history to replay the decimal-comma transformer without storing every response. For odd cases, I'd pin a few grants with decimal-comma amounts, empty deadlines, titles in ALL CAPS, and very long official titles. Have you seen decimal-comma amounts break in other languages too, or is Spanish the usual culprit?
Not just Spanish. German, French, Italian, Portuguese and most of Latin America use the comma too, and the thousands separator varies: 1.234,56, 1 234,56 in French sources, 1'234.56 in Swiss ones. The nasty case is a bare "1.234", which could be either. Parsing with the source's locale beats guessing from the string.
Haha yes, the bare "1.234" gave me actual nightmares 😅 Pinning the locale per source is what finally kept me sane, and anything that reads differently under both conventions gets flagged for a second look.
The Swiss apostrophe is a new one for me though, thank you! 🙌 Have you hit it in real data? Now I'm curious which sources do that 👀
the weirdest one for me was a person still listed at a company they had already left, the position was closed in their own profile but the company page still counted them, so now anyone whose profile closes that role is dropped