DEV Community: Taranpreet Kaur

Scraping PACER Is Not the Hard Part

Taranpreet Kaur — Wed, 09 Sep 2026 20:15:51 +0000

Court records look like a scraping problem for about a week.

The federal docket system is reachable, the pages are simple, the structure is consistent enough to parse, and a competent engineer can pull dockets on day two. Every legal data project we have seen starts with that early win and the confidence it produces.

The confidence is misplaced, and not because access gets harder. Retrieval is the cheapest part of legal data work. What follows retrieval is a chain of problems that are genuinely difficult, largely undiscussed, and where every one of these projects actually spends its time.

The document is not the data

A docket is a list of events pointing at documents, and the useful information is almost entirely inside the documents rather than in the list.

That distinction creates four separate difficulties.

Docket text is written by humans under no schema. Entries are free text, conventions vary by district and by clerk, and the same event is described several ways across courts. Extracting event type, filing party and relief sought from that text is an information extraction task, not a parse.

Documents are scanned, often badly. A meaningful share of what you retrieve is an image of a page that was printed, signed, scanned and filed. Recognition handles most of it. What it does not handle is structure: exhibit boundaries, signature blocks, stamped modifications, handwritten annotations on a scanned order.

Structure spans documents. A motion, its exhibits, the opposition, the reply and the order are five documents that constitute one event, related by references in text rather than by any field. Reconstructing that chain is the difference between a document archive and a litigation dataset.

Sealed and redacted content is invisible but consequential. Portions of the record are not available, and a dataset that does not model absence will present an incomplete docket as a complete one. Modelling absence explicitly is one of the first things that separates a working legal document processing pipeline from a document dump.

None of these are retrieval problems. They are legal document processing problems, and the tooling for them is entirely different from the tooling for fetching pages.

Expert Insight: Sample fifty dockets and hand-label the event type on every entry before you build anything. Two numbers come out of it and both change the plan: how many entries are unambiguous from the text alone, and how many require opening the document. The second number is usually far higher than teams expect, and it is the number that determines whether your project is an extraction project or a document understanding project.

Entity resolution in a domain built to defeat it

The second wall is identity, and legal data is close to a worst case for it.

Party names arrive as free text entered by filers, which produces the full range: abbreviations, punctuation variants, trailing corporate suffixes, misspellings, DBA names, and the fact that the same corporate group litigates under a dozen subsidiary names that share no tokens with the parent.

Three specific traps sit on top of that.

Corporate structure is the point. Parties are frequently subsidiaries chosen deliberately, and rolling them up to a parent requires an external corporate hierarchy that the court record does not contain and will not help you build.
Law firms and attorneys are their own resolution problem. Attorneys move firms mid-case, appear under several bar registrations, and are listed inconsistently. Firm-level analytics depend on getting this right and most datasets quietly do not.
Same-name individuals are common and unmarked. There is no identifier for a natural person in the record, and matching on name alone across jurisdictions produces confident nonsense.

The resolution layer, not the scraper, is what separates a usable legal dataset from a pile of dockets. It also has to be temporal: a subsidiary acquired mid-case belongs to one parent before the acquisition and another after, and a dataset that resolves to today's structure silently rewrites history. This is the layer that most differentiates legal data providers from each other, and it is invisible in any feature comparison.

Expert Insight: Resolve parties with validity dates from the first day rather than retrofitting them. Retrofitting temporal validity into a resolution layer that assumed the present tense is one of the more painful migrations we have watched teams undertake, because every derived analytic silently changes when it lands and nobody can say which version was right.

Coverage, currency and the honest scope conversation

The third wall is the one that reframes the project, and it is worth confronting during scoping rather than in month four.

The federal system is the easy tier. State courts are the majority of litigation and they are a different world: dozens of systems, each with its own interface, its own access rules, its own coverage gaps, some of them not electronic at all. A dataset that covers federal matters and describes itself as covering litigation is describing a slice.

Currency is a per-court property. Filing-to-availability lag varies by court and by document type. A pipeline that treats all courts as equally current produces analytics with a systematic bias toward the fast ones, which is invisible unless you measure it per court.

Access terms vary and change. Every court system has its own rules about automated access, and they are not uniform, not permanent, and not always where you would expect to find them. This is a scoping question with a legal answer, not an engineering one, and it should be settled before the crawler is written rather than after.

The practical consequence is that legal data is an operations commitment rather than an acquisition project. Coverage expands court by court, each with its own access pattern and its own document quirks, and currency has to be monitored per court forever.

At Forage AI the legal work we run is mostly the three layers above rather than the retrieval one: document understanding across scanned filings, temporal party resolution, and per-court coverage and currency monitoring. The retrieval is the part that gets demoed and the part nobody remembers building.

Before scoping one of these, hand-label fifty dockets and count how many courts you actually need. Those two numbers will tell you more about the timeline than any technical spike will.

About the author: This piece was written by the data team at Forage AI, which extracts and structures legal and regulatory documents at scale across 10M+ documents. Learn more about Forage AI's work in intelligent document processing at forage.ai.

Why an LLM Alone Cannot Do Invoice Extraction Yet

Taranpreet Kaur — Wed, 09 Sep 2026 19:50:52 +0000

Feed an invoice to a modern multimodal model and ask for structured JSON. It works. Usually on the first attempt, with no configuration, on a document type that used to require a template.

That result is genuinely new, and it explains why so many teams now believe invoice extraction is finished. It is also the reason a lot of them are about to rediscover the parts of the job that were never about reading.

The distinction worth holding onto is this: **a model that reads an invoice correctly 95% of the time has not automated accounts payable, it has automated the first of five steps. **The other four are where the operational cost lives, and none of them is a reading task.

What the model actually solved

Give the model its due first, because the change is real and worth being precise about.

Template-free extraction is the thing that broke. Rule-based and template-based systems required per-vendor configuration, which meant that a stream of two hundred senders needed two hundred configurations, and every layout change required a person. A capable multimodal model reads a layout it has never seen and returns something sensible, which removes an entire category of work.

It also handles a lot of the messiness that used to defeat OCR pipelines: rotated pages, moderate scan quality, mixed fonts, tables with irregular borders. Reading characters was solved a while ago. Reading a document that was never designed for machines is what changed recently, and it changed substantially.

So the reading step is largely handled. What follows is not.

Step two is validation. Do the line items sum to the subtotal, does the subtotal plus tax equal the total, is the currency consistent, does the date fall within a plausible window? An LLM will happily return a document where the arithmetic does not hold, because it is reporting what it saw rather than checking it.

Step three is matching. The vendor on the invoice has to resolve to a vendor in your system, and the purchase order reference has to resolve to a real purchase order. This is entity resolution, and it fails on abbreviations, legal entity variants, acquired subsidiaries and the fact that your own vendor master was entered by hand over ten years.

Step four is policy. Approval thresholds, tax treatment, cost centre allocation, duplicate detection across the whole history rather than within one document.

Step five is the exception path. What happens to the 5%, who touches it, and whether their correction improves anything.

Expert Insight: The most reliable way to see this distinction in practice is to score a model on documents rather than on fields. A model at 97% field accuracy across roughly thirty fields per invoice will produce a fully clean document a minority of the time, and every document with one bad field still requires a human. Straight-through processing rate is the number that determines staffing, and it is consistently far below the accuracy figure that gets quoted.

Three failure modes that are structural

Beyond the missing steps, three properties of current models make unassisted extraction difficult in a way that is not obviously improving with scale.

Silent confidence. When a rule-based extractor cannot find a field, it returns nothing, which is a signal. When a model cannot find a field, it frequently returns a plausible value, which is not a signal. Self-reported confidence scores correlate weakly with correctness in our experience, and a wrong total delivered with the same fluency as a right one is the specific failure that makes finance teams distrust automation. The same dynamic applies across document types, and it is the property worth testing first when you are comparing invoice extraction tools, because none of them advertises it.

Table and line-item drift. Single-value fields are the easy case. Line items are where invoices actually get complicated: rows that wrap, quantities in one unit and prices in another, discounts applied per line or per invoice, continuation across a page break where the header does not repeat. Models lose rows, merge them, and occasionally invent a plausible one. The failure is quiet, it is per-row rather than per-document, and aggregate accuracy metrics hide it entirely.

Non-determinism. The same invoice submitted twice can produce two different outputs. For a system with a financial audit obligation this is a governance question, not a quality one, and it is answered with a deterministic post-processing and validation layer rather than with a better prompt.

Expert Insight: Before adopting any of this, run a stability test. Submit the same fifty invoices three times each and count how many produce identical output across all three runs. It costs an hour, and the result changes the design conversation, because whatever that number is, it sets an upper bound on how much of the process can run without review.

The shape that does work

None of this argues for going back to templates. The architecture that holds up uses the model for what it is good at and puts deterministic machinery around it.

Model for reading, code for checking. Extraction from the model, then arithmetic validation, format normalisation, currency and date parsing, and duplicate detection in ordinary code. Deterministic checks are cheap, auditable, and they catch the confident-wrong case that the model cannot catch about itself.

Resolution against your own masters, not against the document. Vendor and purchase order matching runs against your systems with explicit fuzzy-match thresholds and a review queue for anything below them. Never let the model assert an identity it cannot verify.

Confidence at the field level, thresholds per field. A misread invoice number and a misread line-item description have different costs. Route on field-specific thresholds rather than on a document-level score.

A review path that produces training data. Every correction a reviewer makes is a labelled example. Treating human review as part of the architecture rather than as a fallback is what separates a system that improves from one that plateaus at whatever the model gave you on day one.

This is roughly the architecture we run at Forage AI across a large document volume, with a 3x QA team sitting on the exception path, because the last few points of document-level accuracy have never been a model property. They are an operating property, and that has not changed with any model release so far.

The next generation will read better. It will still not know whether the vendor on this invoice is the same legal entity as the one in your master file, and that has always been the harder half of invoice automation.

About the author: This piece was written by the data team at Forage AI, which runs intelligent document processing across 10M+ documents with a 3x QA team on every delivery. Learn more about Forage AI's work in intelligent document processing at forage.ai.

Why an LLM Alone Cannot Do Invoice Extraction Yet

Taranpreet Kaur — Thu, 27 Aug 2026 09:29:10 +0000

Feed an invoice to a modern multimodal model and ask for structured JSON. It works. Usually on the first attempt, with no configuration, on a document type that used to require a template.

The distinction worth holding onto is this: a model that reads an invoice correctly 95% of the time has not automated accounts payable, it has automated the first of five steps. The other four are where the operational cost lives, and none of them is a reading task.

What the model actually solved

Give the model its due first, because the change is real and worth being precise about.

So the reading step is largely handled. What follows is not.

Step four is policy. Approval thresholds, tax treatment, cost centre allocation, duplicate detection across the whole history rather than within one document.

Step five is the exception path. What happens to the 5%, who touches it, and whether their correction improves anything.

Three failure modes that are structural

Beyond the missing steps, three properties of current models make unassisted extraction difficult in a way that is not obviously improving with scale.

Expert Insight: Before adopting any of this, run a stability test. Submit the same fifty invoices three times each and count how many produce identical output across all three runs. It costs an hour, and the result changes the design conversation, because whatever that number is, it sets an upper bound on how much of the process can run without review.

The shape that does work

None of this argues for going back to templates. The architecture that holds up uses the model for what it is good at and puts deterministic machinery around it.

About the author: This piece was written by the data team at Forage AI, which runs intelligent document processing across 10M+ documents with a 3x QA team on every delivery. Learn more about Forage AI's work in intelligent document processing at forage.ai.

Why Your AI-Generated Scraper Works in the Demo and Dies by Friday

Taranpreet Kaur — Sat, 08 Aug 2026 12:00:13 +0000

Something changed in 2026, and it is worth naming precisely rather than dismissing.

You can now describe a website to a model and get back a working scraper in under a minute. It handles the pagination. It gets the selectors right. It produces clean JSON on the first run. Ten years ago that was a day of work for a competent engineer, and the fact that it is now a minute is a real advance, not a party trick.

Then it runs for four days and quietly stops being right.

We have watched this cycle enough times in the last year to think the failure is structural rather than a prompt-quality issue. The gap is not between a good scraper and a bad one. It is between the problem that generation solves and the problem that production actually is. Here is where the seam sits, and what the generated code is genuinely good for once you see it.

Generation solves the part that was never expensive

Write down where the hours actually went on the last extraction project you staffed. Not the estimate. The hours.

For every team we have worked with, the distribution looks roughly like this. Writing the initial selectors and parse logic is a small share of total effort, usually well under a fifth. The rest goes to four categories that a code generator does not touch:

Source discovery and scoping. Working out which of the several hundred candidate sources actually carry the fields you need, in a usable form, at a usable refresh rate. This is judgement work against messy reality, and it happens before any code exists.
Anti-bot and access strategy. Not defeating defences, but understanding which sources will tolerate what access pattern, at what rate, from what path, and designing around it so you are a well-behaved consumer rather than an escalating one.
Normalisation and entity resolution. Getting the same company, provider or product to reconcile across sources that disagree about its name, address and identity. This is where most of the real difficulty lives and it is almost entirely invisible in a demo, because a demo has one source.
Continuous monitoring and repair. The permanent tax. Sources redesign, defences escalate, structures change.

Those four are the categories that actually decide the build versus buy question for web data, and a model generates category zero brilliantly. It has no view on the other four, because none of them are expressible as "write me code that reads this page".

Expert Insight: The most reliable predictor of whether a team is about to be surprised by extraction cost is whether their estimate included a line for maintenance. Teams that budget for build-only routinely find the work becoming a permanent one to two engineer commitment inside a year. It rarely arrives as a decision. It arrives as a side project that becomes a quarter of someone's week, then half, then a named owner, then a rotation with a pager.

Why Friday specifically

The four-day figure is a joke with a mechanism behind it.

Generated scrapers are fitted to the page as it existed at generation time, and they are fitted tightly. A model given one rendering of a page will produce selectors that work perfectly on that rendering, which sounds like the goal and is in fact the failure mode. Hand-written extraction code from an experienced engineer tends to be looser on purpose. It anchors on things that change slowly, tolerates missing optional fields, and is built by someone who has been burned by a promotional banner shifting the DOM.

Three specific things go wrong first, in this order.

The A/B variant. The site was already serving two layouts. The model saw one. Your extraction silently covers half the traffic and you have no way to know, because half of something is a plausible number.

The optional-field assumption. The example record had a value in every field. Roughly a fifth of real records do not. The generated parser treats absence as breakage, or worse, grabs the nearest structurally similar element and returns a well-formed wrong answer.

The first redesign. Whenever it lands, the generated code has no mechanism to notice. It returns zero rows, or partial rows, and zero rows is a valid output that fires no alert.

None of these are model failures. A page is a sample of one. Fitting tightly to a sample of one is what you asked for.

Expert Insight: There is a version of this that works, and we have seen teams get real leverage from it: use generation for the first draft, then have someone with production scars deliberately loosen it. Widen the selectors, make every field optional, add the null-rate check, add a golden set. That review takes twenty minutes and it is the difference between a four-day scraper and a four-month one. The teams that skip it are not being careless. They are responding rationally to code that looks finished.

What this actually changes

The honest read is that generation moved the bottleneck rather than removing it, and that is still worth something.

Before, standing up coverage of five sources took a competent engineer a week or two, which was enough friction that most teams scoped down and stayed there. Now it takes an afternoon. The constraint has shifted to the long tail, where it always belonged. We worked with a ticketing marketplace whose engineer had built five scrapers in six months and then discovered the market they were describing had more than five hundred sources in it. Generation makes those first five nearly free. It does not change that the remaining four hundred and ninety-five each carry their own access pattern, their own schema quirks and their own permanent maintenance line.

That is the reframe worth holding: AI collapsed the cost of the first scraper and left the cost of the five-hundredth almost untouched. If your data requirement is genuinely five sources, generation may have just solved your project, and you should take the win. If it is five hundred, generation has made the easy part free and left the expensive part exactly where it was. Most teams that reach for AI-assisted extraction are in the second situation and are pricing themselves as though they were in the first.

This is also, in fairness, the reason a managed extraction layer exists at all. At Forage AI the durable work is not writing extraction code, which we also generate where it makes sense. It is source discovery across the long tail, the access strategy per source, the normalisation, and the monitoring that runs forever. Those four have not become cheaper, for us or for anyone.

Use the generated scraper. Then ask what happens to it on Friday, and whether anyone would find out.

About the author: This piece was written by the data engineering team at Forage AI, which builds and maintains managed extraction pipelines across more than 500M+ websites. We generate a fair amount of extraction code ourselves, and then we go and loosen it. Learn more about Forage AI's work in web data extraction at forage.ai.

The 1-in-3 Rule: What Actually Breaks in a Scraping Pipeline After 90 Days

Taranpreet Kaur — Fri, 31 Jul 2026 09:10:07 +0000

A web scraper that works the day you ship it tells you almost nothing about whether it will work in October.

We have watched this pattern repeat across enough deployments to stop calling it bad luck. Roughly a third of the URLs in a first-generation extraction pipeline stop returning usable data inside the first quarter. Not all at once. Almost never loudly.

The number matters less than the shape of the decay. If you know which parts rot first, you can instrument for them before they cost you a customer demo. What follows is a retrospective on the three decay curves we see most often, why none of them page anyone, and what we would build into the first ninety days if we were starting over.

The three decay curves

Extraction pipelines do not fail on a single axis. They fail on three, and the three run on different clocks.

Selector drift is the fastest and the most visible. A class name changes, a wrapper div appears, a product card gets a new layout for a promotion. Time to first failure is usually two to six weeks. This is the failure everyone budgets for, because it is the one that shows up in a tutorial.
Anti-bot escalation is slower and steeper. Nothing happens for a month, then a source moves behind a challenge, or starts fingerprinting, or quietly begins serving a degraded page to traffic it does not recognise. The curve is a cliff rather than a slope, and it tends to hit several sources at once because vendors sell the same defences to whole industries.
Source-side structural change is the slowest and does the most damage. A site consolidates two sections. A directory adds pagination it did not have. An organisation merges with another and its whole URL scheme is retired. Your scraper still runs, still returns 200, still parses. It is just describing a world that no longer exists.

A healthcare data company we work with was tracking provider directories across several thousand hospital sites. Their scrapers were healthy by every metric they had. What they could not see was that hospital acquisition activity had reshuffled the affiliation graph underneath them, and their affiliation data was lagging reality by weeks. The extraction was fine. The dataset was wrong.

That gap between "the job succeeded" and "the data is true" is where most of the ninety-day loss lives, and it is the reason enterprise extraction pipelines break in ways their own dashboards cannot see.

Expert Insight: Teams consistently over-invest in the first curve and under-invest in the third. Selector drift is annoying but self-announcing, and a junior engineer can fix it in an afternoon. Structural change is silent, compounds daily, and is usually discovered by a customer rather than a monitor. If you only have budget to instrument one curve properly, instrument the slow one.

Why none of it pages you

The uncomfortable part is not that scrapers break. It is that a broken scraper looks identical to a working one from the outside.

Three specific silences account for most of what we see.

The empty 200. The request succeeds, the response is well formed, the body contains a challenge page or a soft error rendered as normal HTML. Your parser finds no matches and writes zero rows. Zero rows is a valid outcome, so nothing fires.

Schema-valid, value-wrong. Extraction returns the right shape with the wrong contents. A price field picks up a strikethrough comparison price. A location field picks up the "nearby" module instead of the record. Every type check passes. Every value is garbage.

Coverage collapse hidden by averages. You scrape four hundred sources. Eleven of them die. Your aggregate row count moves two percent, well inside normal variance, and your dashboard shows a flat green line. The eleven that died happen to be the eleven your largest customer cares about.

None of these throw. That is the entire point. Standard pipeline observability, the kind you get free with an orchestrator, watches whether the task completed and how long it took. It has no opinion about whether the numbers are real. Judging that is the job of an extraction QA workflow, which is a separate discipline with a separate set of checks.

Expert Insight: The single highest-return check we deploy is also the least sophisticated: a per-source row-count floor with a comparison against the same source seven days earlier. It catches the empty 200 and coverage collapse on the same day they happen, and it takes an afternoon to build. Sophisticated anomaly detection can come later. Most teams skip the afternoon version and end up with neither.

What we would instrument in the first ninety days

If we were standing up a new pipeline tomorrow, five checks would go in before any additional source did. They are ordered by return on effort, not by elegance.

Check 1: per-source row-count floors, compared week over week. Absolute thresholds go stale. Relative comparison against the same source's recent history does not. It is the cheapest form of observability for external data pipelines and the one with the highest return per hour spent.

Check 2: field-level null-rate tracking. Not "did we get rows" but "did we get values". A field that was 4% null last week and is 60% null today is the earliest honest signal that a layout changed.

Check 3: a golden set per source. Twenty to fifty records you have verified by hand, re-extracted on every run and diffed. This is the only check that catches schema-valid, value-wrong extraction, and there is no clever substitute for it.

Check 4: response-shape fingerprinting. Hash the structural skeleton of the page, not the content. When the hash moves for a whole source at once, a redesign has shipped and you have days rather than weeks of warning.

Check 5: a freshness clock per record, not per job. Job-level timestamps tell you when you ran. Record-level timestamps tell you how old the truth is. Those diverge the moment a source starts serving cached or partial responses, and only the second one is a number you can put in front of a customer.

None of this is difficult engineering. It is unglamorous, and it competes for time with building the next twenty scrapers, which is why it usually loses. At Forage AI we run these checks as a standing part of delivery rather than an add-on, because the alternative is finding out from the person who bought the data. That is a decision about where the burden of proof sits, and it is worth making deliberately rather than by default.

Expert Insight: Golden sets are the check teams argue about most and regret skipping most. The objection is always maintenance cost. In practice a fifty-record golden set per source costs a few hours to build and about twenty minutes a quarter to keep honest, and it is the only mechanism that distinguishes "we got data" from "we got the right data". Every team we have seen build one has kept it.

What the ninety days actually cost

The cost of skipping this is rarely booked as a data cost. It shows up as headcount.

A scraper starts as a side project. It becomes a quarter of an engineer's week, then half, then a named owner, then a rotation with a pager. We have seen teams cross from "we built a scraper" to "we staff a scraping function" without a single conversation about whether they meant to. One to two full-time engineers, permanently, on maintenance that produces no new capability.

The three curves are going to bend whatever you do. Sources will redesign, defences will escalate, organisations will merge. What you control is whether you find out on day two or day ninety, and the difference between those two numbers is almost entirely a function of five checks you either built early or did not.

Worth asking on your own pipeline: if a third of your sources went quiet this morning, which dashboard would tell you before your customer did?

About the author: This piece was written by the data engineering team at Forage AI, which builds and maintains managed web data extraction pipelines across more than 500M+ websites. We have spent 12+ years watching extraction pipelines decay in production and building the checks that catch it early. Learn more about Forage AI's work in web data extraction at forage.ai.