Quick answer
We impersonate Chrome's TLS fingerprint to avoid being blocked. On CourtListener's API that impersonation caused the failure: curl-cffi sends a browser-style Accept header, Django REST Framework content-negotiates on it, and the endpoint cheerfully returned its HTML browsable API instead of JSON. Every request in production would have died in json.loads.
Twenty-nine unit tests passed. They mocked the fetch layer, so not one of them could see it.
The bug in one line 🔍
# curl-cffi, impersonate="chrome"
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
DRF reads that, decides you are a browser, and serves the human-readable API explorer. Status 200. Content-Type text/html. A perfectly successful HTTP request containing zero of the data you asked for.
The fix is boring:
REQUEST_HEADERS = {"Accept": "application/json"}
The interesting part is why nothing caught it.
Anti-bot armour and API politeness pull in opposite directions ⚔️
Our house rule is that every Actor speaks through curl-cffi with a real browser TLS+HTTP2 fingerprint. That rule exists because it is the difference between reading a page and reading a block page, and it has earned its place many times over.
But a fingerprint is not free of consequences. Impersonating a browser means sending everything a browser sends, including preferences the server is entitled to act on. Against a scraping target that is exactly what you want. Against a content-negotiating REST API it is an instruction to hand you the wrong representation.
So the rule needs a second clause, and now has one: impersonate the browser's fingerprint, then override the headers that change what you get back. Transport-layer disguise, application-layer honesty.
Why the tests were useless here, specifically 🧪
This is the part worth stealing, because it generalises past this bug.
The suite mocked fetch_page and the AsyncSession. That is normal, sensible test design — it makes the parser tests fast and deterministic, and those tests were genuinely good. But look at what the mock encodes: an assumption about what the server returns. Every test downstream of a mock inherits that assumption and can never contradict it.
The defect lived exactly in the seam the mock replaced. No amount of additional unit tests would have found it. What found it was one real apify run against the production API.
We have hit this shape twice recently from opposite directions:
- A pagination parameter the server silently ignored — tests proved we sent it, never that it did anything.
- This one — tests proved we parsed JSON, never that the server sent JSON.
Same root: a fixture proves your code agrees with itself. It cannot prove the server agrees with you. Somewhere in the pipeline, something has to actually talk to the real thing.
What this Actor gets you ⚖️
CourtListener is Free Law Project's open database of American case law — 241,000+ matches on a single common term, and no API key required. The CourtListener Court Opinions Scraper turns its v4 search endpoint into flat rows:
- case name and full case name, citations, citation count
- court, court ID, jurisdiction, citation string
- dates argued and filed
- absolute opinion URL, resolved to a real link rather than the bare path the API returns
Filters: full-text query, court, filed-after / filed-before, and result ordering.
One thing we absorb that matters: v4 paginates by opaque cursor, not page numbers. There is no ?page=2. If you build the obvious offset loop you will re-read page one until your budget runs out — and under pay-per-event, that is not a hang, it is an invoice. We follow the next cursor and stop when it disappears.
Who it is for
- Litigation and IP research — every Federal Circuit opinion mentioning a term, as a table.
- Legal-tech products — a keyless case-law corpus without a PACER contract.
- Academics and journalists — reproducible citation-count and filing-date analysis.
Most competing listings on the Store scrape court dockets through PACER or resell paid data. This reads the free, open, full-text opinion corpus.
The honest limitations 🚧
- Search results, not full opinion text — the v4 search endpoint returns metadata and citations.
- No PACER, no dockets, no sealed material. Public case law only.
- CourtListener is a nonprofit running on donations; we rate-limit ourselves accordingly and you should too.
Pricing
$0.05 per run plus $0.005 per opinion — $5.05 per 1,000. Pay for rows that land, nothing for the ones you asked about.
→ CourtListener Court Opinions Scraper on Apify
Built by Devil Scrapes. We handle the cursors, the retries, the content negotiation and the headers nobody documents, so you get a table instead of a weekend.
Top comments (0)