GBIF — the Global Biodiversity Information Facility — indexes over three billion species occurrence records from museums, herbaria, national surveys and citizen-science platforms. It is keyless, well documented, and generous. It also has a hard wall at exactly 100,001 records that its docs mention in passing, and a name-resolution step that most first drafts skip entirely.
Quick answer
Two things decide whether a GBIF scraper works. First: resolve the species name to a usageKey via species/match before you search — free-text scientificName matching is fuzzy and silently drops synonyms. Second: offset + limit must stay under 100,001 or you get HTTP 400 mid-run. We verified the wall live: offset=200000 answers 400 "Max offset of 100001 exceeded: 200000 + 1".
Why resolve the name first?
You want Panthera leo. So does everyone else — but a species can be recorded under a synonym, a misspelling, a subspecies, or an outdated genus, and taxonomic backbones exist precisely to reconcile that.
GBIF exposes the reconciliation as its own endpoint:
GET https://api.gbif.org/v1/species/match?name=Panthera+leo
That hands back a usageKey — a stable integer for the accepted taxon — plus a matchType and a confidence score. Search by taxonKey=<usageKey> and you get the occurrences GBIF's backbone considers that species, synonyms included. Search by free-text name and you get whatever string-matched.
The response also carries a matchType and a confidence score, and they are worth reading. EXACT is what you want; FUZZY means GBIF guessed, and a fuzzy match on a typo can hand you a completely different organism with a perfectly successful HTTP 200 attached.
Our Actor logs the resolution it actually used — the key plus the canonical scientificName GBIF landed on — so the run log tells you Panthera leo resolved to Panthera leo and not to something adjacent:
resolved 'Panthera leo' -> taxonKey 5219404 (Panthera leo (Linnaeus, 1758))
If species/match finds nothing at all, it says so and falls back to an unfiltered-by-taxon search rather than silently returning zero rows.
The 100,001-record wall
occurrence/search pages with offset and limit. limit maxes at 300. offset is where it gets interesting:
offset=100000 -> 200 OK
offset=200000 -> 400 "Max offset of 100001 exceeded: 200000 + 1"
Verified live, not read off a doc page. So the real constraint is on the sum: offset + limit <= 100001. A loop that pages happily to record 99,900 and then requests offset=99900&limit=300 is asking for 100,200 and gets a 400 — a crash at the very end of a long, expensive run, which is the most infuriating place to put one.
We cap the walk at 100,000 so every request stays inside GBIF's boundary, and we say so in the Actor's own docs rather than letting a customer discover it at row 100,001. If you need more than that, the answer is not a cleverer offset — it is GBIF's asynchronous download API, which is a different tool with a different contract.
Enumerations: read them from the API, do not type them
basisOfRecord (is this a preserved specimen, a human observation, a machine observation, a fossil?) is a closed enumeration. It is tempting to hardcode the four or five values you have seen.
GBIF publishes the authoritative list:
GET https://api.gbif.org/v1/enumeration/basic/BasisOfRecord
We pulled it live and generated the input's allowed values from that, rather than from memory. It costs one request at build time and removes a whole class of "why does this filter return nothing?" support question — the same class of bug as an invented enum value in a platform manifest, which we have shipped before and would rather not again.
What about one malformed record in three billion?
It should cost you one record. GBIF aggregates from thousands of independent publishers with wildly varying data quality — a coordinate that is a string, a date that is a year, an occurrence with no taxonomy at all. All of that is normal.
So parsing is per-record and failures are skipped, logged, and counted:
try:
row = parse_occurrence(item)
except ValidationError:
log.warning(...) # skip this one, keep the run
continue
The single most common cause of a low-success-rate scraper in our fleet has never been a hard block. It is a recoverable per-item error taking down the entire run — one weird row costing the customer all the good ones.
Is scraping GBIF legal?
GBIF exists to publish this data for reuse, and most records carry an explicit open licence (CC0 or CC-BY) which the API returns per record. Honour the per-record licence when you redistribute, cite the datasets, and identify your client in the User-Agent. That is the whole etiquette.
FAQ
Do I need a key or a proxy?
Neither. It is a public keyless API with no anti-bot surface.
How many records can one run return?
Up to 100,000 via the search API, which is the platform's own ceiling. Larger extractions belong to GBIF's asynchronous download endpoint.
Why did my species name return nothing?
Check the match type. A NONE or low-confidence FUZZY match means the backbone did not recognise the name — usually a spelling or an authorship string that needs trimming.
Can I filter by country and year together?
Yes, and you should — it is the cheapest way to keep a query under the offset ceiling while still getting the slice you actually want.
Ready to run: GBIF Species Occurrence Scraper — search by species name, country, year range, or dataset; get full taxonomy, event date, coordinates, basis of record and dataset provenance as JSON, CSV, or Excel.
We do the dirty work so your dataset stays clean. 😈
Top comments (0)