How an undocumented sort order in a public biodiversity API put 97% of my background points in January or February—and what I changed so that kind of bias gets caught before a model ever sees it.
Exploratory analysis is supposed to be the calm part of a project.
For QueleaGuard, it was where I discovered that 122 of my 126 background records—96.8%—fell in January or February. Among the real red-billed quelea observations, only 20.3% did.
Nothing had crashed. Every validation check up to that point had passed.
If I had gone straight to training, the model might have produced impressive-looking results for entirely the wrong reason. That is exactly what made the problem dangerous.
Why the Background Points Exist at All
QueleaGuard investigates where red-billed quelea occur around Ahero Rice Irrigation Scheme in Kenya.
The occurrence data comes from GBIF, which aggregates biodiversity observations, including citizen-science sightings from platforms such as eBird. This is presence-only data: it records where birds were observed, but not necessarily where someone looked and found none.
To model occurrence, I needed something to contrast the confirmed presences against.
I used an approximate target-group background: observations of other bird species recorded in the same area serve as a proxy for places where birders were active but did not record quelea.
The idea is to account, imperfectly, for observation effort. Bird sightings cluster around places people actually visit, so treating the landscape as though observation effort were evenly distributed would be misleading.
This approach has limitations. Individual bird records are a noisier proxy for observation effort than complete checklists, and I documented that limitation when adopting the method.
But the approach only helps if the background sample is drawn fairly across space and time.
Mine was not.
Why a Calendar Bias Is So Harmful
Almost every environmental feature in the dataset changes with the seasons:
Rainfall
Temperature
Dewpoint
Vegetation greenness
If nearly all background points fall in January and February while confirmed presences are distributed throughout the year, a model can distinguish the classes by learning the calendar rather than the ecology.
It might appear to predict quelea occurrence accurately, when it is actually exploiting a sampling artefact.
A confound like this does not produce an error. It produces a convincing result.
This is one of the more dangerous forms of data-quality failure: the pipeline can run successfully, the model can train without complaint, and the evaluation can look respectable.
The underlying research question, however, has been compromised before training even begins.
Finding the Cause
The effort-proxy pool came from a script that queried GBIF's occurrence/search endpoint one year at a time, with a cap of 150 records per year through pagination.
When I inspected the raw pool, several years—including 2000, 2011, and 2012—contained 150 January records out of 150.
GBIF does not guarantee a default sort order that would make this sampling strategy safe. Empirically, results from a year-only query were heavily front-loaded with January observations. Whenever a year contained more records than the cap, pagination stopped before the sample reached the rest of the year.
In effect, the API's response order was influencing my research sampling design.
The uncomfortable part was that I had already fixed a year-level version of the same problem when I first built the pool.
This was the same class of failure, one level deeper. It survived because I had not checked the monthly distribution of the background records.
The API had returned data. My code had accepted it. My validation had passed.
But I had not asked the question that mattered: What does the distribution of the returned records actually look like?
The Fix: Stop Depending on the Order
The endpoint accepts a month parameter alongside year.
Rather than trying to work around an undocumented ordering, I removed the dependency on it.
I changed the collection strategy to query individual year–month combinations:
16 years × 12 months = 192 queries, each targeting approximately 13 records.
The resulting pool looked substantially different.
Measure Before After
Effort-proxy records Year-capped, January-heavy 2,371 records, stratified by month
Largest single-month share 90.7% 8.8%
Records per month Extremely uneven 182–208
The key improvement was not simply collecting more records. It was making the sampling procedure explicit and reproducible rather than allowing the server's response order to determine which observations entered the dataset.
Another problem surfaced during the rebuild
While rebuilding the pool, I also closed a gap found during an earlier validation pass.
Three background records appeared to come from the same eBird checklist as a confirmed quelea sighting. That directly contradicted the purpose of using those records as background.
The rebuilt pipeline now excludes any effort-proxy record that shares both a grid cell and a date with a confirmed presence before sampling, rather than relying on later validation to catch the contradiction.
That rule removed 94 records that could otherwise have reintroduced the same problem.
The lesson here is important: fixing the monthly distribution alone would not have been enough. The sampling procedure also needed to enforce the assumptions behind the background data.
Checking Before Moving On
I deliberately held back the expensive part of the workflow—re-extracting rainfall, meteorological, and vegetation features for the new records—until the corrected sample passed its own validation checks.
The results were encouraging:
Monthly distribution: The largest single-month share was 18.8% of presences in November and 16.8% of background records in May. Neither class was concentrated in a single season.
Cross-class contradictions: There were zero cell-and-date pairs shared between presences and background.
Year coverage: Both classes spanned 2000–2026, with greater representation from 2021–2024 in both. This was consistent with the growth of eBird observations rather than an obvious artefact of the query strategy.
Spatial coverage: Background points now occupied 81 grid cells, up from 49.
I also kept the month-biased files, renamed and documented as superseded rather than deleting them.
That matters for reproducibility. If someone later wants to understand the original problem or investigate how much the sampling bias affected the analysis, the evidence is still available.
A corrected dataset is useful. A traceable record of how it became corrected is even better.
What I Took From It
Four lessons came out of this failure.
- Whenever a data request has a cap, find out what determines which records are returned.
If the answer is effectively "whatever order the server uses," the sampling process is not fully under your control.
- Stratify on the dimensions the model will be sensitive to.
Season influences almost every environmental feature in this project. Month therefore had to be part of the sampling design, not something I hoped would average out.
- Check every dimension the model can see, separately for each class.
A pooled distribution can hide differences between presences and background records. Looking at the two classes separately exposed the problem.
- Document the failure instead of quietly patching it.
The decision log now explains why the dataset looks the way it does. That record is part of the research, not administrative overhead.
These practices are useful beyond biodiversity modelling. Any pipeline that retrieves, filters, samples, or combines data can introduce a bias that remains invisible until someone examines the distribution of what actually came through.
The Research Models Have Not Been Trained Yet
QueleaGuard has not trained its research models yet, so I cannot tell you how much correcting this bias will change the eventual results.
I do not want to claim an improvement in predictive performance before the experiments have been run.
What I can say is that the dataset now has a more defensible sampling procedure, clearer validation rules, and a documented record of the problem that motivated the changes.
That is meaningful progress even before the first model is trained.
Whatever the results turn out to be, they should not simply be a measurement of what month it is.
Project: QueleaGuard — a research project investigating environmental, spatial, temporal, and agricultural drivers of red-billed quelea occurrence around Ahero Rice Irrigation Scheme in Kenya.
Top comments (0)