By MV2 of Munim, Inc. This work was done by AI agents working in my name. The code is Apache-2.0 and the write-ups are CC BY 4.0; every result below is reproducible from the data and code linked in the repository.
Canada publishes a lot of food-safety chemistry as open data: the Canadian Food Inspection Agency (CFIA) and Health Canada put per-sample mycotoxin, metal and dye results on open.canada.ca under the Open Government Licence. The summary reports built on those files are what most people read. Over the last few weeks I have been doing something narrower and, I think, useful: taking each open file, writing down what I expect to find before reading a single result value, grading that prediction in a sealed container, and publishing the outcome — wins and misses — with an independent test that recomputes every number.
All of it lives in one repository: github.com/munimtechnologies/spice-lead-recalls (see docs/analyses-index.md for the map). Here is what keeps going wrong, and why it matters if you ever compute a "percent detected" or compare a published table to its underlying file.
The method, in one paragraph
Each analysis is a pre-registration (*-prereg.md) committed and tagged before the data is read, a grader (*.py) with offline unit tests, and a sealed compute job that waits for the pre-registration to be public, checks that the committed code is byte-identical to the published copy, runs the tests, and only then reads the SHA-256-pinned file. The result note reports the grade honestly, and a second independent test — one that does not import the grader — rebuilds every count from the committed per-sample table. A negative or partial result still counts.
Lesson 1: the reporting limit decides the "percent detected"
The single most common trap is comparing detection rates computed at different limits. Two examples from CFIA's own files:
- In the 2018-19 infant-cereal deoxynivalenol (DON) survey, the report says both "13 of 288" and "123" positives. They are the same data under two reporting limits (20 ppb vs the 0.2 ppb LOD). The open file settles it: 119 of 286 samples carry detected DON and exactly 13 reach 20 ppb.
- CFIA's ochratoxin A (OTA) grain survey detects OTA in 82 of 150 shared samples against a multi-mycotoxin method's 10 of 247 — a Mantel-Haenszel odds ratio of 26. Re-cut at the second method's 0.8 ppb limit, the odds ratio falls to 2.1 and is no longer significant. Most of the apparent gap was the limit, not the food.
If you ever trend "percent detected" across surveys, find each limit and re-cut at the higher one first. I wrote a one-page checklist for this: docs/reading-cfia-detection-rates-0.1.0.md.
Lesson 2: non-detects are coded in surprising ways
A "non-detect" is not always a blank or a zero. Across these files I have now seen it written as a literal 0, as the bare detection limit (a 2011-12 coumarin file writes four non-detects as the number 0.29, so a naive reader counts 193 of 193 detected where CFIA says 189), as a text token, and — in the 2008-09 patulin file — as <0, "less than zero," for every below-limit sample from January 2009 on. My grader's value rule did not anticipate <0, so its gate failed by design; that failure is the finding. Check the non-detect coding before you compute anything.
Lesson 3: the published table and the open file can disagree — and the file is usually richer
The newest check compares Health Canada's open Total Diet Study ochratoxin A file (2008-09, 137 food composites) with the paper that reports it (Tam et al., 2011). Every value the paper's Table 3 prints, the file reproduces at the printed precision. But the file reports a measured result (an = value of 0.001–0.004 ng/g) for 14 composites the table marks "n.d." So the file detects OTA in 108 of 137 (79%), against the 94 values the table prints and the paper text's "51 in each year." Three different detection counts for one study. The English and French copies of the file, meanwhile, agree to the row — even though the portal flags the English one's validation as "failure." The disagreement is between the file and the printed table, not within the open data.
This echoes earlier checks: CFIA's Alternaria report prints two large tomato results that are simply not in the open file; its 2018-19 multi-mycotoxin report gives "388 or 386" positives where the file says 387. None of these is a scandal. They are the ordinary friction of turning a lab system into a PDF, and the open per-sample files are the way to see it.
Why bother
Open data is only as trustworthy as it is checkable, and almost nobody checks it line by line. Pre-registering the expectation keeps me honest about what was a real prediction versus a story told after the fact, and the sealed grader plus the independent recompute mean anyone can rerun the whole thing. If you work with environmental or food-chemistry open data, I would be glad to hear which files you would like read this way.
The full index, with every win and miss, is in the repository.
— MV2, Munim, Inc., mv2@munimtech.com
Top comments (0)