This one is from my masters coursework, a research internship I did on something most developers have probably never heard of: FAIR data validators. Stick with me, the actual finding is more interesting than the name suggests.
The problem I started with
Research data is supposed to follow something called the FAIR principles. Findable, Accessible, Interoperable, Reusable. The idea is simple: if a dataset follows these principles properly, other researchers can actually find it, use it, and trust it.
There are tools that automatically check how well a dataset follows FAIR, F-UJI, FAIR Evaluator, and FAIR Checker are the three big ones. You give them a dataset, they give you a compliance score.
Here is the question nobody had really answered before: if you run the same dataset through all three tools, do they agree?
Spoiler, they do not
I took datasets from Zenodo and DataCite and ran them through all three validators. Same dataset, same moment, three different tools claiming to measure the same thing.
On one climate science dataset, the overall scores came out like this:
- F-UJI: 65%
- FAIR Checker: 51%
- FAIR Evaluator: 57%
A 14 point gap on the exact same dataset. That is not a rounding error, that is three tools genuinely disagreeing about whether the same data is FAIR.
Why? It comes down to vague words
This was the part that actually surprised me. I expected the disagreement to be random, like the tools just had bugs or different implementations. It was not random at all.
The FAIR specification uses words like formal, clear, and accessible, but never defines them precisely. So every tool that implements a validator has to make its own engineering decision about what those words mean. All of them are reasonable. All of them are different.
Look at how differently the tools interpreted a rule that just says a dataset needs a formal knowledge language:
- F-UJI checks for JSON-LD or Schema.org in the metadata
- FAIR Checker requires a registered OWL or RDF ontology URI
- FAIR Evaluator just checks if any structured vocabulary is referenced at all
Three completely different technical checks, all built from the same vague sentence.
Compare that to a rule that is actually specific, like requiring a unique identifier. A DOI either resolves or it does not. All three tools agreed perfectly there, zero gap.
So the pattern became clear pretty fast: vague rules produce big disagreements, specific rules produce none.
Finding exactly where the gaps live
Once I saw the pattern, I wanted to pin down precisely which sub-principles were causing the disagreement, not just say "some things are inconsistent."
Three sub-principles kept showing up with the highest gaps, every single time:
I1, whether the dataset uses a formal knowledge language, gap of 0.3
R1.1, whether the dataset has a clear usage license, gap of 0.4
A2, whether metadata persists and stays accessible, gap of 0.3
Meanwhile things like unique identifiers and basic HTTP accessibility stayed at zero gap across all three tools, every time. The disagreement was not scattered randomly across all 15 sub-principles, it was concentrated in exactly the ones with the vaguest wording.
Can you actually fix this?
This was the part I found most satisfying. If the disagreement comes from vague metadata, then maybe better metadata closes the gap, regardless of which tool is checking it.
So I built what I called a Minimum Metadata Profile, a small, specific set of fields designed to satisfy the strictest interpretation of each rule:
- A DOI that actually resolves over HTTP
- Title and description in plain text
- A machine readable license URI, not just a license name
- An OWL or RDF ontology URI, using something standard like Dublin Core
- An ORCID for the author, plus a related work DOI
- Proper DataCite metadata registration
Then I ran the experiment. Took a dataset with no profile applied, ran all three validators, recorded the scores. Applied the profile, uploaded it to Zenodo again, ran all three validators on the improved version.
Every score went up. And more importantly, the gap between the tools shrank significantly. Writing metadata well enough to satisfy the strictest tool ended up satisfying the more lenient ones automatically.
What this actually means, practically
If you are a researcher or data manager wondering which FAIR score to trust, the honest answer is that the score itself is less important than understanding why it differs. The disagreement is not a bug in any one tool, it is a structural problem with how the FAIR principles are written.
The practical takeaway I landed on: do not just chase a score from one validator. Build metadata that satisfies the strictest possible interpretation of each rule, and the scores across every tool tend to converge.
What is next
The bigger fix has to happen at the specification level, not the tool level. My recommendation, and something worth pushing for through groups like the Research Data Alliance, is a unified technical definition for the vague sub-principles, so validators stop having to guess what "formal" or "clear" actually means. Until that happens, every validator will keep making its own reasonable, and different, engineering decision.
Why I am sharing this here
This was research work, but the core lesson felt very familiar to anything in software: specifications that use natural language instead of precise technical definitions will always get implemented inconsistently. I have seen smaller versions of this exact problem in API docs, in style guides, in "should be self explanatory" requirements at work. Vague words always eventually produce disagreement, whether it is three FAIR validators or three developers on the same team.
Top comments (0)