AI's Blind Spot for African Biodiversity

This is a submission for the Kaggle Benchmarking Challenge

What I Benchmarked

While frontier AI models achieve high scores on standard academic benchmarks, localized ecological and biodiversity knowledge—especially concerning East African avian species—remains largely untested.

I built AI's Blind Spot for African Biodiversity, a Kaggle benchmark suite designed to evaluate how accurately state‑of‑the‑art LLMs process regional ornithological knowledge across three specific tasks:

Hadada Ibis Identification (identify_hadada_ibis) — Tests if the model can accurately identify and reason about the distinct vocalizations and physical traits of the Hadada Ibis (Bostrychia hagedash).

African Fish Eagle Nickname (african_fish_eagle_nickname) — Evaluates knowledge of the iconic call, regional monikers, and cultural identity of the African Fish Eagle (Haliaeetus vocifer).

Ethiopian Highland Birds (identify_native_bird_ethiopian_highlands) — Evaluates knowledge of endemic avian species native to the Ethiopian highlands near the Bale Mountains (such as the Wattled Ibis).

Models Tested

I ran the benchmark suite against Gemini 3.7 Flash on Kaggle. I chose this model because it is a current frontier lightweight model widely used for real‑world API applications and multi‑modal tasks, making it a great baseline to check for gaps in regional training data distribution.

Findings

Gemini 3.7 Flash scored an overall pass rate of 66.67% (passing 2 out of 3 tasks):

PASS — Task 2 (african_fish_eagle_nickname)

PASS — Task 3 (identify_native_bird_ethiopian_highlands)

FAIL — Task 1 (identify_hadada_ibis)

Key Takeaways:

What surprised me: Gemini 3.7 Flash demonstrated accurate knowledge regarding general endemic high‑altitude species and iconic raptors, but failed when evaluated on specific regional vocalizations and constraint checks for the Hadada Ibis.

What I would measure next: Expand the suite to include audio‑based identification tasks (such as recognizing raw field recordings of bird calls) and test additional open‑weights models (like Llama 3 and Gemma 2) to evaluate whether open models show similar regional blind spots.

My Benchmark

Kaggle Benchmark Suite: AI's Blind Spot for African Biodiversity

Comments

[Sameer Qaiser]

The line that hit me: "Frontier AI models achieve high scores on standard academic benchmarks, but localized ecological knowledge remains largely untested."

I'm a beginner — three weeks into Python, writing tutorials about it. I don't have a Kaggle benchmark or a model to test. So most of the technical detail here is beyond where I am.

But this finding made me think about something I've been noticing in my own work.

I use AI constantly to learn Python. And I've started to realize that it's very confident about general knowledge and quietly unreliable about specifics. Ask it what a loop does — perfect answer. Ask it to explain a very specific error from a library version from 2023 — sometimes correct, sometimes confidently wrong.

Your benchmark is measuring exactly that gap, just in a domain where the stakes are higher than whether my Python script runs. Local biodiversity knowledge matters. Regional bird calls matter. The fact that a frontier model fails on those cases isn't a small thing — it's a sign that "smarter model" doesn't mean "knows everything equally."

The 66.67% pass rate is interesting too. Not a failure. But two-thirds means one-third of the test was a blind spot, and that's the part worth investigating.

Great submission. Rooting for you in the challenge.