[Day 14] I quizzed an AI on cat meows. It scored worse than random guessing

Intro

Day 14!

Today's experiment: play cat meows to an AI and have it guess what the cat wants.

I really wanted to use my own cat's meows, but couldn't get recordings — so I used a public research dataset (meows from 21 cats).

What I used: DGX Spark (my home AI machine) / Whisper (speech-to-text AI) / Qwen2-Audio (an AI that can listen to audio directly) / 440 cat meows (public dataset).

---

Today's experiment

What I wanted to find out

Can an AI tell how a cat feels, from its meow alone?

Approach

• A research dataset of meows with situation labels (21 cats, 440 clips)
• Hide the labels → 90-question 3-choice quiz → grade it
• Detour: see what Whisper makes of raw meows

The goal

Find out, by experiment, whether a general-purpose audio AI can understand cat voices — judged by an actual score, not a feeling.

---

Result first: this AI could not read cat feelings

On a 3-choice quiz, its accuracy was 23.3% — below random guessing (33.3%).

As I dug in, it turned out the AI wasn't really listening to the meows in the first place. Here's what happened.

---

Today's material: 440 meows from 21 cats

I used CatMeows, a research dataset of cat vocalizations (credit at the end of this post). Every meow comes with an answer label — the situation it was recorded in.

Label	Situation	Count
Brushing	Being brushed by the owner	127
Waiting for food	Food is being prepared	92
Isolation	Left alone in an unfamiliar room	221

---

Experiment 1: first, a speech-to-text AI

First, a detour: what happens if you hand raw cat meows to Whisper, the speech-to-text AI?

In Japanese mode, 10 of the 12 meows came back as 「ご視聴ありがとうございました」("Thank you for watching!") — plus one 「チャンネル登録をお願いいたします。」("Please subscribe to my channel.")

Whisper is trained on subtitled video audio, so unrecognizable sounds tend to come back as the stock phrases that end videos.

---

Experiment 2: a 90-question quiz for the AI

Next, I handed meows one at a time to Qwen2-Audio (an AI that listens to audio directly) and asked:

This is a recording of a domestic cat meowing.
In which situation was this meow most likely recorded?
(A) The cat is being brushed by its owner.
(B) The cat is waiting for food.
(C) The cat is isolated alone in an unfamiliar room.

90 questions total — 30 meows from each situation. Random guessing would score 33.3%.

The result: 23.3% (21 out of 90). It lost to random guessing.

Here's the per-situation breakdown:

Situation	Correct
Brushing	0 / 30
Waiting for food	9 / 30
Isolation	12 / 30

Brushing: zero. The AI never once picked "(A) brushing" in all 90 questions.

---

Experiment 3: shuffle the option order

"Never picks A" is odd. So I checked: same meows, same question, only the option order swapped — another 90 questions.

If the AI answers by listening, changing the order shouldn't change its answers.

In fact, the answers flipped completely.

	The AI's answers
Before the swap	"isolation" ×50 / "food" ×40
After the swap	almost all "food" (×89)
Option A (first)	0 in both runs (0 out of 180)

The AI was answering by option order, not by the meow.

(The post-swap 32.2% accuracy only looks better because answering "food" every time gets exactly the food questions right.)

---

Meanwhile, a meow-only AI scored 96%

The research team behind this dataset built a purpose-built meow classifier (2019) that scores 95.9% on the same 3-way task (trained on this data, so not the same conditions as my zero-shot run).

General-purpose audio AI seems to be strong at words, still weak at what non-word sounds mean.

---

Today's takeaways

• Quizzes get gamed: an audio LLM may answer by option position instead of listening. Shuffle the options and rerun to catch it
• Don't trust accuracy alone: the 32.2% run was all-in on one answer. Check the answer distribution
• Whisper hallucinates: non-speech sounds come back as stock phrases from its training data ("Thank you for watching!")
• Niche tasks want specialists: for reading meows, a purpose-built model is the right tool

---

The details

Models and dataset

• Speech-to-text: openai/whisper-large-v3-turbo (MIT license)
• Quiz: Qwen/Qwen2-Audio-7B-Instruct (Apache 2.0), run locally via transformers, greedy decoding (do_sample=False) so answers are reproducible
• Dataset: CatMeows (CC BY 4.0). 21 cats, 440 clips, 8kHz mono. The first letter of each filename is the ground-truth label (B=brushing, F=food, I=isolation)
• Machine: DGX Spark. A 7B audio model fits in memory with plenty of room; a few seconds per question

How I asked the quiz (prompt)

This is a recording of a domestic cat meowing.
In which situation was this meow most likely recorded?
Choose exactly one:
(A) The cat is being brushed by its owner.
(B) The cat is waiting for food.
(C) The cat is isolated alone in an unfamiliar room.
Answer with only the single letter A, B, or C.

• 30 clips per label, stratified sampling with a fixed random seed → 90 questions
• The control run swapped the wording of (A) and (C) only; everything else identical
• Grading: extract the first A/B/C that appears in the model's reply

Environment notes

• I reused an existing Python environment and added librosa for audio loading, which bumped numpy past what another library wanted (core stayed fine). Lesson re-learned: separate environments per experiment is the safer way
• The recordings are 8kHz (phone quality); the models expect 16kHz, so audio is resampled on load. Could that hurt? Maybe — but the specialized model hit 96% on the same 8kHz clips, so it's not much of an excuse

---

Sources

• Meow dataset: CatMeows: A Publicly-Available Dataset of Cat Vocalizations (Zenodo, DOI: 10.5281/zenodo.4008297, CC BY 4.0)
• Original paper: Ntalampiras et al., "Automatic Classification of Cat Vocalizations Emitted in Different Contexts," Animals 9(8), MDPI, 2019 (the 95.9% figure is from this paper)

---

Outro

I set out to have an AI listen to cats' feelings, and instead found an AI that answers by option order. The day it understands my own cat's meows still seems a way off.

Next up: a completely different experiment. Thanks for reading!

#100ExperimentsWithDGX #LocalLLM