The first thing my emotion model did was completely ignore grief
I trained a model to detect emotions in text last week, and the first version had a strange habit: it never once predicted "grief." Not "predicted it badly" literally never. Zero. Same story for "relief." I'd feed it a sentence dripping with relief and the model would shrug and pick something else.
At first I assumed I'd broken something. I hadn't. The model was behaving completely rationally. It had just quietly figured out that ignoring those emotions was the smartest thing it could do and fixing that turned into the whole project.
Here's the story, including the parts that didn't work and the two hours I lost to a deployment error that had nothing to do with machine learning.
The dataset that sets you up to fail
I used GoEmotions, a Google dataset of about 54,000 Reddit comments, each tagged with some combination of 28 emotions things like joy, anger, gratitude, curiosity, grief. A single comment can carry several at once ("I'm so grateful but honestly terrified"), which makes this a multi-label problem: for every comment, the model makes 28 separate yes/no calls.
The catch is buried in the numbers. In the test set, "neutral" shows up 1,787 times. "Grief" shows up six times. "Relief" eleven. "Pride" sixteen. That's not a small imbalance, that's a cliff.
And this is where my model's laziness came from. When you train with the standard loss function for this kind of task (binary cross-entropy), every example is weighted the same. So think about what the model learns: if it just always guesses "no grief," it'll be right in something like 99.98% of cases, because grief almost never appears. The penalty for ignoring grief entirely is basically a rounding error. The model isn't broken it's doing exactly what you asked, which is to minimize its loss. Ignoring rare emotions is the optimal strategy under that loss.
That's the trap: a model can look fine on paper while silently failing on a third of the categories.
The metric that hides the problem, and the one that exposes it
When I first measured performance, the overall score looked respectable. That's because I was looking at the wrong number.
There are two ways to average performance across 28 classes. Micro-F1 pools every single prediction together which means the handful of super-common emotions dominate the score, and the rare ones barely register. A model can post a healthy Micro-F1 while being useless on grief, and you'd never know.
Macro-F1 does the opposite: it scores each emotion separately and then averages those scores, so grief counts exactly as much as neutral. The moment I switched to reporting Macro-F1, the real picture showed up and it was ugly. A big chunk of my 28 classes were sitting near zero.
If you take one thing from this: pick the metric that measures the thing you actually care about. Micro-F1 would have let me lie to myself.
The fix: make the model care about rare things
The solution wasn't a bigger model or more training. It was changing the loss function to one that stops treating every mistake as equal specifically, Asymmetric Loss.
The intuition is simple. It does two things: it turns down the volume on the easy, confident "obviously not this emotion" predictions the model already gets right, so the training signal shifts toward the hard cases. And it punishes missing a real emotion harder than it punishes a false alarm which directly attacks the "just say no to everything" shortcut.
The effect was immediate and it landed exactly where the problem was:
- Grief: 0.00 → 0.35
- Relief: 0.00 → 0.31
- Pride: 0.11 → 0.39
- Nervousness: 0.18 → 0.39
Overall Macro-F1 went from 0.48 to 0.53. And crucially, the common emotions didn't suffer gratitude stayed up around 0.92. I lifted the floor without lowering the ceiling. That's the result I was after.
The idea I was sure would work, and didn't
Here's the part most write-ups leave out.
Emotions don't appear randomly they travel in packs. Gratitude shows up with admiration. Grief shows up with sadness. So I figured: instead of treating all 28 emotions as independent decisions, what if I made the model learn those relationships? I added a second training objective (supervised contrastive learning) that pushes the model to represent comments sharing emotions similarly, and comments sharing none differently.
It made things slightly worse.
Not dramatically about a point of Macro-F1 but it lost. I tried tuning its strength; still lost. After digging in, the reason is structural: this technique needs lots of examples-that-share-labels in each training batch to get a useful signal, and I was capped at 32 examples per batch on a free GPU. With 28 sparse emotions, most batches just didn't have enough overlap. The idea wasn't wrong; my hardware couldn't give it a fair test.
I kept it in the final writeup anyway, clearly marked as "tried, didn't beat the simpler approach." A negative result you understand is worth more than pretending you only ever had good ideas.
The two hours that had nothing to do with machine learning
The modeling took an afternoon. Getting the thing deployed is what nearly broke me.
DeBERTa-v3, the model I used, silently produces garbage (NaN values) under half-precision training a known quirk I didn't know about, which cost me a wasted training run that came out as all zeros. Then when I tried to put the demo online, the hosting platform quietly ran it on a brand-new version of Python that PyTorch doesn't support yet, so it kept insisting "torch isn't installed" when torch was right there in my requirements. That one took embarrassingly long to diagnose.
None of this is in the papers. It's the tax you pay for shipping something real instead of leaving it in a notebook. Worth it, but nobody warns you.
What I'd tell myself starting over
Grief is still basically unlearnable at six examples. No loss function fixes that the honest answer is that some problems are data problems, not modeling problems, and cleverness has limits.
But the core lesson held up: before reaching for a fancier model, look at how your current one is failing and fix the specific thing. The imbalance was the disease; the fancy contrastive idea was a cure for a different illness. The boring loss-function swap was the one that mattered.
The model's live if you want to poke at it, and the code and full results are on GitHub. Type in something complicated and mixed — that's where emotion detection gets interesting.
*Model and demo links in the repo. Built on DeBERTa-v3 + GoEmotions, trained on a free Kaggle GPU.
https://github.com/Dev-Abdullah-17/goemotions-emotion-demo
https://goemotions-emotion-demo-cksccxfpdawgisu4esdrzu.streamlit.app/
Top comments (0)