My Model Had 100% Recall — Then I Realized It Was Predicting "Exoplanet" for Everything

Danishh-ux — Sun, 30 Aug 2026 18:05:26 +0000

A few months ago, I set out to build a model that detects exoplanets from stellar brightness data (light curves) collected by NASA's Kepler telescope. When a planet passes in front of a star, it blocks a tiny bit of light — a "transit." The goal: train a model to spot that pattern.

I thought it'd be a straightforward binary classification problem. It was not.

The trap: 100% recall, useless model

My first model was a standard 1D CNN — a few conv layers, batch norm, LeakyReLU, max pooling. Training loss went down steadily. Looked great.

Then I checked the actual metrics:

Metric	Score
Recall	1.0000
Precision	0.0071
F1 Score	0.0141

Perfect recall. Precision near zero. The model was labeling almost every sample as an exoplanet. It hadn't learned anything — it had just found the laziest way to minimize loss.

This is the trap of imbalanced data. The dataset had:

Split	Normal Stars	Exoplanets	% Positive
Train	4292	31	0.72%
Validation	758	6	0.79%
Test	565	5	0.88%

Out of ~5,600 samples, only 42 were exoplanets — less than 1%. A model that predicts "not an exoplanet" every single time gets 99% accuracy while being completely useless. Accuracy was lying to me.

What didn't work

I tried the usual playbook for imbalance:

SMOTE — generates synthetic minority-class examples. Barely helped, because light curves have complex temporal structure that's hard to fake by interpolating between real examples.
SMOTETomek — combined oversampling with cleanup. Slightly better, still unstable.
Threshold tuning (0.30–0.60) — adjusting the decision boundary can't fix a model that's producing garbage probability estimates in the first place.

None of these addressed the actual problem: the model almost never saw a positive example during training.

What actually worked

WeightedRandomSampler. Instead of sampling batches randomly (where most batches had zero exoplanets), I weighted sampling so minority-class examples showed up far more often during training. This alone was one of the biggest single improvements in the whole project.

Focal Loss instead of standard Binary Cross Entropy. BCE treats every sample equally, so the model gets flooded with "easy" majority-class gradient signal and barely learns from the rare hard examples. Focal Loss down-weights easy examples and focuses learning on the difficult ones:

FL = α(1 - pt)^γ * BCE

with alpha = 0.25, gamma = 2.0.

A 1D Residual Network. I'd assumed ResNets were an image-only architecture. Turns out residual connections work just as well for 1D time series — the skip connection (output = F(x) + x instead of just F(x)) helps preserve signal information and stabilizes gradient flow through deeper networks. The final model: an initial conv block, four residual blocks, global average pooling, and a fully connected output layer.

Along with these, I also applied a median filter to remove sensor spikes and normalized each light curve individually (zero mean, unit variance) so the model focused on the shape of the dip rather than absolute brightness.

The results (and the part I almost didn't report properly)

Best single validation run: F1 = 0.9091. Looked great in isolation.

But with only 5-6 positive examples in any given validation/test split, one number from one split doesn't mean much. So I ran Stratified 5-Fold Cross Validation instead:

Fold	F1 Score
1	0.4545
2	0.8421
3	0.7778
4	0.5333
5	0.6667
Average	0.6549

That's a big spread — 0.45 to 0.84. With only ~6-7 positive examples per fold, misclassifying even a single exoplanet swings the F1 score substantially. This variance is the real result. It's a more honest signal than any single number.

For contrast, here's the final test set report:

Class	Precision	Recall	F1
Non-Exoplanet	1.00	1.00	1.00
Exoplanet	1.00	1.00	1.00

A perfect score across the board, on only 5 positive samples. It's tempting to lead with that number — it looks amazing. But with a sample size that small, it's close to meaningless on its own. The cross-validation average is the number I actually trust.

What I took away from this

Class imbalance can be a harder problem than model architecture.
Accuracy is often actively misleading on imbalanced data — check precision/recall/F1 from the start.
A "perfect" test score on 5 samples tells you almost nothing. Cross-validation exists for exactly this situation.
Fixing the data pipeline (sampling, loss function) mattered more here than any architecture change.
ResNets aren't just for images.

Code and full report: [link to your repo]