A few months ago, I set out to build a model that detects exoplanets from stellar brightness data (light curves) collected by NASA's Kepler telescope. When a planet passes in front of a star, it blocks a tiny bit of light — a "transit." The goal: train a model to spot that pattern.
I thought it'd be a straightforward binary classification problem. It was not.
The trap: 100% recall, useless model
My first model was a standard 1D CNN — a few conv layers, batch norm, LeakyReLU, max pooling. Training loss went down steadily. Looked great.
Then I checked the actual metrics:
| Metric | Score |
|---|---|
| Recall | 1.0000 |
| Precision | 0.0071 |
| F1 Score | 0.0141 |
Perfect recall. Precision near zero. The model was labeling almost every sample as an exoplanet. It hadn't learned anything — it had just found the laziest way to minimize loss.
This is the trap of imbalanced data. The dataset had:
| Split | Normal Stars | Exoplanets | % Positive |
|---|---|---|---|
| Train | 4292 | 31 | 0.72% |
| Validation | 758 | 6 | 0.79% |
| Test | 565 | 5 | 0.88% |
Out of ~5,600 samples, only 42 were exoplanets — less than 1%. A model that predicts "not an exoplanet" every single time gets 99% accuracy while being completely useless. Accuracy was lying to me.
What didn't work
I tried the usual playbook for imbalance:
- SMOTE — generates synthetic minority-class examples. Barely helped, because light curves have complex temporal structure that's hard to fake by interpolating between real examples.
- SMOTETomek — combined oversampling with cleanup. Slightly better, still unstable.
- Threshold tuning (0.30–0.60) — adjusting the decision boundary can't fix a model that's producing garbage probability estimates in the first place.
None of these addressed the actual problem: the model almost never saw a positive example during training.
What actually worked
WeightedRandomSampler. Instead of sampling batches randomly (where most batches had zero exoplanets), I weighted sampling so minority-class examples showed up far more often during training. This alone was one of the biggest single improvements in the whole project.
Focal Loss instead of standard Binary Cross Entropy. BCE treats every sample equally, so the model gets flooded with "easy" majority-class gradient signal and barely learns from the rare hard examples. Focal Loss down-weights easy examples and focuses learning on the difficult ones:
FL = α(1 - pt)^γ * BCE
with alpha = 0.25, gamma = 2.0.
A 1D Residual Network. I'd assumed ResNets were an image-only architecture. Turns out residual connections work just as well for 1D time series — the skip connection (output = F(x) + x instead of just F(x)) helps preserve signal information and stabilizes gradient flow through deeper networks. The final model: an initial conv block, four residual blocks, global average pooling, and a fully connected output layer.
Along with these, I also applied a median filter to remove sensor spikes and normalized each light curve individually (zero mean, unit variance) so the model focused on the shape of the dip rather than absolute brightness.
The results (and the part I almost didn't report properly)
Best single validation run: F1 = 0.9091. Looked great in isolation.
But with only 5-6 positive examples in any given validation/test split, one number from one split doesn't mean much. So I ran Stratified 5-Fold Cross Validation instead:
| Fold | F1 Score |
|---|---|
| 1 | 0.4545 |
| 2 | 0.8421 |
| 3 | 0.7778 |
| 4 | 0.5333 |
| 5 | 0.6667 |
| Average | 0.6549 |
That's a big spread — 0.45 to 0.84. With only ~6-7 positive examples per fold, misclassifying even a single exoplanet swings the F1 score substantially. This variance is the real result. It's a more honest signal than any single number.
For contrast, here's the final test set report:
| Class | Precision | Recall | F1 |
|---|---|---|---|
| Non-Exoplanet | 1.00 | 1.00 | 1.00 |
| Exoplanet | 1.00 | 1.00 | 1.00 |
A perfect score across the board, on only 5 positive samples. It's tempting to lead with that number — it looks amazing. But with a sample size that small, it's close to meaningless on its own. The cross-validation average is the number I actually trust.
What I took away from this
- Class imbalance can be a harder problem than model architecture.
- Accuracy is often actively misleading on imbalanced data — check precision/recall/F1 from the start.
- A "perfect" test score on 5 samples tells you almost nothing. Cross-validation exists for exactly this situation.
- Fixing the data pipeline (sampling, loss function) mattered more here than any architecture change.
- ResNets aren't just for images.
Code and full report: [link to your repo]
Would genuinely love feedback — if anyone has dealt with similarly extreme class imbalance (sub-1% positive class) on time-series data, I'd like to hear what worked for you.
`
Top comments (0)