You just retrained your model and R-squared went up. Great! Except it might not have gotten better, you just got lucky with the split. Or worse, R-squared is low and you scrap a model that was actually fine. Either way, you're optimizing the wrong number.
R-squared tells you how much of the variation in your target the model explains.
A 0.844 value means your model explains 84.4% of the variation which is pretty solid. A 0.44 looks like a dud, more than half the variation is unaccounted for.
But a low R-squared doesn't always mean your model is bad and chasing a higher one isn't always the fix.
Here's why:
1. It doesn't measure goodness of fit
R² measures how much of the variance in your outcome the model explains not whether it's captured the relationship correctly. A model can have accurate coefficients and a low R², or vice versa.
Say you're predicting GPA from hours studied. Your model might understand more hours = higher GPA. But GPA also depends on other things like study methods, sleep, and whether the student got lucky with the questions on the exam.
None of that is in your model, so R² comes out low, even though the relationship you modeled is correct.
The reverse happens too.
If you fit a straight line to curved (e.g., exponential) data, R² can still come out high. The line follows the direction of the data even though it's the wrong shape
2. It doesn't tell you how far off your predictions are
Rsquared doesn't tell you how right or wrong your predictions are. A better metric for that is root mean squared error (RMSE) which is even more interpretable. If your GPA model has an RMSE of 0.3, its predictions are typically off by about 0.3 grade points
3. You can't compare R² across transformed outcomes
Sometimes you need to transform your outcome to meet model assumptions(for example a log transform for non-linearity). But once you do, you can't compare R² with the untransformed model.
One measures variance explained in y, the other in log(y). If R² drops after transforming, that doesn't mean the model got worse.
To compare them fairly, back-transform the predictions to the original scale and compare RMSE or MAE.
What to use alongside R²
- Plot your data before modeling: A scatterplot of your outcome against each predictor will tell you whether the relationship is linear. If non-linear you may need a transformation, or a different model.
-
*Plot residual vs fitted values after modelling: *
- Randomly scattered values around 0: The relationship is linear and variance is constant.
- Curved scatter plot: You've missed a non-linear patter. Transform or use another model.
- Funnel shapes: Errors are growing as predictions get larger.
*** Report RMSE and MAE:** Both of these metrics tell you how far off your predictions are, in your outcome's units.
- MAE is the average error
- RMSE squares errors before averaging, so large misses count more. If RMSE is much larger than MAE, a few predictions are badly off.
Before you scrap a model for a low R² score or celebrate a high one, look at the plots and the errors.
Top comments (1)
The straight-line-on-curved-data example is the one that actually burns people — R² looks reassuring while the model is structurally wrong, and no amount of hyperparameter tuning fixes a misspecified form. The residual-vs-fitted habit is the cheapest defense here; it shows you in one glance what the single number hides. I'd add one more habit to the checklist: always report the metric on a held-out split, since in-sample R² is where most of the self-deception happens.