Introduction ๐
Diving into the world of regression and forecasting metrics can be a real headscratcher, especially if you're a newcomer. Trust me, I've been thereโhaphazardly applying every popular metric scholars swear by, only to be left puzzled by the results.
Ever wondered why your MAE, MSE, and RMSE values look stellar, but your MAPE is through the roof? Yep, me too.
That's why I set out on this journey to create an experimental notebook, aiming to demystify how different metrics actually behave.
The Objective ๐ฏ
The goal of this notebook isn't to find the "one metric to rule them all" for a specific dataset. Instead, I want to understand how various metrics respond to controlled conditions in both dataset and model. Think of this as a comparative study, a sort of "Metrics 101" through the lens of someone who's still got that newcar smell in the field. This way, when I'm plunged into realworld scenarios, I'll have a better grip on interpreting my metrics.
Metrics Investigated ๐
To get a comprehensive view, I've opted to explore a selection of metrics that are commonly leveraged in regression and forecasting problems. Here's the lineup:

Mean Absolute Error (MAE):
 Definition: It measures the average magnitude of the errors between predicted and observed values.

Formula:
$MAE = \frac{1}{n}\sum_{i=1}^{n} y_i  \hat{y}_i$where $(y_i)$ is the actual value, $(\hat{y}_i)$ is the predicted value, and $(n)$ is the number of observations.

Mean Squared Error (MSE):
 Definition: It measures the average of the squares of the errors between predicted and observed values. It gives more weight to large errors.

Formula:
$MSE = \frac{1}{n}\sum_{i=1}^{n} (y_i  \hat{y}_i)^2$

Root Mean Squared Error (RMSE):
 Definition: It represents the sample standard deviation of the differences between predicted and observed values. It's the square root of MSE.

Formula:
$RMSE = \sqrt{MSE}$

Mean Absolute Percentage Error (MAPE):
 Definition: It measures the average of the absolute percentage errors between predicted and observed values.

Formula:
$MAPE = \frac{100}{n}\sum_{i=1}^{n} \left \frac{y_i  \hat{y}_i}{y_i} \right$Note: MAPE can be problematic if the actual value $(y_i)$ is zero for some observations.

Mean Absolute Scaled Error (MASE):
 Definition: It measures the accuracy of forecasts relative to a naive baseline method. If MASE is lower than 1, the forecast is better than the naive forecast.

Formula:
$MASE = \frac{\sum_{i=1}^{n} y_i  \hat{y}i}{\sum{i=2}^{n} y_i  y_{i1}}$

Rsquared (Coefficient of Determination):
 Definition: It indicates the proportion of the variance in the dependent variable that is predictable from the independent variable(s).

Formula:
$R^2 = 1  \frac{\sum_{i=1}^{n} (y_i  \hat{y}i)^2}{\sum{i=1}^{n} (y_i  \bar{y})^2}$where $(\bar{y})$ is the mean of the observed data.

Symmetric Mean Absolute Percentage Error (sMAPE):
 Definition: It's a variation of MAPE that addresses some of its issues, especially when the actual value is zero.

Formula:
$sMAPE = \frac{100}{n} \sum_{i=1}^{n} \frac{y_i  \hat{y}_i}{(y_i + \hat{y}_i)/2}$

Mean Bias Deviation (MBD):
 Definition: It calculates the average percentage bias in the predicted values.

Formula:
$MBD = \frac{100}{n}\sum_{i=1}^{n} \frac{y_i  \hat{y}_i}{y_i}$
Severity and Directional Emojis ๐ฅ๐
Let's face it, numbers alone can be dry, and if you're like me, you might crave a more visceral sense of how well your model is doing. Enter severity and directional emojis. These little symbols provide a quick visual cue for interpreting metric results, ranging from "you're nailing it" to "back to the drawing board."
Disclaimer: Keep in mind that these categorizations are userdefined and could vary depending on the context in which you're working.
Standard Error Metrics (MAE, MSE, RMSE) Categorization ๐
To clarify, the concept of Normalized Error Range stems from dividing the error by the range (max  min) of the training data.
Category  Normalized Error Range 

Perfect  Exactly 0 
Very Acceptable  $0 < x \leq 0.05$ 
Acceptable  $0.05 < x \leq 0.1$ 
Moderate  $0.1 < x \leq 0.2$ 
High  $0.2 < x \leq 0.3$ 
Very High  $0.3 < x \leq 1$ 
Exceedingly High  $x > 1$ 
Percentage Error (MAPE, sMAPE, MBDev) Categorization ๐
Category  Error Magnitude (%)  Direction 

Perfect  Exactly 0%   
Very Acceptable  $0 < x \leq 5$  Over/Under 
Acceptable  $5 < x \leq 10$  Over/Under 
Moderate  $10 < x \leq 20$  Over/Under 
High  $20 < x \leq 30$  Over/Under 
Very High  $30 < x \leq 100$  Over/Under 
Exceedingly High  $x > 100$  Over/Under 
R2 Score Categorization ๐
Category  R2 Value Range 

Perfect  Exactly 1 
Very Acceptable  $0.95 \leq x < 1$ 
Acceptable  $0.9 \leq x < 0.95$ 
Moderate  $0.8 \leq x < 0.9$ 
High  $0.7 \leq x < 0.8$ 
Very High  $0.5 \leq x < 0.7$ 
Exceedingly High  $0 < x < 0.5$ 
Doesn't Explain Variability  Exactly 0 
Worse Than Simple Mean Model  $x < 0$ 
MASE Categorization ๐
Category  MASE Value Range 

Perfect  Exactly 0 
Very Acceptable  $0 < x \leq 0.1$ 
Acceptable  $0.1 < x \leq 0.5$ 
Moderate  $0.5 < x \leq 0.9$ 
High  $0.9 < x \leq 1$ 
Equivalent to Naive Model  Exactly 1 
Worse Than Naive Forecast Model  $x > 1$ 
Severity Emojis ๐จ
Metric  Emoji 

Perfect  ๐ฏ 
Very Acceptable  ๐ 
Acceptable  โ๏ธ 
Moderate  โ 
High  โ 
Very High  ๐ 
Exceedingly High  โ 
Doesn't Explain Variability  ๐ซ 
Worse Than Simple Mean Model  ๐ 
Equivalent to Naive Model  โ 
Worse Than Naive Forecast Model  ๐คฌ 
Directional Emojis โก๏ธ
Metric  Emoji 

Overestimation  ๐ 
Underestimation  ๐ 
Nan / None  ๐ โโ๏ธ 
Methodology ๐
For this experiment, I've synthesized datasets using mathematical functions like sine and cosine, which offer a controlled level of predictability. On the modeling end, I've used statsmodels.tsa.ar_model.AutoReg
and OffsetModel
. I chose AutoReg
for its foundational role in time series forecasting, while OffsetModel
serves to mimic good performance by shifting test data. This entire endeavor is laserfocused on forecasting problems, underscoring the fact that all forecasting issues are essentially regression problems, just not the other way around as far as I understand.
Highlights of Findings โจ
To navigate the labyrinth of metrics, I've laid out my explorations in a tree graph, which you can check out below:
The table here provides just a glimpse into the first phase of my deep dive into metrics. For those hungry for the full rundown, it's available right here. Click ๐ for the plot.
Plot  Based on  Variant  Dataset  Model  R2  MAE  MSE  RMSE  MASE  MAPE  sMAPE  MBDev 

๐  Test Size  Small=1  $\cos(x)$  AutoReg  ๐ โโ๏ธ  ๐  ๐  ๐  ๐คฌ  ๐  ๐  ๐๐ 
๐  OffsetModel  ๐ โโ๏ธ  ๐  ๐  ๐  ๐คฌ  ๐  ๐  ๐๐  
๐  $\sin(x)$  AutoReg  ๐ โโ๏ธ  ๐  ๐  ๐  ๐คฌ  โ  โ  โ ๐  
๐  OffsetModel  ๐ โโ๏ธ  ๐  ๐  ๐  ๐คฌ  โ  โ  โ ๐  
๐  Small=2  $\cos(x)$  AutoReg  ๐  ๐  ๐  ๐  ๐คฌ  ๐  ๐  ๐๐  
๐  OffsetModel  ๐  ๐  ๐  ๐  ๐คฌ  ๐  ๐  ๐๐  
๐  $\sin(x)$  AutoReg  ๐  ๐  ๐  ๐  ๐คฌ  โ  โ  โ ๐  
๐  OffsetModel  ๐  ๐  ๐  ๐  ๐คฌ  โ  โ  โ ๐  
๐  Mid  $\cos(x)$  AutoReg  ๐  ๐  ๐  ๐  ๐คฌ  ๐  ๐  ๐๐  
๐  OffsetModel  ๐  ๐  ๐  ๐  ๐คฌ  ๐  ๐  ๐๐  
๐  $\sin(x)$  AutoReg  ๐  ๐  ๐  ๐  ๐คฌ  โ  ๐  โ ๐  
๐  OffsetModel  ๐  ๐  ๐  ๐  ๐คฌ  โ  โ  โ ๐  
๐  Large  $\cos(x)$  AutoReg  ๐  โ  โ๏ธ  โ  ๐คฌ  ๐  โ  ๐๐  
๐  OffsetModel  โ  ๐  ๐  ๐  ๐คฌ  ๐  ๐  ๐๐  
๐  $\sin(x)$  AutoReg  ๐  โ  โ  โ  ๐คฌ  โ  ๐  โ ๐  
๐  OffsetModel  ๐  ๐  ๐  ๐  ๐คฌ  โ  โ  โ ๐ 
Key Insights ๐
Inconsistent R2 Scores: Almost all of the AutoReg and OffsetModel experiments yielded R2 scores that were either nonexistent (
๐ โโ๏ธ
) or poor (๐
). Only one OffsetModel experiment on a large dataset achieved an "Acceptable" R2 score (๐
).Good Performance on Standard Errors: Across various test sizes and datasets, both AutoReg and OffsetModel generally performed "Very Acceptable" (
๐
) in terms of MAE, MSE, and RMSE metrics.Problematic MASE Scores: Every model configuration led to "Worse Than Naive Forecast Model" (
๐คฌ
) MASE scores. This suggests that these models might not be better than a simple naive forecast in certain aspects.Diverse MAPE and sMAPE Responses: The models varied significantly in their MAPE and sMAPE scores, ranging from "Very Acceptable" (
๐
) to "Exceedingly High" (โ
and๐
), especially on sine ( $\sin(x)$ ) datasets.Bias Direction: The Directional Emojis indicate a tendency for the models to either overestimate (
๐
) or underestimate (๐
) the values. The direction of bias appears consistent within the same dataset but varies between datasets.Complexity vs. Error: Larger test sizes didn't necessarily yield better error metrics. In fact, some larger test sizes led to "High" (
โ
) and even "Very High" (๐
) errors, as seen in the last row of the table.Dataset Sensitivity: The models' performance was noticeably different between the sine ( $\sin(x)$ ) and cosine ( $\cos(x)$ ) datasets, showing that dataset characteristics heavily influence metric values.
Best Scenario: If one had to pick, OffsetModel with a large dataset and the sine function ( $\sin(x)$ ) yielded a balanced outcome, achieving "Acceptable" (
๐
) ratings in almost all metrics, barring MASE (๐คฌ
).Limitations & Risks: It's important to remember that these experiments used synthetic data and specific models; thus, the results may not be universally applicable. Caution should be exercised when generalizing these insights.
Please note that these insights are derived from synthetic data and controlled experiments. They are intended to offer a glimpse into the behavior of different metrics and should be used with caution in practical applications.
Points for Critique ๐ค
I'm all ears for any constructive feedback on various fronts:
Did I get the interpretation of these metrics right?
Are there any hidden biases that I might have missed?
Is there a more suitable metric that should be on my radar?
Did you spot a typo? Yes, those bother me too.
Digging into metrics is a lot like treasure hunting; you don't really know what you've got until you put it under the microscope. That's why I'm so eager to get your feedback. I've listed a few questions above, but let's delve a bit deeper.
 Interpretation of Metrics: I've given my best shot at understanding these metrics, but it's entirely possible that I've overlooked some nuances. If you think I've missed the mark or if you have a different angle, I'm keen to hear it.
 Potential Biases: When you're neckdeep in numbers, it's easy to develop tunnel vision and miss out on the bigger picture. Have I fallen into this trap? Your external perspective could provide invaluable insights.
 Alternative Metrics: While I've focused on some of the most commonly used metrics, the field is vast. If there's a gem I've missed, do let me know. I'm always up for adding another tool to my analytical toolbox.
 Typos and Errors: Mistakes are the bane of any data scientist's existence, and not just in code. If you've spotted a typo, I'd appreciate the heads up. After all, clarity is key when it comes to complex topics like this.
So, am I on the right track, or is there room for improvement?
Your input could be the missing puzzle piece in my metrics exploration journey.
Conclusion ๐คโ
So there it isโmy metric safari in a nutshell. It's been an enlightening experience for me, and I hope it shines some light for you too. I'm still on the learning curve, and I'd love to hear your thoughts. Whether it's a critique or a thumbsup, all feedback is golden.
If this sparked your curiosity, let's keep the conversation going. Feel free to drop a comment below, write your own post in response, or reach out to me directly at my link. If you'd like to delve deeper, the full summary of my findings is available here. Better yet, why not conduct your own investigations? I'd be thrilled to see where you take it. You can follow my progress and check out my portfolio repository here
DAIly
A bunch of Data Analysis and Artificial Intelligence notebooks ๐ค I'd worked on almost a daiLY basis ๐จโ๐ป
Ideas
This directory might contain notes or outlines of potential data analysis or AI projects that I'm considering working on in the future. These might be in the form of brainstorming notebooks, rough outlines powerpoint of project ideas, or notes on interesting data sources or tools that I want to explore further
Goodbye Average Rating System Hello Helpful Rating System
Redefining the average rating system by factoring in people's feedback
Regression and Forecasting Metrics Exploration
Navigating the maze of regression and forecasting metrics to understand their behavior and implications
Tips
This directory might contain more practical information, such as code snippets or tutorials that I've found helpful in my data analysis and AI work. These could be tips on how to use specific librariesLet's keep exploring and innovating together!
Top comments (2)
Such a nice article man, thank you so much
Thanks, you could share it so people could find this useful like you do! ๐ค
Here's the Medium version if you like
New to Forecasting: My Take  by Rangga K  Medium
Rangga Kusuma Dinata ใป ใป
Medium