Amazon Forecast : Best Practices and Anti-Patterns implementing AIOps

AIOps leverages artificial intelligence for IT operations. Forecasting is one of the most leveraged use cases in AIOps. It typically involves making predictions or estimates of a dataset based on historical data, patterns, and various quantitative and qualitative factors

Typical Forecasting use cases are:

• Traffic: Predicting traffic or volume fluctuations to anticipate demand and optimize infrastructure accordingly.
• Error Rate: Forecasting error rates to proactively identify and mitigate potential issues, ensuring system reliability.
• Latency: Predicting latency metrics to maintain optimal performance and enhance user experience.
• Resources: Forecasting resource usage patterns to optimize allocation and prevent bottlenecks.
• Business Metrics: Forecasting various business metrics such as sales, revenue, or customer engagement to inform strategic decision-making.
• Error Budget Burn Rate: Forecasting the rate at which error budget is consumed to manage risk and prioritize improvements effectively.
• SLA Adherence: Predicting SLA adherence to ensure service level commitments are met and customer satisfaction is maintained.

Time series data

Amazon Forecast works with time series data. Time series data is a sequence of data points collected, recorded, or observed over a period of time, where each data point is associated with a timestamp or time index. It is characterized by its chronological order, intervals (regular or irregular), trends (such as increasing, decreasing, or cyclical patterns), seasonality (repeating patterns with fixed periodicity), and the presence of noise, all of which influence its analysis and forecasting

Amazon Forecast

It's a fully managed service offered by Amazon for metric forecasting. It's easy to use, as it allows you to integrate historical or related data by uploading them to Amazon Forecast. Once the data is uploaded, Forecast automatically inspects the data, identifies key attributes, and selects the right algorithms needed for forecasting. It then trains and optimizes your custom model. Once generated, forecasts can be visualized via the console or downloaded. Amazon Forecast also provides APIs that allow you to build solutions

Getting started is relatively easy.

Import Your Data

• Create your dataset group.
• Select the forecasting domain. AWS supports multiple domains out of the box:

1. Retail domain
2. Inventory planning
3. EC2 capacity
4. Workforce
5. Web traffic
6. Metric (for forecasting metrics such as revenue, sales, and cash flow)
7. Custom (if your requirement does not match any of the above).

• Then select the frequency of data.
• Create your data schema - the data definition of your sample data. You can use the schema builder or JSON schema.
• Select your data file from S3.
• Create an IAM role providing access to S3.

Train a predictor

• You can select your predictor, essentially the metric you want to forecast.

Generate Forecasts

• You can use the predictor to generate the forecast.
• For all items: Generate forecasts for all items in the input dataset.
• For selected items: Generate forecasts for selected items in the input dataset.
• Query Forecast : You can generate your query to visualize the forecast

Amazon Forecast output snapshot

In Amazon Forecast, P10, P50, and P90 represent the 10th, 50th, and 90th percentiles of the forecast distribution, indicating that there is a 10%, 50%, and 90% probability, respectively, that the actual value will be less than the forecasted value.

Explore Insights

Explore Insights consists of two parts:

Explore Explainability - As per the AWS-provided definition, Explainability insights identify the attributes that impact your forecasts, quantify their impact relative to other attributes, and determine whether they decrease or increase forecast values.

Explore What-if Analysis - As per the AWS-provided definition, What-if analysis explores modifications to the related time series, quantifies the impact of those modifications, and determines how those modifications can impact forecast values

Few things to note about Amazon Forecast:

• It supports forecasting via Console, AWS CLI, or Python notebook.
• Ability to customize forecast parameters.
• Modularized - meaning able to track model drift, what-if scenarios, or forecast explainability.

Pricing:

Pricing is based on imported dataset and the time taken for predictor training, as well as the number of forecast data points.

Best practices to follow when using Amazon Forecast:

• Data quality: Ensure your dataset is clean and accurate, removing outliers or noise that could mislead your forecasts.
• Feature engineering: Include relevant factors in your dataset to improve forecast accuracy.
• Fine-tune model parameters: Adjust forecast horizon, frequency, and other parameters to optimize model performance.
• Select the forecasting algorithm: Choose the appropriate algorithm (e.g., ARIMA, CNN-QR, DeepAR+, ETS, NPTS, Prophet) based on your dataset and needs.
• Continuous evaluation, validation, and monitoring: Regularly assess and improve model performance.
• Enable explainability and predictor monitoring: Activate these features to gain insights and track model performance.
• Updating data: Choose between replacement and incremental updates based on data volume and changes.
• Handling missing data: Use filling methods like middle, back, and future filling to address missing values and ensure accurate forecasting.
• Follow dataset guidelines: Adhere to guidelines to ensure optimal model performance.
• Use predictor monitor: Track model performance over time and make adjustments as needed.
• Leverage Forecast Explainability: Gain insights into how dataset attributes influence forecasts.
• Leverage what-if analysis: Explore the impact of altering related time series on baseline forecasts.

Pitfalls to avoid:

Overfitting Models: Occurs when over-configuring or fine-tuning to prevent noise, instead focus on identifying key patterns.
Complex Algorithms: Stick to Amazon Auto Model selection unless there are specific reasons to use a different algorithm.
Seasonality and Trends: Ignoring seasonality can lead to unsatisfactory forecasts; ensure relevant features and data are included to uncover patterns.
Improper Data Preparation: Prepare data meticulously by clearing, normalizing, and using feature engineering techniques for optimal forecasting results.
Not Spending Enough Time: Forecasting is challenging; allocate sufficient time to understand, evaluate, validate, and make necessary adjustments continuously.
Lack of Business Context: Understand the purpose of forecasting and factors impacting results; if necessary, conduct reverse engineering to clarify data and goals.