1. At what point of time in the designated process, will the team utilize the Model?
Understanding the timeline is very crucial for the model development process because it defines the prediction point. All ML models operate within a specific timeline. If you use the data which is available after a particular timeline, it can result in data leakage. Understanding prediction points also helps to determine what features will be available for the model and would follow the same timeline. All features should follow the same timeline and only include information that would have been available at that point.
For example, if you are trying to flag fraudulent transactions and the transactions take two days to process completely, you would need to determine whether the model should make a prediction when the transaction is initiated or after the transaction is completed, depending on when the decision needs to be made. The model should only use information at the prediction point.
2. What will be the format required by the team(Score, Probability, Category, Yes/No, ranking, alert)?
This defines the output type and formats the output into something that the team can actually use. Different applications in ML may require different output formats. This question can also help identify the appropriate type of ML problem such as classification or regression.
For example, for identifying patients with high risk, a hospital would require a binary flag, for a social media outreach, a team might require a scoring or ranking format, for predicting the cost increase, a team might require a regression format.
3. How frequently will the prediction be used(real-time, daily, weekly, monthly)?
This question identifies the technical architecture and the inference strategy for the model, hence it is very crucial. It establishes latency requirements and determines the frequency of features being refreshed. It drives the computational costs and retraining frequency.
For example, a model that generates predictions in real time would have very different infrastructure and latency requirements from a model that generates predictions once a week. Similarly a model which receives a large amount of new data might require more frequent retaining than a model that is used weekly, although retaining frequency should ultimately depend on model performance and data drift rather than prediction frequency alone.
4. What specific decision will the model help the team make?
ML models exist to support decisions, so this is one of the most important questions to ask. It helps define what success means for the model and can also help determine the approachable target variable. Most importantly, it connects the ML problems to the actual business problem and helps establish how the model’s output will be used.
For example: If the model identifies patients who are at high risk of hospitalization, the actual decision might be whether a care management team should contact those patients and enroll in an intervention program.
5. Does the team already have a baseline metric that they use currently and what is the expected performance?
Without a baseline it is difficult to identify whether the model is providing meaningful value to the team or not. The baseline can be an existing rule based system, a human decision making process or another currently used approach.
For Example: If the current process identifies 60% confidence in the current metrics for identifying high risk patients threshold, a new ml model should be evaluated based on the existing baseline rather than simply looking at the accuracy in isolation.
Thanks for reading!! please feel free to check out my Website
Top comments (0)