Predicting SLA Breaches in Technical Support

Case Study: Predicting SLA Breaches in Technical Support
Project: NovaTech AI Support (hypothetical scenario, synthetic data)
Repo: GitHub-novatech-ai-bottleneck

Executive Summary

In technical support teams, not all tickets represent the same level of operational risk. Some are resolved in minutes; others consume hours of specialized work and end up breaching the Service Level Agreement (SLA).

This case study explores how to build a Machine Learning (ML) system capable of identifying, from the moment a ticket is created, which ones are most likely to breach their SLA.

The project was developed as a hypothetical scenario for NovaTech AI Support, using synthetic data to replicate a real IT support problem. The objective was not only to maximize predictive performance but to design a realistic, explainable, reproducible, and deployable ML workflow.

The most important outcome of the project was not achieving a particular metric. It was discovering that the first model was using information from the future, and redesigning the problem around a more rigorous question:

What can we know about a ticket's risk at the exact moment it enters the system?

The Business Problem

Imagine a technology company that receives hundreds of support tickets every day. Each ticket can represent anything from a simple product question to an integration problem requiring specialized investigation.

For support managers, the challenge is not just resolving tickets. It's deciding:

• Which ones need immediate attention,
• Where to allocate specialized resources,
• Which tickets have the highest risk of breaching the SLA,
• How to move from a reactive to a preventive operation.

Business Hypothesis: If high-risk tickets are identified in advance, support teams can prioritize resources before the problem becomes a breach, improving resource allocation, prioritization, and customer experience.

Success Criteria

A high ML metric was not enough. The system had to meet four conditions:

1. Predictive Performance: Correctly identify tickets with breach risk.
2. Temporal Validity: Use only information available at the moment of prediction.
3. Operational Explainability: Enable understanding of why a ticket was classified as risky.
4. Engineering Readiness: Separate data, training, prediction, and documentation in a reproducible way.

The Alarm Signal: 100% Accuracy

The first result was exactly the type of outcome that generates a false sense of success: the model achieved ~100% on its classification metrics.

In a real operational problem, a perfect prediction should generate more questions than celebration.

The variable audit revealed the problem: temporal data leakage. The model was using information that would only be available after the ticket had been resolved particularly, the actual resolution time.

This creates an almost direct relationship with the target: if we already know how long it took to resolve a ticket, it's trivial to know whether it breached its SLA. But that information doesn't exist when the ticket has just entered the system.

The model wasn't predicting the future. It was using future information to reconstruct the past.

Methodological Correction: Strict T=0 Restriction

The solution wasn't removing a column. It was redefining the problem around the moment when the model would actually be used.

I defined T=0 as the moment of ticket creation (intake). From there, a variable could only be used if it was already available at that instant.

Features available at T=0 (kept):

• Ticket category
• Priority
• Channel
• Target SLA time

Removed (only exist after intake):

• Actual resolution time
• Accumulated waiting time

This decision had an immediate consequence: the ROC-AUC dropped from 1.0000 → 0.9606. At first glance, it seems like a deterioration, but a slightly less accurate model that is valid in production is much more valuable than a perfect model that cannot be used.

Architecture Benchmarking: More Complexity Doesn't Mean Better

With the problem redefined, two approaches were compared under the same T=0 feature set. The system solves a dual task: classification (will it breach the SLA?) and regression (estimating the associated risk/time), which is why both classification metrics (ROC-AUC, F1) and regression metrics (R²) are reported.

Model	Scenario	ROC-AUC	F1-Score	R² (Reg)	Ready for production?
Random Forest	Intake (T=0)	0.9606	0.9095	0.6416	✅ Yes (Recommended)
PyTorch Multitask	Intake (T=0)	0.9617	0.9047	0.5232	️ Optional
Random Forest	Omniscient (Leaky)	1.0000	1.0000	N/A	❌ No (Post-mortem only)

Random Forest was the chosen candidate for production. Both T=0 models are practically tied in classification (RF wins on F1; the neural network wins by a minimal margin on ROC-AUC — 0.0011), but RF has clearly better regression performance (R² 0.64 vs 0.52), plus native integration with TreeSHAP and lower computational cost. The additional complexity of deep learning is not justified for a marginal gain in classification that is also paid for with worse R² and less direct interpretability.

The decision was based on the balance between performance, speed, interpretability, complexity, and required resources, not just the highest metric.

Operational Explainability with SHAP: From Black Box to Support Tool

A prediction alone is not enough for a support operation. If the system flags a ticket as "high risk," the team needs to know why. To answer that question, SHAP (SHapley Additive exPlanations) was incorporated at two levels.

Global explanation

Probability of breach. Positive SHAP values ​​push the prediction toward "high risk."

The SHAP analysis identified the main risk factors:

• Integration/API is the most consistent category that increases risk.
• A short target SLA (8h) significantly increases risk compared to a broader one (72h).
• Technical Issue is the highest-magnitude feature, but it's bidirectional: it can increase or decrease risk depending on context.
• The entry channel (email, chat, or phone) has minimal impact.

This transforms the model from a purely predictive tool into a source of operational information.

Local Explanation

For a ticket with a final breach probability of 99.1%, the analysis decomposes the prediction as follows:

A ticket with a 99.1% probability of a breach.

The question shifts from "Does the model say this ticket is risky?" to "What factors are making this ticket risky?", a fundamental difference when predictions must be translated into human decisions.

Production Implementation and Software Robustness

Another common problem in ML projects is confusing a model trained in a notebook with a usable system. To avoid this, the project explicitly separates training and prediction.

train.py - loads the data, executes the pipeline, trains the model, and saves it for later use. Reproducible via a single command, without depending on manually running a notebook.

predict.py - receives new tickets, executes the transformation flow, generates predictions, and saves the results along with their probabilities.

This creates a clear boundary: Training → Model Artifact → Prediction, leaving the system ready for future integration with other systems.

Results

• First model with leakage: ROC-AUC = 1.0000 / F1 = 1.0000 (invalid, uses future information).
• Corrected model with T=0 features (Random Forest): ROC-AUC = 0.9606 · F1 = 0.9095 · R² = 0.6416.
• Multitask neural network (PyTorch) T=0: ROC-AUC = 0.9617 · F1 = 0.9047 · R² = 0.5232.
• Random Forest selected as production candidate for its balance between performance, simplicity, and interpretability.
• SHAP enabled explanation of both global patterns and individual predictions.

Key Lessons

• A perfect metric can be a warning sign, not a success.
• Define the moment of prediction before choosing features. The question is not "Is this variable predictive?" but "Would this variable exist at the moment I need to predict?"
• Complexity must earn its place. Deep learning is not used just because it's more advanced.
• Explainability matters when it answers concrete operational questions, not when it's included as an additional visualization.
• A notebook is not a production system: it takes data pipeline + validation + model + prediction workflow + logging + documentation + reproducibility.

Conclusion

NovaTech AI started as an SLA prediction problem and ended up becoming a Machine Learning Engineering exercise.

The most valuable result was not moving from one ROC-AUC to another slightly better one. It was identifying that the first perfect result was misleading, redefining the problem around T=0, comparing models considering complexity and operational value, incorporating explainability, and building a structure that could evolve from an experiment to a system.

If you have feedback on the T=0 approach, the choice of Random Forest over deep learning, or ideas to take this further (drift monitoring, calibration, inference API), I'd be glad to read them in the comments. The complete code is in the repo linked above.