Evaluating Machine Learning in Security Products: A Framework

A detection arrives with a score of 0.93. Your SOAR playbook isolates the host because someone set the auto-containment threshold at 0.9. Nobody on the team can say what 0.93 means: 93 percent likely malicious, top 7 percent of events this week, or a sum of weights rescaled to look like a probability.

That gap, between what an ML score is and how your team uses it, is where most evaluations of machine learning in security products never go. The standard checks are well covered elsewhere on this blog: the questions to put to a vendor, how to run the POC, whether the product runs a model at all, and how it holds up against evasion. This framework covers four properties those checks leave out, the ones that decide whether the model is still trustworthy six months after signing.

1. Calibration: Does the Score Mean What Your Playbook Assumes?

A model can rank alerts well and still produce scores that are useless as probabilities. Guo et al. showed in On Calibration of Modern Neural Networks (ICML 2017) that deep networks are systematically overconfident, and a vendor's min-max rescale makes any number look like a probability. That is harmless when analysts only sort the queue. It breaks the moment a playbook treats the score as a likelihood.

Test it with POC output you already collect. Join each vendor score to the analyst disposition and run scikit-learn's calibration_curve:

import pandas as pd
from sklearn.calibration import calibration_curve
from sklearn.metrics import brier_score_loss

df = pd.read_json("poc_scored.jsonl", lines=True)
y = (df["disposition"] == "true_positive").astype(int)
p = (df["vendor_score"] / 100).clip(0, 1)   # drop the /100 if scores are already 0-1

frac_pos, mean_score = calibration_curve(y, p, n_bins=10, strategy="quantile")
for s, f in zip(mean_score, frac_pos):
    print(f"score ~{s:.2f} -> {f:.1%} confirmed malicious")
print("Brier score:", round(brier_score_loss(y, p), 4))

Use strategy="quantile" because security scores pile up near zero, and uniform bins leave the top of the range nearly empty. If the bin around 0.9 shows 30 percent confirmed malicious, your 0.9 containment rule is isolating two clean hosts for every bad one.

The fix does not require the vendor's cooperation. Fit an IsotonicRegression mapping from vendor score to observed rate on one month of dispositions, validate it on the following month, and set playbook thresholds on the mapped value. Fit on the earlier month and test on the later one. A random split leaks future dispositions into the mapping, the same trap described in the post on leakage in SOC training data.

2. Coverage: What Does the Model Decline to Score?

Vendor accuracy figures are computed on the events the model scored. The events it skipped are not in the denominator.

Static file classifiers skip formats they were not trained on. Network models see metadata only for TLS 1.3 sessions. A UEBA model has no baseline for a log source you onboarded last week. Those events usually come back with a null score, or a default value such as 0 or 50 that looks like a real score until you count how often it appears.

cov = df.groupby("source_type")["vendor_score"].agg(
    events="size",
    null_rate=lambda s: s.isna().mean(),
    default_rate=lambda s: s.isin([0, 50]).mean(),
)
print(cov.sort_values("null_rate", ascending=False))

A product with excellent recall on 70 percent of your telemetry and silence on the rest has 70 percent coverage. Write that number into the evaluation next to recall.

3. Updates: How Do You Regression-Test a Model You Do Not Control?

Most teams never evaluate this one, and it bites hardest. A SaaS product can push a new model on a Tuesday, and your thresholds, suppressions, and calibration mapping were all fit to the old one.

Treat a model update like a dependency upgrade. In procurement, demand the ability to pin a model version and replay a frozen labeled set through both versions. A model version string in every detection payload is the minimum; shadow scoring, where the new version runs without generating alerts, is better. Without either, every update is an unannounced change to your detection logic.

With a replay in hand, check two things. First, did the score distribution move? The population stability index, a drift measure borrowed from credit-risk model monitoring, answers that:

import numpy as np

def psi(old, new, bins=10):
    edges = np.unique(np.quantile(old, np.linspace(0, 1, bins + 1)))
    edges[0], edges[-1] = -np.inf, np.inf
    o = np.clip(np.histogram(old, edges)[0] / len(old), 1e-6, None)
    n = np.clip(np.histogram(new, edges)[0] / len(new), 1e-6, None)
    return float(np.sum((n - o) * np.log(n / o)))

print("PSI:", psi(replay["score_v41"], replay["score_v42"]))

The usual credit-risk rule of thumb reads under 0.1 as stable and over 0.25 as a major shift. Second, and more important, recompute recall and daily alert volume at your production threshold on the same frozen events. A large PSI with unchanged recall means you need to re-fit the calibration mapping. A small PSI with a drop in recall on one ATT&CK technique means the vendor's retraining quietly traded away a detection you depend on, and you will not find out from the release notes.

4. Feedback Loops: Who Gets to Write the Training Labels?

Many products retrain on customer signal: "mark as benign" buttons, submitted samples, auto-closed alerts. That makes the retraining pipeline attack surface. MITRE ATLAS tracks this as AML.T0020, Poison Training Data. An attacker who can get their tooling repeatedly marked benign, in your tenant or a shared global model, is teaching the product to ignore them.

Ask three questions, and get the answers in writing:

• Does a disposition in our tenant change only our suppressions, or does it feed a model shared across customers?
• Which roles in our tenant can write labels that reach training, and is that logged?
• Can you identify the training data snapshot behind the model version scoring our events today?

The third question separates vendors with real data lineage from vendors with a retraining cron job.

What the Evaluation Should Produce

Arp et al.'s Dos and Don'ts of Machine Learning in Computer Security (USENIX Security 2022) lists lab-only evaluation and the base rate fallacy among the pitfalls common in published security ML. Vendor datasheets have the same weaknesses with less review. The output of this framework is a short record you can re-run on every model update, not a slide:

# illustrative values for a hypothetical product
product: example-ndr
model_version_evaluated: v41
calibration: {brier: 0.071, observed_rate_at_0.9: 0.34, remap: isotonic_2026-10}
coverage: {overall: 0.78, worst_source: "zeek_ssl (0.41)"}
update_control: {version_pinning: yes, shadow_mode: no}
feedback_loop: {global_model_trains_on_tenant_labels: yes, label_writers: [soc_t2, soc_lead]}

The NIST AI Risk Management Framework asks for ongoing measurement under its MEASURE function. A versioned file like this in your detection-engineering repo is a practical way to meet it.

Where This Framework Is Overkill

The calibration test needs a few hundred confirmed true positives to give stable bins. A small team without disciplined dispositions will not have them, and should lean on update control and coverage, which need no labels. And if the ML in a product is a minor enrichment score attached to a rule-based alert, and nothing automated acts on it, a full evaluation costs more than it returns. Spend the effort where a model score triggers an action without a human in between.

Reading a reliability curve and computing drift on a replay set are the evaluation skills our Applied Data Science and AI for Cybersecurity course builds in Python. Security leaders who need to judge these results without writing the code get the same framing in the executive AI course.