Before You Buy AI SOC Triage, Make It Explain What It Dismisses

An alert can be technically correct and still waste an analyst's time.

A detection fires because an administrator runs a remote command. Another fires because a deployment script changes a service. Both describe real activity. Neither necessarily describes an attack.

Repeat that pattern across an environment, and the security problem changes. The team has detections. What it lacks is enough attention to investigate every result carefully.

That is alert fatigue's failure mode: investigation becomes acknowledgment, acknowledgment becomes bulk closure, and a consequential alert receives the same treatment as yesterday's noise. Buying more detection coverage does not automatically fix it. It can make the queue worse.

AI triage promises relief by sorting alerts before analysts spend time on them. The buying question is whether that sorting preserves the evidence and uncertainty needed to make good decisions.

You should be able to evaluate that without accepting "trust the AI" as an answer.

Start with the decision the AI actually makes

"AI-powered" tells you almost nothing about how a SOC operates.

A system that summarizes an alert has a different responsibility from one that suppresses it. A tool that suggests an investigation step carries different consequences from one that initiates a response.

Before discussing accuracy, ask the provider to walk through one alert from ingestion to disposition:

• What information does the triage layer receive?
• Does it summarize, rank, group, or close alerts?
• Which decisions require human review?
• Can its decisions trigger changes to an endpoint or account?
• What record remains available after an alert leaves the analyst's queue?

Ask for a workflow diagram with the decision boundaries marked. A box labeled "AI analysis" is insufficient.

The most consequential boundary is often where an alert becomes less visible. An incorrect summary might be caught during investigation. An incorrectly suppressed alert may never reach an analyst.

That is where evaluation should concentrate.

Require an explanation an analyst can verify

An explanation is not useful merely because it sounds reasonable.

Consider this disposition:

Low risk. Activity is consistent with normal administrative behavior.

It names a conclusion without showing its basis. A buyer cannot tell whether the system recognized an approved task or simply associated a familiar executable with benign activity.

A useful explanation might look like this hypothetical example:

Proposed disposition: Lower priority; retain for review.

Observed:
- A remote command started on endpoint APP-17.
- The initiating account was svc-deploy.
- The parent process matches the deployment agent.

Supporting context:
- A change record covers this host and time window.

Missing context:
- The destination address has not been validated.

Reasoning:
The deployment context supports a benign explanation.
The unvalidated destination prevents automatic closure.

Decision owner:
An analyst must review the destination before closing.

The exact fields will vary. The important distinction is between observed evidence and an interpretation of that evidence.

Ask whether the analyst can open the underlying events, confirm timestamps, and inspect the relevant process or network context. An explanation should identify unavailable information rather than silently treating its absence as reassurance.

A confidence score alone does not solve this. "97% confident" is difficult to use unless the provider explains what the score measures and how it has been validated. It should not be presented as a probability of maliciousness unless that interpretation is supported.

Also ask whether the explanation reflects the inputs and decision process used at the time. A convincing narrative generated afterward is not necessarily a faithful account of why an alert was downgraded.

Make every false-positive claim show its denominator

"We cut false positives by 90%" is a starting point for questions, not a buying criterion.

First, establish what the provider calls a false positive. It might mean a rule matched incorrectly. It might mean suspicious activity occurred but had an authorized explanation. Some reports also mix in duplicate alerts that added no investigative value.

Those categories need different fixes. Combining them can hide whether AI triage improved judgment or simply grouped repetitive records.

Next, ask for the numerator and denominator.

The formal false-positive rate is:

False-positive rate =
benign cases incorrectly classified as positive
/
all actually benign cases evaluated

Operational dashboards often report something different:

False-discovery proportion =
benign alerts in the reviewed alert set
/
all alerts in that reviewed set

Neither metric is inherently wrong. They answer different questions. The first can also be hard to estimate when a dataset contains only triggered alerts rather than a representative set of benign activity.

Suppose a queue contains 1,000 alerts. Triage removes 900, leaving 100 for analysts. Of those remaining, 80 require no security action.

The queue is smaller. But analysts still spend most of their reviews on non-actionable work, and you have not learned whether the removed set contained important alerts.

Ask the provider:

• Who labeled the evaluation cases, and how were disagreements resolved?
• Did evaluation include suppressed alerts?
• Were ambiguous cases excluded or counted?
• Was the dataset drawn from environments resembling yours?
• What happened to known malicious cases after triage?
• How much analyst time was saved after verification and correction?

Request the counts behind percentages, along with the sample size and evaluation period. A result based on a small, curated demonstration should not be treated like evidence from routine operations.

Run a bounded evaluation before allowing suppression

A useful pilot can begin with AI recommendations recorded while the existing workflow remains in place. This is often called shadow mode.

Agree on the evaluation method before seeing the results.

1. Record a baseline

Use a defined period of your current workflow. Record incoming alert volume, analyst handling time, and the time from alert arrival to meaningful investigation.

Document how cases were resolved. Avoid treating every closed ticket as proof that the activity was benign.

Note unusual conditions such as a large deployment or a staffing shortage. Otherwise, a quieter pilot period can look like a triage improvement.

2. Build a review set that includes difficult cases

Include ordinary operational noise and previously investigated incidents where available. Add authorized administrative activity that resembles attacker behavior.

Deliberately include incomplete records. Missing endpoint context or unavailable identity information can reveal whether a system handles uncertainty sensibly.

For historical replay, check that the system only receives information available at the original decision time. Giving it the final incident notes would make the exercise misleading.

Keep some cases out of tuning so the evaluation measures more than familiarity with the examples.

3. Compare decisions, not just summaries

For each reviewed case, record the proposed priority, evidence cited, and any recommendation to suppress or close it.

Have a qualified reviewer assess the result. Where practical, obtain that assessment before showing the AI recommendation, so the recommendation does not become the default answer.

Count correction effort. A polished summary that takes longer to verify than the original alert does not necessarily reduce workload.

4. Inspect what would disappear

Review a sample of proposed closures and downgrades, including a random component. Give additional attention to consequential systems and cases with missing context.

Do not evaluate only the alerts the AI escalates. That tests what it brings forward while leaving its most consequential omissions unexamined.

Known incidents and controlled exercises can test specific scenarios. They cannot establish that the system will catch every future attack.

5. Set limits before enabling automation

Define acceptable behavior locally rather than borrowing a vendor's headline target.

You might require human review for activity involving privileged accounts, or prevent automatic closure when essential telemetry is missing.

Specify what happens if review finds a consequential miss: pause the affected automation, inspect related dispositions, and correct the underlying issue.

A pilot should produce an operational decision about where automation is justified.

Treat human oversight as an operating process

"Human in the loop" can mean substantive review. It can also mean someone is available if a customer complains.

Ask who reviews which decisions, when that review happens, and what authority the reviewer has to override automation.

Human oversight is the point because triage requires judgment about the environment. An apparently suspicious command might be part of an approved migration. An ordinary administrative tool might be used by an attacker. AI can help assemble and prioritize evidence; responsibility for consequential decisions still needs an owner.

Look for review of suppressed work, not merely approval of escalations. Ask how analyst corrections affect future behavior and whether changes are evaluated before deployment.

There is a tradeoff. Reviewing everything defeats much of the workload benefit. Reviewing nothing makes mistakes harder to discover. A workable design applies stronger review where consequences or uncertainty are higher, while sampling routine dispositions to check whether performance is deteriorating.

The provider should also explain how analysts see original evidence and challenge recommendations. Oversight becomes weaker when people are expected to rubber-stamp a confident answer under queue pressure.

Keep a short acceptance checklist

Before signing, ask the provider to demonstrate these behaviors using representative cases:

• An analyst can trace a disposition back to its evidence.
• Missing context remains visible in the explanation.
• Reported performance includes definitions, counts, and evaluation limits.
• Review covers alerts proposed for suppression.
• A named role owns overrides and consequential decisions.
• The team can pause problematic automation and investigate its prior decisions.

A provider may have legitimate limits on what it can expose about its implementation. You still need enough operational evidence to judge the decisions affecting your environment.

Where our own SOC lands on this

Adaptive IP Services' managed SOC combines endpoint and network telemetry with AI-triaged alerts and expert response. Lightweight endpoint agents report to a hosted analysis stack. The service uses open, proven detection tooling with AI triage and human oversight, with the aim of reducing attacker dwell time and delivering prioritized, real alerts instead of noise.

That describes the approach. It does not establish a particular false-positive rate or answer every workflow question above.

Apply the same evaluation to us: ask to see how a disposition is supported, what happens when context is missing, and where human review changes the outcome. The buying decision should turn on evidence that the service makes useful investigation more likely, including when its initial recommendation is wrong.

Disclosure: I work at Adaptive IP Services, a Dallas based IT and security firm.

David J. Boggs
Founder and CEO, Adaptive IP Services

Adaptive IP Services Managed SOC