12 Questions That Actually Separate Threat Detection Vendors

Hammad Anjum — Thu, 10 Sep 2026 10:45:05 +0000

Most threat detection evaluations are decided by a demo, and demos are designed to succeed. The data is curated, the attack is one the product has seen, and the environment has none of the strange legacy software that makes your estate yours.

The second problem is that every vendor's marketing now says the same words. AI-powered. Behavioural. Zero-day. Reduces alert fatigue. The words no longer discriminate between products, which means the burden falls entirely on the questions you ask.

Below are twelve that do discriminate — in the sense that different products give genuinely different answers, and the answers are hard to fake. I've included what a good answer sounds like and what should worry you.

Full disclosure, since it's relevant: I work on a detection platform. I've tried to write these so they'd be useful against us too, and I've flagged the ones where our own answer is uncomfortable.

Detection quality

1. What is the false positive rate at my event volume — not on your benchmark?

This is the single most important question and the most commonly dodged.

Detection rates are quoted constantly; false positive rates rarely are. But at security's base rates the FP rate is what determines whether the product is usable. A system that catches 99% of attacks and produces 500 false positives a day is not a detection system, it's a queue that your team will learn to ignore.

Good answer: a number, with the event volume it was measured at, and an acknowledgement that it varies by environment.

Red flag: a detection rate offered in place of a false positive rate. Or a percentage with no denominator — "99.9% accurate" is meaningless when the positive class is one in ten million.

2. What was the model trained or tuned on, and how similar is that to my estate?

Real labelled attack data is scarce. Everyone improvises — lab-generated attacks, public datasets, red team exercises, customer telemetry. Each has different failure modes, and a model trained on lab data has learned the artefacts of the lab's generation process alongside the actual attack behaviour.

Good answer: specifics about the data, honest about the gap between it and production.

Red flag: "proprietary" as a complete answer, or "millions of samples" without saying what kind. Ask specifically whether the training data came from lab-executed MITRE ATT&CK techniques, public datasets, or production customer telemetry — the three have very different failure modes.

3. What is your unit of detection — an event, or a relationship?

This distinction determines more about a product's behaviour than any feature list.

If each event is scored independently, an attack made of individually-benign steps produces either nothing or a scatter of low-severity alerts that get closed one by one — which is a large part of why alert fatigue costs what it does. If the product reasons over relationships between events, that same attack can surface as a single correlated incident.

Good answer: a clear description of what gets correlated and how.

Red flag: confusion between correlation and aggregation. Grouping alerts by host and time window is not the same as understanding that one process created another.

4. What happens with activity you've never seen before?

Everyone claims zero-day detection. Push on the mechanism.

Good answer: a specific explanation — anomaly relative to a learned baseline, structural patterns that hold across technique variants, behavioural constraints that any implementation of a technique must satisfy — plus its limitations.

Red flag: "our AI detects unknown threats" with no mechanism. Also worth noting: if the answer is purely statistical anomaly detection, ask how they handle the fact that living-off-the-land attacks are specifically designed to sit inside the normal distribution.

What it's actually like to operate

5. Show me a finding exactly as an analyst receives it.

Ask for a screenshot of the real thing, not a slide. This one question tells you more about daily life with the product than an hour of feature discussion.

If a finding is a severity score and a rule name, your analysts will reconstruct the reasoning themselves from raw logs — which is the work you were trying to avoid. If it carries the evidence chain, they can verify or dismiss it in minutes.

Good answer: they show you one immediately, and it contains enough for you to form your own judgement about the activity.

Red flag: reluctance, or a mockup.

6. Can a mid-level analyst verify a finding without escalating?

Related but distinct. A finding can be detailed and still be unverifiable — buried in jargon, or requiring knowledge of the model's internals.

The practical test: could someone eighteen months into the job look at this and decide whether it's real? If every finding needs your best person, the product hasn't reduced your workload, it's concentrated it.

7. What's the median time from finding to decision?

Ask existing customers, not the vendor. This is the number that converts to money, and it's the one you should be measuring in a POC.

8. What happens when my environment changes?

New software, a migration, a reorganisation, a merger. Baseline-dependent systems degrade as the estate drifts.

Good answer: a clear account of retraining or re-baselining — cadence, who performs it, whether it's included, how long it takes.

Red flag: "it learns continuously" with no operational detail. Ask what happens on day one of a new deployment, and how long until it's useful.

Coverage

9. Which telemetry sources, on which platforms, and what does each give you?

Specificity matters here. "Full endpoint visibility" means nothing. Sysmon, auditd, eBPF, ETW, and vendor-proprietary collectors see genuinely different things — we wrote a detailed comparison of what each endpoint telemetry source actually captures if you want the specifics and have different overheads and kernel requirements.

Ask particularly about Linux, which is where coverage is usually thinnest — and about which kernel versions are supported, since eBPF-based collection has real floor requirements and older estates fall back to something else.

10. What are you not good at?

The most informative question in the list, and the one whose value comes entirely from how it's answered rather than what's said.

Every product has a shape. A vendor who can describe theirs has thought about it; one who claims comprehensive coverage of everything is either inexperienced or managing you.

Good answer: a real limitation, stated without being walked back in the next sentence.

Red flag: "that's a great question" followed by a strength.

Adversarial

11. Which of your detections still work if the attacker knows exactly how the system works?

Assume your vendor's documentation is public and read. Which detections survive?

Detections keyed on things cheap to change — filenames, hashes, specific strings, timing — don't survive. Detections keyed on what an attack must structurally do to succeed are much more durable, because changing them means changing the attack.

Good answer: an explicit distinction between the two, and an honest statement that some detections are in the first category.

Red flag: a claim that everything is durable. Nothing is entirely durable, and saying so signals they haven't modelled an adaptive adversary.

12. What happens when an attacker kills the agent?

They will try. Ask what the product does when telemetry stops — whether agent silence is itself an alert, how quickly it fires, and what tamper protections exist.

An agent going quiet on one host at 3am is a finding. Surprisingly many products treat it as a health metric on a dashboard nobody watches.

Every vendor answers these the same way: "Do you use AI?" · "Do you detect zero-days?" · "Do you reduce false positives?" · "Do you map to MITRE ATT&CK?"

These separate them: "What's your false positive rate at my event volume?" · "Show me a finding exactly as an analyst receives it."

How to run the evaluation itself

The questions matter less than the test. Four things worth insisting on:

Use your own data. A POC on vendor-supplied data tells you the product works on data the vendor chose. Run it against your estate, including the odd legacy application nobody wants to talk about.

Measure false positives over at least two weeks. Detection is easy to demonstrate and false positives take time to accumulate. A three-day POC systematically flatters every product in this category.

Run a real technique, not a sample file. Executing an actual ATT&CK technique — with Atomic Red Team, Caldera, or by hand — tells you far more than dropping EICAR. Have the vendor not know when you're doing it.

Have the analysts who'll use it do the triage. Not the security architect who ran the procurement. The people whose queue this becomes will tell you within a week whether findings are actionable.

The uncomfortable ones

In the interest of the disclosure at the top — the questions above that are hardest for us to answer:

Question 2, on training data. Like everyone in this space, we generate attacks in a lab — MITRE techniques executed on baremetal Windows and Linux VMs — and lab-generated attacks differ from real ones in ways that are genuinely hard to characterise. Anyone claiming otherwise is not describing a situation that exists.

Question 8, on environmental change, is a real cost for any behavioural system and it's worth pressing every vendor on, including us.

Where we do well is 3, 5, and 11 — the unit of detection is a behavioural graph rather than an individual event, findings carry the evidence chain that produced them, and structural features are considerably more expensive for an attacker to evade than name-based ones. Those are the questions we'd want asked. Which is exactly why you should ask all twelve, of everyone, including us.

Common questions

What should you ask a threat detection vendor before buying?

Start with the false positive rate at your own event volume, not their benchmark. Then ask to see a finding exactly as an analyst receives it, and ask what the product is not good at. Those three separate serious vendors from marketing faster than any feature comparison, because none of them can be answered with a slide.

How long should a threat detection POC run?

At least two weeks, on your own data. Detection is easy to demonstrate on day one; false positives take time to accumulate. Short proofs of concept systematically flatter every product in this category, which is why vendors like them short.

What's the difference between detection rate and false positive rate?

Detection rate is the proportion of attacks caught. False positive rate is how much benign activity is wrongly flagged. Vendors quote the first constantly and the second rarely — but at security's base rates, where genuine intrusions are extremely rare against millions of daily events, the false positive rate is what determines whether a SOC can actually use the product.

How do EDR, XDR, SIEM and behavioural detection differ?

EDR collects and analyses endpoint telemetry. XDR extends that across more signal sources. SIEM aggregates logs centrally and runs rules over them. Behavioural detection describes the method rather than the category — reasoning over what processes do and how they relate, rather than matching known-bad artefacts. The categories overlap heavily, which is why question 3 above — what is your unit of detection — discriminates better than asking which acronym a product belongs to.

Does behavioural detection replace signature-based detection?

No, and be sceptical of anyone saying it does. Signatures are fast, cheap, and produce near-zero false positives on known threats. They just have no coverage of anything novel. Production estates run both.

DEV Community: Hammad Anjum