DEV Community

Cover image for Case study: the threat intel feed that never expires (and has no confidence)
cuong do
cuong do

Posted on Originally published at gitlab.com

Case study: the threat intel feed that never expires (and has no confidence)

#azure #security #kql #cybersecurity

A lab company, Contoso, connects a community threat intelligence feed to Microsoft Sentinel over TAXII. The connector is green, indicators flow in every hour, the analytics rules start matching. Two weeks later the SOC lead asks a simple question: how many of these indicators are still worth alerting on? The answer turns out to be "all of them, forever, with no idea how sure anyone is".

Rebuilt in my lab tenant with made-up data, checked against Microsoft Learn (October 2026). All names (Contoso, law-lab-sentinel, LabCommunityFeed) and numbers are lab examples. The diagrams are drawings, not screenshots.

TL;DR

  • Sentinel stores what the STIX feed sends. If the feed leaves out valid_until and the numeric confidence, your indicators have no expiry and no confidence. Sentinel doesn't guess.
  • Feeds built with MISP often carry confidence as a tag (misp:confidence-level="...") instead of the STIX confidence number, and MISP has no "valid until" field of its own.
  • Fix it in three places: check every new feed with one query, derive confidence and an "effective expiry" in your detections, and get the feed (or your own upload pipeline) to send the real fields.

The case at a glance
Situation Contoso lab workspace law-lab-sentinel, a MISP-style community feed LabCommunityFeed connected with the Threat Intelligence - TAXII connector, polling every hour
Goal Use confidence and expiry in detections, so stale indicators stop raising alerts
What went wrong Every indicator from that feed had an empty Confidence and no real ValidUntil, so nothing ever aged out
Fix A health query per feed, confidence and effective expiry derived at query time, ingestion rules for tagging, and the real fields fixed at the source
Result Detections only use indicators that are recent or still valid, and every new feed gets the same two-minute check

Before and after: before, the MISP-style feed arrives with no confidence and no expiry and every indicator stays active forever; after, a health check per feed, a derived confidence from the MISP tag and an effective expiry keep only recent indicators in detections

Table of contents

1. The situation

Item Lab value
Workspace law-lab-sentinel, Sentinel enabled
Solution Threat Intelligence from Content hub (it brings the TAXII connector)
Feed 1 LabCommunityFeed, a MISP-style TAXII 2.1 collection, polled every hour
Feed 2 (for comparison) Microsoft Defender Threat Intelligence connector
Role Microsoft Sentinel Contributor (needed to manage threat intelligence)

The connector setup is the easy part: Data connectors → Threat Intelligence - TAXII → friendly name, API root URL, collection ID, credentials if the server needs them, polling frequency → Add. A few minutes later indicators show up in Intel explorer (Defender portal) or the Threat intelligence page (Azure portal), and in the ThreatIntelIndicators table.

2. What I tried first, and why it failed

Try 1: trust the connector page. It shows data received and the time of the last indicator. All green. That only proves the transport works. It says nothing about the content.

Try 2: filter detections by confidence.

ThreatIntelIndicators
| where Confidence >= 50
Enter fullscreen mode Exit fullscreen mode

Every indicator from LabCommunityFeed vanished. Not because they were weak, but because Confidence was empty on all of them.

Try 3: rely on expiry. Indicators from the feed had no meaningful ValidUntil, so IsActive stayed true and the indicators kept matching, week after week. An IP that was bad last spring is a new coffee shop's Wi-Fi today.

Try 4: compare with a known-good feed. The Defender Threat Intelligence indicators had Confidence and realistic ValidUntil values. Same workspace, same tables, same queries. So the problem was the feed, not Sentinel.

3. Why it happens

Sentinel's threat intelligence tables mirror the STIX object it receives:

STIX property Column in ThreatIntelIndicators Required in STIX?
confidence (0 to 100) Confidence No
valid_from ValidFrom Yes
valid_until ValidUntil No
the whole object Data n/a

Both fields we needed are optional. A feed that leaves them out is still a valid feed.

MISP-based feeds are a classic example. MISP expresses confidence with a taxonomy tag, for example misp:confidence-level="completely-confident", and the tag lands in Tags, not in Confidence. MISP also has no "valid until" field of its own (it has first seen, last seen and decay models instead), so a plain export often has nothing to put there. Sentinel doesn't translate tags into numbers and doesn't invent an expiry date.

The MISP taxonomy does give each confidence label a number, which we can use:

misp:confidence-level Number
completely-confident 100
usually-confident 75
fairly-confident 50
confidence-cannot-be-evaluated 50
rarely-confident 25
unconfident 0

One more thing to know: when an indicator is created or updated, Sentinel writes a new row, and only the newest one counts. Always take the latest row per Id before you count anything.

4. The fix, step by step

Step 1. Run a health check per feed

Query 1 in queries.kql (see Scripts) takes the latest version of each indicator and counts, per source: total, missing confidence, missing or far-future expiry, and still active. Run it after you connect any new feed. Two minutes, and you know what you are dealing with.

In my lab (made-up numbers): LabCommunityFeed had 100% missing confidence and 100% "never expires". The Defender Threat Intelligence source had neither problem.

Step 2. Derive confidence from the MISP tag

Query 2 extracts the misp:confidence-level tag and maps it to the numbers above. It falls back to the real Confidence when the feed does send it. Use this as a function (or a let block) at the top of your TI analytics rules instead of the raw column.

Step 3. Give every indicator an effective expiry

Query 3 builds EffectiveUntil: the real ValidUntil when it exists and is sensible, otherwise ValidFrom plus a window you choose (30 days in my lab for IPs and domains). Detections then use only indicators where EffectiveUntil > now(). Old indicators stay in the table for hunting, they just stop paging people.

Step 4. Use ingestion rules for what they're good at

Threat intelligence ingestion rules run on objects from data connectors before they reach the workspace. They can edit objects (add tags, extend Valid until for chosen sources and confidence levels) or delete them from the pipeline. In my lab I added a rule for LabCommunityFeed that tags everything no-source-expiry, so analysts see the warning in Intel explorer too.

Keep in mind: rules run in order, a Delete stops processing for that object, new or edited rules take up to 15 minutes, and ingestion rules don't apply to the upload API or to manually created objects. I couldn't find in the docs what "extend Valid until" does when the source sent no Valid until at all, so I don't rely on it for this case. Test it in your lab before you do.

Step 5. Fix it at the source

The query-time fix is a bandage. The real fix is that the feed sends confidence and valid_until in its STIX output:

  • Ask the feed provider to include both fields in the export.
  • Or run your own pipeline: pull from MISP and send STIX objects to Sentinel with the upload API, setting confidence and expiry yourself. The open-source misp2sentinel project does exactly that: confidence from MISP's decay score or confidence tag (default 50), and an expiry of days_to_expire (default 50 days).

5. The result

In the lab (made-up numbers):

Before After
Indicators from the feed with a usable confidence 0% Every one that carries a MISP confidence tag
Indicators from the feed that ever expire 0% All, through EffectiveUntil (30 days)
TI matches per day in the lab detections Climbing every day Flat, only recent indicators
Time to vet a new feed "We'll see" One query

Lessons learned

  1. A green connector proves transport, not quality. Check the fields, not the status.
  2. Optional STIX fields stay empty unless someone fills them. Sentinel stores what it gets.
  3. Compare with a known-good source to find out whether the feed or the platform is the problem.
  4. Bandage at query time, fix at the source. Derived values are fine for detections, but the feed (or your own pipeline) should send real values.

Gotchas

  • Old table, old queries. Indicators now land in ThreatIntelIndicators (and ThreatIntelObjects). The legacy ThreatIntelligenceIndicator table stopped receiving new data after July 31, 2025. Old queries against it see nothing new.
  • Duplicates by design. Each update writes a new row. Use summarize arg_max(TimeGenerated, *) by Id before counting.
  • Reingestion costs. Sentinel reingests threat intelligence every 7 to 10 days, billed like other paid tables. A big feed full of never-expiring indicators is also a cost problem.
  • Ingestion rules vs upload API. Rules only touch objects from data connectors.
  • Tags are a string. Tags is text, so extract with a regex and test it on real rows.

Checklist

  • [ ] Health query run for every TI source (confidence, expiry, active count)
  • [ ] Compared with a known-good source
  • [ ] Derived confidence used in TI analytics rules
  • [ ] Effective expiry used in TI analytics rules
  • [ ] Ingestion rule tags feeds that send no expiry
  • [ ] Feed provider asked for confidence and valid_until, or your own upload pipeline sets them
  • [ ] Queries moved to ThreatIntelIndicators

How do you vet a new threat intel feed before it reaches your detections? I'd like to steal your checklist.

Scripts

The full queries used in this post.

queries.kql 

// SEN-CASE-01: the threat intel feed that never expires (and has no confidence)
// Lab names (law-lab-sentinel, LabCommunityFeed). Run in Defender advanced hunting or Sentinel Logs.
// 14 days is enough: Sentinel reingests threat intelligence every 7 to 10 days, so every live indicator has a recent row.

// 1. Health check per source: latest version of each indicator only
ThreatIntelIndicators
| where TimeGenerated > ago(14d)
| summarize arg_max(TimeGenerated, *) by Id
| where IsDeleted == false
| summarize
    Total        = count(),
    NoConfidence = countif(isnull(Confidence)),
    NoExpiry     = countif(isnull(ValidUntil) or ValidUntil > now() + 3650d),
    Active       = countif(IsActive)
    by SourceSystem
| extend PctNoConfidence = round(100.0 * NoConfidence / Total, 1),
         PctNoExpiry     = round(100.0 * NoExpiry / Total, 1)
| order by Total desc

// 2. Derive confidence from the MISP taxonomy tag when Confidence is empty
//    misp:confidence-level="completely-confident" -> 100, and so on
ThreatIntelIndicators
| where TimeGenerated > ago(14d)
| summarize arg_max(TimeGenerated, *) by Id
| extend MispLevel = extract(@'misp:confidence-level=\\?"?([a-z-]+)', 1, Tags)
| extend DerivedConfidence = case(
    isnotnull(Confidence), Confidence,
    MispLevel == "completely-confident", 100,
    MispLevel == "usually-confident", 75,
    MispLevel == "fairly-confident", 50,
    MispLevel == "confidence-cannot-be-evaluated", 50,
    MispLevel == "rarely-confident", 25,
    MispLevel == "unconfident", 0,
    int(null))
| summarize count() by SourceSystem, MispLevel, DerivedConfidence

// 3. Indicators that detections should use: derived confidence + effective expiry
//    Effective expiry = ValidUntil when it is sensible, otherwise ValidFrom + 30 days
let ExpiryWindow = 30d;
let MinConfidence = 50;
ThreatIntelIndicators
| where TimeGenerated > ago(14d)
| summarize arg_max(TimeGenerated, *) by Id
| where IsDeleted == false and Revoked == false
| extend MispLevel = extract(@'misp:confidence-level=\\?"?([a-z-]+)', 1, Tags)
| extend DerivedConfidence = case(
    isnotnull(Confidence), Confidence,
    MispLevel == "completely-confident", 100,
    MispLevel == "usually-confident", 75,
    MispLevel in ("fairly-confident", "confidence-cannot-be-evaluated"), 50,
    MispLevel == "rarely-confident", 25,
    MispLevel == "unconfident", 0,
    int(null))
| extend EffectiveUntil = iff(isnull(ValidUntil) or ValidUntil > now() + 3650d, ValidFrom + ExpiryWindow, ValidUntil)
| where EffectiveUntil > now() and DerivedConfidence >= MinConfidence
| project Id, SourceSystem, ObservableKey, ObservableValue, DerivedConfidence, ValidFrom, ValidUntil, EffectiveUntil
Enter fullscreen mode Exit fullscreen mode

Sources

Top comments (1)

Collapse
 
furqan_ashraf profile image
Furqan Ashraf •

A green connector proves transport, not quality” is going straight into our runbook. To answer your question, the three things I check on any new feed before it touches detections:

Does it have a last_seen, or only a first_seen? Without last_seen there’s no honest way to say “still active.”
Is confidence a number, a tag, or missing? (Your MISP tag mapping is a neat fix for the tag case.)
Is it published as deltas or as full snapshots? With a full daily snapshot, “not in today’s file” is itself an expiry signal.

Disclosure: I work at WhoisFreaks, and our domain threat feeds went the snapshot route. Each feed is rebuilt daily as a full file, with a 0–1 confidence and first_seen/last_seen on every record. One gotcha for your Query 1: our Sentinel connector writes to a custom table (WhoisFreaksThreat_CL) (Whoisfreaks.com/integrations/third-party/sentinel), not ThreatIntelIndicators, so a health check that only looks at the TI tables won’t see feeds like that. It’s worth listing where each feed actually lands.