CrowdStrike's AI Triage Research: How Well Can AI Automatically Judge SOC Alerts?

1. Basic Information

• Article Title: Teaching AI to Reason Through Detection Triage
• Publisher: CrowdStrike
• Publication Date: 2026-08-17
• Severity/Importance: High (Practical value as defense research)
• Original Source: CrowdStrike
• Related Source: Research Paper (arXiv PDF)
• Target: Security alerts generated on Windows endpoints
• Main Models & Methods: Nemotron-3-Nano-30B, Nemotron-3-Super-120B, GEPA, AdaSTaR, LoRA, GRPO

2. Summary in One Sentence

This is research on having AI judge whether Windows endpoint alerts are "real attacks" or "harmless false positives." While it showed certain results, misjudgments increased as time passed, revealing that continuous accuracy checks are necessary to automatically close alerts using AI alone.

Terms Used in This Report

• True Positive: An alert that was genuinely an attack
• False Positive: An alert that was actually not a problem
• Precision: The rate of items judged by the AI as a specific classification that were actually correct
• Recall: The rate of alerts that the AI successfully caught out of all alerts that should belong to that classification
• Confidence: How certain the AI is about its own judgment
• Calibration: Adjusting and checking whether the AI's confidence matches its actual correctness
• Distribution Shift: Changes in operational alerts compared to training data due to new attacks or product updates

3. Defense Challenges to Solve

• SOCs receive a huge volume of alerts, and human review alone takes time
• The team wants to quickly separate real attacks from false positives
• It is necessary not only to output classification results, but also to judge which decisions can be processed automatically
• Accuracy may drop over time or due to environmental differences, even if accuracy is high during training

4. Proposed Methods and Architecture

The research combined an AI that classifies alerts with another AI that checks the certainty of the answers.

1. The first AI reads the endpoint alert and judges whether it is an attack or a false positive, along with a reason.
2. The second AI reads the original alert, the judgment, and the reason, and calculates the probability that the answer is correct.
3. High-confidence results become candidates for automated processing or priority investigation.
4. Low-confidence results are reviewed by human analysts.

5. AI Inputs and Outputs

Inputs

• The process that triggered the alert
• The parent process and its grandparent process
• Execution command, file name, and file path
• How often the file is observed within the specific customer environment
• How often the file is observed across all CrowdStrike customer environments
• Severity assigned by the sensor, action results, and MITRE ATT&CK classification
• Explanations left by analysts in the past
• Whether any input fields are missing

Outputs

• Classification as a real attack or a harmless false positive
• The reason for the judgment
• The probability calculated by a separate model that the answer is expected to be correct
• Information to route the alert to automated processing or human review

6. AI Processing Flow / System Architecture

1. A security alert occurs on a Windows endpoint.
2. The alert and related information are formatted as a JSON with multiple fields.
3. The classification AI reads the information and generates reasons and classification results.
4. The calibration AI calculates the probability that the classification AI's answer is correct.
5. High-confidence results become candidates for automated processing or priority investigation.
6. Low-confidence results are sent to analysts.
7. The false-positive rate is measured even after operations start, and thresholds are adjusted.

7. Evaluation Design

Data

• Training: 388,336 items, 8 consecutive weeks of Windows endpoint alerts
• Tuning: 59,162 items, the following 2 weeks
• Final Test: 42,686 items, 1 subsequent week
• Past human judgments of true positives and false positives are used as ground truth.

Evaluation Methods

• Overall accuracy rate
• For high-confidence attack judgments, the rate of actual correctness and the rate of successful capture (recall)
• For high-confidence no-problem judgments, the rate of actual correctness and the rate of successful capture (recall)
• Accuracy changes when moving from the tuning period to the subsequent unknown period

8. Evaluation Results

• The overall accuracy rate was 82.6%.
• High-confidence "attack" judgments had a precision of 98.9% and a recall of 53.0%.
• High-confidence "no problem" judgments had a precision of 90.8% and a recall of 64.8%.
• The precision of "no problem" judgments dropped significantly from the tuning period to the final test period.
• Pre-tuning aimed for about 98%, but it dropped to 90.8% in the later period.

9. Practical Implications of Evaluation Results

• High-confidence attack judgments can potentially be used for prioritization.
• Human review cannot be completely eliminated because some attacks are missed even with high confidence.
• Automatically closing "no problem" alerts carries the risk of mistakenly closing real attacks.
• Numerical results from testing should not be trusted statically; they must be re-measured by time period, product update, and customer environment.
• Even if clear reasoning is generated, that reasoning itself is not proof that the judgment is correct.

10. Success Conditions

• Information such as processes, commands, files, and observation frequencies must be sufficient and accurate.
• Past human judgments must be accurate and consistent.
• Alert trends during training and operation must not differ significantly.
• Accuracy must be monitored separately for attack judgments and no-problem judgments.
• High-impact automated processing requires human review and a rollback mechanism.

11. Failure Conditions and Limitations

• Input trends change due to new attacks, product updates, or organizational differences.
• Items required for judgment are missing or contain incorrect values.
• Training ground truth labels contain errors or policy discrepancies.
• AI confidence scores or explanations are treated as independent evidence.
• The same results are applied to authentication, cloud, or network domains other than Windows endpoints.
• The auto-closure scope is fixed based solely on initial evaluation results.

12. Implementation and Verification Steps

1. Start with a test run where the AI does not perform actual processing, comparing its output with human judgments.
2. Separate attacks and no-problem alerts, and measure precision, recall, and processable volume.
3. Aggregate misjudgments by OS, sensor version, detection rule, time period, and customer environment.
4. Initially use the AI to prioritize attack-like alerts.
5. Start auto-closing no-problem alerts from limited detection types.
6. Have humans periodically re-examine a portion of auto-closed alerts.
7. If a drop in accuracy is detected, stop automated processing and review the thresholds.

13. Required Records and Audit Trails

• The original alert passed to the AI and any missing fields
• The model version, prompts, and judgment criteria version used
• The AI's classification, reasons, and confidence score
• Executed actions such as auto-closure, prioritization, or endpoint isolation
• Whether a human changed the judgment
• The basis for the final determination of an attack or no-problem
• Misjudgment rates broken down by time period, environment, and detection type

14. Facts / Inference / Hypothesis

Facts

• Windows endpoint alerts divided chronologically were used for training, tuning, and final testing.
• A combination of a classification AI and a calibration AI was used.
• The overall accuracy rate was 82.6%.
• High-confidence attack judgments had 98.9% precision and 53.0% recall.
• High-confidence no-problem judgments had 90.8% precision and 64.8% recall.
• The precision of no-problem judgments decreased in the later period.
• The research scope is limited to binary classification of Windows endpoint alerts.

Inference

• It is safer to introduce AI for prioritizing attack-like alerts rather than for auto-closure.
• In actual operations, precision separated by attacks and no-problems should be prioritized over overall accuracy.
• If humans do not re-examine a subset of auto-closed results, it is difficult to detect drops in accuracy.

Hypothesis

• Adding structured authentication, network, and cloud information might improve the ability to judge post-compromise activity, but this was not verified in this research.
• Adjusting judgment thresholds with each organization's data might suppress accuracy drops caused by organizational differences.

15. Unknowns and Additional Verification

• Can the same accuracy be achieved in other organizations or with other security products?
• Which types of attacks were most easily missed?
• How much does accuracy drop when input fields are missing?
• What is the resistance if attackers mix in information designed to cause misjudgments?
• Can this be applied to non-Windows endpoints, authentication, cloud, and networks?
• Are datasets and models publicly available so third parties can reproduce the results?

16. Impact on Global SOCs and Target-Specific Summaries

Impact on Global SOCs and Enterprises

While AI triage can reduce the number of items to review, its accuracy drops over time. When implementing it, teams must check not only the overall accuracy rate, but also the rate of attacks mistakenly judged as no-problems, changes over time, and the rate of human judgment overrides.

For SOC Teams

Use AI answers as a reference to determine investigation order rather than as final decisions. When auto-closing alerts, have humans re-examine a subset and monitor the misjudgment rate per time period.

For Management

Start with a test run that does not perform live processing. Automate only within ranges where accuracy is stable, and prepare stop and rollback mechanisms.

For End Users

This is not a system directly operated by general users. If an endpoint is mistakenly isolated by an AI judgment, users should report their immediately preceding actions and the business impact to the SOC.