Microsoft's AI Defense Research: Generating Detection Test Logs from Attack Procedures

1. Basic Information

• Article Title: Accelerating detection engineering using AI-assisted synthetic attack logs generation
• Publisher: Microsoft Security Blog / Microsoft Defender Security Research Team
• Publication Date: May 12, 2026
• Importance: High (Practical value for defense research and testing environments)
• Original Source: Microsoft Security Blog
• Related Sources: MITRE ATT&CK / OTRF Security Datasets / ATLASv2 Paper
• Target: Attack logs used for designing and testing detection rules
• Main Methods: Prompt-based generation, multi-AI collaboration, LLM-as-a-Judge, reinforcement learning with verifiable rewards

2. Summary in One Sentence

This research takes MITRE ATT&CK attack techniques and specific attack steps, feeds them to an AI, and creates detection test logs that include process names, parent processes, and command lines. In experiments, a method where multiple AIs share the roles of generation, review, and correction worked best, but synthetic logs are not proof of real-world attack reproduction and are limited to supporting lab tests.

Terms Used in This Report

• Synthetic log: An artificially created log made by AI or other tools, not recorded by a real machine
• TTP: Information showing what an attacker wants to achieve, what techniques they use, and how they execute them
• Ground truth: Logs obtained from real attack reproductions used as a comparison standard
• Recall: How well synthetic logs include important events found in ground truth logs
• LLM-as-a-Judge: A method where another AI evaluates the validity of generated results

3. Defense Challenges to Solve

• Real attack logs happen rarely, making it hard to collect enough volume
• Collecting attack logs, labeling them, and reconstructing attack scenarios take time and money
• Reproducing real attacks in a lab is accurate, but trying many techniques in a short time is difficult
• Real logs from customer environments contain sensitive data, making them hard to share or reuse
• A lack of logs delays the design and evaluation of detection rules

4. Proposed Methods and Mechanisms

The system gives "attack techniques and specific operations" to an AI and generates structured logs that should appear when those operations run.

The goal of this research is not to reproduce real logs word for word. The goal is to create logs that keep the meaning, parent-child process relationships, command contents, and event order needed for detection rules to trigger.

5. AI Inputs and Outputs

Inputs

• MITRE ATT&CK tactics and techniques
• Specific operations executed by the attacker
• Context such as the target OS or attack scenario
• For multi-step attacks, preceding/following operations and host relationships

In the article's example, T1202 (Indirect Command Execution) combined with forfiles, environment variables, hex representation, and Python is shown as the input.

Outputs

• Newly started process name
• Parent process name
• Command line
• Order of multiple events
• Related logs when spanning multiple hosts
• Structured attack logs usable for testing detection rules

6. AI Processing Flow / System Architecture

Method 1: Prompt-Centric

1. An expert gives instructions on the attack scenario and conditions
2. The AI generates logs through multiple interactions
3. Another AI evaluates realism and consistency

This works for simple scenarios, but results became unstable in complex multi-step attacks.

Method 2: Iterative Improvement by Three AIs

1. The Generator AI creates the first log
2. The Evaluator AI points out missing parts and contradictions
3. The Improver AI creates a revised version based on the feedback
4. Repeat generation, evaluation, and improvement

This method helped fill in missing events in complex attacks and kept parent-child processes and relationships in order.

Method 3: Reinforcement Learning Using Ground Truth Logs

1. The AI generates synthetic logs
2. The Evaluator AI compares the meaning and strings with ground truth logs
3. Partial points are given for matching meanings
4. Points are deducted for parts where strings do not match the ground truth
5. Generation methods are improved using evaluation reasons and scores

While this method has potential, it required a large amount of labeled training data.

7. Evaluation Design

Evaluation Data

1. Goal-Driven Campaigns: 10 attack reproductions made by Microsoft researchers in a repeatable format
2. OTRF Security Datasets: Public datasets including attack and normal logs from multiple environments
3. ATLASv2: Windows Security, Sysmon, Firefox, and DNS logs obtained by running 10 multi-step attacks on two Windows virtual machines

For ATLASv2, the evaluation was limited to malicious activity during the attack time window.

Evaluation Method

• The main metric is recall
• Compare synthetic logs and ground truth logs by matching meaning, not exact string matching
• Main comparison items: new process name, parent process name, and command line meaning
• Compared prompt-centric methods with multi-AI collaborative methods
• Used multiple reasoning models and non-reasoning models

For example, even if the synthetic log has forfiles.exe and the ground truth log has the full path D:\Windows\System32\forfiles.exe, it can be evaluated as a match because they represent the same process.

8. Evaluation Results

• The prompt-centric method served as a baseline, but results were unstable
• The method dividing tasks into generation, evaluation, and improvement greatly improved recall across all evaluation data
• Combining reasoning models with iterative improvement generated the most realistic logs
• Synthetic logs successfully expressed multiple events, parent-child process relationships, and realistic command lines
• The reinforcement learning method showed potential, but required a large amount of labeled data to train properly
• The article's tables show recall rates by method, but the text does not show a single number representing all methods

9. Practical Meaning of Evaluation Results

• Synthetic logs are good for speeding up the initial design of detection rules and checking for missing items
• They let organizations create test materials for rare or new attacks without sharing real data
• A structure where another AI checks and corrects logs after generation is important, rather than a single generation step
• Even if process names match, wrong paths, arguments, service names, or event orders will cause detection tests to fail
• Even if a rule triggers on a synthetic log, it does not prove that necessary logs will be collected in a real environment or that real attacks can be detected

10. Success Conditions

• Input TTPs and specific attack operations are detailed enough
• Target OS, log types, and required fields are clearly specified
• Generated results are confirmed using ground truth logs or expert reviews
• Process parent-child relationships, event orders, and host relationships are verified
• The use of synthetic logs is limited to detection design, unit testing, and lab assistance
• Final confirmation is done in a real machine lab or production-equivalent environment

11. Failure Conditions and Limitations

• Passing only TTP names without giving specific attack operations or environment information
• Treating paths, arguments, and service names created by AI as absolute facts
• Adopting generated results without ground truth logs or human review
• Confusing detection success on synthetic logs with successful log collection in real environments
• Not including normal logs, failing to evaluate false positive rates
• Failing to prepare enough labeled data required for reinforcement learning

12. Deployment and Verification Steps

1. Define the target ATT&CK technique and specific attack operations
2. Decide the log types and fields needed for detection
3. Generate synthetic logs using AI
4. Have another AI and humans check parent processes, commands, and timelines
5. Input synthetic logs into the detection engine and check if rules trigger
6. Add normal logs and check if false positives increase
7. Run the same attack operations in a real machine lab
8. Compare actually collected logs with synthetic logs
9. Reflect differences into generation conditions and detection rules

Judgment Categories

• Generated: AI only generated logs
• Reviewed: Humans or another AI checked the contents
• Detection Unit Test Passed: Detection rules triggered on synthetic logs
• Lab Telemetry Confirmed: Necessary logs were confirmed in a real machine lab
• End-to-End Detection Confirmed: Confirmed everything from real attack reproduction to alert generation

13. Required Records and Audit Trails

• Input ATT&CK techniques, specific operations, and target environment
• Models used, prompts, generation settings, and execution timestamps
• Generated logs and each revised version
• Evaluator AI feedback, scores, and evaluation reasons
• Contents modified and approved by humans
• Ground truth logs used and their origins
• Detection rule versions, test results, and false positive test results
• Real machine lab commands, collection settings, real logs, and alerts

14. Facts / Inference / Hypothesis

Facts

• Microsoft evaluated a method to generate structured logs from "TTPs and specific operations"
• Output examples include command lines, process names, and parent process names
• Covered three methods: prompt-centric, iterative improvement by three AIs, and reinforcement learning
• Evaluated using three types of data: 10 Goal-Driven runs, OTRF, and ATLASv2
• Main evaluation items were new process names, parent process names, and command line meanings
• Iterative improvement by multiple AIs improved recall over the prompt-centric method
• The reinforcement learning method required a large amount of labeled data
• Microsoft does not position synthetic logs as a complete replacement for lab testing

Inference

• Good for quickly creating unit test data for detection rules from attack scenarios
• Splitting generation, evaluation, and improvement into separate roles makes it easier to find errors than generating only once
• Differences between synthetic logs and real lab logs serve as material to find missing log collections or scenario descriptions

Hypothesis

• Tuning with organization-specific log formats and ground truth logs could reduce item differences between products
• If normal logs can be generated at the same time as attack logs, detection rates and false positive rates might be evaluated in the same test

15. Unclear Points and Additional Verification

• Can third parties reproduce the numbers in the tables for each model and method under the same conditions?
• Full details of prompts, generation settings, and model versions used
• Application performance outside Windows, such as cloud audit logs, SaaS, and network devices
• False positive rates when normal logs are included
• How many event omissions or order errors occur in long attack chains
• Reproducibility when attackers intentionally add confusing operations
• Standard methods to safely feed synthetic logs into each SIEM and EDR

16. Impact and Role-Based Summaries

Impact on Security Operations

Organizations that do not have enough attack logs can start initial testing of detection rules more easily. Especially for new techniques or low-frequency attacks, teams can organize necessary log fields and detection conditions before running real tests. On the other hand, if detection effectiveness is approved based only on synthetic logs, teams will miss real-world problems such as logs not being collected, field names not matching, or normal operations causing false positives.

For SOC Teams

Use synthetic logs for initial testing of detection rules and coverage checks, but make final decisions using tests that include real machine logs and normal logs.

For Administrators

Keep records of models, prompts, ground truth logs, detection rule versions, and make generated results reproducible. Avoid feeding synthetic logs directly into production; handle them in isolated test environments.

For End Users

This is not a mechanism for general users to operate directly. What gets created is artificial logs for defense testing, which does not mean that user devices have actually been compromised.