Automating Ransomware Detection in Under 5 Minutes (Scripts, Heuristics, and Playbooks)

TL;DR: In penetration testing and blue-team operations, time is your most precious resource — especially with ransomware. This guide shows a 3-step pipeline we use to cut MTTD (mean time to detect) to under five minutes using file integrity monitoring, hybrid detection (signatures + heuristics), and automated containment. Works for internal SOC teams, managed security service providers, and cyber security companies globally.

Why this matters: Whether you’re running SOC services for a client, doing penetration testing as a service, or hardening a product team’s SDLC, early detection prevents mass encryption and reduces MTTR dramatically.

The 3-Step Pipeline

1) File Integrity Monitoring (FIM): Catch abnormal file behavior fast

Early signals of ransomware are almost always behavioral:

• Sudden bursts of file renames or creations
• New extensions (e.g., .locked, .encrypted, random suffixes)
• Rapid entropy increase or unusual write patterns in user shares

What to watch:

• Windows: USN Journal, Sysmon (Event ID 11/12/13), PowerShell 4104, SMB share activity
• Linux: inotify, auditd, access logs on NFS/SMB mounts
• App layer: unusual writes in upload folders, build artifacts, or backup directories

Quick watcher (Python, cross-platform – lab demo):

# pip install watchdog
from watchdog.observers import Observer
from watchdog.events import FileSystemEventHandler
import time, os

SUSPICIOUS_EXTS = {".locked", ".encrypted", ".crypt", ".enc"}

class RansomwareWatcher(FileSystemEventHandler):
    def on_modified(self, event):
        if not event.is_directory:
            _, ext = os.path.splitext(event.src_path.lower())
            if ext in SUSPICIOUS_EXTS:
                print(f"[ALERT] Suspicious extension: {event.src_path}")

    def on_created(self, event):
        if not event.is_directory:
            _, ext = os.path.splitext(event.src_path.lower())
            if ext in SUSPICIOUS_EXTS:
                print(f"[ALERT] Suspicious new file: {event.src_path}")

if __name__ == "__main__":
    path = "/path/to/watch"  # point to a test folder
    observer = Observer()
    observer.schedule(RansomwareWatcher(), path, recursive=True)
    observer.start()
    try:
        while True:
            time.sleep(1)
    except KeyboardInterrupt:
        observer.stop()
    observer.join()

Tip: Start with a test folder and a low-noise path. Expand coverage incrementally to avoid alert fatigue.

2) Hybrid Detection: Signatures + Heuristics for zero-day coverage

Signature scanning (AV engines, YARA rules) is excellent for known families. But new strains and fileless behaviors need heuristics:

Useful heuristics (combine multiple):

• Rename rate: e.g., >50 renames in 60 seconds within a single user home or share
• Entropy spike: suspicious increase across many files in a short window
• Extension burst: many files ending with unusual or randomized extensions
• Process behavior: process touching hundreds of files across disparate directories
• Shadow copy tampering: commands like vssadmin delete shadows or wmic shadowcopy delete

Pseudo-scoring (simplified):

score = 0
if rename_rate_last_60s > 50: score += 40
if entropy_spike_detected: score += 30
if extension_burst_detected: score += 20
if touched_files > 500: score += 20
if shadowcopy_event: score += 40

if score >= 60:
    verdict = "Likely ransomware"
else:
    verdict = "Investigate"

Signature layer ideas:

• Schedule YARA scans for known indicators (IOCs) on high-risk paths
• Integrate with your SIEM/XDR’s detection content for ransomware TTPs
• Keep a curated feed of rules from reputable cyber security companies and research groups

3) Automated Containment: Move faster than the malware

Once the heuristic + signature combo triggers, seconds matter. Your playbook should:

• Isolate the endpoint (EDR API, NAC quarantine, firewall block)
• Kill the offending process (EDR/OS tools)
• Disable SMB/NFS shares temporarily to prevent lateral spread
• Notify on-call and create a ticket with full context (host, user, process tree, paths, hashes)

Windows (PowerShell, lab-only example):

# Block outbound traffic quickly (lab demo)
New-NetFirewallRule -DisplayName "EmergencyContainment" -Direction Outbound -Action Block -Enabled True

# Optionally disable primary NIC (be careful!)
# Disable-NetAdapter -Name "Ethernet" -Confirm:$false

Linux (lab-only example):

# Block all outbound while you triage (lab demo)
sudo iptables -P OUTPUT DROP
# or detach from network manager
# nmcli networking off

Always run containment actions via a controlled SOAR/SIEM workflow with approvals outside the lab.

Deployment Patterns (pick your stack)

• SIEM + SOAR: Send watcher and EDR telemetry → rules/UEBA → playbook triggers containment
• EDR/XDR-first: Lean on vendor detections; add custom rules for your environment
• Lightweight cron/agents: For smaller cyber security firms or air-gapped networks, run agent scripts + syslog

Validation: Test safely before production

• Use harmless test files (e.g., EICAR) and rename bursts to simulate encryption behavior
• Rehearse your incident response: validate that tickets, alerts, and isolation fire within 5 minutes
• Track KPIs: MTTD, MTTR, false-positive rate, coverage (% of endpoints/shares monitored)

What to Log (so triage is painless)

• File paths, old/new names, counts per minute
• PID/PPID, user context, command line, hashes
• Network connections (dest IP/port), SMB share details
• System events: shadow copy deletion attempts, service changes

Example Playbook (YAML-style outline)

name: ransomware_rapid_detection
triggers:
  - rule: rename_rate_over_threshold
  - rule: suspicious_extension_burst
  - rule: shadowcopy_tamper_detected
actions:
  - gather_context: [process_tree, hashes, recent_events]
  - isolate_host: true   # via EDR/NAC
  - kill_processes: ["suspected_ransomware"]
  - disable_shares: ["//fileserver/projects", "//fileserver/home"]
  - notify: ["on_call_slack", "pagerduty"]
  - ticket: { type: "incident", severity: "high" }
post_actions:
  - snapshot_evidence
  - run_av_yara_scan
  - restore_shares_on_clearance

Where this excels

• Web application penetration testing environments (protect build servers and artifact stores)
• Managed security service providers standardizing detection across clients
• Indian/UK/USA cyber security companies building repeatable SOC content without vendor lock-in

For a deeper, step-by-step walkthrough with diagrams, see the full breakdown on Medium: Read the guide.

If you want a compact checklist and worksheets for your next tabletop exercise, we maintain free educational resources here: DefenceRabbit.