Every morning, a DLP analyst logs in to thousands of security alerts. Most are the same pattern as yesterday — the same HR manager printing the same payroll report, the same sales rep emailing the same customer list. Yet each one runs through an expensive LLM pipeline, burns tokens, and produces the same verdict as the last 100 times.
This isn't security work. It's assembly line labor priced at LLM rates.
I spent the last few weeks building something to fix this: a DLP Alert Triage Agent with autonomous memory. The system gets smarter every time an analyst does their job — without requiring them to do anything extra. No "train" button. No rule writing. No prompt tuning.
The Problem With Existing DLP Systems
Traditional DLP tools have a fundamental blind spot: they treat every alert as if it's the first time they've seen it.
An analyst can correct the same false positive 50 times, and the system will happily make the same mistake on attempt #51. There's no feedback loop between the analyst's decision and the model's behavior. Some teams try to work around this with custom rules or fine-tuning, but rules are brittle, fine-tuning is expensive, and neither scales across hundreds of alert types and dozens of departments.
The real issue isn't model accuracy — it's that the system doesn't learn from experience. An analyst shouldn't have to re-litigate the same alert twice.
Architecture: Algorithms Before LLM, Memory After the Analyst
I designed the pipeline around a simple principle: don't pay for what you can compute, and remember what you've already learned.
The pipeline has four stages, each capable of short-circuiting the rest:
Stage 1 — Metadata Analysis. Who initiated the action? What file? What time? What destination? Some alerts are clearly safe or clearly suspicious based on metadata alone. These never proceed further.
Stage 2 — Algorithms. Shannon entropy detects encoded or encrypted payloads. Structural fingerprinting identifies known file types and detects tampering. Encoding detection catches obfuscation attempts. This stage uses zero LLM tokens — pure applied math. In practice, it catches about 20% of clearly malicious content before any model is invoked.
Stage 3 — Memory Recall. This is where the system's learned memory lives. Every past analyst decision has been distilled into a pattern — a Markdown file with YAML frontmatter describing the conditions, the correct outcome, and the confidence level. If a new alert matches an existing pattern, the system returns the cached verdict in approximately 50 milliseconds with zero token cost. This stage is responsible for the bulk of the efficiency gains.
Stage 4 — LLM Pipeline. Only reached for genuinely novel alerts. A Lite LLM performs initial triage, then two Specialist Agents analyze independently. If they disagree, they enter up to two rounds of structured debate — each sees the other's reasoning and may revise its verdict. A Risk Scorer produces the final decision.
How Memory Works
This was the hardest part to get right. The memory system needs to be autonomous, self-organizing, and human-readable.
Every learned pattern is a Markdown file with YAML frontmatter:
---
conditions:
role: HR Manager
action: print
destination: printer
confidence: 0.92
hitCount: 14
created: 2026-07-17
lastMatched: 2026-07-19
outcome: safe
---
HR managers routinely print payroll documents for distribution.
No policy violation detected. Pattern confirmed by analyst 14 times.
The system manages its own memory with zero human intervention:
-
Score-based retrieval. When searching for matching patterns, the system scores each candidate by
hitCount × 0.35 + confidence × 0.35 + recency × 0.30. The most relevant patterns enter the prompt. - Exact match bypass. If conditions match exactly, the verdict is returned instantly without touching the context budget.
- Automatic compaction. When the pattern count exceeds 50, the system runs a maintenance cycle: cull low-confidence stale patterns that haven't matched recently, merge overlapping patterns into a single higher-confidence rule, and refresh the scores of still-active patterns.
- Confidence decay and boost. Confidence drops by 5% per week if a pattern isn't matched. Every correct match adds 0.05. Every analyst contradiction costs 0.20. This ensures the system adapts to changing behavior without manual intervention.
The Hardest Challenge: Memory Merging
The most difficult engineering problem was merging overlapping patterns.
Consider two patterns: one says "HR manager printing payroll is safe," another says "HR manager printing Q4 payroll is safe." Should the system merge them? Keep both? Let the more specific one supersede?
Our first attempt was too aggressive — it merged everything, creating patterns so broad they matched irrelevant alerts. The second attempt was too conservative — it kept every pattern separate, and the memory count grew uncontrollably.
The third iteration finally worked: a dedicated merge agent, powered by Qwen3.7-plus, that analyzes both patterns and decides whether to merge, keep separate, or let one supersede the other. The merge agent considers semantic overlap, confidence scores, hit counts, and recency before making its decision. This runs as part of the compaction cycle and has proven remarkably reliable.
Built Entirely on Qwen Cloud
The system uses Qwen Cloud for everything:
- Qwen3.7-plus handles all reasoning tasks — Lite LLM triage, Specialist Agent analysis, structured debate, memory pattern extraction, and merge decisions.
- Qwen-VL-Plus processes screenshots and images that accompany alerts (compressed via sharp before submission to manage token costs).
- The backend is deployed on Alibaba Cloud ECS, with better-sqlite3 for durable caching and the Qwen API as the sole external dependency.
For document processing, the system uses pdf-parse, mammoth (for .docx), and xlsx (for Excel files), so it supports the full range of file types that appear in real DLP workflows.
Results
In testing across 16 scenarios spanning three companies (Bank Corp, Healthcare+, SaaS Inc) and six employee roles:
- Token savings: 35–75% per alert, with most common patterns short-circuiting before the LLM stage
- Short-circuit response time: ~50ms for exact memory matches
- Memory self-organization: Maintains under 50 active patterns through automatic culling and merging
- Multi-agent debate: Reduced false positives by approximately 60% compared to single-agent analysis in contentious cases
Key Takeaways
If I could share one insight from this project, it's this: most AI systems don't need to be smarter — they need to be better at remembering what they've already figured out.
The obsession with model capabilities often distracts from a simpler truth: the cheapest LLM call is the one you don't make. By investing in algorithmic pre-filtering and persistent memory, you can build systems that are simultaneously more efficient and more accurate than a pure LLM approach.
The code is open source under MIT: github.com/Ra-226/dlp-triage-agent And if you're building something similar, I'd love to hear about your approach to the memory merging problem. It's harder than it looks.
Top comments (0)