DEV Community

Cover image for AUC 0.953 at $0 per 1,000 emails: explainable phishing detection that runs locally (open source and measured)
gj0xv
gj0xv

Posted on

AUC 0.953 at $0 per 1,000 emails: explainable phishing detection that runs locally (open source and measured)

#ai #python #security #opensource

When a security analyst flags an email as phishing, the first question is always "why". A black-box classifier that says "97% phishing" does not answer it. You need the reasons, the specific signals, the things a human can verify.

laya-phishield decomposes the phishing decision into eight narrow yes/no signals, checks deterministic email headers first, and combines everything into a risk score with visible feature contributions. It runs on your machine, for free.

AUC: 0.953 (composite)
Recall at 0.5 threshold: 0.808
Cost per 1,000 emails: $0
Enter fullscreen mode Exit fullscreen mode

How it decides

Two layers, one pass:

Layer 1: deterministic pre-pass. Before any model runs, the tool checks email headers, reply-to mismatches, punycode and IP-literal links, homoglyphs in domains, and SPF/DKIM failures. These are hard signals: no ML needed, no ambiguity.

Layer 2: eight yes/no signals. The laya decision engine (local, CPU) scores each email on:

  • asks_credentials (does it request login or banking details)
  • urgency_pressure (artificial time pressure)
  • brand_impersonation (mimics a known brand)
  • external_link_risk (suspicious outbound links)
  • and four more

A logistic regression head combines them into a final score. The output is not "phishing: yes". It is:

risk_score: 0.87
top_contributors:
  brand_impersonation: +0.34
  urgency_pressure: +0.21
  external_link_risk: +0.18
  asks_credentials: +0.14
Enter fullscreen mode Exit fullscreen mode

An analyst can verify each of those in seconds. That is what explainable means in practice.

The benchmarks

183 emails (105 legitimate, 78 phishing), temporal split, from the Nazario phishing corpus and Enron, deduplicated with MinHash/LSH:

Method AUC Precision @ 0.5 Recall @ 0.5 Cost / 1,000
Keyword rules 0.594 1.000 0.051 $0
Forced choice (laya) 0.944 0.902 0.590 $0
Composite (laya + pre-pass) 0.953 0.913 0.808 $0

The keyword baseline catches almost nothing. The laya model alone is strong. The composite (deterministic checks + ML scoring) is the best, because header failures and homoglyphs are signals a language model should not have to relearn.

Baselines were generated and judged with Z.ai's GLM-5.3-flashX. Local decisions cost $0; only the benchmark pipeline calls a hosted model.

The honest trade-off: at 95% recall, the forced-choice variant has a lower false-positive rate (0.124 vs 0.210 for composite). If you need fewer false alarms and can accept more misses, use forced choice. The README shows the full curve.

Try it 

pip install laya-phishield
Enter fullscreen mode Exit fullscreen mode

Scan an email from the CLI:

laya-scan suspicious.eml
Enter fullscreen mode Exit fullscreen mode

Or serve it as an API:

laya-serve --port 8080
# POST /scan with raw .eml body
Enter fullscreen mode Exit fullscreen mode

Streamlit demo included for visual inspection.

Disclaimer: this is not a replacement for a production secure email gateway. It is an explainable, local signal layer for analysts and researchers. The README says this too.

The rest of the series

This is the last of four open-source tools built on the laya decision engine, the same idea that OpenAI shipped as the Decisions API on Luna and TypeSafe shipped as Jev this month. The difference is where it runs: your machine, your data, $0 per decision.

  • laya-router: route prompts between cheap and frontier models, 54.9% cost reduction
  • laya-compactor: cut 70% of RAG context tokens, same answer quality
  • laya-triage: support ticket triage, fine-tuned from 51% to 90.5% intent accuracy

Every benchmark is committed with the code. Including the numbers that hurt.

Top comments (0)