DEV Community: gj0xv

AUC 0.953 at $0 per 1,000 emails: explainable phishing detection that runs locally (open source and measured)

gj0xv — Thu, 01 Oct 2026 11:10:01 +0000

When a security analyst flags an email as phishing, the first question is always "why". A black-box classifier that says "97% phishing" does not answer it. You need the reasons, the specific signals, the things a human can verify.

laya-phishield decomposes the phishing decision into eight narrow yes/no signals, checks deterministic email headers first, and combines everything into a risk score with visible feature contributions. It runs on your machine, for free.

AUC: 0.953 (composite)
Recall at 0.5 threshold: 0.808
Cost per 1,000 emails: $0

How it decides

Two layers, one pass:

Layer 1: deterministic pre-pass. Before any model runs, the tool checks email headers, reply-to mismatches, punycode and IP-literal links, homoglyphs in domains, and SPF/DKIM failures. These are hard signals: no ML needed, no ambiguity.

Layer 2: eight yes/no signals. The laya decision engine (local, CPU) scores each email on:

asks_credentials (does it request login or banking details)
urgency_pressure (artificial time pressure)
brand_impersonation (mimics a known brand)
external_link_risk (suspicious outbound links)
and four more

A logistic regression head combines them into a final score. The output is not "phishing: yes". It is:

risk_score: 0.87
top_contributors:
  brand_impersonation: +0.34
  urgency_pressure: +0.21
  external_link_risk: +0.18
  asks_credentials: +0.14

An analyst can verify each of those in seconds. That is what explainable means in practice.

The benchmarks

183 emails (105 legitimate, 78 phishing), temporal split, from the Nazario phishing corpus and Enron, deduplicated with MinHash/LSH:

Method	AUC	Precision @ 0.5	Recall @ 0.5	Cost / 1,000
Keyword rules	0.594	1.000	0.051	$0
Forced choice (laya)	0.944	0.902	0.590	$0
Composite (laya + pre-pass)	0.953	0.913	0.808	$0

The keyword baseline catches almost nothing. The laya model alone is strong. The composite (deterministic checks + ML scoring) is the best, because header failures and homoglyphs are signals a language model should not have to relearn.

Baselines were generated and judged with Z.ai's GLM-5.3-flashX. Local decisions cost $0; only the benchmark pipeline calls a hosted model.

The honest trade-off: at 95% recall, the forced-choice variant has a lower false-positive rate (0.124 vs 0.210 for composite). If you need fewer false alarms and can accept more misses, use forced choice. The README shows the full curve.

Try it

pip install laya-phishield

Scan an email from the CLI:

laya-scan suspicious.eml

Or serve it as an API:

laya-serve --port 8080
# POST /scan with raw .eml body

Streamlit demo included for visual inspection.

Disclaimer: this is not a replacement for a production secure email gateway. It is an explainable, local signal layer for analysts and researchers. The README says this too.

The rest of the series

This is the last of four open-source tools built on the laya decision engine, the same idea that OpenAI shipped as the Decisions API on Luna and TypeSafe shipped as Jev this month. The difference is where it runs: your machine, your data, $0 per decision.

laya-router: route prompts between cheap and frontier models, 54.9% cost reduction
laya-compactor: cut 70% of RAG context tokens, same answer quality
laya-triage: support ticket triage, fine-tuned from 51% to 90.5% intent accuracy

Every benchmark is committed with the code. Including the numbers that hurt.

From 51% to 90.5%: fine-tuning a local triage model on 10,003 support tickets (open source and measured)

gj0xv — Thu, 01 Oct 2026 11:09:49 +0000

When OpenAI launched the Decisions API on Luna this week, and TypeSafe launched Jev two weeks before that, they were both making the same bet: the future of AI in production is not bigger models, it is smaller, faster decision models embedded in your workflow.

I agree. And I have the numbers to prove it works.

laya-triage routes support tickets through a local decision model: intent (77 categories), department (8), urgency, frustration, churn risk, refund flag, and a confidence-gated escalation to a human. All in one forward pass, on CPU, for free.

Hierarchical intent accuracy: 51.0% -> 90.5% (after fine-tuning)
Coarse-cluster accuracy: 68.0% -> 96.0%
Urgency MAE: 0.81 -> 0.70
Cost per ticket: $0

The architecture

Flat 77-way classification is hard for a small model. So I broke it into two stages:

Ticket -> [coarse stage: 12 clusters] -> [fine stage: 3-10 candidate intents] -> decision
              |
              +-> urgency, frustration, churn, refund (same forward pass)
              |
              +-> confidence check -> escalate to human if below 0.84

The hierarchy alone added 14.5 percentage points over flat routing. The fine-tune added another 39.5.

What I fine-tuned on

10,003 BANKING77 tickets, split into two sequences per ticket, trained on Kaggle's free 2x T4 GPUs. The checkpoint is published on HuggingFace: Gjusev/laya-triage-banking77.

The eval harness that tracked the improvement ran on Z.ai's GLM models, with GLM-5.3-flashX as the workhorse. When the judge model changes, the numbers move a little. The fine-tune gain does not.

The results, before and after

Measurement	Zero-shot	Fine-tuned
Hierarchical intent accuracy	51.0%	90.5%
Macro-F1	not measured	0.848
Coarse-cluster accuracy	68.0%	96.0%
Urgency MAE	0.81	0.70
Frustration MAE	1.07	1.07 (untouched heads)

At the 0.84 confidence threshold (zero-shot): 75.2% accuracy with 60.5% of tickets auto-handled. The rest escalate to a human with a recorded reason.

The escalation policy (and why it matters)

The threshold gates only on cluster and intent confidence. Signal confidence (urgency, frustration, churn) is deliberately not used, because score-type questions have structurally lower confidence and would trigger near-universal handoffs.

On an 8-ticket out-of-domain smoke test (IT-ops tickets against banking intents): 7 escalated correctly. The one that did not, an SSO lockout, mapped to "unable to verify identity". Not wrong, exactly, but the reason it matters is that the behavior is visible and fixable, not hidden behind an opaque binary.

Try it

The demo is live on Streamlit, in 6 languages (English, Spanish, French, German, Hindi, Arabic):

Live demo

Or run it yourself:

pip install laya-triage

Fine-tuning notebook included for Kaggle (2x T4, ~4 hours).

The rest of the series

laya-router: route prompts between cheap and frontier models, 54.9% cost reduction
laya-compactor: cut 70% of RAG context tokens, same answer quality
laya-phishield: explainable phishing detection, $0 per 1,000 emails

All four tools use the same laya decision engine, Apache 2.0, local, with published benchmarks. If OpenAI's Decisions API and TypeSafe's Jev are the proprietary versions of this idea, this series is the open one. Run it on your machine, keep your data, pay nothing per decision.

Cutting 70% of RAG context tokens and keeping the answers identical (open source and measured)

gj0xv — Thu, 01 Oct 2026 09:30:00 +0000

Your RAG pipeline retrieves 12 chunks because the retrieval score said "maybe". Your LLM reads all of them. You pay for all of them. And the answer quality was decided by chunks 2 and 7 anyway.

On September 29, OpenAI launched the Decisions API built on Luna, and on September 15, TypeSafe launched Jev. Both are fast decision models for exactly this kind of problem: deciding what matters before the expensive model runs. But they are APIs. Your documents leave your machine, and you pay per decision.

laya-compactor does the same thing locally, for free. It scores your entire retrieval batch in a single forward pass and decides which documents stay in context and which get cut.

70% fewer input tokens
Zero answer-quality loss on SQuAD (exact match: 0.345 vs 0.345)
94.5% of gold documents retained on HotpotQA
$0 per compaction decision

How it works

One principle drives the design: delete, do not rewrite.

The tool never summarizes or paraphrases your documents. It keeps the ones that matter verbatim and removes the rest, with an auditable reason (low score or exhausted budget). The evidence stays intact for the LLM to reason over.

+--------------------------------------------------+
|  Retrieval batch (12 docs, ~3,200 tokens)         |
|                                                   |
|  laya decision engine (one forward pass, local)   |
|  scores each doc: 0=irrelevant .. 3=essential     |
|                                                   |
|  Budget: keep top-scoring docs until limit        |
|                                                   |
|  Output: 4 docs, ~970 tokens, same answer         |
+--------------------------------------------------+

Python API and CLI:

from laya_compactor import compact

result = compact(
    question="What is the refund policy for annual plans?",
    documents=retrieved_chunks,
    token_budget=1000,
)
# result.kept: the documents that survived
# result.dropped: [{doc, score, reason}] for the ones cut

The benchmarks (all of them)

200 questions per dataset, BM25 retrieval, generator and blind judge powered by Z.ai's GLM-5.3-flashX:

Dataset	Full context	Compacted	Tokens saved
SQuAD	EM 0.345, 3,214 avg tokens	EM 0.345, 973 avg tokens	69.7%
HotpotQA	EM 0.230, 3,193 avg tokens	EM 0.200, 947 avg tokens	70.3%

On SQuAD the exact match is identical: the compactor removed noise without touching signal. On HotpotQA it dropped 3 points but kept 94.5% of the gold documents, meaning the loss came from multi-hop reasoning, not from cutting the wrong chunks. Truncation baselines (head-only, tail-only) scored worse on both.

Caveats, because they exist: CPU latency is significant (6.3 to 10 seconds p50 for a batch). Multi-hop questions remain hard. The rubric sensitivity study is committed with a hand-labeled 100-row dataset so you can see exactly where the scoring is fragile.

Integrations

Drop-in for the two frameworks you already use:

# LangChain
from langchain.retrievers import ContextualCompressionRetriever
retriever = ContextualCompressionRetriever(
    base_compressor=LayaCompactor(token_budget=1000),
    base_retriever=your_retriever,
)

# LlamaIndex
from laya_compactor.integrations import LayaNodePostprocessor
query_engine = index.as_query_engine(
    node_postprocessors=[LayaNodePostprocessor(token_budget=1000)],
)

Try it

pip install laya-compactor

CLI included: laya-compact --question "..." --docs docs/ --budget 1000

The rest of the series

This is the second of four tools built on the same idea: local, measured decisions in LLM pipelines.

laya-router: route prompts between cheap and frontier models, 54.9% cost reduction
laya-triage: support ticket triage, fine-tuned from 51% to 90.5% intent accuracy
laya-phishield: explainable phishing detection, $0 per 1,000 emails

Each one ships with its benchmarks in the README, including the numbers that hurt.

How I cut my LLM bill by 54.9%: a local router that costs $0 per decision (open source and measured)

gj0xv — Thu, 01 Oct 2026 06:30:00 +0000

This week, two of the biggest names in AI shipped the same idea within days of each other. On September 15, TypeSafe launched Jev, the first "System One model": an AI that outputs typed decisions instead of text. On September 29, OpenAI answered at DevDay with the Decisions API built on Luna, their fast, cheap model for real-time decision-making.

The premise is right. The pricing usually isn't.

That is where open source matters. When the decision layer runs on your machine, with your data, at $0 per call, the economics flip: every prompt gets the right-sized model instead of the one that maximises someone else's margin. You can audit it, fork it, and turn it off when it does something stupid. No vendor slide can promise you that.

So here is mine: laya-router, the first of four open-source tools that put local, measured decisions between your app and your LLMs, built on the laya decision engine. Benchmarked with Z.ai's new GLM-5.3-flashX as the cheap tier, which is itself a fitting story: a fast, cheap model deciding which prompts it can handle itself.

54.9% estimated cost reduction
80.6% of prompts routed to the cheap tier
$0 per routing decision
460 ms warm latency (p50)

The idea

You keep your code exactly as it is. You only change the base_url. The proxy ignores the model you asked for and picks one itself:

client = OpenAI(base_url="http://localhost:8000/v1", api_key="...")
# from here on, everything is standard

Three layers decide:

Regex fast path: trivial prompts ("hi", "translate: merci") skip the model entirely
A local decision model (the laya engine, Apache 2.0, runs on CPU) classifies the prompt as simple, standard or complex, with calibrated confidence
A confidence gate: uncertain prompts escalate to the frontier tier. The system is allowed to say "I don't know" and pay more

Every response carries inspection headers, so you can audit what happened and why:

X-Laya-Route: cheap
X-Laya-Model: glm-4.6-flash
X-Laya-Confidence: 0.91
X-Laya-Reason: classified simple, above threshold

The benchmarks (all of them)

180 prompts, one cheap and one frontier model, blind LLM judge. I publish the wins and the caveats:

Measurement	Result
Routed to cheap tier	80.6%
Estimated cost reduction	54.9%
Cheap win/tie rate vs frontier	79.3%
Routing latency p50 / p95 / p99	460 ms / 1.4 s / 2.7 s

Caveats, because they exist: one judge, one model pair, and the judge was noisy on trivial prompts. The backtest pipeline is committed (make backtest) with the dataset, the answers and the judgements, so you can re-run it on your own traffic. If your prompts are all hard, you will save nothing. Measure first.

What it deliberately does not do

No silent failover. If routing fails, you get a structured 503, not a surprise GPT-4 call on your bill
No rewriting. Your prompt arrives at the upstream model byte-identical. The proxy decides, it does not mutate
No vendor lock. Works with OpenAI, vLLM, Ollama, OpenRouter, Z.ai, anything OpenAI-shaped

Try it

pip install laya-router

Define your tiers in YAML (models + prices), point your base_url at it, done. Docker image included, /metrics endpoint for Prometheus, JSONL decision log if you want to analyze your routing later.

The rest of the series

laya-router is one of four tools I'm building around the same idea: local, measured decisions in LLM pipelines.

laya-compactor: cut 70% of RAG context tokens with zero answer-quality loss (measured)
laya-triage: support ticket triage, fine-tuned from 51% to 90.5% intent accuracy
laya-phishield: explainable phishing detection, $0 per 1,000 emails

Each one ships with its benchmarks in the README, including the numbers that hurt.

If your LLM app ever died in production after a perfect demo, we have something to talk about :)

Judge cheap, audit confidence: a CI gate for LLM evals (open source and measured)

gj0xv — Thu, 01 Oct 2026 06:00:00 +0000

Your LLM passes the demo every time. Then a customer sends a prompt you didn't test, and the answer is garbage. The gap between "it worked in the eval" and "it works in production" is almost always the same: you measured accuracy, but you never measured whether the model's confidence was trustworthy.

laya-evals closes that gap. It is the fifth tool in my open-source series on the laya decision engine, and the one that keeps the other four honest.

Judge accuracy: 0.8968 (Cohen's kappa 0.7938)
Calibration: ECE 0.0253, Brier 0.0779
Cost per 1,000 judgments: $0 (local inference)
CI gate: fails the build when accuracy or calibration regresses

What it does

Three things, in one pipeline:

1. Rubric-based LLM-as-judge. Score your eval sets with a local laya decision model instead of paying for GPT-4 as a judge. Choice questions, score questions, or no-ground-truth mode. 6.4 decisions per second on CPU.

2. Calibration audit. Accuracy tells you how often the model is right. Calibration tells you whether you can trust its confidence. laya-evals computes Expected Calibration Error, Brier score, and reliability bins, then tells you which confidence threshold to use for each question shape.

Why per shape? Because a 2-option question needs a different gate than a 20-option question. The SST-2 benchmark (2 options) needs a threshold of 0.865 for 95% accuracy. The MASSIVE intent benchmark (20 options) needs 0.994. One global min_confidence is not a policy, and laya-evals proves it with your own numbers.

3. CI regression gate. Wire it into GitHub Actions. Exit code 0 means pass, exit code 1 means a measured regression, exit code 2 means the run itself was invalid. Your eval suite becomes a build gate, not a suggestion.

The numbers

872 SST-2 sentences, 2-option rubric, local laya judge:

Measurement	Result
Judge accuracy vs gold	0.8968
Cohen's kappa	0.7938
ECE (15 bins)	0.0253
Brier score	0.0779
Throughput	6.4 decisions/s
API cost	$0

Cross-checked against Z.ai's GLM-5.3-flash as reference judge: 95.1% accuracy, 90.7% agreement, kappa 0.814. The local judge is cheaper and nearly as reliable.

I also re-measured four public laya benchmark claims with laya-evals and got values within 0.0003 of the published numbers. The reproduction pack is committed.

Try it

pip install laya-evals

from laya_evals import Judge, ece, brier_score, advise_thresholds

judge = Judge()
results = judge.run(eval_set, rubric)
print(ece(results.confidences, results.correct))
print(advise_thresholds(results))

The series

This is the fifth and last tool in the laya series, built on the laya decision engine by NandhaKishorM. The same engine that TypeSafe shipped as Jev and OpenAI shipped as the Decisions API on Luna this week. The difference: this one runs on your machine, for free, and the benchmarks are committed.

laya-router: 54.9% cost reduction, $0 per routing decision
laya-compactor: 70% fewer RAG tokens, same answer quality
laya-triage: support triage from 51% to 90.5% after fine-tuning
laya-phishield: explainable phishing detection, $0 per 1,000 emails

Every tool ships with its benchmarks, including the numbers that hurt.