DEV Community: Ikkun

Jev vs a 310M encoder I trained myself: 750 rows, three tasks, two different winners

Ikkun — Mon, 21 Sep 2026 02:20:55 +0000

TL;DR: I ran three Japanese classification tasks (250 rows each, same gold labels) through six systems. A 310M encoder I fine-tuned on 250 labels beat TypeSafe's Jev on topic classification by +12.0 points (McNemar p=0.00007) and was 4–20× faster — but on the two polarity tasks it only tied Jev. The zero-shot open-source models everyone is writing about lost everywhere: a GLiNER-family classifier scored 62.8 / 87.2 / 44.8 against Jev's 76.8 / 94.4 / 74.0. The same architecture family, trained on my labels, scored 88.8 / 92.8 / 75.2.

One-line summary: A decision API wins when you have no labels; a small trained encoder wins when you have a few hundred — and which one wins on a given task is decided by the shape of the task, not by which model is trending.

The question behind the hype

Jev (TypeSafe AI's System One) is genuinely good, and it is cheap: $0.042 per 1M input tokens with output free, roughly 32k of context, a confidence number in the response, and no training data required. If you read the last two weeks of posts about it, the conclusion is "use Jev."

I use Jev. But I had a pile of results from my own tasks that did not fit that conclusion, and one of them was embarrassing enough that I re-ran it twice:

A char n-gram + logistic regression baseline beat Jev on long-document topic classification by 11.6 points (0.884 vs 0.768).
The same baseline lost to Jev by 20.8 points on short-text polarity (0.736 vs 0.944).

A 5-minute linear model beating a decision API on one task and losing badly on the next is not a story about either model. It is a story about task shape. So I designed the experiment I should have run first: hold the answers fixed, hold the rows fixed, and separate the two axes that actually differ — labels or no labels, and topic or meaning.

What I measured, and how

Three Japanese datasets, 250 rows each, frozen:

Task	Shape	Classes	Chance
livedoor news corpus	long-document topic (mean 1,174 chars)	9	11.1%
rakuten reviews	short-text polarity (mean 138 chars)	2	50.0%
chabsa	financial-sentence polarity (mean 92 chars)	3	33.3%

Six systems, two conditions:

Zero-shot (0 labels): Jev, a local 26B MoE (Gemma 4 26B-A4B, Q4_K_M, resident on the same mini-PC), SemIf (Qwen3.5-4B), GLiClass multilang-mini (a GLiNER-family classifier), Laya (typed decisions).
Trained (250 labels): sbintuitions/modernbert-ja-310m with a classification head, and a char n-gram + logistic regression baseline.

The trained numbers are 5-fold cross-validated out-of-fold predictions: train on 200, predict the held-out 50, five times, so all 250 reported predictions come from a model that never saw that row. That is the same protocol I use for the linear baseline, which means the trained row and the baseline row are directly comparable.

Everything is compared on identical row ids with McNemar's test. This matters: at n=250 the independent-confidence interval is about ±6 points, so "0.888 vs 0.944" cannot be called a difference without a paired test.

Results

System	Condition	livedoor (9)	rakuten (2)	chabsa (3)	p50 latency
ModernBERT-ja-310m	trained, 250 labels	88.8%	92.8%	75.2%	0.10–0.45s
Jev 1.13.0	zero-shot	76.8%	94.4%	74.0%	1.9–2.4s
Gemma 4 26B-A4B (local)	zero-shot	65.2%	94.4%	75.2%	1.24s
SemIf (Qwen3.5-4B)	zero-shot	68.8%	92.0%	64.0%	1.3–6.8s
GLiClass multilang-mini	zero-shot	62.8%	87.2%	44.8%	0.04–0.42s
Laya (typed decisions)	zero-shot	29.2%	60.0%	40.0%	—
char n-gram + LogReg	trained, 250 labels	88.4%	73.6%	64.0%	~0s

McNemar, same ids:

Comparison	livedoor	rakuten	chabsa
trained encoder − Jev	+12.0pt (p=0.00007)	−1.6pt (p=0.50)	+1.2pt (p=0.83)
trained encoder − local 26B MoE	+23.6pt (p<1e-15)	−1.6pt (p=0.45)	±0.0pt (p=1.00)
trained encoder − SemIf	+20.0pt (p<1e-15)	+0.8pt (p=0.83)	+11.2pt (p=0.007)
trained encoder − GLiClass (zero-shot)	+26.0pt (p<1e-15)	+5.6pt (p=0.016)	+30.4pt (p<1e-15)
trained encoder − char n-gram LogReg	+0.4pt	+19.2pt	+11.2pt

Finding 1: the same family wins or loses depending on whether you trained it

This is the part I did not expect, and it is the most useful thing in the table.

GLiClass is a GLiNER-family classifier — same lineage as the models currently being pitched as open-source Jev alternatives. Zero-shot, it scored 62.8 / 87.2 / 44.8: it lost to Jev on all three tasks, and on the 3-class polarity task it was barely above chance (44.8% against 33.3%).

Trained on 250 of my labels, the same generation of model scored 88.8 / 92.8 / 75.2. The gap between zero-shot and trained is +26.0 / +5.6 / +30.4 points, all significant.

So "is there an open-source model that replaces Jev?" was the wrong question. The models are not the variable — the labels are. A 310M encoder with 250 labels beats a decision API on one task and ties it on two; the same architecture with zero labels loses to everything.

I also paid for this lesson twice. Before this experiment I ran a proper evaluation of the two models most often named as Jev replacements — SemIf and Laya — on these same rows. Both failed to replace anything. Laya scored 29.2% on a 9-class task (chance 11.1%) with a median confidence of 0.029, i.e. it was visibly not deciding. Its zero-shot English number in its own documentation is 0.362, so this is not a Japanese-specific defect — it is a zero-shot defect. SemIf was closer but lost on all three tasks, two of them significantly.

Independent benchmarks showed the same thing. On a frozen-question benchmark, Laya's "hard" score was 34.1% while its model card implied a win over Jev, because the model had been fine-tuned on that benchmark's training split. Self-reported comparisons of fine-tuned models are not evidence about zero-shot behaviour.

Finding 2: training pays on topic tasks and does not on meaning tasks

Look at where the 12-point win came from and where it did not.

livedoor (9-class topic, 1,174 chars): trained 88.8% vs Jev 76.8%. The labels are in the text — if you have a few hundred examples, "what topic is this" is a vocabulary problem, and a trained encoder learns the vocabulary. It also beat the char n-gram baseline by a hair (+0.4pt), which tells you the topic signal itself was already largely lexical.
rakuten (2-class polarity, 138 chars): trained 92.8% vs Jev 94.4% — a statistical tie (p=0.50). Same for the local 26B MoE (p=0.45). Adding data does not fix this, because the problem is not "I don't know the vocabulary", it is "I need to judge sentiment".
chabsa (3-class financial polarity, 92 chars): trained 75.2%, Jev 74.0%, local MoE 75.2%. A three-way tie. On this task no one is good — 74–75% is the ceiling for all three approaches, and that is the real finding. If you need better here, you need a third stage (a stronger model on the low-confidence tail), not a different single model.

The practical rule I now use: if the label is visible in the words, train something small; if the label is a judgement about the words, use a decision API; if both plateau around 75%, stop shopping for models and fix the data or add an escalation stage.

Finding 3: the trained encoder is dramatically cheaper to run

Latency, single-threaded, same machine:

	p50 per item
trained encoder (310M, CPU)	0.10–0.45s
Jev	1.9–2.4s
local 26B MoE	1.24s
SemIf (4B)	1.3–6.8s

That is 4–20× faster than the API, with zero marginal cost, no network dependency, and no data leaving the machine. The confidence score is a softmax you can calibrate yourself rather than a vendor's number.

This is the part that changes architecture. Once the local model is both faster and not worse, the API becomes the fallback rather than the primary path: trained encoder first, Jev only on the low-confidence tail, the novel-looking rows, and as a drift detector.

Because you will ask: can I train it on the GPU in the box?

I tried that too, on the same mini-PC (Ryzen 9 7940HS, Radeon 780M iGPU, gfx1103). Short answer: it works, and it is not worth switching to.

Getting PyTorch onto that iGPU is now possible through AMD's device-specific wheels, but there are three traps in a row:

Installing with the AMD index as the only index makes pip fail to resolve ordinary dependencies, because you have just replaced PyPI.
Adding PyPI back as an extra index lets PyPI's newer torch win the version comparison — you silently get a CUDA build and a pile of nvidia-* packages.
The nightly index's newest torch requires a rocm-bootstrap version that the same index does not ship.

Once pinned properly, the iGPU is real: torch 2.8.0+rocm7.14.0a20260519, arch=gfx1103, fp16 matmul fine. The microbenchmark is impressive — 1.21 s/step against 16.41 s/step on CPU, 13.6×.

The real workload does not deliver that:

Task (5-fold CV, 310M encoder)	CPU fp32	iGPU fp16
long documents (512 tokens)	112 min	44.5 min (2.5×)
short sentences	21 min	23 min (0.9×)

Two reasons the microbenchmark lies. Short sequences leave the GPU idle while the CPU is comparatively efficient at small matrices, and per-fold fixed costs (model load, tokenisation, one-at-a-time evaluation) can exceed the training itself. Only long-document training showed a real gain.

And it is not stable by default. Roughly every few minutes the driver resets the GPU — amdgpu: ring gfx_0.0.0 timeout → GPU reset(8) → then the process dies with HIP error: unspecified launch failure. The GPU edge temperature was 49°C, so this is a driver problem, not a thermal one. Serialising kernel dispatches and pinning the performance level plus running one fold per process with retries did get a full 5-fold run through cleanly, but fp16 also cost 2.8–4.4 points of accuracy (not significant at n=250, but negative on both tasks).

So: CPU for training, GPU for what already works — Vulkan inference. If you are training on your own private data, a documented 30 minutes per dataset on CPU is better than a 2.5× speedup that crashes.

Where the labels come from is the whole game

If trained beats zero-shot this decisively, the obvious move is to generate labels with Jev and train on those. That works, with one trap I want to be explicit about, because it is the reason I did not just ship the first version:

If Jev alone produced the labels, Jev's accuracy is the ceiling of the student. You are teaching a model to reproduce another model's mistakes. The fix is to spend the labels where they matter: run Jev over everything, then take the low-confidence rows and have a stronger model or a human relabel them. That is where the ceiling comes off, and it is also where the marginal cost is, because high-confidence rows can be accepted as-is.

At 250 labels the trained encoder is already at parity with Jev on the tasks Jev is strong at, and ahead where Jev is weak. Labels are the lever, and the second and third hundred labels are worth more than the first.

What I run now

Situation	Use
No labels, short text, judge-shaped task	Jev (74–94% zero-shot, no setup)
No labels, must stay offline or in-machine	local 26B MoE (ties Jev on both polarity tasks)
A few hundred labels, topic-shaped task	train a small encoder (beats Jev by 12pt, 4–20× faster)
A few hundred labels, meaning task	trained encoder, but accept parity with Jev
Everything plateaus at ~75%	stop model-shopping; escalate the low-confidence tail
Labels to be generated	Jev for volume, a stronger model or a human on the low-confidence rows

Caveats

The trained numbers are 5-fold CV over 250 rows, so 200 rows of training data per fold. That is a floor, not a ceiling — more labels should widen the topic-task margin.
livedoor was truncated at 512 tokens, so the longest documents are not fully represented.
The trained side reports raw argmax; I have not temperature-scaled it, so its confidence is not yet comparable to Jev's for thresholding.
The char n-gram baseline is a single configuration. It may not be the strongest non-neural baseline available.
The iGPU accuracy comparison is at n=250 per task, which is not enough to make a 3-point effect significant. "Not significant" is not "no effect."

Takeaway: Zero-shot and trained are different products, not better and worse versions of the same one. If you have no labels, a decision API is the only thing that works today. If you have a few hundred labels and the label is visible in the text, a 310M encoder will beat it, cost nothing to run, and answer in a tenth of a second. Measure your task's shape before you pick a winner — and be suspicious of any "open-source Jev replacement" benchmark where the model was trained on the benchmark.

The API used here: TypeSafe AI's Jev (System One, jev-latest) — text plus a list of questions in, a 0–1 score per question out, own confidence included, roughly 32k of context, $0.042 per 1M input tokens with output free. Both controlled and randomised question sets were frozen and committed before the runs, and every comparison above is on identical row ids.

If you have run the same shape of experiment on your own data, I want to know where a trained small model beat the API and where it did not — especially on tasks where both plateaued.

Jev: one judgment call, or twelve dimension scores? I measured both on three classification tasks

Ikkun — Thu, 17 Sep 2026 16:13:40 +0000

The obvious way to use Jev (TypeSafe AI's System One judgment API) is one question per row, then threshold the score. I wanted to know whether the other shape earns its extra tokens: 12 to 14 narrow questions per row, cached scores, weights fitted locally on my own labels. Three tasks, both shapes, identical rows: 25,174 API calls, 5,477 test rows, 34.1M input tokens, $1.43.

The two shapes

Jev takes a text plus a list of questions and returns a calibrated 0–1 score per question. It is $0.042 per 1M input tokens, output free, ~32k of context, and it reports its own confidence per question.

Direct judgment. One question per row, then threshold the score.
Feature extraction. 12–14 questions per row in one request, raw scores cached to JSONL, an L2 logistic regression fitted on my own labels; multi-class is one-vs-rest plus argmax.

I wanted the second shape for operational reasons first: extraction costs money and needs the network, so it caches and resumes; fitting is local and free, so I can change weights, thresholds, or the whole stack without spending another token. Worth nothing unless the decomposition beats one well-written question.

Experiment one: a task my own question could ace

200 synthetic rows of B2B replies built from 32 template families, deliberately poisoned: "yes" rows containing "we'll pass", "no" rows containing "absolutely" and "happy to meet".

The direct question scored 100%, and swapping the choice names also scored 100%: no position effect. The noul form at a 0.5 threshold got 96.0%. My decomposed pipeline scored 98.0%, losing four rows and winning none. The test was worthless: the answer sits in front of one question. Those 200 rows cost 292,924 input tokens in 5.7 seconds, zero failures.

The split that turned 98% into 90%

That 98.0% (AUC 0.998) came from a random 5-fold split. Holding out whole template families instead of single rows dropped the same features to 90.0% (AUC 0.970): that apparent skill was the model recognising phrasing it had already been trained on.

A character-bigram naive Bayes on the same grouped folds scored 93.5% (AUC 0.989); a word-level one scored 47.5%. My API-scored dimensions lost to a free model whose sibling was worse than a coin flip.

Two rules stuck: print the majority baseline before believing any number, and choose the split — grouped by family or generator, never random rows — before building the dataset.

Experiment two: a task where one call should fail

300 rows, 150 per class, no names, no keyword, no single sentence that decides the label; the label exists only as a residue of many weak stylistic cues across the whole text. I handed the Jev call both class style descriptions, so it knew what to look for.

Direct choice: 64.7%. Choice order swapped: unchanged. The noul form scored 62.0% (AUC 0.676).

Dimensions plus fitted weights, 5-fold out-of-fold: 74.0% (AUC 0.811). McNemar exact on the same rows with a paired bootstrap of 20,000 resamples: +9.0 points over direct choice, 95% CI [+3.0, +15.0], 57 wins against 30 losses, p=0.0050; +11.7pt over noul, CI [+5.0, +18.3], p=0.0012; +7.7pt over the single best dimension, p=0.0140.

I reimplemented the cross-validation in plain numpy, not trusting my own code: 73.7% (AUC 0.809). And the CLI prints the in-sample score beside the out-of-fold one — 100% adjacent to 74%.

Confidence came apart from correctness

On 126 of the 300 rows (42%) the Jev call reported confidence of 0.9 or higher, and on exactly those rows it was 72.2% accurate. Confidence and correctness separated where the task got hard.

The obvious ensemble failed too: the Jev call's probability as a 13th input beside the 12 dimensions scored 73.0% against 74.0% — its information was already inside the dimensions.

Then a character-bigram naive Bayes on the raw text, same folds, scored 74.0% (AUC 0.824) — the same number as the Jev-scored dimensions, for zero API calls.

The real ledger export broke the script twice

Third task: my own bookkeeping export. 4,907 train rows, 2,101 test rows, labelled with the debit account and restricted to the top 12 debit accounts, which cover 87.4% of rows (about 55 accounts exist in total). The input was the description text, the amount, and the credit-side account with its sub-account; I dropped the debit sub-account because it gives the label away.

One direct question carrying 12 answer choices scored 0.3998: it reads the words correctly and still cannot choose among twelve labels when asked once. Comprehension was never the bottleneck; the output shape was.

14 dimensions with fitted weights: 0.9105. Word-bigram naive Bayes: 0.9491, beating both by +3.9pt over the dimensions (p<1e-4). Their errors differed, though: n-grams alone rescued 138 rows, the dimensions alone 63. Stacking the 12 n-gram class probabilities beside the 14 dimension scores reached 0.9695 — +2.33 points over n-grams alone (95% CI [+1.57, +3.14]) and +5.90pt over dimensions alone. The question was never which model wins, but whether they are wrong on the same rows.

For a detector, the headline should be false positives

339 rows of hard benign text: text that mentions injection techniques while being entirely benign — security write-ups, red-team notes, a post like this one. The single Jev question flagged 1.5% of them; my 12 dimensions flagged 37.2%; word-bigram naive Bayes flagged 43.4%. The gap between 1.5% and 37.2% is 35.7 points (p<1e-4) — about 25× the review load. The external set runs the other way: the direct call held 0.9558 on 2,060 rows where the dimension model managed 0.8398 and the n-gram baselines collapsed to 0.35–0.38, because the labels there run against the surface.

Four fixes, four failures

(a) Add hard benigns to training — 170 rows: 37.2% → 34.3%, detection held at 0.9554. Barely moved. (b) Drop the surface-level dimensions: 40.7%, worse. (c) Set the threshold on fold A for roughly 5% FPR: fold B landed at 8.3%, but attack detection fell from 0.9522 to 0.770, an 18-point loss. (d) Feed the Jev call's probability in as a 13th feature: benign FPR 46.0% and the external set fell from 0.9558 to 0.7883.

The fitted weights explain it: the two largest positives were obfuscated_encoding (+1.80) and hidden_in_content (+1.62), both high for an attack string and for a security document that quotes one. This is a dimension-design problem, not a weights problem: none of my 12 questions asks whether a text uses the technique or merely discusses it, and that is the dimension to write next.

The per-source table taught me the most

source	n	dims	direct	word bigram
hackaprompt	1,383	0.977	0.665	0.994
neuralchemy_v1	186	0.968	0.942	0.989
original	191	0.895	1.000	0.842
wildguard_judgecomp	325	0.674	0.651	0.628
harmbench (benign)	264	0.864	0.909	0.909
harmbench (attacks)	16	0.875	0.875	0.062

On competition-style attack strings the single Jev call collapses to 0.665 while the dimensions hold 0.977. On wildguard_judgecomp every method sits between 0.63 and 0.67 — a shared ceiling, not a method difference, so I stopped treating that slice as evidence. Across the test set, dimensions beat the single Jev call by +11.6pt (p<1e-4) but only tie word bigrams (+0.2pt, p=0.90).

Japanese NLI, where decomposition finally earned its keep

8,000 train, 2,434 test rows, three classes one-vs-rest, neutral 55% of the test set. Every n-gram baseline landed at or below majority: 0.5534, premise-only word bigram 0.5423, word bigram 0.5251, character bigram 0.4996. The hypothesis-only probe, the standard label-leak test, scored 0.5394 — so no label is hidden in the hypothesis.

14 dimensions reached 0.9076 (macro-F1 0.8845) against 0.8373 for the single Jev call (0.8198): +7.03 points, 95% CI [+5.67, +8.38], p<1e-4, with 234 rows solved only by the dimensions and 63 only by Jev. Leak check two: zero fully duplicated premise+hypothesis pairs between train and test, 21 duplicated premises only.

The single Jev call stalls at 0.675 in the 0.60–0.80 Jaccard band while the dimensions get stronger as overlap rises (0.987 in the 0.80+ band). The gap is all in contradiction (0.921 against 0.654); on entailment the single Jev call wins, 287/353 against 271/353. The labels are noisy — reverse-direction pairs carry different labels (jne-00015, jne-00016) — so 0.9076 is accuracy against human labels, not a ceiling.

Nine minutes, 25,174 calls, one error — mine

All extraction for all three tasks — 8 runs, 25,174 calls, 20,139 rows — finished between 22:59:51 and 23:09:05 JST: nine minutes and fourteen seconds, zero failed calls. The 4,391 injection training rows took 76 seconds; 8,000 JNLI rows 129; 4,907 ledger rows 83.

The one error was mine: at 23:06:24 a question object named rel_choice was missing its instructions field, and the CLI refused it with exit code 1 and the field name rather than judging something I had not asked. A criteria field passed alongside noul is the same class of usage error; a missing dimension stops with exit 1 and names it, while --allow-missing mean-imputes it.

Two things came from experiment, not documentation: noul points the way I did not expect (a true statement scored 0.96, a false one 0.02, so it is the probability the proposition is true), and --out merges rather than appends — re-running the same file reported skipped=300 / extracted=0, so a long job can resume without re-paying.

The bill is the cheap part

34.1M input tokens: 27.9M for feature extraction ($1.17) plus 6.23M for direct questions ($0.26) — $1.43 at $0.042 per 1M input tokens. Output was 846,765 tokens, counted separately and not priced.

Per row, one direct question costs 444–619 tokens, 12 dimensions 1,010–1,036, and 14 dimensions 1,136–1,154. Each dimension adds 40–50 tokens, mostly the wording of the question, so twelve dimensions run 1.6–2.3× one question, and fourteen run 1.8–2.6×: $0.019–0.026 per 1,000 rows against $0.042 and $0.048, or $19–26 per million rows against $42 and $48. Extraction is paid once: 4,391 training rows cost $0.19, and every later fit or threshold change reuses that cache for nothing.

At a million rows a month with 20% benign traffic, a 37.2% false-positive rate sends about 74,000 rows to human review; at 1.5% it sends about 3,000. That is roughly 71,000 rows — about 590 hours at 30 seconds per row — against a monthly token bill of $42 versus $19–26.

The ladder I actually run

Step	Do	Stop when
0	200 labelled rows, real ones if you have them	—
1	n-gram / TF-IDF baseline	It meets the target: no judgment API needed
2	One direct question per row	≥0.95 accuracy: keep it, decomposition is a downgrade
3	Dimensions + fitted weights	Only where there is no lexical shortcut
4	Stack dimensions with the n-gram probabilities	Their errors are orthogonal (+2.3pt measured)
5	Guardrails: direct call primary, dimensions as second opinion	Always train on hard benigns

Guardrails get one direct call with a strict threshold plus a small dimension model consulted where that call is unsure; classification gets n-grams first, with dimensions stacked on the residual.

What I got wrong, and the counterexample I still want

In order of how much it cost me: believing a random-split CV; treating a judge's confidence as calibrated; testing on data my own generator had made too easy; and reporting accuracy for a detector whose real risk is false positives.

What I still want is a counterexample: some task where the dimensions beat a direct call that was already competent — what did they know that one question could not? I have not found one in three tasks and I would like to be wrong.

The API used here: TypeSafe AI's Jev (System One, jev-latest) — text plus a list of questions in, a 0–1 score per question out, own confidence included, roughly 32k of context, $0.042 per 1M input tokens with output free. Each dimension costs 40–50 tokens, mostly the wording of the question; that is where the 1.6–2.3× comes from.

Two caveats: I hand-wrote all 12–14 dimensions, so these numbers describe my dimension design as much as feature extraction; and the ledger task covers only the top 12 account classes and took the credit-side account as input — the import-with-context case, not a bare bank statement. The JNLI labels carry noise and a 55% neutral skew, the external injection set has contradictory labels on near-identical rows, and output tokens are not priced.

The full tables and threshold sweeps are at https://agentjournal.dev/blog/llm-judge-vs-feature-extraction/.

Auditing AI agent activity: structured logs for credential access

Ikkun — Sun, 16 Aug 2026 13:20:00 +0000

AI coding agents are the most privileged new user on developer machines in a
decade: they read source, run commands, and access credentials. And unlike
every other privileged user, they're almost always unaudited. If you had to
answer "which API keys did my agent touch this week?" right now, could you?

Why audit logs matter more with agents

Traditional secret access has a human in the loop: a developer runs a
command, and the human's intent is the context. Agents change the equation:

Attribution is unclear. A command run by an agent was decided by a
model — influenced by its context, its tools, and potentially by injected
instructions. "Who accessed this key" needs to include "what the agent
was doing at the time".
The blast radius is session-wide. One agent session touches many
credentials through many invocations. Without logs, a compromise is
invisible until the damage shows up elsewhere.
Compliance surfaces. If your setup ever needs to answer "was this
secret exposed?" — for an incident, a client, or your own peace of mind —
logs are the difference between an answer and a guess.

What good credential audit logs look like

Structured. JSON lines, not prose. Each event is a record with fields: timestamp, the credential key, the operation (resolve / refresh / scrub), the invoking context, and the outcome.
Append-only. The log can't be rewritten by the thing being audited. File-based logs get 0600 permissions and reopen-on-rotation (so logrotate doesn't drop events); journald is a solid alternative.
Complete. Every credential resolution is logged — not just failures. The absence of an event is information too.
Machine-readable. --json output for cron, SIEM, or alerting: "three failed Bitwarden unlocks in a row" should be something you can detect, not something you notice.

What to log

The useful baseline for agent credential access:

Resolutions — which key was resolved, when, by what invocation. This is the "who touched what" record.
OAuth lifecycle — login, refresh, reauth_required. Token refresh failures are early indicators of revoked credentials or policy changes.
DLP hits — outbound pattern matches, with mode (log vs mask). A pattern hit in log mode is a secret that almost left the machine.
Scrub operations — what was scrubbed, from where. The audit trail for your erasure layer.

The test: can you answer the question?

A good test for whether your agent setup has adequate auditing: an incident
review asks "did the agent access the production key during that session?"
If the honest answer requires digging through shell history, you don't have
audit logs — you have hope.

trustless writes structured
credential audit events (append-only JSONL at 0600, file or journald) for
every resolution, OAuth refresh, DLP hit, and scrub — plus trustless doctor --json for setup health in machine-readable form. The audit command is in
the command reference.

Agents will keep getting more capable, which means more credentials touched
per session. The question isn't whether to log — it's whether you want the
answer to "what happened" to be a query or a mystery.

OAuth device flow explained: how CLIs and AI agents approve access

Ikkun — Sun, 16 Aug 2026 12:30:49 +0000

You've seen it on a smart TV: "Go to example.com/device and enter code
ABCD-1234". That's the OAuth device authorization flow, RFC 8628 — and it's
the right way for headless tools, CLIs, and AI agents to get OAuth tokens.
Here's how it works and why it matters for agent security.

The problem with OAuth on a headless machine

OAuth's normal web flow needs a browser redirect: authorize → the provider
redirects to http://localhost:8976/callback?code=.... On a headless server
or inside an agent session there's no browser to redirect to — and no user
watching the address bar.

The workaround people fall into is worse: paste a personal access token into
a config file, or run a long-lived refresh token into the environment where
the agent can read it. Both put credentials where agents can exfiltrate them.

How the device flow works

RFC 8628 splits authorization into two steps that don't need to happen on
the same device:

The CLI asks the provider for a device code and gets back a verification URL + a short user code.
The CLI prints "open this URL, enter ABCD-1234" — and starts polling the provider for approval.
The user opens the URL on any device — phone, laptop, anywhere — and approves. No browser on the server needed.
The CLI's poll succeeds, and the provider returns tokens.

The tokens land on the machine that requested them, in the process that
requested them — not in a browser, not in a shared log.

Why it fits agents

For AI agents, the device flow has a structural advantage: the approval
happens outside the agent entirely. The agent runs the login command, prints
the URL, and the human approves on their own device. The tokens are stored by
the credential layer, and the agent never handles them.

That's the same principle as the rest of agent credential hygiene — the agent
is an untrusted caller, so credentials should be resolved and refreshed
outside its reach.

Token handling done right

Getting tokens is step one; keeping them safe is step two:

Store the entry compactly. The OAuth entry lives in the credential store as a single-line JSON record — one entry, resolved like any other secret.
Refresh without exposure. Access tokens expire; the broker refreshes on expiry, and the new value is never printed to output the agent can capture.
Rotate refresh tokens. Refresh tokens are single-use in this design — rotated on every refresh, so a leaked one is already dead.
Report state. status tells you valid / expired / reauth_required, so automation can detect a revoked token (invalid_grant) and ask the human to re-approve instead of failing silently.

The audit trail

Every resolution and refresh lands in an append-only audit log. You can
answer "which key was accessed, when, and by what" — including OAuth entries,
which most setups can't audit at all because the tokens live in browser
profiles and config files.

trustless implements the device
flow for Google and Lark providers (trustless oauth login google api/google), with automatic refresh, single-use rotation, and status
reporting — documented in the command reference.

The device flow is one of those rare security mechanisms that's also more
convenient: no browser juggling on the server, no callback URL gymnastics,
and a human approval step that's exactly where it should be.

pass vs Bitwarden for AI agent credentials: which works with agents

Ikkun — Sun, 16 Aug 2026 12:30:48 +0000

When you decide to stop putting API keys in .env files and start keeping
them in a real secret store, the first question is: which store? For AI
agent setups, the two realistic candidates are pass (the unix password
manager) and Bitwarden. Both work — but they make different trade-offs, and
the right choice depends on your existing workflow.

pass: the unix-native store

pass is a shell script over a directory of GPG-encrypted files. That's the
whole architecture.

Strengths for agent use:

Zero infrastructure. No server, no sync, no daemon. The store is files.
Scriptable by nature. Everything is files and GPG, so tooling around it is trivial — a broker reads a path, decrypts, injects.
GPG is already everywhere. If you sign commits or use SSH with GPG keys, the tooling is already installed.
Transparent. pass show <name> and you see exactly what a tool will get.

Weaknesses:

Sync is your problem. No built-in sync across machines; you roll your own (git remote, Syncthing, etc.).
No native web vault / mobile app ecosystem. The CLI is the interface. For a single-user developer setup, that's fine.

Bitwarden: the managed vault

Bitwarden is a full password manager: client apps, web vault, sync, sharing,
and a CLI (bw) that agents can drive.

Strengths for agent use:

Sync and recovery built in. Vault is on your terms (Bitwarden cloud or self-hosted Vaultwarden).
Rich entry model. Login items with URIs, fields with hidden values, notes — better structured than a flat file tree.
Access control. Collections and organizations, if you ever share credentials with a team.

Weaknesses:

Session management is fiddly. The bw CLI needs a session key (BW_SESSION) that must be passed to every invocation — get it wrong and the tool either fails or prompts interactively. This is the classic integration trap for agent tooling.
More moving parts. Server sync, unlock state, API rate limits.

What the agent setup actually needs

For a credential broker pattern — the agent references by name, the broker
resolves at process/transport time — the store is an implementation detail.
What matters is that the broker can read it non-interactively:

With pass: the broker decrypts with the GPG key; no session state to manage.
With Bitwarden: the broker unlocks once, holds the session key in an environment variable (never argv — that would leak it into process lists), and fails closed when the session is invalid. A dedicated unlock command handles the interactive step.

Both backends can power the same injection, sanitization, and DLP layers.
The pragmatic rule: use pass if you're a single-machine, CLI-first user;
use Bitwarden if you already live in it or need sync across devices.

trustless supports both as first-
class backends — pass by default, Bitwarden via the bw CLI with
BW_SESSION handling and fail-closed behavior. The backend design is
documented in the architecture page.

Whichever store you pick, the goal is the same: the agent gets access, never
the plaintext. Which one are you already using — or are you still on .env?

SQLite forensics: why deleting rows doesn't erase secrets (FTS, free pages, VACUUM)

Ikkun — Sun, 16 Aug 2026 12:22:52 +0000

You deleted the row. The secret is gone from the app, the queries return
nothing, and the dashboard is clean. In SQLite — the database behind most
session stores, browser profiles, and agent state files — that delete is a
fiction. The bytes are still in the file.

Three ways deleted data survives

1. Free pages. SQLite doesn't zero out the space a deleted row occupied.
The page is marked free and added to the freelist; the old bytes stay until
they're overwritten by a future write. A file that's been deleted-from is a
forensics goldmine: recover the freelist pages and the "deleted" rows come
back.

2. FTS virtual tables. If the database uses SQLite's full-text search
(FTS5), the FTS index keeps its own copies of the indexed text, maintained
separately from the source tables. Delete the row from the source table and
the FTS index still contains the tokens — searchable. This is the one that
catches people: their app shows the secret is gone, and the FTS index still
has it.

3. WAL and journal files. In WAL mode, recent writes live in the
-wal file; transactions in the -journal file. Both can retain pre-delete
content until checkpointed or cleaned.

"Deleted" in SQLite means "no longer referenced", not "no longer present".

What erasure actually requires

Making a secret physically disappear from a SQLite database takes three
operations, in order:

Replace the value everywhere it lives. Known secret values get
replaced across all tables; pattern matches (API key formats) get masked.
Two layers, because you can't enumerate every secret that leaked.
Rebuild the FTS indexes. INSERT INTO t(t) VALUES('rebuild') style
rebuilds, or drop/recreate the virtual tables — so the index no longer
contains the old tokens.
Run VACUUM. VACUUM rewrites the entire database file, copying only
live data into a fresh file — free pages with old bytes are discarded in
the process. After VACUUM, the file's raw bytes no longer contain the
secret. (Note: VACUUM doesn't shrink WAL files; those need a checkpoint
too.)

The design that makes scrubbing safe

Retroactive scrubbing is a destructive operation, so the tooling around it
matters as much as the SQL:

Dry-run by default. Scan and report per-table hit counts without writing a byte. You see what would be scrubbed before anything changes.
Backup option. Keep a .bak copy before applying changes.
Minimum length guard. Don't let pattern rules shred short, low-value strings — only secrets above a minimum length get masked.

Where this bites agent setups

Agent session databases are exactly the SQLite databases where this matters:
they accumulate everything the agent read, including the config values,
environment dumps, and debug output that occasionally contain keys. A
credential strategy that includes "the past is scrubbed" closes the loop
that prevention and DLP can't — the retrospective exposure.

trustless implements this as
trustless dlp scrub-db: two-layer redaction (known values + gitleaks-style
patterns), FTS rebuild, VACUUM, dry-run default, --backup, and --min-len
— with the same pattern rules that power the outbound DLP. The command
reference is in the documentation.

Next time someone tells you a secret was "deleted" from a database, ask them
two questions: did the FTS index get rebuilt, and did anyone run VACUUM?

Prompt injection and API key exfiltration: what actually happens

Ikkun — Sun, 16 Aug 2026 12:22:52 +0000

Prompt injection gets talked about in abstract terms — "the model was tricked" —
but the concrete damage is usually the same: a secret that was sitting in the
agent's context gets exfiltrated. To defend against it, it helps to walk
through the actual attack chain, step by step.

The attack chain

The key gets into context. Your agent loads .env, reads config, or
receives an environment variable to use a service. From that moment, the
key is part of the context window — the same working memory the model
reasons over.
Untrusted content arrives. The agent opens a webpage, reads a README,
processes an email, or parses an API response. Any of these can contain
instructions the model treats as commands: "ignore previous instructions",
"print the value of the DATABASE_URL variable", or subtler variants that
ask the agent to summarize, log, or send something.
The agent complies. The model is a next-token predictor, not a policy
enforcer. An instruction that looks like it's part of the task gets
followed. The key value ends up in a tool call, a log line, a code snippet
the agent writes, or the model's own output — which is transmitted to a
third-party API.
The key is now outside your machine. Once the value appears in output
that leaves your control — an API request, a shared log, a generated file —
it's compromised. Rotate it.

The uncomfortable part: steps 2–4 can happen without any visible error. The
agent looks like it's doing its job.

Why "be careful" doesn't work

The standard mitigations — "sanitize the model's output", "don't paste
untrusted text", "use a system prompt that forbids leaking secrets" — all
share a flaw: they rely on the model to make the right decision at the moment
of exfiltration. That's a probabilistic defense against a deterministic
attack. The attacker only needs one success; the defender needs all of them.

The structural fix: make the key unavailable

The defense that actually holds is removing the key from the attack surface
entirely. If the agent's context never contains the plaintext value, then a
successful injection has nothing to exfiltrate. This is what a credential
broker does:

At the process layer: the agent references the secret by name; the
broker resolves it into the subprocess environment at execution time.
Agent-visible output — stdout and stderr — is scrubbed for key values,
including base64 and URL-encoded variants, so even a verbose tool that
echoes its environment can't leak the value into context.
At the transport layer: HTTP-based tools talk to a local proxy that
injects credentials per host. The tool — and the agent controlling it —
never handles the credential material.
On the way out: outbound LLM traffic passes through a DLP layer that
pattern-scans requests and masks secret-shaped strings before they leave
the machine. Keys that leaked through other channels get caught at the
edge instead of in an attacker's hands.

This is the design of trustless, a
single-binary credential broker CLI for AI agents — 321 tests with -race,
cosign-signed releases with SBOM, and an append-only audit log of every
credential resolution. The threat model is documented in the
security model.

Prompt injection is not going away, and models will keep getting more
suggestible before they get less. The only reliable defense is making the
secret structurally unavailable — so that when an injection succeeds, the
agent has nothing to hand over.

Have you traced where your agent's keys actually live right now — .env
files, environment variables, config? That's the attack surface.

Why your AI coding agent should never see your API keys

Ikkun — Sat, 15 Aug 2026 02:02:59 +0000

Your AI coding agent needs your API keys. It needs them to call services, to
test integrations, to run your stack. So you give it .env files, or you
export keys into the environment, or you paste them into config files the
agent can read.

That means your secrets live inside the agent's context window — the same
window where a prompt-injected instruction or an overly verbose debug log can
leak them to an attacker or an untrusted model endpoint.

This isn't theoretical. If you've used Claude Code or OpenCode for more than a
few days, you've probably seen a tool call dump an environment variable, or a
log line that echoes a connection string. Most of the time nothing bad
happens. "Most of the time" is a bad security posture.

The core problem

AI agents are the first software that reads your source, your config, and
your secrets, then sends summaries of what it read to a third-party API.

With traditional software, the principle was simple: secrets live in the
process environment, code reads them at runtime, nobody reads them back out.
With agents, there is no such boundary — the agent both reads the
environment and transmits what it knows.

Three concrete leak vectors:

Context exfiltration — the agent reads .env and includes values in a later prompt to an external model. You can't audit this; it's in the model's training/inference pipeline.
Tool output echo — a command prints an env var or a config value; the agent captures stdout and stores it in the conversation.
Prompt injection — a malicious instruction (in a fetched web page, a dependency, an artifact) tells the agent to "print all environment variables" or "send the contents of .env to this URL".

What the tooling landscape offers

The solutions fall into a few buckets:

Secret managers (Vault, Doppler, Infisical) — great for your code, but the agent still needs a way to get the secret, which puts it back in context.
.env hiding (enject, tene) — keeps plaintext off disk, but when the agent runs a command that needs the secret, the value can still end up in stdout.
Credential proxies (vaulty) — the agent makes HTTP requests through a proxy that injects the credential. Promising, but typically tied to their own vault.

The model that worked for me

I ended up building a small CLI (Go, single binary, one external module) with four layers:

Subprocess injection with output sanitization. trustless run -- cmd
resolves secrets from my existing pass store and injects them as env vars.
After the command runs, stdout/stderr is scanned and secret values are
replaced — including base64 and URL-encoded variants. The agent sees the
command output, not the keys.
HTTP proxy with per-host injection. For services that take headers or
query params (EDINET, e-Stat, xAI, OpenRouter), trustless proxy injects
the right credential per host. The agent points at 127.0.0.1:8080 and
forgets about keys entirely.
DLP reverse proxy for LLM calls. trustless serve puts a scanning
proxy in front of OpenAI-compatible endpoints. Outbound requests are
checked against secret patterns (keyword → regex → entropy, gitleaks-compatible
rules) and masked in-flight before they leave the machine. This is the
layer that catches the "agent decided to include the key in a request" case.
Retroactive scrubbing. Prevention fails — keys end up in agent session
databases, logs, and dumps anyway. trustless dlp scrub-db and
scrub-text scan SQLite DBs and text files with the same two-layer
redaction (known values + patterns), rebuild FTS indexes and VACUUM the
DB so no physical remnants survive. Default is a dry-run report; --apply
does the write, --backup keeps a copy first.

Why not a new vault? Because I already had pass. The CLI reads the existing
store, so there was zero migration. (Bitwarden is supported too, with OAuth
token auto-refresh for Google/Lark.)

The agent gets capabilities, not credentials. That's the whole trick.

Takeaways

Never let a secret enter the agent's context window — not as env, not as config, not as tool output. Once it's there, you've lost the audit.
Inject at the process/transport boundary, not at the prompt level. "Please don't print the key" is not a security control.
Sanitize output, not just input. The leak vector is often the command's stdout, not the agent's intent.
Scan outbound requests if your agent calls external APIs directly. A DLP layer is the difference between "we hope it didn't leak" and "we know it didn't".
Assume leakage happened and scrub retroactively. Run a periodic scan of agent session DBs and logs; a dry-run scrub report tells you what would be found, --apply cleans it, and a backup keeps the recovery path.

If you want to look at the code: trustless is MIT-licensed at
https://github.com/ikkun1222/trustless — 321 tests, race-detector clean,
cosign-signed releases. It's one implementation of this model; the threat
model discussion is more valuable than the tool itself.