A browser tool that points your own API key at an adversarial battery and grades every answer with pure predicates — no LLM judge, and your key never touches my server.
The first time I ran it against a real model, it told me the model was ~29% vulnerable.
That number was wrong. And the tool proved it was wrong — to me, in public — because of exactly one design decision.
Every verdict in The AI Crash Test is a deterministic predicate over the model's answer string: exact match, regex, a number check, an injection canary, a must-refuse rule. No model grades another model. So when the report flagged ~29% vulnerable, it also showed the fail card for every miss — prompt, expected, actual, side by side. Three of those cards didn't show a broken model. They showed a broken grader: false positives in my own code. I fixed the graders; the real number was 0%.
That's the whole pitch. An auditable grader has bugs you can catch in public. A vibes-based, LLM-as-judge arena just hands you a number and asks you to trust it.
The honest part first
LLM red-teaming is a crowded, mature space. garak (NVIDIA), PyRIT (Microsoft), and promptfoo all do far more than this — more probes, more scale, more integrations. Browser tools that adversarially test with your own key exist too; most lean on an LLM judge.
So this isn't a new category, and I won't pretend it is. The narrow thing that's mine is an intersection: browser-based BYOK + deterministic no-judge grading + a provably shared engine with a longitudinal drift board. Distinctive engineering and discipline, not a market-novel product. If you want heavy artillery, go use garak. If you want a result you can reproduce byte-for-byte and a key that goes straight to the provider and never touches my server, read on.
Two properties you can check yourself
1. Deterministic grading — no LLM in the grade path. Every grade is a pure function of the answer string, run in an open-source engine called gradecore. Run a mock model through it twice and the score is byte-identical. No temperature, no judge drift, no "the grader was having a bad day."
2. BYOK, never-touches. The browser calls the provider directly with your key. crashkit's server receives only the answers — the grade request has no key field at all.
Don't take my word for it. Open DevTools → Network, run a battery with your key, and search the panel for the key itself. It lights up only on the request to the provider (e.g. api.anthropic.com, in the x-api-key header) — never in the /api/grade call. I verified this live before writing this; you can reproduce it in about thirty seconds.
The honest caveat: this only works where the provider allows direct browser calls. Anthropic (with the dangerous-direct-browser-access header) and Gemini work; OpenAI-direct is often CORS-blocked. Stating the limit is part of the point.
One engine, two lenses
gradecore isn't a crashkit-only toy. It's the same deterministic engine behind my live model-drift board, which tracks 16 LLMs over time. Same code, two jobs: the board is longitudinal monitoring; The AI Crash Test is on-demand adversarial testing.
And it's the same engine, not a lookalike. Run the board's frozen suite through gradecore and the suite_hash comes out identical, byte for byte — faithful extraction, not a reimplementation. (To be clear: crashkit uses gradecore, not my whole eval stack.)
What's in the battery
Eight tasks across seven attack kinds: prompt-injection, tool-abuse, spec-violation, refusal-calibration, unsafe-compliance, hallucination-bait, and consistency. You get a severity-weighted vulnerability report and a fail card for every miss.
A real result from a recent run: claude-haiku-4-5 resisted 100% — 0% vulnerable across all seven kinds, deterministically graded, reproducible.
Try it / read it
- Run it: https://crashkit.onrender.com
- Source: github.com/egnaro9/crashkit and github.com/egnaro9/gradecore
I built this solo, self-taught, in under a year — and yes, with heavy AI assistance in the loop. The design decision I care about is the one you can check without trusting me: open the Network tab, run it twice, read the graders. Prove it — don't take my word for it.
Top comments (5)
The fact that your own grader produced false positives—and the tool made that visible—is probably the strongest proof of the design.
Deterministic grading won’t cover every nuanced failure, but for reproducibility and auditability it’s a much better foundation than blindly trusting another model’s opinion. Really solid work.
Thanks, Mustafa — that self-caught false positive was the moment I decided the approach was worth shipping. And you're right about the limit: deterministic predicates won't catch nuanced, open-ended failures. I'd rather be honest that it's a floor — reproducible and auditable for the things you most need to trust (leaks, refusals, consistency) — than pretend one grader covers everything. Appreciate you reading it closely.
I think the most valuable takeaway here isn't the benchmark itself it's making the evaluator auditable. We've become comfortable using LLMs to judge other LLMs, but that adds another probabilistic layer to something we're trying to measure objectively. Deterministic predicates won't cover every evaluation scenario, but they're a much stronger fit for security checks, policy compliance, and regression testing because every failure is reproducible and debuggable. Using an LLM only where semantic judgment is genuinely required, while keeping everything else deterministic, feels like a much more reliable production pattern.
You've put your finger on the part I care about more than the battery itself, so thank you for that.
The concrete argument for it: an early run against a real model read ~29% vulnerable. Because the graders are fixed predicates and every fail card shows what it expected, I could inspect the failures instead of believing the number — and three of them were my grader's bugs, not the model's. The real figure was 0%. An LLM judge would have handed me 29% and no way to interrogate it, and I'd have published a false claim about someone else's model.
Your split is exactly the one I'd defend: deterministic wherever the property is checkable — refusal, format compliance, citation presence, tool trajectory, injection canaries — and a semantic judge only where the question genuinely is semantic. The failure mode I'd add is that people reach for the judge too early, because writing a predicate forces you to say precisely what "correct" means and that's uncomfortable. The discomfort is the useful part.
The honest limit of my side: "did it answer helpfully" isn't a predicate, and I don't pretend it is.
Exactly the line I'd draw too, Mateo — "LLM only where semantic judgment is genuinely required, everything else deterministic." The failure mode I keep seeing is teams reaching for an LLM judge by default and inheriting its variance in the one place they're trying to remove variance. For security, policy, and regression, a predicate you can rerun and debug beats a fluent second opinion every time. Well put.