Before anyone ships an AI system, it helps to know how systems like it have already failed. That record exists, in the AI Incident Database, and the bigger thing I want to build is a way to turn those real past failures into red-team tests and risk checklists for a new system. The catch is that using the database that way means first sorting every incident by the kind of harm involved, and that sorting is done by hand. It doesn't keep up as the pile grows.
So this post is the first piece of that, something narrow and testable: how much of the sorting can a small, locally-run model do on its own, and how close does it land to the humans who normally do it?
What I used
The AI Incident Database (AIID) is the biggest open catalog of real-world AI failures, somewhere around 1,600 incidents. Each one has a title, a description, links to the original news reports, and the part I cared about, taxonomy labels applied by human editors. Those human labels were my answer key.
For the taxonomy I used the MIT AI Risk Repository, which AIID already applies: seven top-level risk domains (Privacy & security, Misinformation, Malicious actors & misuse, AI system safety and limitations, and a few more) that split into about two dozen finer subdomains.
The model was Llama 3.1 8B, running on my own machine through Ollama.
The pipeline is about as plain as it gets:
- pull the weekly AIID export and flatten it into a clean table
- for each incident, hand the model the title, the description, and the list of allowed labels, and ask for exactly one
- compare its pick to the human label across the ~1,500 incidents that carry one
I scored it two ways, plain accuracy and Cohen's kappa, which discounts the agreement you'd get by luck. That second number matters here because some harm types are far more common than others.
What came back
Across 1,495 incidents, with not one malformed answer, here's where it landed:
| Granularity | Accuracy | Cohen's kappa |
|---|---|---|
| 24-way subdomain | 47.6% | 0.41 |
| 7-domain (collapsed) | 62.1% | 0.51 |
Two things jumped out.
The model is much better at the broad domain than the specific subdomain. Kappa of 0.41 on the fine-grained labels is fair agreement. At the domain level it climbs to 0.51, solidly moderate. Put plainly, it usually knows what kind of harm an incident is about, but it fumbles the exact flavor inside that kind. In the confusion matrix, most of the misses were these near-misses within the right domain, not wild swings across the board.
The other thing was how uneven it was. The concrete categories held up well. Fraud and scams, cyberattacks, plain safety failures, discrimination, all showed a clean diagonal. The fuzzy ones fell apart. Misinformation, privacy, the socioeconomic bucket, the human-computer interaction bucket, all scattered. Some of that is thin data. Some of it is real ambiguity that trips up people too. Is a surveillance case a privacy harm or a misuse harm? I could argue it either way, and so could the editors.
The part I got wrong
I figured a few worked examples in the prompt would tighten up the fine-grained calls. Standard few-shot prompting. So I ran it again with three example incidents pasted in ahead of each one, held out from the scoring so I wasn't grading on them.
It didn't help. It made things slightly worse.
| Config | Subdomain kappa | Domain kappa |
|---|---|---|
| Zero-shot | 0.41 | 0.51 |
| Few-shot (3) | 0.36 | 0.52 |
Fine-grained agreement dropped from 0.41 to 0.36. The domain number barely moved. My best guess is label bias. Three examples can only show three of two dozen possible labels, and a small model leans toward whatever it just saw. On a task with this many classes, that nudge does more harm than good. I only know that because I measured it, which is the whole reason to measure.
What I'm building on this
The classifier is step one. Reading incident descriptions won't teach a model to behave, so the goal isn't to feed these reports to a system and hope it learns. An incident report tells you what broke, not the input that broke it or the right way to handle it, so there's nothing in it to train avoidance from.
What the data can do, and this is the part I'm building, is drive a tool that turns past failures into tests. Every incident links back to its source articles, so you get the real detail behind each case, not just a one-line summary. The tool takes a new system, or even a narrow custom skill someone is building, uses the classifier to name the harm domains it's most likely to touch, then pulls the real incidents in those domains and turns them into two things: a risk checklist a person can actually read ("systems like yours have failed in these ways, here are the cases"), and a set of seed scenarios for red-teaming ("try to make it do this"). The linked reports give enough substance to write those tests from real failures instead of imagined ones.
A person stays in the loop. The classifier is only fair at the fine grain, so it suggests the top couple of domains rather than silently routing anything. And a checklist built from past incidents only covers harms someone already reported, so a clean run isn't a safe system, just one that dodged the known traps.
What I took away
A small model on a laptop can already do the coarse sorting, putting incidents in the right broad domain most of the time. The finer calls are less reliable and vary a lot by category. So the practical read is fairly narrow i.e. this is good enough to pre-sort or triage incidents for a human reviewer, not to replace the labeling. Triage is most of the tedious part, so that's still worth having.
The other thing worth noting is how much the headline "is X type of harm rising?" depends on the labeling underneath it, which is more subjective than it looks. That's not a reason to distrust the numbers, just a reason to know how they're made before leaning on them.
I’d love to hear any thoughts on what other use cases or considerations we should explore with this kind of incident-driven testing?
Data: the AI Incident Database. Taxonomy: the MIT AI Risk Repository. Code and the full analysis are on GitHub
Top comments (0)