Black Box: finding yesterday's warning signs in 225,000 maintenance reports

On 1 April 2011, Southwest Flight 812 lost a five-foot strip of fuselage skin at 34,000 feet. Everyone survived. The NTSB's final report described fatigue cracks along one lap joint, stringer S-4L.

Here's the part that keeps you up at night: mechanics had filed reports about cracked clips on that exact stringer before the accident. Two of them, on the same tail number, twenty-four inches from where the roof later tore.

Nobody reads those records together. The investigation lives in one database, the maintenance paperwork lives in another, and the person who could act on the pattern is a structures engineer with a queue full of work orders. The datasets aren't secret. They're just apart.

That's the bet behind Black Box, our project for the FalkorDB Graph Hacks hackathon: this is a graph problem. If maintenance reports, NTSB findings, and FAA airworthiness directives all live in one graph that understands where on the aircraft a report points, then the precursor pattern shows up as a query, not a coincidence.

One graph, two filing cabinets

We ingested two public datasets into a single FalkorDB graph — NTSB accident reports and FAA maintenance reports, ~225,000 records — plus the FAA's Airworthiness Directives, the official "fix this" mandates, and, in a later pass the same week, the actual text of those directive documents and the NTSB brief themselves, sectioned and dated.

The core design decision is an address system for parts of an aircraft. Every report attaches to a Location node that refines in three steps:

• a zone — fuselage section 53, the lower skin
• a line — stringer S-4L, one specific structure inside it
• a bin — a 20-inch stretch of that stringer: the literal segment where the failure happened

Two datasets that never knew about each other now meet at the same address, down to the foot. A "find the warning sign" question stops being an analytics project and becomes one walk of the graph:

// Reports on one tail, at one address, before the clock we're standing at
MATCH (r:Report)-[:ON]->(:Aircraft {tail:'N648SW'}),
      (r)-[:AT]->(:Location)-[:WITHIN*0..2]->(line:Location {level:'line'})
WHERE r.date < $asOf
RETURN r.id, r.date, r.text

For followers of this case, the output of the fleet version of that query is the famous number: Southwest 737 Classics reporting skin cracks on stringer 4 went 9 → 18 → 31 between 2008 and 2010, three months before the accident. And it wasn't the loudest line overall — other stringers had more total reports. What stood out was the rise: second of 37 lines on our checked date. A lead worth review, not a prediction.

The time machine

Every investigation in the app carries a date, and the graph obeys it. Set the clock to 31 March 2011 and the NTSB's report about the accident is not there. Documents dated after the cutoff are invisible, down to the emergency AD issued five days after the accident.

Why so strict? Because most "we saw it coming" demos cheat without meaning to: the accident's own report leaks into the evidence pool, or the model quietly reads documents published afterwards. Ours is overfitting-resistant by construction — the rule is a date clamp on the graph, not a narrative.

And because one memorable example is a trap, we built a backtest harness and ran our naive lead-signal across every NTSB event that had earlier reports. It initially scored three times better than chance. Then we closed two leaks — the accident's own report in scope, and aircraft families that always report the same systems — and the advantage fell to 0.9, no better than chance. A system-level match is not a warning sign. What mattered was the exact location and the fleet trend. That negative result is the honest center of this project, and the app shows it.

The agent: useful, but on a leash

The investigation board hosts an AI agent (a free-tier model on OpenRouter) that investigates through an MCP server we expose over the graph. The agent picks which tool to call next and writes the conclusions. Every one of those tools is a fixed Cypher query — the model never writes a query; it chooses among queries we wrote and verified, so a hallucination can't become a fake datapoint.

Two deliberate guardrails:

1. There is no approve tool on the agent's server. If its evidence is strong, it may draft a failure mode — linked to the reports and accident that support it, status proposed. Only a person can confirm, defer, or merge it, in the UI. When our agent's first live answer drifted into causal language ("this caused the failure"), our harness made it rewrite. We surface warning signs, never causes. A lead is a lead.
2. Human decisions get routed like real decisions. A confirmed risk becomes an alert, then a work-order draft — routed to the team that owns that ATA chapter, and to a specific engineer picked by who worked that exact line before the investigation date, with the fewest open jobs. The graph ends where responsibility begins.

The third layer of rigor is the paper trail itself: the AD paragraphs and the NTSB brief live in the graph as dated sections, linked to the locations they concern, so "the records said this" can be checked against the paper it came from — while still respecting the date clamp.

What we learned shipping it

• Model the address, not the abstraction. The best decision we made was treating "where on the aircraft" as a sharp, ladder-shaped property (zone → line → bin) instead of free text. Everything else — heat maps, trends, documents, agent tools — slotted into that shape.
• Graph queries read like the domain. Our judge-facing pitch is one Cypher query, not an architecture diagram: reports ON a tail, AT a location, WITHIN a stringer, before a date.
• A date clamped graph is a cheap honesty engine. One WHERE r.date < $asOf everywhere beat any amount of prompt engineering about "don't use future knowledge".
• Deployment is where the war stories live. The whole stack — FalkorDB v4.14.6 plus a Next.js app — runs as one warm container on Modal, with the graph shipped as a 63 MB RDB in a volume, cold boot under a minute. Which sounds smooth now. It took two genuinely fun bugs to get there: the database image's own startup script quietly hijacked container boot (running its bundled Redis and a browser UI instead of our code), so Modal kept killing "initializing" containers — fixed by blanking the image's ENTRYPOINT; and Modal refusing to mount a volume over a path the image has files baked into — fixed by moving the mount. Everything was our own Docker-free setup flowing through docker compose locally first, which is why none of this showed up until the first deploy.