Rainbone at Hack the Valley 11: We Built a 3D Bone Stress Test and Made the AI Prove Every Number

This is a submission for the MLH x DEV Writing Challenge

What I Built

A bone can look fine on a grey 2-D CT slice and still be close to breaking. The people looking after an older woman in senior living rarely get a plain answer to "how strong is her bone?" until a fall gives them one.

At Hack the Valley 11, my team of four built Rainbone: a 3D stress test for bones, explained by AI agents. It takes a real research CT scan of a thighbone and solves it with finite-element physics. It shows in 3D where the femur carries the most strain, and then Gemini explains the result in plain words for the care team: nurses, CNAs, wellness directors, doctors and admin staff. We built it for PointClickCare's Catch It Early challenge.

Try it: rainbone.health. Press Enter the lab; the demo seat (Dr. Rowan, care team) needs no account.

Rainbone is a research prototype built on public, de-identified research data. It is not a medical device, and it never says whether or when a fracture will happen.

One real femur, solved end to end

We found HFValid, a dataset from the University of Bologna. It covers 101 women aged 56 to 86, with a femur CT for each woman, the study authors' own fall-fracture model results and a record of who fractured over the following five years. The lab opens on subject Pat001: the complete right femur of a woman who was 78 at her scan.

The femur pipeline runs offline:

1. HFValid CT and the authors' segmentation go into 3D Slicer, which exports a closed surface.
2. TetGen turns the surface into 60,819 tetrahedra and 17,361 nodes.
3. SfePy runs a linear elastic solve: 800 N on the femoral head, the distal 15 mm fixed, one assumed material (E = 10 GPa, ν = 0.3). The support reaction balances the load to better than one part in a billion.
4. The results become VTU and VTP files, rendered live in the browser with vtk.js and coloured by von Mises stress (0 to 15.83 MPa) or displacement.

On top of that:

• Cut-away view. It slices the solved tetrahedral mesh in the browser, so you read the stress inside the bone and not just on its surface. We built it from scratch because the 3D library couldn't read the solved volume mesh natively.
• Where she stands in the cohort. Her absolute risk of fracture at time 0 (ARF0, the study authors' model) is 47.1%, higher than 56 of the 101 women. A "What do these numbers mean?" panel tags every number as published data, computed by Rainbone, derived or assumed.
• A virtual press test. It replays the same femur in a virtual hydraulic press over 15 solved frames. The force peaks at 20,079 N, then falls as modelled damage spreads. The page labels it as exploratory.
• Print the break. At any step after the first break you can download the exact missing region as an STL and 3D-print it at 1:1 scale.
• A live engine. Each visit sends a small synthetic CT volume to our FastAPI back end on Vultr. The back end keeps voxels at or above 150 HU, maps CT values to stiffness, meshes the bone and solves an 800 N load with SfePy. Every run is saved to Tiger Cloud. Import scan runs the same solve on your own .nii or .nii.gz CT.

The part I'm proudest of: AI that has to prove every number

I was a full-stack and AI engineer on the team and its #1 GitHub contributor. I spent most of my time on the AI layer, because in a clinical setting the hard part isn't the model. It's the boundary: what the model is allowed to say.

Our rule: code computes, and the model only puts the results into words.

• Facts first. The API builds every number a model may use: Pat001's ARF0, her two ranks, the cohort median and fracture range, and the fall-force ratios. Models receive only those numbers, never a scan or a file name, and are told to calculate nothing.
• Pinned facts. The API reads the facts from a private folder in Vultr Object Storage. The index's sha256 must equal the hash pinned in our repository, and the object's bytes must hash to that value. If the bucket is slow, unreachable or doesn't match, the API answers from the copy bundled with it and says so. A changed bucket can never quietly change the facts.
• A checker on every answer. Before any text reaches the screen, code rejects it if it uses a number that isn't a fact, uses a fact as the wrong quantity (an age as a percent, a rank as a risk), states a rank the wrong way round, calls ARF0 a "chance", says when or whether a fracture will happen, or gives advice. Our live log shows the checker catching a real answer that stated a rank backwards and serving a checked answer instead.
• Honest fallbacks. The chain runs gemini-3.8-flash, then gemini-3.5-flash-lite, then a saved Gemini answer for exactly the same facts (checked again), then Groq's openai/gpt-oss-120b, and finally a deterministic template that always passes the checker. Free-tier rate limits hit us during the build. When the main model returns HTTP 429, the API remembers that for 300 s and the lighter model answers instead. Each card names where its text came from.

Ask Rainbone: agents you can watch

Ask Rainbone is a chat drawer that answers questions about the case, the pages and the data, and it shows each step as it runs. The five stages run as a LangGraph StateGraph. Triage works by rules, or Gemini's native function calling picks the tools. Evidence calls read-only tools. The writer may only pick statements from a fixed, checked catalogue. A code critic can send the draft back for one rewrite. The audit stage writes a row to Tiger.

This is the real wiring from our repo:

def compile_graph(execution: Execution):
    from langgraph.graph import END, START, StateGraph
    graph = StateGraph(GraphState)
    for name in STAGES:
        async def node(state: GraphState, name=name):
            return await execution.run_stage(name)
        graph.add_node(name, node)
    graph.add_edge(START, "triage")
    graph.add_edge("triage", "evidence")
    graph.add_edge("evidence", "writer")
    graph.add_edge("writer", "critic")
    graph.add_conditional_edges("critic", lambda state: state["next_stage"], {"writer": "writer", "audit": "audit"})
    graph.add_edge("audit", END)
    return graph.compile()

We chose LangGraph over a single chain because the critic needs to send drafts back to the writer. An explicit state graph makes that loop, and the audit step after it, visible and testable. If the framework itself fails, a built-in runner carries on from the same stage without re-running finished ones.

We also red-teamed our own agent: 32 hostile questions to the live Ask Rainbone, scored by fixed rules. 30 passed, and the two misses are now fixed and pass live (red-team report).

Demo

Full walkthrough (2:39):

Highlights cut (1:59):

Try it in 60 seconds at rainbone.health:

1. Press Enter the lab and take the demo seat.
2. Patient overview: rotate the femur and turn on Cut-away to see the stress inside the bone.
3. Physics simulations: scrub the press, press Go to first break, and download the missing region as an STL.
4. Ask Rainbone: click What does the 47% mean? and watch the agents run: triage, evidence, writer, critic, audit.
5. Secure datasets → Recent AI answers: the audit rows Tiger Cloud stored for those answers.

The full method and the maths are in our research paper: 69 numbered equations and the LangGraph state machine.

galitau / Rainbone_prod_hackthevalley11

Rainbone

Live site · Demo video (2:39) · 1:59 highlights · Devpost · Research paper · Architecture · Business process flow

A 3D stress test for bones, explained by AI agents.

Built for PointClickCare's Catch It Early challenge at Hack the Valley 11: catch bone weakness early, so prevention can start before the fall.

Who it's for: the care team in nursing homes and senior living: nurses, CNAs, wellness directors, doctors and admin staff.

Rainbone combines CT-derived 3D models, physics simulations and checked AI to help the care team see bone weakness early, before a fall becomes an emergency.

Rainbone starts from real research CT scans of the thighbone, each with a recorded follow-up It shows, in 3D and in colour, where a real femur takes the most strain under load. Then Gemini explains in plain words, for the care team, how that femur compares with the published research cohort.…

View on GitHub

Partner Technologies

We used four MLH partner technologies, and each one does a real job in the product.

Gemini API. Gemini writes every plain-language explanation in the app, and it runs the triage step of Ask Rainbone with native function calling to pick read-only tools. Both providers return structured JSON, and every answer goes through the checker before it's shown. Running two Gemini models (gemini-3.8-flash, then gemini-3.5-flash-lite) with a 300 s cooldown on HTTP 429 kept the app answering through free-tier rate limits. Gemini's answers are cached for an hour, and page loads that arrive together share one model call.

Vultr. The FastAPI and SfePy back end runs on a Vultr Compute VM behind Caddy HTTPS at api.rainbone.health. Our first build hit a wall: the 3D meshes made huge files that lagged the browser. Moving storage to Vultr Object Storage (S3-compatible, via boto3) and the heavy solvers to Vultr Compute fixed it. The pelvic meshes sit in a private bucket. The browser loads each one through a 15-minute pre-signed link and checks its sha256 before drawing it. The clinical facts the AI reads are sha256-pinned in the same storage.

Tiger Data. Tiger Cloud (TimescaleDB on PostgreSQL) holds two hypertables. simulation_runs stores every live solve and feeds a real-time hourly continuous aggregate that the dashboard reads back. ai_analyses audits every AI answer: the endpoint, provider, model, the fallbacks taken and why, latency, where the facts were read from, and the sha256 of the facts and of the text. It never stores the text itself. Anyone can read the latest rows at rainbone.health/api/ai-log.

GoDaddy Registry. Our domain is rainbone.health. The .health extension is run by GoDaddy Registry, and we registered the name through Porkbun. The apex serves the Vercel site, and api. points at the Vultr VM. The lesson: a registry is not a registrar.

The rest of the stack: Next.js 15, React 19, TypeScript and Tailwind CSS on Vercel, Groq as a backup model, TotalSegmentator for the pelvic CTs, and an optional C++23 mesh kernel. That kernel measured 15.5 to 16.1x faster than the NumPy path on the production API host, with a NumPy fallback if it can't be built.

Hackathon Experience

Hack the Valley 11 ran at the University of Toronto Scarborough. 408 hackers took part in teams of four, and 133 projects were submitted. My team was Galit Tauber, Camila Rojas Molina, Jisara Wuttikreekiat and me.

Rainbone wasn't our first idea. We started on a smart scrunchie, but it needed an accelerometer we couldn't get our hands on, so we pivoted the whole project on Saturday afternoon. Everything above was built after that.

What I'll remember most is the honesty work. Red on a stress map doesn't mean "about to break", so we built a separate failure-proximity view. A stress ratio isn't a probability, so we labelled it as an engineering ratio. Our first AI prompt used solid-bone stiffness for spongy bone, so we checked everything against the literature and stopped the model from ever guessing a number. Our biggest lesson was that a language model follows rules most of the time, and a checker is what decides what reaches the screen.

And I'll remember the last minutes: we were still adding the demo video links to Devpost with two minutes on the clock.

What's next: CT-density-calibrated material instead of one assumed stiffness, a mesh-convergence study, and the femur pipeline run on all 101 HFValid femurs, so we can compare our solve with the recorded outcomes.

Links: Devpost · Live app · Research paper · GitHub

If this was useful, share it with someone who builds health tools. Thanks for reading!