FirstHour : 2nd Prize Winners for Espressif and StepFun

#mlhacks #devchallenge #hackathon

This is a submission for the MLH x DEV Writing Challenge

What I Built

Imagine an older adult slipping in the bathroom while home alone. They’re conscious but can’t get up. The phone is in another room, and the emergency pendant is on the nightstand. That’s the moment FirstHour is built for. More than 1 in 4 adults over 65 fall every year, and often the most dangerous part isn’t the fall itself. It’s the time spent on the floor waiting for someone to notice.

FirstHour is a small, camera-free safety device for the bathroom. An Espressif ESP32-S3-BOX-3 listens for the thud and feels the vibration of a fall. A Raspberry Pi 5 running QNX confirms it and immediately starts a local alarm and a time-down clock, which keep running even if the device loses its connection. A calm voice then asks, “Are you okay?” The person’s spoken answers are transcribed on the Pi with whisper.cpp, and the family gets an instant summary by iMessage, with a phone call from a voice agent if needed. Meanwhile, StepFun’s step-5-preview refines that summary in the background. To let help in, a caregiver texts back a one-time code, and the Pi double-checks every request before the door latch opens (simulated in our prototype), so a stranger can’t get in.

The biggest thing we learned is that safety can’t wait on AI or the network. The device always raises the alarm first, and the AI only refines the summary afterward. We also learned that hardware is humbling: getting a Raspberry Pi on QNX to talk to the BOX-3 over USB took far more patience than any code we wrote. FirstHour is a prototype, not a medical device, but we’re proud of it and would love to keep building it alongside clinicians and older adults.

Demo

Here’s what happens, minute by minute, when someone falls in a bathroom with FirstHour.

The fall: We knock on the table to stand in for a fall. Nobody needs to actually fall for this. The Espressif ESP32-S3-BOX-3 hears the thud and feels the impact at the same moment. The instant they do, the box’s screen flips to TIME DOWN 00:00 and an alarm starts. That decision is made right there, by a Raspberry Pi 5 running QNX, without waiting on the internet.

The check-in: A calm voice says it’s an automated assistant and asks, “Are you okay?” The person just answers out loud, no buttons and no phone. The Pi transcribes the answer itself using whisper.cpp. If they clearly say “I’m fine”, everything stands down. If they ask for help, or say nothing at all, it escalates.

The alert: Their family gets an iMessage within seconds: how long the person has been down, what’s wrong, and a note that they’re on a blood thinner. That last part comes from their profile, never from the AI. If needed, a voice agent calls the family too. Meanwhile, StepFun’s step-5-preview quietly refines the summary in the background. On the dashboard you can see the rule-based summary and the AI’s version side by side, right next to the person’s exact words.

The door: A stranger texts the unlock code and is turned away, because they’re not on the list. Then the real caregiver texts the one-time code. The Pi checks that the command is signed, fresh and never used before, and only then does the latch open (simulated in our prototype). It locks itself again a few seconds later.

Under the hood: Every incident becomes one Sentry trace, so we can see exactly what happened and how long each step took.

Partner Technologies

Yes. ElevenLabs is the voice of FirstHour. When someone falls, the first thing they hear is ElevenLabs, and if their family needs a phone call, that’s ElevenLabs too. We used it in three ways.

The voice that checks in (text-to-speech)

After a fall, the device has to talk to the person right away. If they’ve hit their head or are scared, a slow or robotic voice makes things worse.

Pre-cached for speed: We call ElevenLabs’ text-to-speech API with the low-latency eleven_turbo_v2_5 model and a calm stock voice. Every fixed line is generated ahead of time and cached to disk: “This is an automated assistant”, “Are you okay?”, and the follow-up triage questions. When a fall happens, the first question plays instantly, with no API call in the critical moment.
One line at a time: A single playback worker plays lines in order so they never overlap. If the API is slow or unavailable, playback falls back to the Mac’s built-in voice so the check-in never goes silent.
The silence timer starts after the question finishes. Silence means “escalate”, so the timer only begins once the question has finished playing. A long sentence never eats into someone’s time to answer.
Honest about being a machine. The voice always introduces itself as an automated assistant. It never pretends to be a family member.

The escalation phone call (Conversational AI agent)

If the person needs help, a text isn’t always enough, so we built a voice agent that calls the caregiver.

How the call is routed: the call goes out from a Photon phone number to our ElevenLabs agent over SIP.
What the agent knows: it gets the incident context as variables: who fell, where, how long they’ve been down, the triage summary, and whether they’re on a blood thinner. A get_incident tool lets it look up fresh facts mid-call.
What it asks: whether the caregiver can come, and their ETA. The answer is sent back to our backend after the call.
Strict guardrails: the agent never guesses or gives medical advice, and it never reads, accepts or invents a door code. Voice can’t unlock the door. The code is only ever sent by text, after the call.
Verified for real: the backend placed two real escalation calls. The agent collected an ETA and refused to unlock.

A companion conversation (experimental)

We also built a second agent, “FirstHour companion”, that keeps talking to the person on the floor after the fixed questions. Its tools let it read the incident facts and relay a message to the caregiver. It’s limited to six relays per incident, and it rejects any door code.

Our experience:
The good: the stock voice sounded natural and calm on the free tier, and turbo model plus caching made the first words instant. Building an agent that could take a real phone call in a few hours was the highlight of the weekend.

The hard parts:
Our webhook ran through a temporary tunnel that kept dropping, and one finished call’s result never arrived. We fixed it by also polling ElevenLabs for finished calls, and wrote a script that re-points the webhook whenever the tunnel changes.
Writing the agent prompts carefully took real iteration. For a safety product, what the agent won’t do (diagnose, promise delivery, touch the door code) mattered as much as what it does.

Free-tier limit: voice cloning isn’t available on our tier, so we used a stock voice. That’s arguably the right call anyway: the assistant shouldn’t sound like someone’s daughter.

Hackathon Experience

I attended LA Hacks 2026 at UCLA, and it was one of the most energizing weekends I’ve had. From the opening ceremony on, the whole place buzzed. There were people sprawled across tables with laptops and tangled cables, hardware piled up everywhere, and the hum of ideas being tested at 3 AM.

Two of my teammates and I built FirstHour, a camera-free safety device that helps older adults in the first hour after a fall. None of us had much hardware experience going in, so the weekend became a crash course in Raspberry Pis, QNX and the Espressif ESP32-S3-BOX-3. We failed, debugged, laughed and tried again more times than I can count.

Beyond building, I loved the events around the hackathon: the workshops, the sponsor booths, and the late-night activities that gave us a much-needed break. I met so many fellow hackers working on completely different ideas, and swapping stories with them was half the fun. Talking with engineers and recruiters from the sponsor companies taught me a lot about how their technology works in the real world. Walking the judges through our project, and answering their tough questions about safety and privacy, pushed us to think more deeply about what we’d built.

What I’ll remember most is the moment it all came together on real hardware: a drop on the table, the box lighting up with TIME DOWN, and a calm voice asking “Are you okay?” Winning second place in both the StepFun and Espressif challenges was the perfect ending to a weekend full of learning, new friends and very little sleep. 💙