Building FireSight at LA Hacks: Bringing a Drone’s View to Smart Glasses

Building FireSight at LA Hacks: Bringing a Drone’s View to Smart Glasses
The view from above, delivered to the people on the ground. Eyes up, hands free.
What happens when a drone can see a hazard, but the person approaching it is busy watching their surroundings?
That question became the starting point for FireSight, our LA Hacks 2026 project. We wanted to connect aerial footage, AI analysis, a command dashboard, and smart glasses so that useful information could reach someone without requiring them to stop and check another screen.
Wildfire awareness was our starting point. Our broader vision includes hikers, residents, and field teams who could benefit from a better understanding of what is happening nearby.
Over the hackathon, the project grew from a drone-to-glasses concept into a prototype with geographic hazard annotations, wearer location reporting, a glasses mini-map, and SOS and man-down features.
💻 GitHub repository
🌐 FireSight dashboard

1. We started with the information gap Aerial video contains a lot of information, but someone still has to interpret it and communicate what matters to the person on the ground. We focused on three questions:
2. Where is the nearest mapped hazard?
3. Approximately how far away is it?
4. Is the wearer moving closer to it or farther away? We also wanted information to travel back. A coordinator should be able to see the wearer’s reported position, device status, and requests for help. That led us to a two-way system: aerial information travels toward the wearer, while location and status reports travel back toward command.
5. We divided the system into connected parts Our architecture connects several environments:

DJI drone
↓
Beagle Eyes: video, visual odometry, and geographic telemetry
↓
FireSight backend: ingestion, analysis, and hazard mapping
↓
Even Realities phone app ↔ G2 smart glasses
↓
Wearer location, telemetry, and acknowledgements
↓
Shared command dashboard

The backend uses Python and FastAPI, with OpenCV, NumPy, and Google’s Gemini SDK for image processing and analysis. The dashboard uses MapLibre, while the glasses app uses TypeScript, Vite, and the Even Hub SDK.
We kept GPU-dependent drone processing on a separate computer and deployed the backend and dashboard together on Render.

1. We worked through the drone integration The drone side uses Beagle Eyes, which processes video and uses deep visual odometry to estimate camera motion. One of our biggest integration problems was agreeing on what a “frame” actually included. An image arriving near a telemetry message does not necessarily mean they describe the same instant. That matters when the system uses camera position and orientation to place an annotation on a map. We built a push integration around paired image and telemetry bundles. Each upload includes a source frame ID, sender session, timestamp, JPEG data, and checksum. The receiver can reject duplicate or older frames and preserve source identity through analysis. This gives us a clearer way to trace an alert back to its input. We also used outbound HTTPS uploads so the drone computer could send data to the hosted backend without requiring an inbound tunnel for that path.
2. We designed the pipeline to avoid falling behind Video capture and AI inference run at different speeds. Processing every captured frame can create a growing queue, leaving the system describing something that happened several seconds ago. We used bounded, latest-only buffering. The sender keeps one pending frame, and analysis can skip intermediate frames to work on newer information. That was a useful engineering lesson: for this kind of prototype, freshness can matter more than processing every frame. We also added timestamps and stale-source handling so an old image or alert would not silently appear current.
3. We added AI analysis and operator-driven annotations FireSight includes an OpenCV heuristic analysis path and a Gemini vision-language-model path. The Gemini integration supports structured alerts formatted for the glasses. We also added dashboard controls where an operator can request annotations such as “fire, smoke, people.” The annotation pipeline takes detected image regions and uses camera geometry and geographic telemetry to estimate their positions on the ground. This introduced an important distinction: detecting something in an image and locating it geographically are separate problems. Map placement depends on the camera pose, geographic transform, and scale calibration. We therefore treat projected hazard positions as approximate. Annotations also expire rather than remaining on the map indefinitely.
4. We built the command dashboard We developed a map-first dashboard that brings together:
5. Drone video and frame information.
6. Geographic annotations with categories and visibility controls.
7. Reported wearer positions.
8. Motion and battery status.
9. SOS and man-down banners with reported coordinates.
10. A G2 display card showing the text the glasses app reported rendering. We refined the map styling, added geographic-origin handling, and made new annotations easier to locate. A surprisingly useful feature was the display-reporting loop. Publishing an alert on the backend does not tell us whether the glasses app accepted it. The app reports display status back to the server, making it easier to inspect the connection between backend, phone, SDK, and dashboard. A separate wearer tap can acknowledge an alert; SDK acceptance alone does not prove someone read it.
11. We made the information personal to the wearer Next, we added phone GPS reporting through the Even app. The backend uses that reported position to calculate distance and bearing to mapped hazards. This lets the glasses show messages such as: Demo fire ~160 m NE Closing in

We added distance-based alert priorities, including a critical state below 100 metres, and a north-up mini-map centered on the wearer. The map scale changes with hazard distance.
Those thresholds are prototype behavior, rather than validated safety boundaries.
We also had to work within the hardware’s limits. The glasses do not provide reliable head-facing compass orientation, so we use geographic directions such as northeast rather than claiming a hazard is directly to the wearer’s left or right.

1. We added SOS and a man-down check Once the glasses could send location and telemetry, we explored how the wearer could request help. Holding a temple sends an SOS, which appears on the dashboard with the wearer’s reported position. We also implemented an experimental fall-check sequence:
2. Detect a strong acceleration event.
3. Check for stillness afterward.
4. Ask, “Are you OK?”
5. Allow the wearer to tap to confirm.
6. Report MAN DOWN if there is no confirmation within 20 seconds. A single movement spike was not enough. We needed a sequence that considered impact, continued motion, stillness, and user response. We added automated tests for scenarios including impact followed by stillness, continued movement after impact, and cancellation. This remains a prototype heuristic that needs much more physical testing.
7. We debugged the hardware and installation details Some of our most practical challenges came from SDK behavior and deployment. The glasses documentation describes two SDK issues we worked around: input events arriving through a different event field than the starter example, and hot reload attempting to recreate a startup page that can only be created once per session. We also learned that package installation needs careful version management. Replacing a package without increasing its version can leave the phone running an older copy. Networking introduced another familiar trap: 127.0.0.1 means the device opening the connection. A hosted dashboard pointing there cannot reach a teammate’s drone computer. These issues pushed us to document installation, permissions, backend configuration, package updates, and connection troubleshooting.
8. We consolidated the project and made it easier to demonstrate Different branches held different pieces of the application. Bringing them together required preserving compatible API contracts, the newer glasses behavior, and the map-first dashboard. We deployed from main and added a compatibility fix so deployment would honor the setting that pauses drone-alert delivery. We also built simulator inputs and an opt-in, explicitly labeled demo hazard. That lets us exercise GPS distances, map updates, and wearable interactions without needing a drone flight for every development cycle. The repository includes automated tests covering ingestion, analysis, annotations, wearer reporting, display behavior, and safety-state logic. These checks help verify individual components; they do not replace a continuous test with a real drone, live model responses, phone, and physical glasses. What we learned—and what comes next The hardest part was making the connections between components trustworthy: pairing images with telemetry, keeping information fresh, distinguishing display acceptance from human acknowledgement, and translating geographic data into something readable on a small display. The repository now contains those connected components, but a repeatable, fully verified drone-to-glasses field demonstration remains an important next step. We also want to improve geographic calibration, evaluate false alerts, add access control for private location data, persist state across restarts, and explore offline behavior. Voice capture exists behind a feature flag, but its backend receiver is still unfinished. FireSight is a hackathon prototype, and we have more work to do before it could support real emergency use. Building it helped us explore how aerial perception, wearable interfaces, and two-way field communication can work together. Our goal is to make aerial information easier to understand—and easier to reach while keeping your eyes up. Explore the code, architecture, installation guides, and limitations in our GitHub repository.

https://github.com/lily1c/la-hacks-2026

LAHacks #AI #Drones #WearableTech

Comments

[Tyler N]

This is an awesome project! I’ve never seen anything like it!