I built a browser app that turns a live camera feed into 3D sound, so a visually impaired person can hear where things are. It runs entirely on-device — no cloud, no network. What exists today is a working prototype in a web browser; the larger goal is a native phone app running on dedicated AI silicon.
This post walks through the idea, what actually works right now, and the honest gap between the two.
Live demo: https://saartheye-ai.vercel.app
Code: https://github.com/MaybeSomeone-arc18/saartheye-ai
Why a phone in the cloud is too slow to keep you safe
A safety warning that arrives a second late is useless. Most AI vision apps send a camera frame to a server, run the model there, and send the answer back. That round trip takes roughly 1.2–3.5 seconds. A person walking at a normal pace covers nearly five feet in that window — long enough to have already hit the obstacle the app is warning about.
There are three problems with the cloud approach for real-time navigation:
- Latency. The round trip is slower than the thing it's meant to prevent.
- Dead zones. Elevators, basements, subways, parking garages — the places you most need help are the places with no signal. A cloud app stops working the moment the connection drops.
- Privacy. A navigation aid you wear all day streams a 24/7 video of your entire life — your home, your workplace, the strangers around you — to someone else's servers.
Doing everything on the device sidesteps all three at once: no round trip, no signal required, and camera frames never leave the phone. That premise is the whole reason this project exists.
What I actually built
Everything happens inside the web page — there's no backend server at all. The prototype opens your camera, spots objects in the video, and plays them back to you as spatial audio in real time.
- Vision: a YOLO-based object detector running through TensorFlow.js, so inference happens on your machine, not a server.
- Audio: the browser's Web Audio API, which can place a sound at a point in 3D space around your head.
- Interface: a React layer that draws the detection boxes and mode controls on screen.
Why the browser first? It was the fastest way to prove the core idea is real. It let me test the full loop — camera → detection → sound — with zero install, on any laptop or phone, in a few days of building. It's the proof of concept, not the finished product.
From pixels to sound
1. Finding things: the detector
Each video frame goes through an object detector that returns a list of what it sees and where. For every object, the model gives back a label, a confidence score, and a box (its position and size on screen). That box is the single most important piece of data — everything downstream is built from it.
interface DetectedObject {
class: string; // e.g. 'person', 'car', 'chair'
score: number; // confidence, 0.0 to 1.0
bbox: [
number, // x (top-left)
number, // y (top-left)
number, // width
number // height
];
spatialVelocity?: number; // how fast it's approaching (see below)
}
2. Turning a box into a sound you can locate
The trick is echolocation: map where an object sits in the frame to where its sound sits around your head. Two simple mappings do most of the work, and because they're just arithmetic on the box, they're effectively instant.
| What you hear | Comes from | So that… |
|---|---|---|
| Left / right (stereo pan) | The box's horizontal position | An object on your left sounds on your left |
| Pitch (high / low) | The box's size on screen | A bigger, closer object sounds lower and nearer |
| Urgency of the pulse | How fast the object is approaching | A fast approach sounds more insistent |
Under the hood this uses the Web Audio API's PannerNode, which is designed to position sound in 3D. The result: you don't get a robotic voice saying "chair, two o'clock" — you just hear the chair, roughly where it is.
3. Telling a threat from a table: velocity vectoring
Distance alone is a bad alarm. A chair one meter away isn't dangerous; a cyclist one meter away and closing is. The distinction isn't how close something is, but whether it's getting closer — and how fast.
The prototype keeps a short memory of each object's box across recent frames and watches whether that box is growing. A box that swells quickly frame-over-frame means something is rushing toward you, so it escalates to a sharp, panned warning tone. A box that holds steady is treated as furniture or a standing person and fades into a soft, occasional ping.
4. Reading the room: contextual modes
An aid that screams at everything gets switched off within a minute. Avoiding that "false-positive fatigue" is a feature, not an afterthought.
- Outdoor / navigation: high sensitivity to anything moving or in your path — bikes, poles, stairs, people.
- Social / conversation: a person standing still in front of you is a friend, not a hazard, so their alarm is suppressed to a gentle ambient ping. Only a sudden approach breaks through.
The honest gap: prototype vs. blueprint
The architecture I designed is more ambitious than the thing I've shipped, and it's worth being clear about which is which. The browser prototype proves the concept end to end. The native version is the target, not a claim about today.
| Prototype (today) | Blueprint (the goal) | |
|---|---|---|
| Runs on | Web browser tab | Native Android app |
| Detection | YOLO via TensorFlow.js | Quantized YOLOv12 on the phone's neural chip |
| Reasoning | Bounding-box math in JavaScript | On-device small language model (Phi-3.5) |
| Feedback | 3D spatial audio | Spatial audio + haptic vibration |
| Latency | Fast, browser-limited (unmeasured) | Sub-15ms target on dedicated silicon |
| Offline | Yes, once loaded | Yes, full airplane mode by design |
The prototype already answers the risky question: can bounding boxes become useful, locatable sound in real time? Yes. The remaining work — native memory handling, offloading the model to a dedicated neural processor, adding haptics — is a systems-engineering climb, not an unknown. The concept is de-risked; the performance is the next mountain.
Where it goes next
- Port the pipeline to native so camera frames flow straight into the model without the browser's overhead.
- Offload detection to the phone's neural chip to chase the sub-15ms target and free the main processor.
- Add haptics so a warning you can feel backs up the one you hear, for loud environments.
- Test with actual users, because the real measure isn't latency — it's whether someone trusts it enough to leave the cane at home.
You can build the hard, scary part of an idea in a browser in a weekend and learn whether it's worth pursuing. This started as a question about whether sound could stand in for sight, closely enough to be safe. The prototype says it's possible. The rest is engineering.
The name comes from the Sanskrit Saarthi — a charioteer, the one who guides — joined with Eye.
Top comments (0)