A few years ago I would have told you AR was a graphics problem. You get a 3D model, you get a camera feed, you draw one on top of the other. Ship it.
Then I built location-based AR for real, and the graphics turned out to be the least interesting part of the whole thing.
The build that taught me this was Planes XR, one of the location-based AR experiences I worked on for ARCortex. The pitch is simple to say and brutal to deliver: point your phone at the sky, and the actual aircraft flying overhead show up, positioned where they really are. Not a canned demo of a plane. The plane that is genuinely up there right now, pulled from live flight data, rendered in the correct patch of sky.
If you have only ever done marker-based or surface AR, that one requirement changes almost every engineering decision you make. So I want to walk through why, because the lesson generalizes well past AR.
The three flavors of AR are not the same job
It is worth being precise, because "AR" gets used for three genuinely different builds:
- Marker-based AR shows content when the camera recognizes a specific image or QR code. Reliable, but something physical has to be in frame.
- Surface AR drops content onto a detected flat surface near you, a floor or a table. This is your furniture-in-your-living-room app.
- Location-based AR anchors content to real-world coordinates, so it works across a whole site, a city, or the open sky, with no marker at all.
Marker and surface AR are, engineering-wise, mostly solved problems with good SDK support. The moment your experience has to span something bigger than a tabletop, you cross into location-based territory, and the difficulty steps up hard.
Location-based AR is really three systems pretending to be one
A location-based AR app has to know three things at the same instant: where the user is, which way they are facing, and where the content is supposed to live in the world. Get all three aligned and a digital object sits convincingly in a real street or a real sky. Get any one wrong and the illusion collapses instantly, because humans are shockingly good at noticing when something floats in the wrong spot.
That splits into three subsystems, and each one is a place the whole thing can quietly fall apart.
Positioning
The app fuses several signals to place you in the world: GPS for a rough fix, the compass and motion sensors for heading, and the camera itself for fine correction. The thing nobody tells you up front is that raw GPS is only accurate to several meters. That is completely fine for "you are near this landmark" and completely useless for "this arrow points at that specific doorway." Closing that gap, from good-enough-for-a-map to good-enough-to-overlay, is one of the core engineering problems in the entire category. You do not solve it with a better model. You solve it with sensor fusion.
Anchoring
Once you know where the user is, you have to lock the content to the world and hold it there as they walk, turn, and pan around. If the anchor drifts even slightly, the object slides off its spot and the experience reads as broken. Stable anchoring is the whole difference between an overlay that feels physically present and one that visibly wobbles, and it gets harder outdoors, at distance, and across large areas, which is to say exactly the conditions location-based AR lives in.
Data
Here is the part that ate the most of my time. The interesting location-based experiences are almost never showing a static model. They are showing something tied to what is happening right now at that place, which means a live-data pipeline feeding the immersive layer. For Planes XR that meant pulling live aircraft positions from OpenSky, keeping them current, and rendering each plane where it actually is in the sky relative to where the user is standing and pointing.
That pipeline, fetching the data, cleaning it, keeping it fresh, and handling the moments it is slow or simply unavailable, was as much work as the AR rendering itself. The overlay is the part users see. The data plumbing is the part that decides whether they trust it.
The failure modes live in the seams, not the scene
That sentence is the thing I actually took away from the build, and it is why I keep bringing it up in other work.
Every part of Planes XR that was hard to get right lived in a seam between two systems, not inside any one of them. The 3D plane model rendered fine on day one. What broke was the plane rendering in the correct spot only when positioning, heading, and the data feed all agreed at the same moment. When a GPS reading was stale, or the compass drifted a few degrees, or the OpenSky feed lagged, no single component was "wrong," but the plane ended up in the wrong piece of sky and the whole thing felt fake.
You debug those problems by staring at the boundaries between subsystems, not by staring at any subsystem alone. That is a very different debugging mindset from a normal app, where a bug usually lives in one function you can put a breakpoint in.
How I scope one now, so it does not become a science project
The most expensive mistake in this category is commissioning a broad, city-scale, multi-feature experience before proving the core alignment even feels right. So the way I approach it now is deliberately narrow at the start:
- Prototype the hardest alignment in week one. If the whole thing hinges on content landing accurately at a real place, build exactly that first, at one single location, and then go physically stand in the real spot with a real device. Everything else is comparatively predictable once positioning and anchoring feel right.
- Prove one location before you build a hundred. Nail a single site, or a single sky, measure whether the overlay holds and the data stays trustworthy, then expand. Coverage is a multiplier, not a starting point.
- Treat the data pipeline as its own project. If live data is part of the value, budget for it as a first-class workstream, including the unglamorous parts: freshness, error handling, and graceful failure when a feed drops. It will be bigger than you think.
- Start on phone AR unless a headset is truly essential. Phones reach the widest audience with no hardware to buy, which makes them the right place to validate the concept before you commit to a headset's economics.
The test for whether you even need location-based AR is one honest question: does being at the real place matter? If yes, this is the tool, and it is worth the extra difficulty. If the content would work just as well on your coffee table or from scanning a printed marker, use a cheaper AR mode and do not pay for this one.
The part that transfers
I do not build AR every week. But "the hard, valuable engineering is in the alignment and the data, not the thing you can see" turned out to be one of the more portable lessons I have picked up. It is true of AR. It is true of most systems that touch the physical world, and it is true of a surprising number that do not. The demo shows you the scene. The product lives or dies in the seams.
Planes XR stopped being a novelty the moment those seams held. That is the whole game.
If you want the longer, scoping-oriented version of this with the full cost breakdown, I wrote it up on the studio blog at nullstud.io.
Top comments (0)