Building Spatial Memory Part 3: What I Learned Building a Mobile AR App with Go Backend

Honestly, I didn't expect to be writing the third article about this project so soon. After getting spatial search working pretty well (we went from 8 seconds to 20ms in case you missed the last article), I thought "cool, now just slap a mobile AR frontend on it and we're done."

Spoiler: it wasn't that simple.

I learned the hard way that building an AR app that actually works in the real world is way more than just dropping some ARCore frameworks in and calling it a day. After three weeks of debugging weird issues, throwing away half my code, and staring at my phone walking in circles around my neighborhood, I want to share what actually worked and what didn't.

The Idea: "Pinterest for the Physical World"

Just to catch you up if you're new here: Spatial Memory is an app where you "pin" notes, photos, and memories to physical GPS coordinates. You can only see the pin when you're actually physically near that location. So you could:

• Leave a note for your friend at their apartment door for their birthday
• Pin a hiking trail guide at the trailhead
• Leave a "hidden treasure" photo hunt around the city for people to find
• Remember where you parked your car (okay, that's been done a million times, but it's still useful)

I built the entire backend in Go with PostGIS for spatial search and Redis GEO for caching. That's what I covered in the first two articles. Now it was time for the mobile frontend. I decided to go with native Android and ARCore. Why? Because ARCore does a lot of the heavy lifting for plane detection and position tracking, right?

The First Surprise: AR Positioning Isn't That Accurate

Okay, I knew GPS wasn't perfect. I wrote about that in the first article. But what I didn't expect was how bad ARCore's position tracking drift gets over time.

Here's what happens: you start the app, ARCore finds a plane. It gets your initial GPS position from your phone. Then you start walking. ARCore does visual SLAM to track how you've moved relative to your starting point. But every step adds a tiny bit of error. After walking 50 meters, you can be off by 2-3 meters. After 100 meters, that drift can be 5+ meters.

That's a problem when your entire app relies on pins being at the correct physical location. A 5-meter drift means the pin could be floating in mid-air across the street instead of where you actually placed it.

So what did I do to fix it? Here's the compromises I ended up with:

1. Regular GPS Re-syncing

Every 10 seconds, I take the current GPS reading and blend it with ARCore's position. It's not perfect, but it keeps the drift bounded. Here's roughly how the blending works in Kotlin:

fun updatePosition(arPosition: Vector3, gpsPosition: GpsCoordinate): GpsCoordinate {
    // Convert AR relative movement to GPS coordinates
    val arOffset = calculateGpsOffsetFromMovement(arPosition, initialGpsPosition)
    val rawPosition = initialGpsPosition + arOffset

    // Blend: 70% AR tracking, 30% GPS. GPS keeps us from drifting too far.
    val blendFactor = 0.3f
    return blend(rawPosition, gpsPosition, blendFactor)
}

It's simple, but it works. The AR keeps the position smooth when you're moving around, and GPS keeps it from wandering too far from where you actually are.

2. Require "Fresh GPS" Before Placing Pins

If the app hasn't gotten a GPS update with accuracy better than 10 meters in the last 5 seconds, I don't let the user place a pin. I show a little warning:

"Waiting for better GPS signal... Try walking to an area with clearer sky view."

It's annoying, but it prevents people from placing pins way off where they intended. Bad data in = bad data out. Better to wait a few seconds than have a pin that's useless because it's across the street.

3. Accept That It's Not Perfect

Honestly? Some drift is still going to happen. GPS isn't perfect indoors or in city canyons. AR drift is real. I've accepted that this is a limitation of current hardware. I just document it in the FAQ: "If your pin looks off, just move closer and the app will correct it eventually."

Users are surprisingly understanding when you're honest about limitations.

The Second Surprise: Rendering Pins at the Right Distance Is Tricky

ARCore gives you a camera view with a pose matrix, and you put your anchors where you want them. That works great when pins are within 5-10 meters. But when you have pins that are 100+ meters away (which you do in a location-based app), things get weird.

The further a pin is from the camera, the more small errors in position are amplified. A 1-meter position error at 100 meters doesn't sound like much, but it makes the pin look like it's floating way off where it should be.

Also, rendering too many pins in AR kills your frame rate. If you're in a park with 50 pins nearby, you don't want all 50 rendered at once. That's just visual clutter and it slows everything down.

Here's what I ended up doing:

1. Two Different Rendering Modes

Close pins (< 20 meters): Full 3D rendering anchored to the physical world. People can interact with them, tap them, see the details.

Distant pins (20 - 200 meters): Just a 2D billboard in the direction of the pin, drawn on top of the camera feed. It tells you "there's a pin that way" without trying to anchor it perfectly in 3D space. Perfect 3D isn't needed when you just need direction.

Here's a simplified version of the distance check:

fun shouldRenderAsBillboard(pin: Pin): Boolean {
    val distance = calculateDistance(currentGps, pin.gpsCoordinate)
    return distance > 20.0f // meters
}

Simple, effective. It keeps the frame rate up and avoids weird 3D floating pin syndrome at long distances.

2. Cull Everything Beyond 200 Meters

If a pin is more than 200 meters away, I don't show it at all. What's the point? You can't interact with it anyway. Just show it when you get closer. Reduces rendering load, keeps the scene clean.

Again, it's a compromise, but it's the right compromise for this kind of app.

3. LOD Scaling for Distant Pins

Smaller pins when they're further away. Duh, right? But you'd be surprised how many AR apps I've seen that don't do this. A pin that's 2 meters big at 5 meters should be 10 meters big at 50 meters to look the same size on your screen. If you don't scale it, distant pins look tiny and you can't see them.

fun getPinScale(distance: Float): Float {
    val baseSize = 1.5f // meters
    // Scale proportional to distance so apparent size stays constant
    return baseSize * (distance / 10f)
}

This was a game-changer for discoverability. Before I added this, people would walk right past pins because they couldn't see them. After this, they pop even when they're far away.

The Third Surprise: Battery Drain Is Real

Oh wow. ARCore eats battery. Like, really eats battery. If you leave AR mode on with the camera open for 30 minutes, your phone is going to be dead. I know, you're probably thinking "all AR apps drain battery, duh." But I didn't expect how much.

My daily driver is a Pixel 7. With Spatial Memory open in AR mode, I was getting about 3 hours of battery life. That's just not acceptable for an app you might use while hiking for a day.

So what optimizations did I make? Here are the ones that made the biggest difference:

1. "Low Power Mode" When The App Is In Background

Android gives you a callback when your app goes into the background. I immediately pause ARCore session and stop all rendering when that happens. Don't just let it run in the background. Users forget to close apps, and that's how you end up with a dead battery.

override fun onPause() {
    super.onPause()
    if (arSession != null) {
        arSession!!.pause()
    }
}

override fun onResume() {
    super.onResume()
    if (arSession != null && arSession!!.isPaused) {
        arSession!!.resume()
    }
}

This seems obvious, but I forgot to do it the first time. Big mistake. Don't be me.

2. Option to Disable AR for Just GPS Mode

A lot of times, people just want to find pins, they don't need augmented reality. I added a toggle: "AR Mode On / Off". If AR is off, you just get a map and compass pointing to the pin. No camera, no ARCore, battery lasts 5x longer.

This is my go-to when I'm just hiking around looking for pins. Why drain the battery when you don't need the AR effect?

3. Throttle Location Updates When Not Moving

I use the Android Activity Recognition API to detect when the user is stationary. If they haven't moved in 30 seconds, I drop the GPS update rate from 1Hz to 0.1Hz (every 10 seconds). Doesn't affect usability, but saves a surprising amount of battery.

Again, simple optimization, big impact.

Code Snippet: My Simplified AR Position Fusion

Let me show you the actual (simplified) code I use to fuse AR tracking with GPS. It's not rocket science, but it took me a while to get the blending right.

First, we keep track of the starting point:

data class GpsCoordinate(val latitude: Double, val longitude: Double)

class ArGpsFuser {
    private var initialGps: GpsCoordinate? = null
    private var initialArPosition: Vector3? = null
    private val blendFactor = 0.3f // 30% GPS, 70% AR

    fun startNewSession(startGps: GpsCoordinate, startArPosition: Vector3) {
        initialGps = startGps
        initialArPosition = startArPosition
    }

    fun getCurrentPosition(currentArPosition: Vector3, latestGps: GpsCoordinate?): GpsCoordinate {
        checkNotNull(initialGps) { "Session not started" }
        checkNotNull(initialArPosition) { "Session not started" }

        // Calculate how far we've moved in AR space relative to start
        val deltaX = currentArPosition.x - initialArPosition.x
        val deltaZ = currentArPosition.z - initialArPosition.z // forward/backward in ARCore

        // Convert meters movement to GPS delta (approximation)
        val deltaLat = deltaZ / 111000.0 // 111km per degree latitude
        val deltaLon = deltaX / (111000.0 * Math.cos(Math.toRadians(initialGps.latitude)))

        val arEstimatedLat = initialGps.latitude + deltaLat
        val arEstimatedLon = initialGps.longitude + deltaLon

        // If we don't have a fresh GPS, just return AR estimate
        if (latestGps == null || !isFresh(latestGps)) {
            return GpsCoordinate(arEstimatedLat, arEstimatedLon)
        }

        // Blend them
        val blendedLat = arEstimatedLat * (1 - blendFactor) + latestGps.latitude * blendFactor
        val blendedLon = arEstimatedLon * (1 - blendFactor) + latestGps.longitude * blendFactor

        return GpsCoordinate(blendedLat, blendedLon)
    }

    private fun isFresh(gps: GpsCoordinate): Boolean {
        // Check that GPS timestamp is within last 10 seconds
        return (System.currentTimeMillis() - gps.timestamp) < 10000
    }
}

This is the core of my solution. It's simple enough that I can debug it when something goes wrong, which is way more important than being fancy.

Pros & Cons: How's It Working After Three Weeks?

I'm going to be honest with you—this is my side project, and it's not a finished product. Let me break it down straight:

Pros ✅

1. It actually works most of the time. When you're outside with good GPS reception, pins stay where you put them within a meter or two. That's good enough for the use case.

2. The two-tier rendering (full 3D close, billboard far) works really well. No more floating pin weirdness, frame rate stays above 60fps even with 20 pins nearby.

3. Battery life is acceptable with the optimizations. I get about 6-7 hours with AR mode on and off, which is enough for a day of hiking. Low power mode goes way longer.

4. Go backend scales really well for a side project. I'm hosting on Fly.io with the free tier, and it handles everything I throw at it. Cold starts are annoying, but for a side project with a handful of users, it's fine.

Cons ❌

1. Indoor AR is still pretty bad. GPS doesn't work inside, and visual drift gets crazy. I just don't advertise indoor use right now. Maybe when ARCore gets better with global localization.

2. Different phones have wildly different AR performance. My Pixel 7 works great. My old Samsung A50 struggles to hit 30fps. It's just the nature of Android, but it's still frustrating.

3. Cold start is a problem. Like all location-based social apps, you need users to have enough pins in an area for it to be useful. Right now, all the pins are in my neighborhood because I'm the only one testing it. That's just a chicken-and-egg problem that only growth would fix.

4. Apple support doesn't exist yet. I only built for Android because I'm an Android guy and I only have one Mac that's super old. If someone wants to help with iOS/ARKit, let me know!

Would I Do It Again?

Yeah, honestly. I learned so much building this. When I started, I just had this silly idea: "what if you could leave notes in physical space?" And now I actually have a working app that does that. Sure, it's got rough edges, but that's okay for a side project.

The biggest lesson I keep learning over and over: solving hard problems with modern hardware still requires a lot of small compromises. All the tutorials show you how to get something basic working, but they don't tell you about all the little adjustments you need to make to get it working well.

AR positioning isn't perfect, so you blend it with GPS. Full 3D rendering doesn't work for distant pins, so you use 2D billboards. AR drains battery, so you add low-power modes. It's all tradeoffs.

If you're thinking about building an AR location-based app yourself, my advice is: start simple. Don't try to boil the ocean. Get one thing working well, then add the next thing. And test outside in real conditions, not just in your living room. I can't tell you how many problems only showed up when I started walking around the block.

---

The project is fully open source on GitHub if you want to check it out or contribute:

👉 https://github.com/kevinten10/spatial-memory

I've got the backend all there, the Android app is coming along, there are still plenty of TODOs if you're looking for a side project to contribute to.

Question For You

I'm curious—have you ever built an AR app with location-based positioning? What surprised you the most? Did you run into the same drift issues, or did you find a better way to handle it? Drop a comment below—I'd love to hear your experiences.