DEV Community

Sam Ellis
Sam Ellis

Posted on

What we learned measuring a location game: the metrics that actually tell you if it works

Most of the advice you get about measuring an app is written for apps that happen when people sit down. Open it, do a thing, close it. Session length, daily actives, retention curve. You measure a funnel and you optimise the top of it.

A location game does not work like that. Your player is walking, in the real world, holding a phone in one hand, often in low light or poor signal, and they will leave. Not because they are bored — because they have reached a place where the hunt cannot continue, or because it is raining, or because their phone died.

That changes what you should be measuring, quite a lot.

Session length is the wrong headline number

The instinct is to watch how long people play. For a walking game this is actively misleading, because a long session can mean the opposite of what you want. A player who walks a hunt for ninety minutes because they could not find the third spot is not having a good time; they are stuck and stubborn.

We track hunt completion rate as the headline instead: of the hunts people start, what proportion do they finish? A completion rate is a much better proxy for "the game works", because every step of a hunt is a place where the game can lose the player, and the completion rate is the product of all of those steps.

If completion is high, the difficulty is right. If it is low, you have a funnel problem, and now you know to look at the funnel.

Where people drop off tells you which thing is broken

The single most useful measurement is the distribution of abandon points. If you log the index of the last spot a player reached, you get a curve. The shape is more informative than the average.

Different shapes mean different bugs:

  • A cliff at one specific spot means that spot is unfair. It might be visually ambiguous, or physically unreachable, or the camera match fails on a specific image. This is a content bug and it is easy to fix once you know.
  • A gentle decline across all spots means the hunt is simply too long or too hard, and the fix is pacing rather than content.
  • A cliff at the first spot means the very first unlock is confusing — the player does not understand what "match the scene" means, and they walk away. That is an onboarding problem disguised as a difficulty problem, and no amount of tuning the difficulty will fix it.
  • A spike of abandonment right after a hint is the interesting one. It suggests hints are working but the player has decided the hunt is no longer worth finishing. If your hints are that effective, they may be giving away too much, or revealing that the spot was never findable in the first place.

We keep a per-spot abandon list and treat any spot above a threshold as needing a look. It is unglamorous and it is the single highest-leverage thing we do.

Hint usage is a difficulty instrument

Every product that can be helped has a help button, and most teams treat help usage as a failure signal. For a location game it is much closer to a difficulty gauge, and the pattern matters more than the rate.

  • High hint usage on the first spot of a hunt usually means the player is new to the format, not that the spot is bad.
  • High hint usage spread evenly means the hunt is hard.
  • High hint usage on one spot means that spot is unclear.
  • Hint usage that rises every hunt for the same player means they are stuck in a loop of starting and failing, and they need something different — an easier hunt, a different area, or a nudge to come back later.

That last one is worth watching, because it is a small group of people who will never tell you they are frustrated.

Time to first unlock is the onboarding metric

If there is one number I would put on a dashboard for a location game, it is time to first unlock — how long from starting a hunt until the player successfully matches their first spot.

It is high-leverage because it captures everything that can go wrong at the beginning: the player cannot find the first spot, they do not understand the camera match, the GPS has not converged, they have the wrong permissions, or they simply walked the wrong way. Every one of those has a completely different fix, and they all show up in the same number.

Compare it against the median time to first unlock across all hunts. If a player takes three times the median, something is wrong with that player, and the most common causes are worth checking in order: is the first spot visible from the start point, did the camera permission get granted, and is the matching tolerance set so that a slightly different angle still counts.

Radius of play tells you about the world, not the player

I like one more unusual metric: how far players actually go from the start of a hunt, and how it varies by area.

It tells you things about the map that no engagement metric will. A neighbourhood where players travel much further than expected to reach spots is one where the imagery is thin or the spots are badly spread. A neighbourhood where players barely move is one where the spots are clustered and the hunt is over too quickly. Neither shows up as a drop-off, but both are design problems, and you cannot see them without this number.

It also tells you something about the product: if players consistently stay within a few hundred metres, that is evidence the hunt is working as a walk rather than turning into a trip across town — which is what you designed it for.

Be careful what you ask the player

A location game is unusual in that it can collect a lot of data about a person's movements, which means the easy metrics are also the invasive ones. Logging exact coordinates in analytics is both a privacy problem and a mediocre measurement.

We log aggregates: hunt started, spot unlocked, hint requested, hunt abandoned, hunt completed. Nothing that says where anyone was. It is enough for every measurement above, and it means the analytics pipeline never holds a movement history that could leak.

The practical rule we use: if a log line could embarrass someone if it leaked, it does not belong in the log. A dashboard that shows "session started at 55.6761, 12.5683" is not a measurement, it is a surveillance product with a chart on top.

The metrics that are worth a dashboard

If I were starting a location game from scratch, this is the list I would put up:

  1. Hunt completion rate — the headline.
  2. Abandon-point distribution — the funnel, and it tells you content from design.
  3. Time to first unlock, against its own median.
  4. Hint usage per spot, and hint usage per player over time.
  5. Radius of play, by area.
  6. Spots per hunt completed — difficulty in one number.

None of them are session length, and none of them are daily actives, and that is deliberate.

What I would tell someone starting this

  • Do not optimise for time in app. It rewards being stuck.
  • Plot where people quit, not just how many. The shape of the curve names the bug.
  • Watch the first unlock. It is where every category of failure shows up.
  • Read hints as a difficulty gauge, not as failure.
  • Keep the logs free of coordinates. You do not need them for any of this, and not having them is worth a lot.

The general point is that a game played in the physical world has a different failure surface from one played on a couch, and most of the standard measurement advice assumes the couch. Measure the walk, not the sitting.

If you are building something similar, I am happy to compare notes - Ping Hunt is out, and the awkward parts are mostly written down.

Top comments (0)