Image tracking vs 3D object recognition in AR: when a printed marker isn't enough

Most AR projects start with image tracking. It's built into ARKit and ARCore, it's well documented, and you can have content floating above a poster in an afternoon.

Then someone asks: "Can it work on the machine itself, not on a sticker we put on the machine?" And the project gets a lot harder.

We build NoxVision, an SDK for recognising real 3D objects in AR apps, so we see this moment a lot. This post covers how image tracking works, the five situations where it breaks, and how to decide whether you actually need 3D object recognition. Most of it applies whatever SDK you use.

Image tracking in 20 lines

In Unity with AR Foundation, image tracking needs three things:

1. A reference image library (XRReferenceImageLibrary) with your images and their real-world size
2. An ARTrackedImageManager on your XR Origin, pointing at that library
3. A script that reacts when an image is found

Here's a minimal script that places content on a poster and hides it when the poster goes out of view (AR Foundation 5.x):

using UnityEngine;
using UnityEngine.XR.ARFoundation;
using UnityEngine.XR.ARSubsystems;

[RequireComponent(typeof(ARTrackedImageManager))]
public class PosterContent : MonoBehaviour
{
    [SerializeField] GameObject contentPrefab;
    ARTrackedImageManager manager;

    void Awake()     => manager = GetComponent<ARTrackedImageManager>();
    void OnEnable()  => manager.trackedImagesChanged += OnChanged;
    void OnDisable() => manager.trackedImagesChanged -= OnChanged;

    void OnChanged(ARTrackedImagesChangedEventArgs args)
    {
        foreach (var image in args.added)
        {
            if (image.referenceImage.name == "SpringPoster")
                Instantiate(contentPrefab, image.transform);
        }

        foreach (var image in args.updated)
        {
            bool visible = image.trackingState == TrackingState.Tracking;
            foreach (Transform child in image.transform)
                child.gameObject.SetActive(visible);
        }
    }
}

In AR Foundation 6, trackedImagesChanged is replaced by the trackablesChanged event, but the idea is the same.

The content is a child of the tracked image's transform, so it moves with the poster. Simple, fast and reliable, as long as you're tracking the right kind of thing.

A quick check before you commit to an image

Not every image tracks well. Tracking works by finding distinctive feature points, so:

• Photos and busy patterns track well
• Plain colours, simple logos, text on white and repeating patterns track badly

Google's arcoreimg command-line tool scores a candidate image from 0 to 100, and Google recommends a score of at least 75. Run it before you print 5,000 flyers.

Also set the physical size in the reference library accurately. ARKit relies on it to work out how far away the image is, and a wrong size puts your content in the wrong place.

Five situations where image tracking breaks

1. The thing isn't flat

Image tracking assumes a flat, rigid surface. A label wrapped around a bottle, a curved dashboard or a fabric bag bends the image, and tracking becomes unstable or fails.

2. People approach from the side

Tracking works best when the camera faces the image. At steep angles, the features are squashed and the tracker loses them. A poster on a wall is fine. A part inside an engine bay, seen from above and the side, is not.

3. The real object has no printed surface

Machine parts, tools, furniture, sculptures and most products on a shelf have no flat, high-detail image on them. You can stick a marker on, but now your app depends on the sticker being there, clean and undamaged.

4. The same object looks different from different sides

A shoe from the front and from the side are two different images. You can add both to the library, but you're now maintaining many images per object, and you still have gaps between the angles.

5. You need to know which object, not just that there's an image

If you're identifying one of 50 similar machine parts, image tracking only helps if each one has its own marker. The question you actually want answered is "which part is this?", and that's object recognition.

The options for real 3D objects

When image tracking isn't enough, teams usually end up with one of three approaches.

Platform-specific object detection. ARKit can detect real objects on iOS using a scanned reference object. It works, but it's Apple-only, so your Android and Unity-on-Android users need something else.

Train your own model. Collect hundreds or thousands of photos of each object from different angles and in different light, label them, train a detection model, and repeat for every new object. Powerful, but it needs ML skills and weeks of work per batch of objects.

Use a 3D model of the object as the reference. If you have a 3D model, from a scan or from CAD, the model already describes what the object looks like from every angle. This is the approach we took.

How to decide

Answer these before writing code:

Question	If yes	If no
Is the target flat and printed?	Image tracking	Keep going
Do you control the design of the target?	Image tracking, designed to track well	Keep going
Will people see it from many angles?	3D recognition	Image tracking may be enough
Do you need to tell similar objects apart?	3D recognition	Either
Do you need iOS and Android?	Avoid single-platform options	Platform-native is fine

A rule of thumb: if you're about to stick a printed marker on a real object so your app can find it, you probably need 3D recognition.

Where NoxVision fits

NoxVision is built for the "use a 3D model as the reference" approach:

1. Scan the real object with an app like Polycam or Luma AI
2. Upload the model as .glb, .ply or .obj
3. Detect it through the phone camera in real time, using the SDK for Android, iOS or Unity

There's no custom model training, and detection can start within minutes of uploading. SDK calls use a personal API key from your dashboard's Developer Settings; models are private to your account, and there's offline model caching for places with poor connectivity, such as factory floors.

A good way to test the idea: pick one object you know well, scan it, and see how detection holds up from different angles and in different light. That tells you more about your use case than any demo.

And if your target is a flat poster, use image tracking. It's simpler, and it's built in.

---

What are you tracking? If you've built AR, have you hit a case where image tracking wasn't enough? What did you switch to? Tell us in the comments.