DEV Community: Vishnu JP

Image tracking vs 3D object recognition in AR: when a printed marker isn't enough

Vishnu JP — Mon, 09 Nov 2026 10:35:46 +0000

Most AR projects start with image tracking. It's built into ARKit and ARCore, it's well documented, and you can have content floating above a poster in an afternoon.

Then someone asks: "Can it work on the machine itself, not on a sticker we put on the machine?" And the project gets a lot harder.

We build NoxVision, an SDK for recognising real 3D objects in AR apps, so we see this moment a lot. This post covers how image tracking works, the five situations where it breaks, and how to decide whether you actually need 3D object recognition. Most of it applies whatever SDK you use.

Image tracking in 20 lines

In Unity with AR Foundation, image tracking needs three things:

A reference image library (XRReferenceImageLibrary) with your images and their real-world size
An ARTrackedImageManager on your XR Origin, pointing at that library
A script that reacts when an image is found

Here's a minimal script that places content on a poster and hides it when the poster goes out of view (AR Foundation 5.x):

using UnityEngine;
using UnityEngine.XR.ARFoundation;
using UnityEngine.XR.ARSubsystems;

[RequireComponent(typeof(ARTrackedImageManager))]
public class PosterContent : MonoBehaviour
{
    [SerializeField] GameObject contentPrefab;
    ARTrackedImageManager manager;

    void Awake()     => manager = GetComponent<ARTrackedImageManager>();
    void OnEnable()  => manager.trackedImagesChanged += OnChanged;
    void OnDisable() => manager.trackedImagesChanged -= OnChanged;

    void OnChanged(ARTrackedImagesChangedEventArgs args)
    {
        foreach (var image in args.added)
        {
            if (image.referenceImage.name == "SpringPoster")
                Instantiate(contentPrefab, image.transform);
        }

        foreach (var image in args.updated)
        {
            bool visible = image.trackingState == TrackingState.Tracking;
            foreach (Transform child in image.transform)
                child.gameObject.SetActive(visible);
        }
    }
}

In AR Foundation 6, trackedImagesChanged is replaced by the trackablesChanged event, but the idea is the same.

The content is a child of the tracked image's transform, so it moves with the poster. Simple, fast and reliable, as long as you're tracking the right kind of thing.

A quick check before you commit to an image

Not every image tracks well. Tracking works by finding distinctive feature points, so:

Photos and busy patterns track well
Plain colours, simple logos, text on white and repeating patterns track badly

Google's arcoreimg command-line tool scores a candidate image from 0 to 100, and Google recommends a score of at least 75. Run it before you print 5,000 flyers.

Also set the physical size in the reference library accurately. ARKit relies on it to work out how far away the image is, and a wrong size puts your content in the wrong place.

Five situations where image tracking breaks

1. The thing isn't flat

Image tracking assumes a flat, rigid surface. A label wrapped around a bottle, a curved dashboard or a fabric bag bends the image, and tracking becomes unstable or fails.

2. People approach from the side

Tracking works best when the camera faces the image. At steep angles, the features are squashed and the tracker loses them. A poster on a wall is fine. A part inside an engine bay, seen from above and the side, is not.

3. The real object has no printed surface

Machine parts, tools, furniture, sculptures and most products on a shelf have no flat, high-detail image on them. You can stick a marker on, but now your app depends on the sticker being there, clean and undamaged.

4. The same object looks different from different sides

A shoe from the front and from the side are two different images. You can add both to the library, but you're now maintaining many images per object, and you still have gaps between the angles.

5. You need to know which object, not just that there's an image

If you're identifying one of 50 similar machine parts, image tracking only helps if each one has its own marker. The question you actually want answered is "which part is this?", and that's object recognition.

The options for real 3D objects

When image tracking isn't enough, teams usually end up with one of three approaches.

Platform-specific object detection. ARKit can detect real objects on iOS using a scanned reference object. It works, but it's Apple-only, so your Android and Unity-on-Android users need something else.

Train your own model. Collect hundreds or thousands of photos of each object from different angles and in different light, label them, train a detection model, and repeat for every new object. Powerful, but it needs ML skills and weeks of work per batch of objects.

Use a 3D model of the object as the reference. If you have a 3D model, from a scan or from CAD, the model already describes what the object looks like from every angle. This is the approach we took.

How to decide

Answer these before writing code:

Question	If yes	If no
Is the target flat and printed?	Image tracking	Keep going
Do you control the design of the target?	Image tracking, designed to track well	Keep going
Will people see it from many angles?	3D recognition	Image tracking may be enough
Do you need to tell similar objects apart?	3D recognition	Either
Do you need iOS and Android?	Avoid single-platform options	Platform-native is fine

A rule of thumb: if you're about to stick a printed marker on a real object so your app can find it, you probably need 3D recognition.

Where NoxVision fits

NoxVision is built for the "use a 3D model as the reference" approach:

Scan the real object with an app like Polycam or Luma AI
Upload the model as .glb, .ply or .obj
Detect it through the phone camera in real time, using the SDK for Android, iOS or Unity

There's no custom model training, and detection can start within minutes of uploading. SDK calls use a personal API key from your dashboard's Developer Settings; models are private to your account, and there's offline model caching for places with poor connectivity, such as factory floors.

A good way to test the idea: pick one object you know well, scan it, and see how detection holds up from different angles and in different light. That tells you more about your use case than any demo.

And if your target is a flat poster, use image tracking. It's simpler, and it's built in.

What are you tracking? If you've built AR, have you hit a case where image tracking wasn't enough? What did you switch to? Tell us in the comments.

Letting an AI agent query your production database? Do these 6 things first

Vishnu JP — Mon, 02 Nov 2026 09:51:54 +0000

"Just ask the database in plain English" is one of the most useful things an AI agent can do. A sales lead types "Which 10 customers spent the most this quarter?" and gets an answer in seconds, instead of waiting two days for an analyst.

It's also one of the scariest things to set up. You're giving software that writes its own SQL a connection to your data.

We build AI Agent Studio, a no-code platform where agents can answer questions from a live database, so we think about this a lot. Here's the setup we recommend before connecting any AI tool to a database, ours included.

The examples use PostgreSQL, and we ran every snippet on PostgreSQL 16 before publishing. The ideas carry over to MySQL, SQL Server and the rest.

The short version

Step	What it protects against
1. A dedicated read-only role	The agent changing or deleting data
2. Views instead of raw tables	The agent reading personal data it doesn't need
3. Timeouts	One bad query tying up the database
4. Row limits	Huge result sets and multi-statement tricks
5. A read replica	Analytics slowing down your app
6. Logging	Wrong answers you can't explain

1. Create a dedicated read-only role

Never reuse your app's database user. Give the agent its own role that can only read, and only from one schema:

-- A separate schema for everything the agent may see (more in step 2)
CREATE SCHEMA reporting;

CREATE ROLE ai_agent LOGIN PASSWORD 'use-a-long-random-password' CONNECTION LIMIT 5;

GRANT CONNECT ON DATABASE shop TO ai_agent;
GRANT USAGE ON SCHEMA reporting TO ai_agent;
GRANT SELECT ON ALL TABLES IN SCHEMA reporting TO ai_agent;

-- Make views and tables created later in this schema readable too
ALTER DEFAULT PRIVILEGES IN SCHEMA reporting
  GRANT SELECT ON TABLES TO ai_agent;

-- Extra safety: every transaction starts read-only
ALTER ROLE ai_agent SET default_transaction_read_only = on;

The SELECT-only grants are the real protection. default_transaction_read_only is a second layer, but a session can switch it off with one SET, so don't rely on it alone. CONNECTION LIMIT stops a runaway agent from opening hundreds of connections.

Two details that catch people out:

ALTER DEFAULT PRIVILEGES only applies to objects created by the role that ran it. Run it as the same user that will create your views.
Store the password in your secrets manager, not in the agent's prompt or config file.

2. Expose views, not raw tables

The role above can only see the reporting schema. Put views there that contain only what the agent needs, and leave out personal data:

CREATE VIEW reporting.orders_summary AS
SELECT
  o.id,
  o.created_at,
  o.status,
  o.total_amount,
  c.country            -- no names, emails or addresses
FROM public.orders o
JOIN public.customers c ON c.id = o.customer_id;

By default a view runs with its owner's privileges, so ai_agent can read reporting.orders_summary without any access to public.orders or public.customers. (If you create views with security_invoker = true, that changes, and the agent would need access to the underlying tables.)

Then describe the data. Many AI tools read the database's comments when they work out what to query, and plain-English descriptions make their SQL much more accurate:

COMMENT ON VIEW reporting.orders_summary IS
  'One row per order. total_amount is in USD, including tax.';
COMMENT ON COLUMN reporting.orders_summary.status IS
  'pending, paid, refunded or cancelled';

Clear names help just as much. total_amount_usd gets fewer wrong answers than amt2.

3. Set timeouts

An agent will eventually write a query that joins three big tables with no filter. Make sure it can't run for an hour:

ALTER ROLE ai_agent SET statement_timeout = '5s';
ALTER ROLE ai_agent SET idle_in_transaction_session_timeout = '30s';

Five seconds is plenty for most business questions. If a question needs more, that's a sign it belongs in a proper report or a pre-computed view, not a live query.

Role settings apply to new sessions, so reconnect after changing them.

4. Limit rows in the tool layer

Nobody needs 2 million rows in a chat answer. Wrap the agent's query before running it:

import psycopg2

MAX_ROWS = 500

def run_agent_query(conn, sql: str):
    sql = sql.strip().rstrip(";")
    if ";" in sql:
        raise ValueError("Only one statement is allowed")

    with conn.cursor() as cur:
        cur.execute(f"SELECT * FROM ({sql}) AS agent_q LIMIT {MAX_ROWS}")
        columns = [d.name for d in cur.description]
        return columns, cur.fetchall()

conn = psycopg2.connect(dsn)  # credentials for the ai_agent role
conn.set_session(readonly=True, autocommit=True)

Wrapping the query in a subquery also means anything that isn't a SELECT fails with a syntax error before it runs. A DELETE inside SELECT * FROM (...) simply isn't valid SQL.

This is a guardrail, not a security boundary. The read-only role from step 1 is what actually protects your data. The wrapper keeps answers small and stops the agent sending several statements at once. (The semicolon check will also reject a query with a ; inside a string, which is a trade-off we're happy with.)

5. Point it at a replica

If you have a read replica, connect the agent there instead of your primary database. Even a well-behaved query can be heavy, and you don't want a curious question about last year's revenue slowing down checkout.

A replica is also naturally read-only, which gives you one more layer on top of step 1.

No replica? Lean harder on steps 3 and 4, and consider a lower CONNECTION LIMIT.

6. Log every question, query and answer

When the agent gives a wrong number, you need to know why. Log three things together, with a timestamp and the user who asked:

The question the person asked
The SQL the agent generated
The answer it returned

On the database side, a superuser can log every statement from the agent's role:

ALTER ROLE ai_agent SET log_statement = 'all';

And show the SQL to the person, next to the answer. Anyone who knows the data can then sanity-check it. "Revenue: $1.2M" is easy to trust blindly; "Revenue: $1.2M, from SUM(total_amount) WHERE status = 'paid'" invites someone to say "that should include refunds".

Test it before you trust it

Before you connect an agent, log in as ai_agent yourself and check the guardrails actually hold:

-- Should work
SELECT country, SUM(total_amount) FROM reporting.orders_summary GROUP BY country;

-- Should all fail
SELECT * FROM public.customers;   -- ERROR: permission denied for table customers
SET default_transaction_read_only = off;
DELETE FROM public.orders;        -- ERROR: permission denied for table orders
SELECT pg_sleep(6);               -- ERROR: canceling statement due to statement timeout

That's exactly what we saw when we ran this setup on PostgreSQL 16: the agent can read the view, and nothing else.

Then test the answers. Ask the agent 10 questions you already know the answers to, such as last month's revenue, the number of active customers or your top product. Compare. Wrong answers usually point to a confusing column name or a missing view, which you fix in step 2.

Where AI Agent Studio fits

AI Agent Studio lets you connect a database and ask questions in plain English, alongside document search (with sources) and agents that act in 100+ apps. When you connect it, or any similar tool, give it the ai_agent role and the reporting schema from this post, not your app's credentials. The setup is the same whichever tool you use.

How are you handling this? If you've connected an AI tool to a real database, what guardrails did you add, and what went wrong first? Tell us in the comments.

The AI agent made my Xcode build green. Here are 6 ways it might have cheated

Vishnu JP — Mon, 26 Oct 2026 07:57:59 +0000

"Six common 'fixes' AI coding agents make to get a Swift build or test passing, what the real fix looks like, and a pre-commit hook that catches them."

AI coding agents are very good at making a red build go green. They're less reliable at making it go green for the right reason.

We build Phoenix.vu, an AI coding agent for Xcode, so we read a lot of agent-written diffs. The same handful of shortcuts comes up again and again, from every model and every tool, including the agents now built into Xcode. None of them are malicious. The agent was asked to fix an error, and these are the fastest ways to make the error disappear.

Here are six to watch for, with the real fix for each.

1. `@unchecked Sendable` to silence a concurrency error

The error (Swift 6 strict concurrency):

Capture of 'cache' with non-sendable type 'ImageCache' in a '@Sendable' closure

The shortcut:

final class ImageCache: @unchecked Sendable {
    private var storage: [URL: UIImage] = [:]

    func image(for url: URL) -> UIImage? { storage[url] }
    func insert(_ image: UIImage, for url: URL) { storage[url] = image }
}

It compiles. It also tells the compiler "trust me, this is thread-safe" when it isn't. You still have a data race on storage. The compiler has just stopped warning you about it.

The real fix: make the shared state safe. An actor is the simplest option:

actor ImageCache {
    private var storage: [URL: UIImage] = [:]

    func image(for url: URL) -> UIImage? { storage[url] }
    func insert(_ image: UIImage, for url: URL) { storage[url] = image }
}

// Callers now await:
let cached = await cache.image(for: url)

@unchecked Sendable has legitimate uses, for example a type protected by a lock. But if the diff adds it without adding a lock, ask why.

2. `nonisolated(unsafe)` on global state

The error:

Static property 'shared' is not concurrency-safe because it is nonisolated global shared mutable state

The shortcut:

final class Settings {
    nonisolated(unsafe) static var shared = Settings()
    var theme: Theme = .system
}

The real fix: decide who owns this state. If it never needs to change, make it a let of a Sendable type. If the UI owns it, isolate it to the main actor:

@MainActor
final class Settings {
    static let shared = Settings()
    var theme: Theme = .system
}

3. Force unwraps to get past optionals

The error:

Value of optional type 'URL?' must be unwrapped to a value of type 'URL'

The shortcut:

let url = URL(string: endpoint)!

The build passes, and the app crashes the first time endpoint is malformed.

The real fix:

guard let url = URL(string: endpoint) else {
    throw APIError.invalidURL(endpoint)
}

4. `try?` that throws away the error

The error:

Call can throw but is not marked with 'try'

The shortcut:

let user = try? decoder.decode(User.self, from: data)

Now a decoding failure silently becomes nil, and your UI shows an empty profile with no clue why.

The real fix: let it throw, or handle it on purpose:

do {
    let user = try decoder.decode(User.self, from: data)
    show(user)
} catch {
    logger.error("Failed to decode User: \(error.localizedDescription)")
    showError(.couldNotLoadProfile)
}

try? is fine when nil really is an acceptable result. It's a problem when it was added only to make the compiler stop complaining.

5. Disabling the failing test

The failure:

Expectation failed: (order.total → 108.0) == 110.0

The shortcut:

@Test(.disabled("Flaky"))
func totalIncludesTax() {
    let order = Order(items: [.init(price: 100)], taxRate: 0.10)
    #expect(order.total == 110)
}

Or the test is deleted, or the expected value is changed to match the wrong output. The suite is green, and the bug is now invisible.

The real fix: fix the logic the test is checking. Then, if the test really is flaky, fix the flakiness, for example by removing the dependency on time, the network or ordering.

6. Making a property optional to "fix" decoding

The failure:

keyNotFound(CodingKeys(stringValue: "price", intValue: nil), ...)

The shortcut:

struct Product: Decodable {
    let id: String
    let name: String
    let price: Decimal?   // was: let price: Decimal
}

It decodes, and now every screen that uses price has to deal with nil, even though a product without a price is probably a bug in the API response.

The real fix: find out why the key is missing. If the API really does omit it sometimes, decode it explicitly with a documented default:

init(from decoder: Decoder) throws {
    let c = try decoder.container(keyedBy: CodingKeys.self)
    id = try c.decode(String.self, forKey: .id)
    name = try c.decode(String.self, forKey: .name)
    price = try c.decodeIfPresent(Decimal.self, forKey: .price) ?? 0 // API omits price for free items
}

A 30-second review checklist

Before you accept an agent's fix, search the diff for:

@unchecked Sendable or nonisolated(unsafe)
New ! force unwraps, try! or as!
New try?
.disabled(, XCTSkip, or removed test functions
Properties that became optional
Large deletions in a "small" fix

Bonus: a pre-commit hook that flags them

Save this as .git/hooks/pre-commit and make it executable (chmod +x .git/hooks/pre-commit):

#!/bin/sh
# Flags common "make it compile" shortcuts in staged Swift changes.
PATTERNS='@unchecked Sendable|nonisolated\(unsafe\)|try!|as!|\.disabled\(|XCTSkip'

if git diff --cached -U0 -- '*.swift' | grep '^+' | grep -E "$PATTERNS"; then
  echo ""
  echo "These added lines often hide a real problem. Review them,"
  echo "or commit with --no-verify if they're intentional."
  exit 1
fi

It won't catch everything, but it catches the most common cases, whoever wrote the code.

Where Phoenix fits

We built Phoenix around this review step. It runs the build-fix loop alongside Xcode, but every change is shown as a diff that you approve or reject before it's applied, and it snapshots each file before writing so you can undo without Git. It's pay-per-task with no subscription, and your code stays on your Mac.

If you already pay for Claude or ChatGPT, Xcode's built-in agents are a good place to start, and everything above applies to them too.

What's the worst "fix" you've seen an AI agent make to get a build passing? Share it in the comments. We'll add the best ones to the checklist.

Spotted something that behaves differently in your Xcode version? Let us know in the comments and we'll update the post.

We build 5 AI products. All of them follow one rule: the agent shows you before it acts

Vishnu JP — Wed, 21 Oct 2026 12:30:00 +0000

How a small team designs AI agents that do real work (code, marketing, phone calls, video, AR) while people keep the final say.

We're iBoson Innovations, a small product company. We started in augmented reality, and today we build five AI products:

Phoenix.vu: an AI coding agent that works alongside Xcode
Lyks.ai: AI product videos for Shopify stores
AI Agent Studio (aiagents.dev): no-code agents that answer calls and act in business apps
Blooming.sh: 40+ AI agents for marketing, sales and support
NoxVision.ai: real-time 3D object recognition for AR apps

This is our first post on DEV, so instead of a feature tour, I want to talk about the one design rule all five products share. We think it's the most important decision in any agent product.

Agents fail differently from chatbots

There's a lot of scepticism about AI agents here right now, and much of it is fair.

When a chatbot is wrong, you get a wrong answer. You read it, frown, and ignore it. When an agent is wrong, something changes: code gets rewritten, an email goes to a client, a ticket gets closed, a call ends with the wrong booking. The cost of a mistake moves from "I wasted 30 seconds" to "now I have to find out what it did and undo it".

That's why "fully autonomous" was never our goal.

Our rule: show, then act

Every iBoson product follows the same loop:

The agent does the work and produces a concrete proposal: a diff, a draft, a script, an action.
A person reviews and approves it.
Only then is it applied, and it can be undone.

Here's how that looks across the five products:

Product	What the agent does	What you review before it counts
Phoenix.vu	Writes Swift/SwiftUI, runs the build, fixes errors	A diff for every change, plus file snapshots before each write
Blooming.sh	Drafts posts, emails, sequences and support replies	An approval step per client brand before anything is published or sent
AI Agent Studio	Answers calls, queries data, acts in 100+ apps	Approval steps on the actions you choose, plus handover rules for calls
Lyks.ai	Writes a script from your product listing and generates a video	The script, which you can edit before anything is rendered
NoxVision.ai	Detects real objects through the phone camera	Your app decides what happens on detection; models stay private to your account

NoxVision is the odd one out. It's a perception SDK, not an agent that takes actions. But the same idea applies: it tells your app what it sees, and your code decides what to do with that.

Four things we learned building it

1. Make the proposal the unit of work

"The agent did some stuff" is impossible to review. A diff, a draft, or a numbered list of actions is easy to review. We design every agent around producing something with a clear before and after.

2. Make undo cheap

If undo is expensive, people stop trusting approvals, because one bad click costs too much. Phoenix snapshots each file before it writes to it, so you can roll back without relying on Git. Blooming keeps drafts separate from what's published.

3. Show the cost of every action

Agents can burn money quietly. Phoenix and Lyks both use pay-as-you-go credits instead of subscriptions, and Phoenix shows the exact cost after each request. Seeing "this fix cost 4 credits" changes how people use an agent. It also keeps us honest about whether each step was worth it.

4. Don't ask for approval on everything

Approval fatigue is real. If you ask someone to approve every tiny step, they start clicking "yes" without reading, which is worse than no approval at all. So the rule is about risk, not every step:

Read-only work runs freely: searching documents, answering questions from a read-only database connection, reading build logs.
Changes need approval: writing code, publishing content, sending messages, updating records.
Users choose the line: in AI Agent Studio you decide which actions need a human.

Something we got wrong

We built Phoenix into a gap: an Xcode-native coding agent, just as a well-liked tool in that space shut down. Then Xcode 26.3 shipped its own built-in agents from Anthropic and OpenAI. A big part of our pitch became something Xcode does by default.

We learned to position against what's coming, not just against what's missing today. We now tell developers who already pay for Claude or ChatGPT to try Apple's agents first. Phoenix is for people who want pay-per-task pricing and a diff-first workflow. Being fair about that has made the rest of what we say more believable.

What we'll write about here

One practical post a week, rotating through our products:

Reviewing AI-made fixes in Xcode, so a green build isn't hiding a bad change
Giving an AI agent database access safely
AR object recognition without training an ML model
AI video for e-commerce: what works and what doesn't
Running marketing with multiple agents without losing a brand's voice

We'll share code, trade-offs and mistakes, and say when another tool is the better choice.

Over to you

Where do you draw the line? Which actions would you let an AI agent take without asking you first, and which should always need a human? Tell us in the comments. We're using the answers to tune our defaults.

DEV Community: Vishnu JP

Image tracking vs 3D object recognition in AR: when a printed marker isn't enough

Image tracking in 20 lines

A quick check before you commit to an image

Five situations where image tracking breaks

1. The thing isn't flat

2. People approach from the side

3. The real object has no printed surface

4. The same object looks different from different sides

5. You need to know which object, not just that there's an image

The options for real 3D objects

How to decide

Where NoxVision fits

Letting an AI agent query your production database? Do these 6 things first

The short version

1. Create a dedicated read-only role

2. Expose views, not raw tables

3. Set timeouts

4. Limit rows in the tool layer

5. Point it at a replica

6. Log every question, query and answer

Test it before you trust it

Where AI Agent Studio fits

The AI agent made my Xcode build green. Here are 6 ways it might have cheated

1. @unchecked Sendable to silence a concurrency error

2. nonisolated(unsafe) on global state

3. Force unwraps to get past optionals

4. try? that throws away the error

5. Disabling the failing test

6. Making a property optional to "fix" decoding

A 30-second review checklist

Bonus: a pre-commit hook that flags them

Where Phoenix fits

We build 5 AI products. All of them follow one rule: the agent shows you before it acts

Agents fail differently from chatbots

Our rule: show, then act

Four things we learned building it

1. Make the proposal the unit of work

2. Make undo cheap

3. Show the cost of every action

4. Don't ask for approval on everything

Something we got wrong

What we'll write about here

Over to you

1. `@unchecked Sendable` to silence a concurrency error

2. `nonisolated(unsafe)` on global state

4. `try?` that throws away the error