That mouse on the rug isn't really there. The real one is sitting on my desk. What you're looking at is a 3D copy of it, dropped into my room in AR, and it was built from photos I took on my phone.
The part I care about most is where it was built. Not on someone's GPU farm. On my MacBook, over my home Wi-Fi. The photos never left the house, and there was no sign-up, no upload and no credits to buy.
The app is called ThreeD. It's a React Native app for iPhone and Android, plus a small Node server that runs on your Mac and drives three engines: Apple's Object Capture, Tencent's Hunyuan3D (through ComfyUI) and a small vision model in Ollama. It's open source. In this post I'll show you what it does, how it's put together, and the things that bit me while building it.
Repo: github.com/neel-turbo/app-image-to-3d-local-ai-rn
Three ways to make a model
Open the app and the first thing it asks is how you want to capture. There are three answers, and each one trades accuracy for effort.
Photo set is the one I recommend. You walk around the object three times (eye level, from above, from low down) and take 12 photos on each ring, roughly 30° apart. That's 36 photos, although the Build button unlocks once you have 20 good ones. The Mac runs Apple's Object Capture on them and you get the object's real shape with its real colours. Small things like a logo or a button survive.
4 photos (AI) is the middle ground. Front, back and both sides go to Hunyuan3D-2's multi-view model, so the AI isn't guessing what the back looks like. It saw it.
1 photo (AI) is the quick try. One clear photo, and Hunyuan3D-2 mini imagines the rest. Front and sides look good. The back is a guess, and sometimes a funny one: details from the front, like a face, get mirrored onto the back of the head.
If you just want to play, the single photo is fine. If you want something you'll keep, take the extra two minutes and walk around it.
Most of the work is stopping bad photos
The hard part of photogrammetry isn't the maths. It's getting a human (me included) to take decent photos. A blurry shot, two photos from the same spot, the object half out of frame: any of those and Object Capture either hands you a broken mesh or gives up with "not enough features". So most of the phone-side code exists to catch bad photos before they ever reach the Mac.
First, the camera tells you where to stand. A guide grid sits over the preview with a top-down dial of the 12 spots around the object, a 3 × 12 grid of every photo you still need (Above, Eye level, Low), and one plain line like "Step about 30° to your right". Once you get the hang of it you can turn it off.
Auto-capture is my favourite trick in the app. When you walk around an object, your phone turns by the same angle you walked. So the app reads the motion sensors (CoreMotion on iPhone, the rotation vector on Android) and waits until you've moved about 30° to a new spot, at the right height for the ring. Hold still and it takes the shot. No two photos from the same place, and no tapping.
Every photo is then checked in C++: sharpness (variance of the Laplacian), exposure, contrast, and overlap with the previous shot using a 64-bit dHash. It's one pure C++ Turbo Module, so the same code runs on iOS and Android, and I can unit-test it on my laptop. JavaScript only decides what to do with the numbers ("too blurry, try again"). A bad shot gets rejected on the spot, with a tip.
The phone also cuts the object out of the background, with Apple Vision on iPhone and ML Kit Subject Segmentation on Android. For the AI modes that cut-out is centred on white with a 15% border, because that's the kind of input Hunyuan3D was trained on.
Finally there's an "Ask local AI" button. A 4B vision model (qwen3.5:4b, running in Ollama on your Mac) names the object, rates the photo Good, OK or Poor, and gives you tips. I didn't want a small model overruling a Laplacian filter about whether a photo is sharp, so the phone sends its own measurements along with the image and the gateway drops any tip that contradicts them. It runs at temperature 0 with a fixed seed too, so the same photo always gets the same advice.
Then you get to play with it
When a build finishes, it shows up in the Gallery tab. Every model your Mac has built is there.
Tap one and you get a proper 3D viewer. Drag in any direction (up and down too, so you can check the underside), pinch to zoom, or jump straight to Front, Back, Top and the rest. The viewer is Google's <model-viewer>, but it's served by the Mac rather than a CDN, so it works with no internet at all. It also uses soft studio lighting, so the colours look like the real object.
View in AR is the fun bit. On iPhone it opens AR Quick Look with a .usdz file. On Android it hands a .glb to Google's Scene Viewer. Either way you can put the model on your floor and walk around it like it's actually there.
Sharing goes through the normal share sheet. You can send the 3D file, the AR file (iPhone), a picture of the current view, or a view-only link. The link works in any browser on the same Wi-Fi for 24 hours, needs no app and no pairing, and only ever exposes a random link id, never your device token.
How it's put together
There are two halves. The phone does the capturing and all the per-photo maths. The Mac does the heavy 3D work. They talk over plain HTTP and a WebSocket on your Wi-Fi, and the phone finds the Mac with Bonjour (mDNS).
📱 Phone app 💻 Your Mac (same Wi-Fi)
───────────── ────────────────────────
1. Take or choose ── upload ───────▶ 2. ThreeD gateway (Fastify)
photos (checked ├─ Apple Object Capture (photo set → 3D)
and cut out ├─ ComfyUI + Hunyuan3D (1 or 4 photos → 3D)
on the phone) └─ Ollama (capture tips)
│
4. View, AR, share ◀──── 3D model ───── 3. Builds and cleans the model
The phone app is a tiny super app
If you write React Native, this is probably the interesting bit. The app is a shell plus two mini apps (Gallery and Profile) that are downloaded over the air. That means I can ship a Gallery fix without waiting on an App Store review.
Metro can't load separately built bundles at runtime, so the project uses Re.Pack (Rspack) with Module Federation 2 instead. The shell doesn't hard-code any remotes. On launch it reads a manifest from the Mac and registers each mini app with registerRemotes(), so the version can change without a new binary.
A few rules made this hold together:
- Every native module lives in the shell. The stores don't let you download native code, so mini apps ship JavaScript only. VisionCamera, Firebase, the WebView, my C++ module: all of it is in the shell and shared as Module Federation singletons. There's a Jest test that fails if a native dependency isn't shared or its version drifts.
- One Redux store for everything. The Gallery mini app injects its own slice when its bundle runs, so its state is code-split along with its code.
- Every bundle is signed. Re.Pack signs each chunk with RSA and the shell refuses anything unsigned or modified. Rollouts are staged (say, 20% of phones first), and if a version crashes while rendering, that phone blacklists it and falls back to the previous one. The shell itself never goes down.
The Mac runs a small gateway
The phone never talks to Ollama or ComfyUI directly. Neither has any authentication, and on an 8 GB Mac they can't both be loaded at once anyway. So there's a Fastify server in the middle that does four jobs:
- Pairs phones with a 6-digit code (the device token is only stored as a SHA-256 hash).
- Takes the uploads, re-hashes them and throws away duplicates or anything that isn't really a JPEG or PNG.
- Runs one heavy job at a time and evicts the other models from memory first.
- Cleans up the mesh afterwards and colours it from your photos.
Don't underestimate the clean-up step. Raw AI meshes come out with floating bits, small holes and far too many triangles. The gateway welds the mesh, removes the floaters, fills holes, simplifies it with meshoptimizer to around 80,000 triangles and smooths the normals. Then it projects colour from your photo (or all four photos) with a depth test, so colour never "shoots through" to the wrong side.
Things that bit me
8 GB of memory is a hard wall. My test machine is an M1 MacBook Air with 8 GB. Object Capture, a Hunyuan3D diffusion model and a vision LLM simply don't fit in unified memory together. So before every 3D job the gateway unloads the Ollama model, and after the job it asks ComfyUI to free its memory. Unloading Ollama turned out to be one tiny request:
/** keep_alive: 0 evicts the model from unified memory immediately. */
async unload(): Promise<void> {
await this.fetchImpl(`${this.cfg.url}/api/generate`, {
method: 'POST',
headers: {'content-type': 'application/json'},
body: JSON.stringify({model: this.cfg.model, keep_alive: 0}),
signal: AbortSignal.timeout(5000),
}).catch(() => undefined);
}
While a build owns the memory, "Ask local AI" gets an HTTP 409 ("Your Mac is building a model") instead of a crash. Not glamorous, but it's the difference between a slow app and a Mac that locks up.
Shiny, thin and dark things don't scan. Photogrammetry needs texture to match points between photos. A laptop or a phone screen comes out broken no matter how careful you are. Matte, solid objects on a textured table work best. When Object Capture fails with "not enough features", the gateway retries once at high sensitivity before giving up.
Android AR went black, and it wasn't my bug. At one point, tapping View in AR on Android just showed a black screen and bounced back to the app. Google's Scene Viewer (it lives inside the Google app) was crashing, and it crashed with Google's own sample models too. Updating or clearing the cache of the Google app fixes it. Separately, Scene Viewer was dropping big textures, so the gateway now makes a lighter Android AR copy with textures capped at 2048 px, and colours always show.
AI models came out grey. Hunyuan3D-2 mini gives you shape only, no colour. The gateway now paints the mesh from your photo. For models built before that fix there's a repair script, pnpm --filter @threed/gateway fix-ar-textures, which colours the old grey ones.
Model files lie about what they are. People rename checkpoints, and ComfyUI fails with a cryptic Value not in list: ckpt_name. Now the gateway opens the .safetensors header and works out what's actually inside: Hunyuan3D mini, full v2, 2.1, a multi-view model, or a plain 2D model like SD 1.5 that can't make meshes at all. Then it picks the right settings by itself.
OTA is not a loophole. Apple and Google allow interpreted-code updates that don't change what the app does, so I use OTA for fixes and tweaks. When sharing needed new native modules (react-native-share and a file-system module), that meant a new shell release, 1.1. The Gallery 1.1 update is still safe to push to 1.0 shells: they just fall back to React Native's built-in Share for links.
What to expect on an 8 GB M1
These are measured on the base M1 MacBook Air, so pretty much any Apple Silicon Mac should do at least this well.
| Mode | Build time | What you get | Tip |
|---|---|---|---|
| Photo set (Object Capture) | 1–2 min for 24–37 photos | The best: true shape, real colours, small details like logos and buttons | The underside stays open unless you photograph it too |
| 4 photos (Hunyuan3D multi-view) | About 35 min | Good all round; back and sides follow your real photos | Needs about 7 GB, so close every other app first |
| 1 photo (Hunyuan3D mini) | About 6 min | Good from the front and sides; the back is a guess | Use a clear three-quarter view on a plain background |
The 35 minutes for multi-view is the memory talking, not the GPU. On a 16 GB Mac it's much faster. More memory also buys more detail automatically: Macs with 16 GB or more get full-detail photo sets and 4K textures, without you changing a setting.
Try it yourself
You'll need:
- An Apple Silicon Mac on macOS 14 or newer (an 8 GB M1 is enough)
- An iPhone or Android phone on the same Wi-Fi
- Node.js 22.13+ and pnpm 10+
- Xcode 16+ for iOS, Android Studio for Android
- Ollama for the capture tips, and ComfyUI only if you want the two AI modes
Then:
# 1. Install
pnpm install
# 2. Get the model for capture tips (keep the Ollama app running)
ollama pull qwen3.5:4b
# 3. Check that everything is set up (it tells you how to fix what's missing)
pnpm check-setup
# 4. Start the Mac gateway and the app's dev servers
pnpm start --real
# 5. In a second terminal, run the app
pnpm pods && pnpm ios # iPhone simulator
pnpm android # Android over USB
On a real iPhone, start the dev servers with pnpm start --real --device so the phone can reach them over Wi-Fi, and run from Xcode. Then on the phone go to Profile → Your Mac → Find Macs on this Wi-Fi and type the 6-digit code the gateway prints in its terminal.
One thing I'm glad I added: if you don't have any AI installed yet, plain pnpm start (without --real) runs a fake gateway that still produces real 3D files. You can click through the whole app before downloading a single model. And pnpm check-setup checks Node, Ollama, ComfyUI, Object Capture, the gateway and your Wi-Fi address, and tells you the exact fix for anything that's off.
The repo also comes with about 400 Jest tests, 39 C++ tests and Detox end-to-end tests, plus setup, architecture, technology and security guides in the docs/ folder.
That's it
I wanted to know how far you can get with the hardware that's already on your desk. Pretty far, it turns out. A phone that's fussy about its photos and a base-model Mac that's careful with its memory can turn a real object into something you can spin around, drop on your floor and send to a friend. None of it touches the cloud.
The code is all on GitHub: neel-turbo/app-image-to-3d-local-ai-rn. If you try it, I'd love to see what you scan. If something breaks, open an issue and tell me what you were pointing the camera at.
And if this was useful, a star on the repo helps other people find it.
Top comments (1)
Auto-capture off the rotation vector is a neat trigger. One thing I'd test: the phone's yaw changes by 30° when someone pivots in place or sidesteps without circling, so rotation alone can't tell an orbit from a spin. Since you already compute a dHash against the previous shot, I'd use it as a second gate with both bounds: reject near-duplicates, but also flag a frame that differs too much, because Object Capture needs enough overlap to match features. Do the "not enough features" failures you hit come mostly from gaps between rings, or from shiny objects?