Why Shade Matching Online Keeps Failing (It's a Lighting Problem, Not a Photo Problem)

When I started building Mirrrd, I assumed the hard part would be the AR rendering — getting a lipstick to follow lip geometry in real time. That turned out to be the second hardest thing. The first was understanding why the same foundation shade looks completely different on three different phones showing the same product image.

It's a lighting problem disguised as a color problem, and once you see it you can't unsee it.

The actual problem with online shade matching

Every product photo you see in a beauty store is lit under controlled studio conditions, usually 5500K daylight-balanced strobes. Your skin, right now, is lit by whatever's in your environment — a warm 2700K LED, fluorescent overheads, a window with afternoon sun, a monitor's blue glow. Color is not an intrinsic property of an object. It's a relational property between a surface, a light source, and a detector (your eye, or a camera sensor).

Foundation shade "Warm Beige 230" photographed under studio lighting has a specific spectral reflectance curve. That curve will look different under your bathroom lighting, different again on your phone screen (which has its own white point calibration), and different still against your actual skin under your actual light. This is called metamerism, and it's why you can buy what looks like a perfect match online and receive something that reads completely orange under fluorescent light.

Most beauty e-commerce tries to solve this with better product photos or "shade finder" quizzes. Both help at the margins. Neither addresses the core issue: the color rendering environment is completely outside their control.

What on-device processing actually buys you

The approach I took with Mirrrd is to stop trying to predict how a shade will look and instead simulate it directly in your environment, on your device, under your current lighting.

This requires a few pieces working together:

Facial geometry estimation — you need a rough 3D mesh of the face to know where to apply color, how it follows contours, and how shadows fall. MediaPipe's Face Mesh gives you 468 landmarks in real time on device, which is enough to do decent contour-aware blending.

Skin tone detection and segmentation — before you can simulate a product, you need to isolate skin pixels from background, hair, eyes, and lips. This is where most implementations fall apart. A naive color-range mask breaks instantly across different skin tones and lighting. You need a segmentation model that's actually trained on diversity across the full Fitzpatrick scale.

Lighting estimation — this is the piece most AR beauty apps skip, and it's the piece that makes results look fake. If you don't account for the dominant light direction and color temperature in the scene, your virtual product will never look like it actually belongs on the face. Even a rough spherical harmonic lighting estimate — something you can compute from the image itself — makes a significant difference.

Color space math — makeup doesn't layer like digital paint. Lipstick has translucency. Foundation has a skin-interaction model. Blush scatters. You can fake most of this with LAB color space blending and adjustable opacity, but it requires careful tuning per product category.

The key thing about doing all of this on-device: the facial geometry data and skin segmentation never leave the phone. I made this a hard constraint early on, not primarily for marketing reasons but because storing biometric geometry is a liability I didn't want to take on as a solo founder. GDPR, BIPA, and the emerging patchwork of state-level biometric laws are complicated enough without building a central store of face meshes. The on-device approach sidesteps that entirely.

Where it breaks

I'll be honest about the limitations because they're real.

Extreme lighting conditions — very low light, direct harsh backlighting — degrade the lighting estimation and the segmentation quality simultaneously. The simulation still runs, but the result is less reliable.

Glasses are a pain. The face mesh handles them reasonably but any product that overlaps the frame area (eyeshadow, eye liner) clips in ways that look wrong.

Very dark or very light skin tones at the extremes of the Fitzpatrick scale still need more training data than I have. The segmentation is decent but the product simulation parameters were tuned more heavily on mid-range tones and I'm actively working on this.

Real-time performance on older or lower-end devices is a genuine concern. The pipeline runs at 24–30fps on a mid-2022 iPhone. On older Android devices you can drop below 20fps, which makes the try-on feel laggy and breaks the illusion.

The shade matching payoff

Despite the limitations, the approach does solve the core problem I set out to fix. When you try a foundation shade in Mirrrd, you're seeing it rendered on your face, under your lighting, against your skin tone. The metamerism issue doesn't go away entirely — your screen still has its own white point — but it collapses from a four-variable problem (product photo lighting × product photo camera × your screen × your skin) to a two-variable one (your screen × your skin).

Return rates for wrong shades are one of the dirtier secrets in beauty e-commerce. I've seen estimates from 15–25% for foundation and concealer specifically. Not all of those are shade mismatches — fit, formula, texture matter too — but shade is consistently cited as the top reason. That's a fixable problem with sufficient rendering fidelity, and getting there without requiring biometric data storage seems like the right path.

If you're building in this space or have questions about the on-device CV pipeline, I'm happy to get into specifics in the comments. Mirrrd is currently in beta at mirrrd.com — free to try if you want to see the simulation in practice.