In online shopping, you can only imagine how a product actually feels.
- "I won't really know until it arrives."
- "Since I can't tell, I'll just go with whatever has good reviews."
A lot of purchases get put on hold like this.
What if customers could feel a product's texture through the screen?
This article shows a way to let customers see a product's texture on iPhone.
We deliver it right in front of the customer, in AR.
The value of AR commerce: real texture
The value of AR commerce is being able to bring a product into the customer's own space. This project goes one step further: it lights the product with the real light of the customer's room and reflects their environment in it.
Texture comes alive in the room's light: the environment's lighting is captured and the product is re-lit. Glossy surfaces reflect the surroundings; a dark room produces calm, subdued shading — the texture "reacts to the environment."
Placed on the floor at true scale: the floor is detected and the product is placed at real size. Walk around it and you instantly grasp its size and its look from every angle.
A real "it's actually there" feeling: a contact shadow at its base, plus occlusion (people or furniture in front hide the product), give a placement with no composited look.
The goal: let customers experience, in their own environment, "the texture you couldn't tell from photos."
Why Gaussian Splatting — how the model is made
First, you need to capture the product as 3D. Here we use 3D Gaussian Splatting (3DGS).
- It reconstructs a realistic 3D model from photos/video taken from multiple viewpoints (no manual modeling).
- It's good at view-dependent representation, where appearance changes with viewing angle — well-suited to reproducing gloss and highlights.
- Being point-based, it can be rendered in real time even on a phone.
For this project I built the 3D with Ref-Gaussian, which learns materials (normal, roughness, reflectance, albedo) separately. That makes it possible to "strip off the light from capture time and re-light with the light of wherever you place it (relighting)." This is the decisive difference from ordinary photogrammetry, which bakes lighting in.
Implementation: real-time AR on iPhone with Metal
Rendering is hand-written in Metal (based on scier/MetalSplatter, with split-sum IBL relighting and deferred PBR added). AR is ARKit. Main features:
-
Relight & reflect with the room's light (feeding
AREnvironmentProbeAnchor's environment cubemap into IBL) - True scale, floor placement, walk-around (plane detection + raycast, ARCamera 6DOF)
-
Contact shadow / depth occlusion (LiDAR
sceneDepth) - Color / finish try-on (configurator): switch the same model's color and finish (matte / gloss / mirror / metallic), correctly re-lit by the room's light. Change the color and the white logo or pattern stays white (only the hue is replaced).
- Turntable display: hands-free, slow 360° auto-rotation.
The technical key is that lighting is computed at render time. That's why texture reacts correctly, in real time, to "the customer's room light" and "the color the customer chose."
Applicable to many products
The more a product depends on texture and size, the better this lands.
- Cars: body-paint reflections, showroom-style presentation, fit-check for a parking space.
- Furniture / interior: gloss and color under the room's light, sense of size when placed.
- Jewelry / watches: metal and glass reflections (products where texture differences map directly to price).
- Appliances / kitchenware: the feel of stainless steel or glossy plastic.
- Apparel accessories: the sheen of leather and patent.
In all of them, "texture that photos can't convey" and "a size you can't judge until you place it" are solved in the user's own environment.
Summary
- For e-commerce's "size and texture don't come across" problem,
- capture the real product with Gaussian Splatting, and
- display it in AR on iPhone at true scale, under the customer's room light,
- to create a "check the texture before you buy" experience that applies to many products.
The code (relightable GS viewer / Metal + ARKit) is public:
https://github.com/john-rocky/MetalGaussianSplatRelighting
Originally published in Japanese on Qiita. I build apps with machine learning and AR, and write about both. GitHub / X
Top comments (0)