Daisuke Majima

Posted on Jun 2 • Originally published at qiita.com

Real-time relighting of Gaussian Splatting reflections on iPhone (Metal)

#ios #metal #graphics #3d

I built a Metal viewer on iPhone that re-lights an already-captured 3D scene with any lighting you like. Swap or rotate the HDR environment map, and the object's reflections follow in real time, with that environment also drawn into the background.

Why relighting is valuable (the practical, commercial meaning)

Ordinary Gaussian Splatting has the lighting from capture time baked in. So if you place the captured object somewhere else, the highlights and shadows clash with the new surroundings and look fake. Relighting strips that light off and returns it to material, so you can place the captured object under any lighting. This matters in practice:

E-commerce / product visuals: capture a product once and show it under any light — showroom, outdoors, the customer's room (AR). Strong for "texture sells" goods like furniture, cars, jewelry, sneakers.
Film / virtual production: place a real capture into a new scene, consistent with that scene's lighting (HDRI / LED wall). No reshoot.
AR / spatial computing: an object placed in a real room only blends in once it's lit by the room's light. Relighting is a precondition for realistic AR placement.
Games / real-time 3D: instead of a baked-in fixed look, you get a photoreal asset that reacts to dynamic in-game light (day/night, moving lights).
Cost: an asset captured in minutes becomes "usable under real production lighting," like manual modeling + material authoring that takes days.

In short, relighting turns a captured 3D into an actually usable asset. This article is a record of bringing it from desktop research (CUDA-assumed) to a real-time Metal implementation on iPhone.

The first half collects background knowledge; the second half covers the implementation and four bugs I hit. I define jargon as it appears, so you can follow even without a Gaussian Splatting / PBR background.

Repo: https://github.com/john-rocky/MetalGaussianSplatRelighting

Background

1. What is 3D Gaussian Splatting

A method that reconstructs a 3D scene from a set of photos and renders it in real time. The scene is represented as a huge set of translucent ellipsoids called splats. Each splat has "position, shape & orientation (rotation), color, opacity." Rendering projects each splat to the screen as an ellipse and alpha-composites them front-to-back in depth order. Color changes with viewing angle (view-dependent).

Key point: ordinary Gaussian Splatting holds the appearance (color) itself, with the capture-time lighting baked in. So you can't change the lighting afterward.

2. Lighting and relighting

Ordinary GS directly learns "the color of that spot photographed under that light." So lighting is fixed.

Relightable GS thinks differently. Per splat, it holds not "color" but a decomposed material:

Albedo: the base color of the material itself, with lighting removed
Normal: the direction the surface faces
Roughness: surface micro-roughness
Reflectance: strength of specular reflection

With the material, you can recompute the color on the fly under any environment light. That's relighting. The Ref-Gaussian I used learns this material decomposition.

That's enough if you get "splats hold material, and we want to re-light them in a new environment." The rest, #3–#5, just answer one question:

Given that material and the environment, how do we compute the color of one pixel?

3. Reflection is split into two and added

Light hitting a surface returns in two ways:

Diffuse: returns light evenly in all directions → you see the albedo color itself. No reflections.
Specular: returns only in a specific direction (the reflection of the incidence) → the environment is reflected.

So color = diffuse + specular. The splat's reflectance decides the blend (how strong the specular is). And roughness decides how blurry the specular is:

low roughness → the environment reflects crisply, like a mirror
high roughness → blurry; the bright parts of the environment just appear as a vague blob

This matters later. In fact, a glossy car (roughness ~0.2) just shows the environment lighting as a blurry white blob that moves, without resolving beams or window shapes. It's not "no reflection," it's a blurry reflection, and that's physically correct. Drop the roughness way down and the same car becomes mirror-like, clearly reflecting the room.

4. How to hold the environment (light source)

Instead of placing point bulbs, use a 360° image as the light = a list of what color light comes from each direction.

HDR: an image that can hold brightness above 1 (needed because windows and lights are orders of magnitude brighter than paper)
equirect / cubemap: names for the storage format of that 360° image. Both contents are "direction → light color."

Swapping the environment map = swapping the lighting = relighting.

5. Getting diffuse and specular from the environment (split-sum)

We want to compute #3's "diffuse" and "specular" from #4's environment map. Done naively, you integrate the environment per pixel — heavy every frame. So UE4's split-sum precomputes two images:

For diffuse (irradiance): the environment averaged over all directions → one lookup in the normal direction gives diffuse light.
For specular (prefiltered): the environment blurred per roughness level (stored progressively in mips) → one lookup in the reflection direction gives specular light (blur level = roughness).

At runtime you just sample these two textures. Replacing a heavy integral with "look up a pre-blurred image" is the heart of split-sum. (Auxiliary: a small table that fine-tunes specular strength by angle — the BRDF LUT — is also precomputed.)

I implemented this precompute kernel in Metal.

6. Deferred shading (a splat-specific issue)

Splats are translucent and overlap, so neighboring splats' normals vary and get noisy. If you shade each splat individually and then composite, that noise comes straight through.

So change the order: first accumulate each splat's material (color, normal, roughness, etc.) into a screen buffer (G-buffer) and blend = average, then shade once per pixel. Computing after normals are averaged reduces noise. In Metal, use tile memory (imageblock) to keep this buffer on the GPU and process it fast.

7. Normals and coordinate systems

Normal: the unit vector of the surface direction. The reflection direction depends on it, so if it's off, all reflections are off. It's reconstructed from the splat's orientation (rotation quaternion).
Up-axis convention: there's Y-up (many viewers) and Z-up (Blender-family). A mismatch between data and viewer tips the object over (→ bug 3).

Implementation: the shading equations

With the background in place, read the equations Ref-Gaussian's deferred surfel renderer (render_surfel) computes per pixel:

F0       = 0.04*(1 - reflectance) + albedo * reflectance
specular = prefiltered(reflect(V, N), roughness) * (F0 * fg.x + fg.y)
final    = (1 - reflectance) * base_color + specular

reflect(V, N): view direction V reflected about normal N. Look up prefiltered (#5) here = the environment reflected in the specular.
fg: a lookup into the BRDF LUT (#5).
final: uses base_color directly for diffuse, and only computes specular from the environment and adds it (= #3's "diffuse + specular"). This matters in bug 2.

Whole pipeline:

Ref-Gaussian .ply --> load --> per-splat material (normal, roughness, reflectance, albedo)
                                       |
HDR env --> IBL precompute --> prefiltered (specular) + irradiance (diffuse) + BRDF LUT
                                       |
                         +-------------+--------------+
                         v                            v
                 G-buffer pass               postprocess pass
        (blend color/normal/material      (per-pixel split-sum IBL
          into tile memory = #6)            + skybox compositing)

— sounds clean. Until you run it. Now the real story: four bugs.

Bug 1: the normal map is a rainbow sandstorm

The shading was patchy and flickering. The "Normal" debug view (normals visualized as color) was rainbow noise, not a smooth gradient.

My first thought — "2D-surfel normals are just inherently noisy" — was wrong, and I nearly wasted a stack of device builds on it.

What saved me was discipline: first draw the normals offline and verify. Compositing each splat's geometric normal (#7: reconstructed from the rotation quaternion and flipped to face the camera) with a small numpy script gave a smooth result (median gradient 0.006). So the data was correct and the renderer was buggy.

Culprit: at load time MetalSplatter reorders splats for cache efficiency (Morton order, sortByLocality). It reorders the splat buffer and the SH-coefficient buffer, but the material buffer (which holds the normals) I'd added later was a separate buffer and wasn't reordered.

So after sorting, splats[i] corresponded to materials[some other j], and every splat held someone else's normal. Color (the view-dependent color of #1) was in an already-reordered buffer, so it stayed consistent, and only the normals and specular looked broken — which made it hard to isolate.

The fix was one line:

materialsBuffer.values.reorderInPlace(fromSourceIndices: sorted)

Lesson: when you bolt a "per-element parallel buffer" onto someone else's pipeline, fix every place the source data gets reordered.

Bug 2: I was re-lighting the diffuse with environment light (don't)

Even after fixing normals, the body was a "watery," patchy yellow.

I'd written the diffuse term by the textbook as albedo × irradiance (multiplying by #5's diffuse image). But Ref-Gaussian's equation uses base_color directly for diffuse — only the specular is relit. I was painting a pattern onto the cream-colored body with my own irradiance. Worse, I was tinting the specular F0 with the view-dependent color (capture-time reflections baked in) instead of the learned albedo.

Matching the reference equation exactly fixed it. Lesson: before improvising "correct" PBR, read the reference implementation's source and match it line by line.

Bug 3: the car is on its side (Z-up vs Y-up)

When I trained and loaded a reflective car, it rendered 90° on its side.

Instead of guessing, I measured the point cloud's bounding box: the shortest axis was Z (= height), the longest was Y (= length), and the dark tire splats were on the −Z side. The data was Z-up (#7, Blender-family). The viewer assumed Y-up. The 90° offset that didn't show on a round helmet was suddenly exposed by the car.

Adding a Z-up→Y-up correction (−90° about X) to the camera fixed the car. Then a second head sprouted: now the background environment was 90° off. equirect (#4) assumes Y-up, but the skybox rays and reflections are computed in the scene's Z-up frame.

The fix: convert the sample direction for the environment into the environment's Y-up frame, and rotate the slider about the scene's up-axis:

environmentRotation = Rx(-90°) * Rz(slider angle)

The skybox and reflections sample with the same matrix, so they always match. I verified the mapping numerically before building, and validated the skybox itself by an offline render that reconstructs rays from the inverse view-projection. That also caught an old top-bottom flip (double-flipped) I'd previously baked into the HDR.

Bug 4: a "successful" build that runs old code

After the orientation fix, a report came in: "the car is still on its side." The offline render had already proven the math correct, so the running binary must be stale — but why?

I'd been type-checking with xcodebuild -destination platform=macOS. That only compiles the #if os(macOS) branch and Mac architectures. When I built for the iOS simulator, existing code revealed a compile error: I was assigning Float16(x).bitPattern into a [UInt16] array.

arm64 (device): native Float16 exists, bitPattern is UInt16 → compiles
the simulator's x86_64 slice: Float16 falls back, bitPattern is UInt32 → type error

The iOS build was failing, so the device kept running the previous binary. Holding [Float16] directly fixed it.

Lesson: type-check for the platform you ship to. A macOS-only xcodebuild will happily lie about your iOS app. And "nothing changes on device" is almost always a sign the binary isn't new — suspect that before re-debugging your logic.

The methodology that actually worked

What all four bugs share: establish ground truth before judging your output. Early on, my offline numpy preview was overly smooth and misled me. What worked:

Run the reference renderer (Ref-Gaussian's eval.py or training-time visualizations) on the same asset and compare. If the reference is clean and yours is dirty, it's a bug in your renderer.
Reproduce the transforms (normals, skybox rays) exactly offline and look at them before building for device.
Verify on a clean synthetic asset (a hand-made chrome sphere) to separate "renderer bug" from "asset bug."

Every time I skipped this and settled for "looks fine," I was wrong.

Aside: is the look "correct"?

After finishing, I felt "the car's reflections are dull, it doesn't look like it's reflecting the environment." That's not a bug — it's the material's nature. To be clear:

The trained car is roughness ~0.2 = semi-gloss. Semi-gloss reflects the environment blurrily (not a mirror). So bright lighting appearing as a blurry blob is correct. Any renderer, lit by the same environment, shows it equally dull. Even the official renderer's output (ground truth) shows a clean reconstruction (sharp car, smooth normals); the material is fine.
Turning off the app's "Use trained material" and lowering roughness lets you override all splats to a uniform specular, clearly reflecting the room. That's flashy but not present on a real car. ON = the learned real material, OFF = a manual override.
Note that material decomposition has an inherent ambiguity: a blue car's albedo can decompose as yellowish (splitting blue into "blue light + yellow material"). This is normal in inverse rendering; the rendered result itself is clean, so the harm is small, but it's not a "physically perfect decomposition."

So "dull look = the material behaving correctly," not an error in the iOS renderer.

Result

A reflective Ref-Gaussian scene, relit in real time on iPhone: switch and rotate the HDR environment and the reflections and skybox follow. The base is MetalSplatter, the relighting model is Ref-Gaussian, and split-sum IBL is from UE4.

Code, demo, details: https://github.com/john-rocky/MetalGaussianSplatRelighting

Next: light the splats from the actual environment via ARKit's environment probe — place a relightable object in your own room and reflect the room.

Implemented in Swift + Metal / iOS. Credits: MetalSplatter (Sean Cier, MIT), Ref-Gaussian, HDRIs from Poly Haven (CC0). Originally published in Japanese on Qiita.

DEV Community