A car can be cut out cleanly and still look pasted on.
The outline is right. The tires touch the ground. The mirrors have no halo. The giveaway is light: the vehicle still carries the room where it was photographed.
That is the image-compositing problem after segmentation. Once the boundary is good, the remaining lie is lighting. The reusable rule is simple: separate factual detail from derived shading, apply a bounded low-frequency gain, then reject the frame if protected detail moves beyond tolerance.
In the AutoLensAI automotive image pipeline, the recent relighting work follows four threads: per-pixel roughness from the material stage, numerical fixes in the Trowbridge-Reitz GGX specular term, a corrected glass reflectance estimator, and a physically based relight path inside the compositor.
1. The matte pipeline fixed edges, not illumination
The deterministic composite stage still matters. compose.py shifts colour statistics toward the plate, matches defocus, places a depth-anchored contact shadow, and wraps background colour around the silhouette. Those steps are fast and predictable.
They also treat most of the vehicle as a flat photograph. A grade can warm the whole car, but it cannot make the hood catch light from the new direction or stop the windshield from carrying the old studio.
flowchart TD
input[Vehicle photo] --> matte[Subject matte]
matte --> screen[Screen-space composite]
plate[New background] --> screen
screen --> fixes[Colour shift, blur match, shadow, light wrap]
fixes --> oldFail[Clean edge, stale lighting]
input --> pbr[Physically based composite]
matte --> pbr
plate --> illum[Scene illumination estimate]
depth[Depth prior] --> normals[Visible normal estimate]
material[Per-pixel roughness map] --> pbr
illum --> pbr
normals --> pbr
pbr --> final[Clean edge, changed shading]```
## 2. Relighting changes the derivable part
Physically Based Rendering (PBR) sounds large, but the decision in `carsegnet/pbr.py` is narrow: estimate the plate illumination, estimate visible surface normals, and replace broad shading while preserving photographed detail.
The Bidirectional Reflectance Distribution Function (BRDF) answers one question: for a view direction, a light direction, and a material finish, how much light reaches the camera? The relight path splits that answer into diffuse response, specular response, and glass handling, then applies the result as a low-frequency gain over the original pixels.
The cost is scope. This stage can move illumination. It cannot author badges, tire lettering, panel gaps, plates, or trim. Those live in the high-frequency band, and `relight` reports the detail-energy ratio so callers can refuse a frame that sharpens, erases, or invents protected evidence.
```mermaid
flowchart LR
albedo[Observed vehicle pixels] --> final[Final subject pixel]
illum[Estimated illumination] --> diffuse[Diffuse term]
normals[Normals from depth prior] --> diffuse
illum --> spec[Specular term]
rough[Material-stage roughness map] --> spec
glass[Glass reflectance bound] --> pane[Glass term]
diffuse --> gain[Low-frequency shading gain]
spec --> gain
pane --> gain
gain --> clamp[Gain clamp and detail check]
clamp --> final```
The underlying decomposition is the useful pattern:
`image = albedo * old_shading`
`output = image * (new_shading / old_shading)`
The gain ratio is low-passed by construction. That makes relighting a bounded edit to derived signal instead of a redraw of customer-owned pixels.
## 3. Roughness made highlights local
A single roughness value gives paint, rubber, trim, and glass the same highlight behaviour. The material stage can now supply a per-pixel roughness map, so clearcoat can stay sharper while tire sidewalls and black trim respond more broadly to the same scene light.
That benefit adds a constraint: roughness may enter only through the specular term, then through the low-frequency gain, then through the verification check. A roughness map cannot become a brush.
The implementation accepts scalar roughness or an array broadcastable against the normal field. When an array arrives, the path keeps per-pixel mode in the report and summarizes the range with real NumPy calls such as `np.min(roughness)` and `np.max(roughness)`. The earlier assumption that `ggx_specular` already handled arrays was false; coercing roughness through `float` would have crashed on a multi-element map, so the fix removed that scalar-only path and covered it with a regression test.
## 4. GGX needed stable inputs and stable math
The specular lobe uses the Trowbridge-Reitz GGX microfacet distribution, shortened to GGX. It controls how glossy highlights form from tiny surface facets facing the light.
The boundary is now explicit in `ggx_specular`: callers pass perceptual roughness in `[0.02, 1.0]`, scalar or map, and the function squares it internally to get GGX alpha. Pre-squaring at the material table would make clearcoat behave like a mirror.
The tradeoff is compatibility with the tuned renderer. The Cook-Torrance specular term uses Schlick Fresnel and separable Smith-Schlick masking with the Unreal Engine 4 Karis remap, `k = (r + 1)^2 / 8`. That is not height-correlated Smith. The label was corrected because changing the visibility term would alter grazing fenders and roof edges enough to require retuning.
The related numerical fix matters most at automotive clearcoat values. Near-mirror roughness makes highlights narrow, and small floating-point errors show up as visible sparkle or missing energy. The corrected path keeps the perceptual roughness meaning intact and applies geometric specular antialiasing so curvature widens the lobe only as much as needed.
## 5. Glass forced a reflectance bound
Glass exposed a separate estimator bug. A smoothed achromatic floor was being counted as reflection, so grey cabin transmission looked like environment reflection.
The corrected estimator uses a physical bound for a uniform source. A uniform environment contributes a constant reflected veil across the pane, while transmitted cabin light is non-negative. Therefore `R * L_env` must be no greater than the darkest observed pane luminance. On the Z/28 door glass case in `carsegnet/glass.py`, that brought the bound back near the Schlick prediction instead of the inflated value.
The cost is refusal. A windshield often mixes cabin, far-side studio, and structured reflected shapes. When the separation is unsafe, the pipeline should decline the glass adjustment rather than make a confident edit from ambiguous pixels.
## 6. Reports keep the edit accountable
Once the compositor changes subject pixels, the caller needs a record. The relight path writes whether it ran, the estimated illumination, confidence, gain range, roughness mode, and detail tolerance result.
There is also a split between `alpha` and `blend_alpha`. The subject matte remains authoritative for harmonisation, defocus matching, shadow placement, and light wrap. A separately validated alpha can be used for the final foreground-background blend. That plumbing costs complexity, but it keeps measurement from being contaminated by edge cosmetics.
```mermaid
flowchart TD
before[After segmentation] --> flat[Flat paint from source lighting]
before --> wrongGlass[Windshield carries old scene]
before --> overpaint[Generated-looking edits to customer pixels]
flat --> fixPaint[Illumination plus normals adjust broad shading]
wrongGlass --> fixGlass[Reflectance bound separates veil from cabin]
overpaint --> guard[Gain clamp, confidence gate, detail tolerance]
fixPaint --> after[Physically based composite]
fixGlass --> after
guard --> after```
Segmentation decides which pixels belong to the car. Relighting decides which part of those pixels is allowed to change. In visual AI products, that distinction is the engineering line: derive the new shading, preserve the evidence, and make every pixel edit leave a report.
---
🎧 **Listen to the audiobook** — [Spotify](https://open.spotify.com/show/4ABVd5yDVfbX9HlV5JjT7D) · [Google Play](https://play.google.com/store/audiobooks/details/How_to_Architect_an_Enterprise_AI_System_And_Why_t?id=AQAAAECafz8_tM&hl=en) · [All platforms](https://www.craftedbydaniel.com/audiobook)
🎬 [Watch the visual overviews on YouTube](https://youtube.com/playlist?list=PLRteDbGJPYDb9XNjecvHplGlgW7tIv_q6)
📖 [Read the full 13-part series](https://www.craftedbydaniel.com/blog/series/how-to-architect-an-enterprise-ai-system-and-why-the-engineer-still-matters)
Top comments (0)