A Perfect Fit Can Recover the Wrong Physics

AutoLensAI moves photographed vehicles into virtual studios for dealership listings. I need to change the surroundings without changing the car: its equipment, its cabin, even the damage in its paint. A reflection across a headrest puts those requirements in the same pixels.

On October 4, I had to change what a glass estimate was called. The code could fit a factor relating two photographs. It had not established that the factor measured how much light the glass transmitted. Illumination and camera processing could affect it too.

Calling the value transmission would have made downstream code treat an uncalibrated estimate as a measured property of the pane. I kept the appearance estimate and made the missing evidence explicit.

That distinction is easy to dismiss while the reconstructed image looks right. It becomes harder to dismiss when two explanations fit every observed sample, yet one predicts the wrong result as soon as the lighting changes.

The number the photographs actually identify

Rendering starts with geometry, materials, and light, then predicts an image. Inverse rendering works backward from images to those causes. My glass problem belongs to the second direction, but a good reconstruction alone does not tell me how much of the explanation I have recovered.

Consider one intensity channel with perfect alignment, a linear camera response, no reflection, no clipping, and no noise. Let a reference sample be $x_i$, the observed sample be $y_i$, transmission be $\tau$, and the illumination ratio between conditions be $g$:

$$
y_i = \tau g x_i.
$$

The observations determine the product $\tau g$. They do not, by themselves, separate its factors.

Explanation	Transmission $\tau$	Illumination ratio $g$	Product
Synthetic ground truth	0.40	1.50	0.60
Alternative	0.60	1.00	0.60

Both rows produce the same image. Assuming unchanged illumination picks the second row and makes the transmission estimate 50% too high.

These numbers are chosen for a reproducible counterexample, not taken from a vehicle. The ambiguity is exact: replacing $(\tau,g)$ with $(c\tau,g/c)$ leaves the image unchanged for any positive $c$ that keeps both parameters admissible.

An optimizer can return one answer from that family. A prior can favor one answer, and regularization can make the choice stable. None of those choices adds an observation. I need to know which part of the result came from the photograph and which part came from an assumption.

Removing reflection from this example makes the point more specific, not less relevant. Transmission and illumination are already inseparable here. Adding an unknown reflection field does not automatically supply the missing information.

Held-out pixels can have the same blind spot

The script at the end generates 64 positive reference samples. It fits the product on 32 and reserves the other 32 for evaluation. Both explanations reproduce all the reserved samples with zero error in the recorded run.

Nothing leaked from the held-out set. The split did its job. It tested reconstruction under a condition that both explanations could reproduce.

The distinction I care about appears under a different test. Set a new illumination ratio independently to $g_{\mathrm{new}}=1$, and do not allow another fit. The true transmission predicts $0.4x$. The alternative predicts $0.6x$. Their agreement on the source has become a 50% relative prediction error.

Allowing the second explanation to adjust its illumination again would hide the mistake. The changed condition has to remain fixed by evidence outside the fit.

More pixels are useful when estimating the product in noise. They do not separate the factors in this model. The accompanying tests check 8, 64, and 4,096 samples and find the same missing degree of freedom. More images have a similar qualification: if every image brings another unconstrained illumination multiplier, the absolute scale can remain ambiguous.

This is why I separate source replay from a test under changed lighting. Source replay asks whether the explanation can reproduce what it was given. The second test asks whether it predicts something it was not allowed to adjust itself to match.

Architecture of the public experiment

flowchart TD
  truth[Known synthetic parameters] --> source[Source observations]
  source --> fit[Fit on training samples]
  fit --> alternatives[Image-equivalent explanations]
  alternatives --> replay[Reconstruct held-out samples]
  source --> replay
  replay --> sourceError[Source reconstruction error]
  alternatives --> changed[Predict under known new lighting]
  lighting[Independent lighting condition] --> changed
  truth --> target[Separate target ground truth]
  lighting --> target
  changed --> compare[Compare changed-condition predictions]
  target --> compare
  compare --> parameterError[Error hidden by source agreement]

The target comes from the known synthetic scene, not the candidate estimate. Using the candidate's own parameters to build its reference would make the comparison circular. The diagram shows the evaluation; the private recovery method is outside its scope.

What would let me separate them?

The Jacobian makes the missing information visible. It describes how the predicted samples change as each parameter changes:

$$
J=\begin{bmatrix}g\mathbf{x}&\tau\mathbf{x}\end{bmatrix}.
$$

The columns are proportional. There are two parameter columns but only one independent direction of change, so the matrix has rank one. An increase in transmission can be canceled by a decrease in illumination.

Now add a separate measurement of $g$. Its derivative with respect to $(\tau,g)$ is $[0,1]$. Appending that observation raises the rank to two, provided the reference has nonzero energy and $g$ is nonzero. In this model, that is enough to separate the factors.

An illumination value computed from the same unresolved image ratio adds no independent information. Neither does another region that carries the same unknown optical factor merely because I give it a different label.

Uncertainty matters after the rank problem is resolved. Suppose the product is exactly $0.6$, but the new measurement bounds illumination between $1.425$ and $1.575$. Then

$$
\tau\in\left[\frac{0.6}{1.575},\frac{0.6}{1.425}\right]
\approx[0.381,0.421].
$$

That interval uses deliberately chosen bounds. It shows what happens when the independent reading is imperfect. On a real capture, uncertainty in the fitted product, camera response, alignment, and image model would also have to be accounted for.

Camera calibration cannot be skipped by calling the working pixels linear. Inverting an sRGB transfer function removes that transfer function; it does not undo every operation that produced a processed photograph. In Debevec and Malik's radiance-recovery method, known exposure relationships are used to recover the imaging response up to scale. That response is part of the recovery, not a name assigned to the decoded array.

The October 4 audit had not established the corresponding calibration for my capture path. Its illumination diagnostics also lacked uncertainty bounds. Those are reasons to retain an appearance estimate, even when the structural test and image fit look encouraging. They are not reasons to discard the work, but they change what I can use its output to claim.

608 images, five matched openings

I had already encountered a different way for a good image score to conceal a bad result.

In August, I evaluated a trained alpha refiner on 608 images from 38 vehicles. Alpha matting determines how much of the source to retain at each pixel. The refiner's job was to improve vehicle boundaries, and its boundary score did improve over the upstream semantic mask.

Then I counted the enclosed background openings.

Output	Region IoU	Boundary IoU	Matched openings	Extra openings
Semantic mask	0.9904	0.4233	1 of 1,193	72
Fallback refinement	0.9815	0.2596	28 of 1,193	2,492
Trained refinement	0.9908	0.4762	5 of 1,193	111

From the August evaluation record. These images were used for checkpoint selection; they were not an untouched test set.

IoU measures overlap with the reference region. Body pixels dominate it. A filled wheel opening occupies little area beside an entire door or hood, but it still carries the old background into the composite. Boundary IoU looks more closely at contours. The opening comparison checks whether enclosed background components have survived at all.

The trained model had the best boundary score and still missed 1,188 reference openings. The fallback matched more openings but created 2,492 extras. The extra count prevented me from treating that apparent improvement as useful recovery.

This failure had a different cause from the glass ambiguity. The refiner could modify only its allowed boundary region. A confidently filled opening outside that region was fixed with respect to the learned weights. Training that same stage longer could not give it control of pixels its output definition excluded.

In the glass example, different parameters explain the same observations. Here, the desired correction can sit outside the model's permitted outputs. The first problem needs evidence or assumptions that separate the explanations. The second needs a prediction target and output region capable of representing the correction, plus tests for the new false openings that freedom can introduce.

I retained the checkpoint for its boundary improvement. Its clean studio images and binary glass labels did not establish performance on dealer lots or recovery of light through glass. The opening counts were enough to stop the broader acceptance claim without denying what the model had learned.

I rejected the lowest error

On September 3, the relighting work presented another trap. I had no photograph of the actual truck under the destination lighting. Judging a correction only by how little it disturbed that truck could favor a method that barely changed anything.

I used a synthetic calculation with a known destination response to compare the alternatives. The approximate geometry used for lighting could leave its own pattern in the result. Suppressing that pattern was useful until the suppression also weakened the broad shading I was trying to add.

Candidate	Target RMS log error	Unwanted-pattern correlation	Protected-detail damage	Intended-shading correlation
Baseline	0.3869	0.9994	0.0594	0.9472
A	0.3242	0.0276	0.0042	0.9557
B	0.3257	0.0134	0.0000	0.9530
C	0.3335	0.0109	0.0000	0.8955

Recorded September 3 synthetic comparison. Candidate letters replace the private settings. Damage is a fraction of protected detail; the two correlations compare the unwanted pattern and the intended shading.

A had the lowest error against the target in log brightness, but it damaged protected structure. C removed more of the unwanted pattern than B, but it also weakened the intended shading. I selected B. A single ranking by either target error or artifact suppression would have selected something else.

The damage test needed checking too. Brightening a badge is not the same as changing its lettering. In the retained controls, smooth illumination changes registered zero structural damage. Replacing “F-150” with “F-250” registered 0.0629. That gave the rejection a specific meaning: the test was responding to an identity change, rather than any difference in RGB.

The calculation isolated the operator's behavior. It left out indirect illumination, render noise, real geometry gaps, and material boundaries. I can use it to compare those candidates, not quote its error as an expected result on a dealership photograph.

Run the optical check

The transmission example is intentionally small enough to inspect. It is standard multiplicative ambiguity, not a claim to a new theorem. The broader problem has a long history: Belhumeur, Kriegman, and Yuille's bas-relief result shows how compensating changes in shape, reflectance, and light can preserve images under different assumptions from this glass example. Modern methods such as NeRFactor make their representations and priors explicit when recovering shape, reflectance, and illumination.

Here the useful result is that even a perfect held-out image fit cannot tell the two rows apart. A known change of lighting can.

Save the following as optical_identifiability.py. Install NumPy with python -m pip install numpy, then run python optical_identifiability.py.

import numpy as np


def estimate_product(reference, observed):
    x = np.asarray(reference, dtype=np.float64)
    y = np.asarray(observed, dtype=np.float64)
    if x.ndim != 1 or y.shape != x.shape or x.size == 0:
        raise ValueError("expected matching nonempty 1D arrays")
    if not np.isfinite(x).all() or not np.isfinite(y).all():
        raise ValueError("samples must be finite")
    energy = float(x @ x)
    if not np.isfinite(energy) or energy <= 0:
        raise ValueError("reference must have finite positive energy")
    product = float(x @ y) / energy
    if not np.isfinite(product):
        raise ValueError("fitted product must be finite")
    return product


x = np.linspace(0.1, 0.8, 64)
true_tau, true_light = 0.4, 1.5
observed = true_tau * true_light * x
training = np.arange(x.size) % 2 == 0
heldout = ~training
product = estimate_product(x[training], observed[training])

for tau in (0.4, 0.6):
    fitted_light = product / tau
    prediction = tau * fitted_light * x[heldout]
    rmse = float(np.sqrt(np.mean((prediction-observed[heldout])**2)))
    assert rmse < 1e-14
    print(f"tau={tau:.1f}, light={fitted_light:.1f}, RMSE={rmse:.3g}")

# The new light is fixed independently; it cannot compensate for wrong tau.
new_light = 1.0
expected = true_tau * new_light * x[heldout]
wrong = 0.6 * new_light * x[heldout]
relative_error = np.linalg.norm(wrong-expected) / np.linalg.norm(expected)
assert np.isclose(relative_error, 0.5)
print(f"Changed-light relative error: {relative_error:.1%}")

# Check whether the observations distinguish the two parameters.
jacobian = np.column_stack((true_light*x, true_tau*x))
augmented = np.vstack((jacobian, [0.0, 1.0]))
assert np.linalg.matrix_rank(jacobian) == 1
assert np.linalg.matrix_rank(augmented) == 2

recovered_tau = product / true_light
assert np.isclose(recovered_tau, true_tau)

light_low, light_high = 1.425, 1.575
assert 0 < light_low <= light_high
interval = (product / light_high, product / light_low)
assert interval[0] < true_tau < interval[1]
print("Ranks:", np.linalg.matrix_rank(jacobian),
      np.linalg.matrix_rank(augmented))
print(f"Recovered tau with known light: {recovered_tau:.3f}")
print(f"Tau bound with uncertain light: [{interval[0]:.3f}, {interval[1]:.3f}]")

The run produced:

tau=0.4, light=1.5, RMSE=0
tau=0.6, light=1.0, RMSE=0
Changed-light relative error: 50.0%
Ranks: 1 2
Recovered tau with known light: 0.400
Tau bound with uncertain light: [0.381, 0.421]

RMSE is the root mean square difference between the held-out prediction and observation. The assertions allow floating-point roundoff. The companion ten-test suite also checks invalid inputs, unchanged input arrays, rank at different sample counts, and propagation of the illumination bounds. The example and all ten tests passed with Python 3.13.5 and NumPy 2.3.5.

The private model, capture procedure, training recipe, and recovery settings are not needed to reproduce this calculation. The August and September tables report retained experiments; the code above is the separate synthetic example run for this article.

For the actual glass estimate, the next empirical test needs an independently known change of condition and an uncertainty budget for the measurements. I need the estimate to survive that change of light, with no second fit to rescue it.

Cover: generated editorial artwork, not an image from the experiments.

---

🎧 Listen to the audiobook — Spotify · Google Play · All platforms
🎬 Watch the visual overviews on YouTube
📖 Read the full 13-part series