Which Colors to Save

The Color After White

Last time, AWB restored white to white. But white being right doesn't mean the rest of the colors are. However precisely you fix the white point, the reds and greens and blues beside it are still off somewhere. This installment looks at the seat that deals with that mismatch, CCM, through the same frame.

CCM, the color correction matrix, is the job of moving the color a sensor recorded closer to the color a person sees. The root of the mismatch is that sensor and eye divide color differently. The human eye splits light with three kinds of cone cells; the sensor splits it with color filters. Because the two divide color differently, even for the same red apple, the red the sensor recorded is a little different from the red a person perceives. If AWB fixed a single point, white, CCM pulls all the remaining color toward the human side.

What CCM Decides

What CCM does comes down to a single 3×3 matrix. Multiply the sensor's (R, G, B) by this matrix and you get an (R, G, B) closer to what a person sees. The diagonal terms keep each channel as it is; the off-diagonal terms mix the channels together. Subtracting some green and blue when making red, for instance, spreads back apart the colors the sensor lumped together.

How is the matrix determined? You photograph a reference color chart whose true values are already known, under a defined light, and measure what (R, G, B) the sensor actually records for those colors. Then you fit the nine coefficients of the matrix so that those measurements are moved as close as possible to the true values. White and neutrals were already fixed by AWB, so CCM constrains each row to sum to one in order not to disturb them. And this fit changes with the light. Even for the same sensor, the color response under daylight differs from the response under incandescent, so matrices are kept per illuminant and the space between them is filled according to color temperature. Measurement → matrix → application → switching per illuminant. That is the skeleton of CCM.

The Core Is a Fit That Can Never Be Perfect

Within that skeleton, what most reveals CCM's character is that this fit can never, in principle, be perfect.

The way a sensor captures color and the way an eye perceives it are mismatched from the start, so a single 3×3 multiplication cannot bring the two into complete alignment. Whatever matrix you choose, you cannot hit every color at once, and some colors will always remain slightly off. So fitting the matrix is not the job of eliminating error. It is the job of deciding where the error that must remain will go. You give up a little on other colors to keep skin tones accurate; you fix the colors people hold in memory, like sky and grass, first and push the rest back. Which colors to save and which to give up is the actual content of this matrix.

In the Two Seats, This Work Diverges

The two seats grip this color-moving matrix from different ends.

In the ISP seat, the asset is the CCM algorithm itself. What error criterion the matrix is fit against, how neutrals are constrained and protected, how many matrices are held per illuminant and how the space between them is filled by color temperature, and how the choice of which matrix to use interlocks with the estimate of the light's color. These are the handles this seat moves. But this seat does not choose which sensor it will be paired with. Sensors capture color differently, so the same color is recorded as a different (R, G, B). That characteristic is a given input, and the algorithm must be able to extract a usable matrix across a wide range of sensors, not one. In this seat, the sensor's color characteristics are not a handle you can move but a wall you cannot cross.

In the module seat, that handle and that wall swap places. The CCM algorithm is now a given: it arrives with the ISP this module will run on. And the module, already designed and in production with its sensor, color filter, and infrared cut filter, is fixed. What can be moved are the values the algorithm leaves open: the coefficients of the matrix. You photograph the reference chart under each illuminant and extract a matrix suited to this sensor. But deciding which colors the residual error lands on, and how saturated to push color, is not this seat's call. What the module seat does is establish how far color can be saved and pushed with this sensor, the achievable range, and hand that to the client; where to place the error is decided by the client, looking at that range. The module seat then realizes that choice as actual coefficients on this sensor. The sensor color characteristic that one seat cannot cross as a wall, the other measures directly under the reference chart and uses as the starting point of the matrix. The same color characteristic is a wall in one seat and a handle in the other.

So "It Went Well" Means Different Things

Both seats share the goal of passing spec. What diverges is what each protects beyond it.

What the ISP seat protects is breadth. An algorithm that extracts a good matrix on one sensor isn't enough. It has to extract usable matrices across sensors whose color characteristics vary widely, fill the space between illuminants smoothly, and offer a clean interface where the integrating side can seat a matrix for its own sensor. Only then does that algorithm get chosen again on the next project. In this seat, CCM "going well" is closer to a generality reused across sensors.

What the module seat protects is repeatability. This one module passing its color spec (reference color errors within tolerance under defined illuminants, natural skin tones) isn't the end. In production, unit-to-unit variation in the color filter and infrared cut filter shifts the sensor's color characteristics, and then a matrix fit to one unit is off on the next line, with skin tones starting to go sallow or ruddy. Fit the matrix too tightly to a single golden sample and it won't survive that variation. In this seat, CCM "going well" is closer to a specificity that repeats on this module.

After Color Finds Its Place

By the time you've come through CCM, the amount of light, the evenness across the frame, white, and the color beyond white have all found their places. The big decisions of color that make up an image are largely done here. Across four blocks, the same scene has repeated. The same decision diverged into breadth in the ISP seat and into repeatability in the module seat.

I have watched that divergence from both seats, across all four blocks. Next time I step back from the individual blocks and move the frame to tuning as a job, and to the movement between the two seats.