ISP Tuning as a Job

Same Word, Different Work

In camera module work, the word "tuning" comes up constantly. But it points to entirely different jobs depending on which seat you're sitting in.

For one person, tuning means refining algorithms and parameters so that the ISP they built behaves consistently across many kinds of sensors and lenses. For another, tuning means making one specific module they designed and are about to mass-produce deliver the image a client asked for, on top of an ISP they were handed.

Both say they "went through hell tuning it." But what one moved and what the other moved are not the same thing. What they hold in their hands and what they accept as an immovable wall differ, and so does the standard by which they can say it's finished.

This series follows that difference. It looks at how the same word, "tuning," names two different jobs in two different seats, by opening up the core image-formation blocks one at a time.

What Separates the Seats Is the Asset

"Seat" here doesn't mean a job title or an employer. What separates the seats is what you hold as the asset you can sell.

One seat's asset is the ISP: the processing blocks that turn light into an image, the algorithms inside them, and the ability to run all that on many different pieces of hardware. What this seat sells, in the end, is a promise: "our ISP produces a good image no matter which sensor and lens it meets." In this seat, the module is not an asset. It is one of countless inputs the ISP must withstand.

The other seat's asset is the module: binding lens, sensor, and actuator into a single physical component, designing it, and mass-producing it at a defined quality. What this seat sells is the promise that "this module meets spec and can be produced reliably at volume." In this seat, the ISP is usually not an asset. It arrives with the SoC or comes from outside. It is a given condition.

So the same "tuning" becomes different work in the two seats. One refines its own asset, the ISP, to fit someone else's hardware. The other fits its own asset, the module, to someone else's processing chain. The direction the refining hand points is opposite from the start.

From here on, I'll call the first the ISP seat and the second the module seat.

Handles and Walls Swap Places

Tuning is, in the end, moving something in order to satisfy something. So: what can you move, and what can you not? The arrangement of handles and walls swaps between the two seats.

In the ISP seat, what moves is the ISP itself. There is freedom in how the processing blocks behave and in their parameters. The module, conversely, is a wall. And this seat isn't fitting to a single module. It assumes a range of modules as input and looks for behavior that won't collapse anywhere across that range.

In the module seat, the arrangement is reversed. Once design is finished and volume production begins, the module hardware is effectively frozen. The lens characteristics and the sensor's behavior are already settled. What can be moved are the ISP's settings sitting on top of it, and even those have less freedom than in the ISP seat, since the ISP itself is a condition handed down from outside.

Both seats aim at a spec the client has defined. Once a project is locked and the client and target product are fixed, a spec that must be met exists for either seat. The mere existence of a spec does not separate them.

What separates the two seats is not whether a spec exists, but what each can move toward it. Facing the same spec, what was a handle in one seat becomes a wall in the other.

"It Went Well" Means Different Things

If handles and walls are arranged differently, then what it means to say "the tuning went well" differs too. Both seats must pass the spec, as we've seen. Where they diverge is beyond that passing. Past the spec, each seat has something else it must protect.

What the ISP seat protects beyond passing is generality. Meeting this project's spec isn't the end. The ISP must not collapse anywhere across the range of hardware it is expected to carry. Only then can it say, to the next customer and the next project, "just use ours." Tuning in this seat hits the spec in front of it while simultaneously holding up a reusable breadth.

What the module seat protects beyond passing is repeatability. One unit passing the spec isn't the end. Every unit coming off the line must hold that result. Tuning in this seat is optimized toward binding fixed hardware stably to a defined spec.

Holding up breadth and holding up repeatability are different jobs. Even when touching the same block, the two seats touch it to pass the same spec while protecting different things.

The Cross-Section This Series Examines: Four Blocks

Many processing blocks are involved as an ISP turns light into an image. This series doesn't cover all of them. I chose only the cross-section that most sharply reflects the difference between the two seats.

There are four blocks. The order follows the pipeline, the order in which light becomes an image.

• AE (Auto Exposure) — decides how much light to take in.
• Lens Shading Correction — corrects the light falloff and the color shift that grow toward the edges of the frame.
• AWB (Auto White Balance) — estimates the color of the light source and makes what people perceive as white come out white.
• CCM (Color Correction Matrix) — moves all the colors beyond white closer to what a person sees. There's a reason lens shading sits before AWB. If the frame hasn't been flattened, the color cast at the edges muddies the very evidence AWB uses to estimate the illuminant. You have to flatten the frame before you can ask what white is.

Two criteria drove the selection. First, the block had to make the division of labor between the seats clearly visible. Second, it had to belong to the core of image formation: the amount of light, how evenly that light lands across the frame, the color of the illuminant, and the reproduction of color. All of these are work at the stage where the image is formed, which is different in character from post-processing that polishes an image already formed.

It's only honest to state why the other blocks are left out. This is not a ranking of importance. It's a question of whether they're cleanly captured by this series' frame.

AF (Auto Focus) belongs more to mechanics and calibration than to ISP tuning. Bringing something into focus is a different kind of problem from what happens inside the chain that turns light into an image.

Blocks like NR (Noise Reduction), Sharpening, and Gamma/Tone vary enormously in structure from one ISP to the next, and "how much is good" is deeply subjective. These blocks are handled differently in the two seats as well, but the difference comes less directly from asset structure and gets buried in per-ISP implementation and taste. The "two seats" frame doesn't cleanly separate them.

So this series looks at four blocks only. Not everything that could be covered, but only what this frame makes sharp.

The Distance Varies by Block, and That Belongs to Each Installment

One thing to nail down in advance: the distance between the two seats is not the same across the four blocks. In some, the seats do nearly the same work. In others, nearly opposite work.

But how that distance varies by block is not something I'll settle here in Part 1. If I concluded up front which block is farthest and which is closest, each installment that followed would become an exercise in retroactively justifying that conclusion. The distance comes not from the seats but from the nature of the block itself, and that nature only reveals itself when you actually look inside. So I leave that judgment to each installment.

About the seat I'm writing from, one line is enough. I have passed through both of these seats and watched, up close, how the same work is carried out on top of different decision structures. What this series rests on is that observation.

From the next installment, I open the blocks one at a time. The first is AE: how much light to take in. The decision at the very front, where the image begins.