I've been building browser-based image tools for a while now, and color replacement is one of those features that seems simple on the surface but hides a lot of interesting problems. I want to write about the main one: why the obvious approach (RGB distance matching) fails on real photos, and how HSV fixes it. Plus some notes on shading preservation, edge detection, and where intelligent scissors fit in.
Why RGB matching fails on photos
When most developers implement "replace this color," the first instinct is something like:
const dist = Math.sqrt((r - r0)**2 + (g - g0)**2 + (b - b0)**2);
if (dist < threshold) replacePixel();
This works for flat, uniform colors on a white background. It breaks completely on photos with lighting variation.
The problem is shadows. A red shirt in bright light might have pixels around (220, 40, 40). The same shirt in shadow is more like (80, 15, 15). Both are "red." Their Euclidean distance in RGB space is massive, so either you set a high threshold and accidentally replace dark brown/maroon pixels you didn't want, or you set a low threshold and miss everything in the shadows.
You end up in a losing tradeoff that doesn't resolve cleanly no matter how much you fiddle with the slider.
Switching to HSV
HSV separates the color information (hue) from the brightness (value). That red shirt in shadow still has a hue of around 0 degrees regardless of how dark the pixel is. So you can match on hue with a tight tolerance and not worry about lighting variation at all.
In practice: convert each pixel to HSV, check if the hue falls within N degrees of your target hue AND saturation exceeds a minimum threshold (this excludes near-white and near-gray pixels where hue is essentially undefined). Both conditions met? That's a pixel to replace.
The result on photos is significantly better. Deep shadows and bright highlights of the same hue all get picked up, and the selection boundary follows the actual color rather than the lighting.
One thing that took some iteration: a single "tolerance" slider doesn't work well for HSV. You really want separate H, S, and V tolerance controls. A common case is wanting a tight hue band (only this shade of red, nothing pink or orange) but a wide value tolerance (catch all the dark and light versions of it). Collapsing that into one number forces a tradeoff that's impossible to tune correctly. Three independent sliders is more UI but actually usable.
Shading preservation
Once you correctly identify the pixels to replace, there's a second problem: how do you fill them?
Naive approach: overwrite with the new color's RGB values. This makes it look fake instantly. A red jacket replaced with flat blue loses all the depth. The folds, creases, and fabric texture disappear into a uniform blob.
What actually looks good: replace only the hue component and keep the original pixel's saturation and value. In code:
// target hue comes from the new color
const resultH = targetHSV.h;
const resultS = originalHSV.s; // keep original
const resultV = originalHSV.v; // keep original
This preserves all the lighting information: shadows still dark, highlights still bright, texture intact. Only the color itself changes. The output looks like someone actually repainted the object rather than pasted a layer mask over it.
You can add an optional saturation scale on top of this if the new color should be more or less vivid than the original, but the core keep-S-and-V trick does most of the work.
Polygon selection and edge snapping
Global color replace is fine when the background is clearly a different color. But in most real photos, background pixels share hue values with the foreground. You need region selection.
We implemented a polygon selection tool where the user clicks to define a boundary around the area to recolor. That's straightforward. The interesting part is the magnetic lasso behavior: when the user clicks two anchor points, instead of a straight-line segment between them, the path finder uses edge detection to trace along the nearest object boundary.
The edge detection is a Sobel operator, two 3x3 convolution kernels that compute intensity gradients in X and Y, then combined as gradient magnitude. High gradient = edge. These values become the traversal cost for a shortest-path search.
The path finding itself (Livewire / intelligent scissors) is Dijkstra's algorithm on the pixel grid, where traversal cost is inversely proportional to edge strength. Following a strong edge is cheap; cutting across flat regions is expensive. So the algorithm naturally snaps to object boundaries. When it works well (objects with distinct edges against a contrasting background) it feels like the cursor is magnetic, which makes precise selection much faster than clicking every vertex manually.
Where we shipped this
All of this is in a free browser tool at https://irrationaltools.com/color-replacer. No uploads, no server processing, everything runs in-browser. There's also support for multiple color pairs in one pass (change red to blue AND green to orange simultaneously), full undo/redo, and zoom up to 10x for pixel-level work.
If you're implementing something similar and have questions on the HSV matching, the Livewire implementation, or anything else in here, happy to dig into it in the comments.
Top comments (1)
This is super insightful! I've always struggled with