3D parallax from one photo and a depth map, in one WebGL fragment shader

sss — Tue, 10 Nov 2026 16:07:03 +0000

You have a photo and a grayscale depth map of it (white = near, black = far, or the other way round). You want the "2.5D" effect: the camera drifts a little and near things slide past far things. No mesh, no Three.js, just a full-screen quad and one fragment shader.

This is the approach I ended up shipping, with the details that are easy to get wrong.

The naive shader and why it tears

The obvious version samples the depth at the output pixel and shifts the lookup by it:

float d = texture2D(uDepth, vUv).r;
gl_FragColor = texture2D(uImg, vUv + uOffset * (d - uFocus));

This is backwards. It asks "how far does this output pixel move?", but depth belongs to the source pixel. At every depth edge you get the foreground smeared into a double image, because pixels next to the edge read the depth of the wrong surface.

Solve for the source pixel instead

What we actually want is the source position p that lands on output uv:

p = uv + offset * (depth(p) - focus)

p appears on both sides, so iterate. Start at p = uv and repeatedly plug it back in. Plain fixed-point iteration oscillates at sharp edges, so damp each step by mixing half of the new estimate in:

precision mediump float;
uniform sampler2D uImg;
uniform sampler2D uDepth;
uniform vec2 uOffset;   // camera offset, fraction of the image
uniform float uFocus;   // depth that stays still, 0..1
uniform float uInvert;  // 1.0 if the map is black = near
uniform float uScale;   // zoom so no edge ever shows
varying vec2 vUv;

float depthAt(vec2 p) {
  float v = texture2D(uDepth, p).r;
  return mix(v, 1.0 - v, uInvert);
}

void main() {
  vec2 uv = (vUv - 0.5) / uScale + 0.5;
  vec2 p = uv;
  for (int i = 0; i < 16; i++) {
    vec2 q = uv + uOffset * (depthAt(clamp(p, 0.0, 1.0)) - uFocus);
    p = mix(p, q, 0.5);
  }
  gl_FragColor = texture2D(uImg, clamp(p, 0.0, 1.0));
}

Notes:

Focus plane. Pixels whose depth equals uFocus don't move; nearer and farther ones move in opposite directions. Putting focus on the subject makes it feel anchored while the background swings. That reads much better than everything sliding one way.
16 iterations is a fixed loop count because GLSL ES 1.0 needs a constant bound. It's cheap: one texture read per step.
uInvert exists because depth models disagree. Some output white = near, some white = far. A flipped map makes the effect look "inside out", which is a confusing bug to chase.

Don't let the edges show

When the image shifts, empty space appears at the borders. Zoom in just enough that the worst-case shift is covered. The farthest any pixel moves is strength × max(focus, 1 − focus) on each side, so:

function coverScale(strength, focus) {
  return 1 + 2 * strength * Math.max(focus, 1 - focus) + 0.01;
}

This is also why a focus plane near the middle of the depth range lets you use more strength for the same crop.

Blur the depth map a little

Depth maps from neural models have hard, slightly misaligned edges around hair and outlines. Feeding them in raw gives a visible stretched seam. Blurring the depth (not the photo) by about 1.5 px before uploading it as a texture smooths the transition. A 2D canvas does it in one line:

const ctx = depthCanvas.getContext("2d");
ctx.filter = "blur(1.5px)";
ctx.drawImage(depthImage, 0, 0, w, h);

WebGL 1 texture gotchas

Photos are almost never power-of-two sized. In WebGL 1, a non-power-of-two texture renders black unless you use no mipmaps and clamp wrapping:

gl.pixelStorei(gl.UNPACK_FLIP_Y_WEBGL, true);
gl.texImage2D(gl.TEXTURE_2D, 0, gl.RGBA, gl.RGBA, gl.UNSIGNED_BYTE, source);
gl.texParameteri(gl.TEXTURE_2D, gl.TEXTURE_MIN_FILTER, gl.LINEAR);
gl.texParameteri(gl.TEXTURE_2D, gl.TEXTURE_MAG_FILTER, gl.LINEAR);
gl.texParameteri(gl.TEXTURE_2D, gl.TEXTURE_WRAP_S, gl.CLAMP_TO_EDGE);
gl.texParameteri(gl.TEXTURE_2D, gl.TEXTURE_WRAP_T, gl.CLAMP_TO_EDGE);

If the depth image comes from another origin, set img.crossOrigin = "anonymous" before src, or the upload throws a security error.

A loop that records cleanly

The camera follows a closed path driven by one angle, so any whole period loops seamlessly:

const PERIOD_MS = 4000;
function motionOffset(kind, ms, s) {
  const a = ((ms % PERIOD_MS) / PERIOD_MS) * Math.PI * 2;
  return kind === "circle"
    ? [s * Math.cos(a), s * 0.5 * Math.sin(a)]  // flattened ellipse
    : [s * Math.sin(a), 0];                      // left-right swing
}

The vertical half of the ellipse is halved on purpose. Equal vertical motion looks like the photo is bobbing rather than the viewer moving.

To export a video, record canvas.captureStream(30) with MediaRecorder for exactly one period, after resetting the clock to zero. Two details:

Pick a type the browser can actually write. Support differs: Safari records MP4, Firefox records WebM, and Chrome depends on the version. Try a list in order and wrap each check in try, because some browsers throw on unknown codec strings instead of returning false:

   const TYPES = [
     ["video/mp4;codecs=avc1", "mp4"], ["video/mp4", "mp4"],
     ["video/webm;codecs=vp9", "webm"], ["video/webm;codecs=vp8", "webm"],
     ["video/webm", "webm"],
   ];

Make the canvas size even. H.264 encoders reject odd widths and heights, so round both to multiples of 2 when you cap the long edge.

I also create the WebGL context with preserveDrawingBuffer: true, so the canvas still holds the last frame whenever the recorder reads it.

Small UX things that mattered

Pointer control: map the pointer to [-1, 1] and use it as the offset while hovering. Set touch-action: pan-y on the canvas so vertical swipes still scroll the page on phones.
Start paused when prefers-reduced-motion: reduce is set.
Cap the canvas long edge (I use 1280 px). A 4000 px phone photo at 16 iterations per pixel is wasted work on a laptop GPU.

Where the depth map comes from

Any monocular depth model works. I use Depth Anything V2 on the server to produce the map, and everything above runs in the browser, so the photo doesn't have to be uploaded again just to preview or record the effect.

If you want to try the result without writing code, it's the 3D preview in the free depth map tool I build: imagic-ai.com/tools/image-depth-map-generator. Happy to answer questions about the shader in the comments.

DEV Community: sss