DEV Community: Kim Brandwijk

LiDAR Depth Maps on the GPU in React Native — And a Dawn Upstream Contribution

Kim Brandwijk — Wed, 18 Mar 2026 15:24:59 +0000

In Part 1, I built a zero-copy GPU compute pipeline for camera frames. In Part 2, I got Apple Log HDR working with LUT color grading. This post covers what happened next: getting LiDAR depth data into the same GPU pipeline, the format wars that ensued, and submitting my first upstream patch to Dawn (Google's WebGPU implementation).

The Goal

iPhone Pro models have LiDAR sensors that produce real-time depth maps at 320×180 at 60fps. I wanted to feed that depth data into the same WebGPU compute pipeline as the camera video — one shader that reads both the camera frame and the depth map, producing a depth-colored overlay blended with the live video.

The API I wanted:

const { currentFrame } = useGPUFrameProcessor(camera, {
  resources: {
    depth: GPUResource.cameraDepth(),
  },
  pipeline: (frame, { depth }) => {
    'worklet';
    frame.runShader(DEPTH_COLORMAP_WGSL, { inputs: { depth } });
  },
});

Declare a dynamic depth resource. Bind it as a shader input. The native side handles AVCaptureDepthDataOutput, frame synchronization, and per-frame texture updates. The user writes a WGSL shader and gets LiDAR data.

Getting there took longer than expected.

The Apple Log Color Space Saga

Before depth, I had to fix something from Part 2: Apple Log wasn't actually Apple Log.

"Why Doesn't It Look Washed Out?"

Apple Log footage should look flat and desaturated — that's the whole point of a log encoding. But our output looked identical to standard sRGB. Same contrast, same saturation. The Apple Log → Rec.709 LUT produced oversaturated garbage because it expected flat input.

I spent hours adjusting the YUV→RGB shader math. Full range vs video range. BT.709 vs BT.2020 matrix. Different normalization constants. Nothing changed the look. Then I added CVPixelBuffer attachment logging:

[DawnPipeline] CVPixelBuffer transfer=ITU_R_2100_HLG, matrix=ITU_R_2020, primaries=ITU_R_2020

HLG. Not Apple Log. iOS was delivering HLG-encoded frames despite activeColorSpace = .appleLog on the capture device. The camera was converting Apple Log to HLG before handing frames to AVCaptureVideoDataOutput.

Spying on Blackmagic Camera

I had a theory but no proof. Then I did something useful: opened Console.app, filtered for Blackmagic Camera's process, and looked at their AVCaptureSession configuration:

VC: <SRC:Wide back x422/3840x2160, ColorSpace:3, ...>

ColorSpace:3 — that's Apple Log. Our session showed ColorSpace:2 — HLG. Same device, same format, different color space. What were they doing differently?

Two things:

automaticallyConfiguresCaptureDeviceForWideColor = false on the session — this prevents iOS from overriding the color space
autoConfig: 0 in their session configuration

Adding one line fixed it:

session.automaticallyConfiguresCaptureDeviceForWideColor = false

Suddenly the footage was flat, washed out, and beautiful. The LUT worked perfectly. I saved a feedback memory: "Disable automaticallyConfiguresCaptureDeviceForWideColor for Apple Log; session overrides color space otherwise."

The Format Wars: 4:2:0 vs 4:2:2

But the color space wasn't the only issue. The camera's native format was x422 (10-bit 4:2:2 YUV) — half chroma subsampling horizontally, full vertically. We were requesting x420 (4:2:0 — half in both dimensions), forcing AVFoundation to convert. That conversion was likely applying the HLG transform.

Dawn's multi-planar support has separate feature flags per format:

DawnMultiPlanarFormats — enables 8-bit NV12 (R8BG8Biplanar420Unorm)
MultiPlanarFormatP010 — enables 10-bit 4:2:0 (R10X6BG10X6Biplanar420Unorm)
MultiPlanarFormatP210 — enables 10-bit 4:2:2 (R10X6BG10X6Biplanar422Unorm)

We had P010 enabled but not P210. Requesting the native 4:2:2 format required enabling P210 — and adjusting the UV plane dimensions in the shader (half-width, full-height instead of half-both).

Lesson I saved as permanent feedback: "Verify ALL required Dawn feature flags when adding new GPU format support; Dawn silently returns zeros for unsupported formats."

Upstream Progress

The multi-planar feature flags (P010, P210) and the getRecorder() API I needed for canvas overlays were both merged into react-native-skia upstream (PR #3753 and PR #3751). The 1×1 throwaway surface hack from Part 2 — creating a dummy offscreen surface just to extract the thread-local Recorder* — is now replaced with a clean ctx.getRecorder() call. Small wins that clean up the codebase.

Adding LiDAR Depth

With Apple Log solid, I moved to depth. The design was straightforward:

When GPUResource.cameraDepth() appears in resources, add AVCaptureDepthDataOutput to the session
Use a separate DepthDelegate that caches the latest depth frame
The video FrameDelegate grabs the cached depth and passes both to processFrame
Import the depth IOSurface as a R16Float texture via Dawn's SharedTextureMemory
Bind it as a dynamic per-frame input in the depth shader

Each step had its own set of problems.

Problem 1: The LiDAR Device

The LiDAR depth camera is a separate AVCaptureDevice — .builtInLiDARDepthCamera. It captures both video and depth, but its video output is 8-bit NV12 (420v) instead of the wide-angle camera's BGRA. The pipeline needed a new code path: NV12→RGB conversion with BT.709 matrix and video range expansion.

But the shader selection was tied to useDepth, not the actual pixel format. When the JS-side depth resource parsing had a timing issue, useDepth was false even though the LiDAR camera was delivering NV12. The BGRA passthrough shader ran on YUV data — bind group mismatch, silent black output.

The fix was adding a lidarYUV flag set in Swift based on the actual camera device, passed through the ObjC++ bridge to C++, independent of whether depth data is also being captured.

Problem 2: Dawn Doesn't Know About Depth

This was the big one. Everything worked: frames arrived, the depth CVPixelBuffer had a valid IOSurface, SharedTextureMemory imported it, CreateTexture with R16Float succeeded, BeginAccess succeeded. But textureLoad in the shader returned all zeros.

Sound familiar? Same pattern as Apple Log without the P010 feature. Dawn silently accepts the import but can't actually read the data.

I fetched Dawn's source code from dawn.googlesource.com and found GetFormatEquivalentToIOSurfaceFormat() in SharedTextureMemoryMTL.mm. It's a switch statement mapping CVPixelFormat codes to Dawn TextureFormats. 17 formats are mapped — every video format you'd expect: BGRA, NV12, P010, P210, etc.

kCVPixelFormatType_DepthFloat16 (hdep) is not in the list.

The depth IOSurface has format code 0x68646570. Dawn doesn't recognize it. It imports the IOSurface handle successfully (it's still a valid IOSurface), creates the texture object, and even succeeds at BeginAccess. But the underlying Metal texture can't be properly configured because Dawn doesn't know the pixel format — so textureLoad returns zeros.

The irony: Dawn already maps kCVPixelFormatType_OneComponent16Half → R16Float. The depth format DepthFloat16 has identical memory layout — single-channel, 16-bit half-float. Only the FourCC tag differs.

The CPU Upload Workaround

For now, I bypass SharedTextureMemory entirely for depth:

// CPU readback + WriteTexture (115KB at 320x180 — negligible)
CVPixelBufferLockBaseAddress(depthBuffer, kCVPixelBufferLock_ReadOnly);
const void* data = CVPixelBufferGetBaseAddress(depthBuffer);
size_t bpr = CVPixelBufferGetBytesPerRow(depthBuffer);

wgpu::TexelCopyTextureInfo dst{};
dst.texture = depthTexture;  // persistent, created once

wgpu::TexelCopyBufferLayout layout{};
layout.bytesPerRow = (uint32_t)bpr;
layout.rowsPerImage = (uint32_t)depthH;

device.GetQueue().WriteTexture(&dst, data, bpr * depthH, &layout, &extent);
CVPixelBufferUnlockBaseAddress(depthBuffer, kCVPixelBufferLock_ReadOnly);

320×180 × 2 bytes = 115KB per frame. At 60fps, that's 6.9MB/sec of CPU→GPU transfer. On a device that does 25GB/sec bandwidth, this is invisible. The texture is persistent (created once, reused every frame), so there's no allocation overhead either.

Zero-copy would be nicer. Which brings us to...

The Dawn Upstream Contribution

Four lines of code. That's all it takes:

case kCVPixelFormatType_DepthFloat16:
    return wgpu::TextureFormat::R16Float;
case kCVPixelFormatType_DepthFloat32:
    return wgpu::TextureFormat::R32Float;
case kCVPixelFormatType_DisparityFloat16:
    return wgpu::TextureFormat::R16Float;
case kCVPixelFormatType_DisparityFloat32:
    return wgpu::TextureFormat::R32Float;

Added to the switch in GetFormatEquivalentToIOSurfaceFormat(), right before the default case. Same file, same pattern as every other format mapping.

Dawn uses Chromium's Gerrit for code review instead of GitHub PRs. I filed a bug at issues.chromium.org under Blink>WebGPU, then submitted the CL.

The CL is at dawn-review.googlesource.com/c/dawn/+/297995. Four case statements. If accepted, every iOS app using Dawn can zero-copy import LiDAR depth data.

The Depth Colormap Shader

With depth data flowing (via CPU upload for now), the shader is satisfying:

@group(0) @binding(0) var inputTex: texture_2d<f32>;
@group(0) @binding(1) var outputTex: texture_storage_2d<rgba16float, write>;
@group(0) @binding(3) var depthTex: texture_2d<f32>;
@group(0) @binding(4) var depthSampler: sampler;

fn depthColormap(t: f32) -> vec3f {
  let r = clamp(2.0 * t - 0.5, 0.0, 1.0);
  let g = clamp(1.0 - 2.0 * abs(t - 0.5), 0.0, 1.0)
        + clamp(2.0 * t - 1.0, 0.0, 1.0);
  let b = clamp(1.0 - 2.0 * t, 0.0, 1.0);
  return vec3f(r, g, b);
}

@compute @workgroup_size(16, 16)
fn main(@builtin(global_invocation_id) id: vec3u) {
  let outDims = textureDimensions(outputTex);
  if (id.x >= outDims.x || id.y >= outDims.y) { return; }

  let color = textureLoad(inputTex, vec2i(id.xy), 0).rgb;

  // Rotate UV 90° CW — depth texture is landscape, output is portrait
  let rotU = (f32(id.y) + 0.5) / f32(outDims.y);
  let rotV = 1.0 - (f32(id.x) + 0.5) / f32(outDims.x);
  let depth = textureSampleLevel(depthTex, depthSampler, vec2f(rotU, rotV), 0.0).r;

  let t = clamp(depth / 5.0, 0.0, 1.0);  // 0-5m range
  let blended = mix(color, depthColormap(t), 0.6);

  textureStore(outputTex, vec2i(id.xy), vec4f(blended, 1.0));
}

The depth texture is 320×180 (landscape), but the output is 1080×1920 (portrait, after the rotation pass). The UV rotation maps portrait coordinates to landscape depth coordinates. textureSampleLevel with a linear sampler gives free bilinear interpolation, upsampling the 320×180 depth map smoothly to the 1080×1920 output.

The normalize distance (depth / 5.0) sets the colormap range — 5 meters maps to the full blue→green→yellow gradient. LiDAR range is ~5m on iPhone, so this captures the useful range. Closer objects are blue, farther objects yellow.

GPU-Side Portrait Rotation

One thing that helped across the board: moving the landscape→portrait rotation from Skia to the GPU.

Previously, the camera delivered landscape textures (1920×1080), the compute pipeline processed them in landscape, and Skia applied a 90° rotation transform when drawing to the portrait screen. This rotation was expensive — Skia had to do a rotated blit of a 4K F16 texture every frame, costing significant GPU time.

Now the first compute pass (passthrough or YUV→RGB) does the rotation:

// 90° CW: output(x, y) reads from input(y, inH - 1 - x)
let yDims = textureDimensions(yPlaneTex);
let srcCoord = vec2u(id.y, yDims.y - 1u - id.x);

The output texture is portrait-sized (1080×1920). Every downstream shader and Skia sees portrait coordinates. The Skia Canvas draws with an identity transform — no rotation, no scaling, just a straight blit. This matters at high resolutions where the rotated blit was the bottleneck.

It also fixed the onFrame canvas coordinate problem: previously, Skia draws happened in landscape coordinates and then got rotated, making positioning unintuitive. Now (0, 0) is top-left of what the user sees.

What I Learned

Dawn's format mapping is strict and silent

When Dawn doesn't recognize an IOSurface pixel format, it doesn't error — it imports successfully but reads zeros. We hit this three times: once with Apple Log (missing P010), once with 4:2:2 (missing P210), and once with LiDAR depth (missing DepthFloat16 mapping entirely). The symptoms are always the same: "everything succeeds but the output is black."

Check how other apps configure their sessions

Spying on Blackmagic Camera's AVCaptureSession via Console.app was the breakthrough for Apple Log. Their ColorSpace:3 vs our ColorSpace:2 immediately showed the misconfiguration. System-level logging is an underused debugging tool.

CPU upload is fine for small data

The instinct to zero-copy everything is strong, but 115KB at 60fps is nothing. The CPU upload for depth data is simpler, more debuggable, and just as fast as the zero-copy path would be. Save the optimization for when it matters — the 8MB camera frame, not the 115KB depth map.

Contributing upstream is surprisingly accessible

Dawn's Gerrit workflow is different from GitHub but not harder. CLA + clone + hook + push. For a four-line format mapping, the total effort was: file a bug (5 min), clone the repo (2 min), make the edit (1 min), push (1 min). The hardest part was finding GetFormatEquivalentToIOSurfaceFormat() in the source — once found, the fix was obvious.

The Full Pipeline

Here's what's running on an iPhone 16 Pro:

Camera (LiDAR, 1920×1080 NV12 420v)
  + Depth (320×180 DepthFloat16, CPU upload to R16Float)
  → Pass 0: NV12→RGB (BT.709, video range) + 90° CW rotation
  → RGBA16Float ping-pong (1080×1920 portrait)
  → Pass 1: Depth colormap (bilinear upsample, blue→green→yellow blend)
  → RGBA16Float output
  → Skia Graphite MakeImageFromTexture → SkImage → Canvas

Or with Apple Log:

Camera (Wide, 1920×1080 10-bit 4:2:2 YUV, Apple Log)
  → Pass 0: YUV→RGB (BT.2020, video range, no clamp) + 90° CW rotation
  → RGBA16Float ping-pong (1080×1920 portrait)
  → Pass 1: Apple Log → Rec.709 LUT (65³ 3D texture)
  → RGBA16Float output
  → Skia Graphite MakeImageFromTexture → SkImage → Canvas

Same pipeline architecture, different first-pass shader. The user writes WGSL, declares resources, and the native side handles format detection, IOSurface import, YUV conversion, and rotation.

What's Next

Pipeline/overlay/onFrame split — separating what goes into recordings vs. what's display-only
Recording — ProRes output with Apple Log preserved, overlays burned in
Dawn upstream — waiting on the CL review, hopefully zero-copy depth soon
More depth effects — focus peaking, background blur, depth-based segmentation

This is Part 3 of an ongoing series. Part 1 covers the initial spike. Part 2 covers Apple Log HDR and debugging. Code: react-native-webgpu-camera

From Flickering Histograms to 10-Bit HDR: Debugging a WebGPU Camera Pipeline in React Native

Kim Brandwijk — Mon, 16 Mar 2026 16:21:35 +0000

In the last post, I built a zero-copy GPU compute pipeline for live camera frames in React Native — IOSurface import, Dawn compute shaders, Skia Graphite rendering, 4K at 120fps. That post ended with buffer readback and histogram overlays working.

What I didn't cover was everything that broke between "it works" and "it actually works." The histogram flickered. The canvas overlay disappeared. A 33MB dead memcpy hiding in the camera delegate tanked performance by 5.5x. A confident theory about thread-local recorders turned out to be completely wrong.

This post covers the debugging journey from multi-pass rendering through to Apple Log — iOS 17's 10-bit log-encoded camera format, with LUT-based color grading running entirely on the GPU.

The 33MB/Frame Dead Memcpy

After multi-pass compute was working, I noticed something wrong: 22fps at 4K. The GPU profiler showed 2.88ms compute time — plenty of headroom at 120Hz. So where was the time going?

I added per-step timing instrumentation to the native pipeline:

auto t0 = std::chrono::high_resolution_clock::now();
// ... import IOSurface ...
auto t1 = std::chrono::high_resolution_clock::now();
// ... create bind groups ...
auto t2 = std::chrono::high_resolution_clock::now();
// ... dispatch compute ...

The instrumentation showed the problem immediately: the camera delegate's captureOutput:didOutputSampleBuffer:fromConnection: was copying the entire pixel buffer before passing it to the pipeline. At 4K BGRA, that's 3840 × 2160 × 4 = ~33MB of memcpy per frame. For no reason — the pipeline imports the IOSurface directly from the CVPixelBuffer. The copy was dead code left over from an earlier approach.

Removing one line restored 120fps. The timing metrics became a permanent part of the pipeline, exposed to JS via JSI:

[FPS] 120fps | lock=0.02ms import=0.31ms bind=0.08ms compute=2.88ms buf=0.01ms skImg=0.12ms

Lesson: Before debugging the GPU, profile the CPU. The most expensive operation in the pipeline was happening before the GPU was even involved.

The Histogram Flicker

With performance fixed, the next problem was visual: the histogram overlay flickered. Every few frames, it would vanish for one frame and reappear.

The setup: a compute shader runs a histogram pass, writing both to the output texture (passthrough) and to a storage buffer (bin counts). A Skia canvas draws the histogram bars directly onto the compute output texture via WrapBackendTexture. The composited result — video frame with histogram overlay — becomes the final SkImage.

The flicker happened because the compute pipeline and Skia canvas were sharing the same GPU texture. Here's the race:

Frame N:
  Camera thread: processFrame() dispatches compute → writes to texA
  Camera thread: Skia canvas draws histogram bars on texA
  Camera thread: MakeImageFromTexture(texA) → outputImage
  UI thread: reads outputImage, renders to screen ✓

Frame N+1:
  Camera thread: processFrame() dispatches compute → writes to texB
  UI thread: still rendering frame N's outputImage (backed by texA) ✓
  Camera thread: next processFrame() dispatches compute → writes to texA  ← BOOM
  UI thread: texA just got overwritten mid-render

The UI thread was reading a texture that the camera thread was simultaneously overwriting with the next frame's compute output. With ping-pong textures, the overwrite happens every other frame — hence the flicker, not a solid corruption.

Fix: Separate Canvas Texture + Split Mutex

The fix had two parts:

1. Dedicated canvas texture. Instead of drawing Skia overlays directly onto the compute output texture, copy the compute result to a separate canvasTex first, then draw overlays on canvasTex. The compute pipeline never touches canvasTex, so the UI thread can safely read it.

// Copy compute output to canvas texture (GPU-side, fast)
encoder.CopyTextureToTexture(&src, &dst, &copySize);

// Now draw Skia overlays on the stable copy
auto surface = SkSurfaces::WrapBackendTexture(recorder, canvasBackendTex, ...);
// ... draw histogram bars ...
surface->flush();

2. Split the mutex. The original implementation held a single mutex for the entire processFrame() — import, bind group creation, compute dispatch, buffer readback, SkImage creation. That's ~3ms of lock hold time, during which the UI thread blocks on every nextImage() call.

The fix: run all GPU work unlocked, then take the lock only to publish results:

void processFrame(CVPixelBufferRef pixelBuffer) {
  // All GPU work runs WITHOUT holding the lock
  auto ioSurface = CVPixelBufferGetIOSurface(pixelBuffer);
  // ... import, bind groups, compute dispatch, buffer copy ...
  // ... canvas draw, MakeImageFromTexture ...

  // Brief lock to swap published state (~0.05ms)
  {
    std::lock_guard<std::mutex> lock(_mutex);
    _impl->outputImage = newImage;
    _impl->generation++;
  }
}

Lock hold time went from ~3ms to ~0.05ms. The UI thread no longer stalls waiting for compute to finish.

Lesson: GPU pipeline debugging is hard because the symptoms (visual flicker) don't obviously point to the cause (shared texture + lock contention). Timing instrumentation and understanding the frame lifecycle are essential.

The Theory That Was Wrong

During the multi-pass rewrite, rendering broke completely — solid nothing on screen. The SkImage existed (non-null), had the right dimensions, but rendered as empty.

I had a confident theory: Skia Graphite uses thread-local Recorder objects for GPU command recording. processFrame() runs on the camera thread. The Skia <Canvas> renders on the Reanimated UI thread. If the SkImage was created on the camera thread's recorder, maybe the UI thread's recorder couldn't use it.

This led to an elaborate deferred-creation system:

// processFrame() — camera thread
_impl->imageDirty = true;  // mark for lazy recreation

// getOutputSkImage() — UI thread
if (_impl->imageDirty) {
  _impl->outputImage = ctx.MakeImageFromTexture(
    *finalTex, width, height, format);
  _impl->imageDirty = false;
}
return _impl->outputImage;

The theory sounded right. Thread-local recorders are a real Graphite concept. The code was clean. It compiled. It produced frames.

The frames were empty.

I spent time adding synchronization, trying different creation points, ensuring the texture was valid. Nothing worked. Then I looked at the single-pass version — the one that was working perfectly — and realized it created the SkImage directly in processFrame(), on the camera thread. The exact thing my theory said shouldn't work.

I reverted to the simple approach: MakeImageFromTexture() in processFrame(), return the cached image in getOutputSkImage(). Rendering worked immediately.

The thread-local recorder theory was either wrong or not the actual problem. I documented this in the architecture decisions log:

AD-1: SkImage creation in processFrame (camera thread)
We initially believed Skia Graphite's thread-local Recorder would prevent an SkImage created on the camera thread from rendering on the UI thread. The deferred approach produced frames but they rendered as nothing. Reverting to create the SkImage directly in processFrame() — matching the working single-pass implementation — fixed rendering immediately.

Lesson: Confident theories about GPU threading are still just theories. When debugging, always have a known-working reference to diff against. I'd made two changes simultaneously (deferred SkImage creation + RGBA view format override), couldn't tell which one caused the problem, and the one I was most confident about wasn't the fix.

Obtaining a Graphite Recorder (The 1×1 Hack)

For the canvas overlay feature, I needed a skgpu::graphite::Recorder* to call WrapBackendTexture. But DawnContext::getRecorder() is private in react-native-skia. getWGPUDevice() and getWGPUInstance() are public — the recorder just wasn't exposed.

The workaround: create a 1×1 throwaway offscreen surface and extract its recorder:

auto tempSurface = ctx.MakeOffscreen(1, 1);
auto* recorder = tempSurface->recorder();

This works because the recorder is a thread-local singleton — all surfaces on the same thread share the same instance. The 1×1 surface is created once during setup().

I filed PR #3751 to make getRecorder() public, consistent with the other Dawn accessors. Once merged, the hack can be replaced with ctx.getRecorder().

Depth Estimation: ML on the Same GPU

Before diving into Apple Log, I took a detour: running a depth estimation model on camera frames using the same Dawn GPU device.

Transformers.js (Hugging Face's JS ML library) supports WebGPU as an execution backend. Since Skia Graphite already provides navigator.gpu via its Dawn JSI bridge, the model can run on the same GPU context — no separate GPU initialization, no data copies between contexts.

The demo runs Depth Anything V2 Small on camera frames. The interesting part isn't the ML itself — it's the plumbing. Transformers.js expects browser APIs (Canvas, Blob, Image) that don't exist in React Native. The workaround: take the model's raw output (Float32Array of depth values), normalize to grayscale bytes, and construct an SkImage directly:

const rgba = new Uint8Array(pixelCount * 4);
for (let i = 0; i < pixelCount; i++) {
  const v = depthData[i];
  rgba[i * 4 + 0] = v;
  rgba[i * 4 + 1] = v;
  rgba[i * 4 + 2] = v;
  rgba[i * 4 + 3] = 255;
}
return Skia.Image.MakeImage(info, Skia.Data.fromBytes(rgba), width * 4);

This is the "don't do this in production" pixel copy approach from Part 1 — acceptable for a demo that updates a few times per second, but not for 120fps video. The real solution is ONNX Runtime's WebGPU execution provider, which accepts an external Dawn device via string-encoded pointers and keeps everything GPU-side. That's next.

Apple Log: 10-Bit HDR Camera Frames

Now the main event. Everything above was the prologue — fixing bugs, adding features, and building the infrastructure that makes Apple Log possible.

What is Apple Log?

Standard iPhone footage is "finished" by the ISP before it reaches your app — tone mapped, color graded, dynamic range compressed. Apple Log is the raw-ish alternative: a flat, log-encoded format with ~15 stops of dynamic range. Highlights that would be clipped are preserved. Shadows that would be crushed are recoverable. The footage looks awful until you apply a LUT (Look-Up Table) to map it to a displayable color space.

Professional video apps have supported this since iOS 17. I wanted it in the WebGPU compute pipeline — capture in Apple Log, convert YUV→RGB on the GPU, apply a .cube LUT, render through Skia Graphite. All zero-copy.

The Problem: 10-Bit Bi-Planar YUV

Standard frames come as kCVPixelFormatType_32BGRA — four bytes per pixel, one plane, simple. Apple Log frames come as kCVPixelFormatType_420YpCbCr10BiPlanarVideoRange:

420: Chroma subsampled — UV is half resolution in each dimension
YpCbCr: Luminance and chrominance are separate
10Bi: 10 bits per component in 16-bit containers
Planar: Y and UV in separate memory planes
VideoRange: Values don't span full 0–1023 (Y: 64–940, UV: 64–960)

Dawn Multi-Planar Textures

Dawn supports this natively, but needs two device features enabled:

if (adapter.HasFeature(wgpu::FeatureName::DawnMultiPlanarFormats)) {
  features.push_back(wgpu::FeatureName::DawnMultiPlanarFormats);
}
if (adapter.HasFeature(wgpu::FeatureName::MultiPlanarFormatExtendedUsages)) {
  features.push_back(wgpu::FeatureName::MultiPlanarFormatExtendedUsages);
}

DawnMultiPlanarFormats enables the bi-planar texture format. MultiPlanarFormatExtendedUsages allows using it as a TextureBinding — without this, you can import the texture but can't sample it in a shader.

Importing the IOSurface

The IOSurface is imported once as a multi-planar texture, then split into per-plane views:

inputTexDesc.format = wgpu::TextureFormat::R10X6BG10X6Biplanar420Unorm;
inputTexture = sharedMemory.CreateTexture(&inputTexDesc);

// Y plane: full resolution, single channel
wgpu::TextureViewDescriptor yViewDesc{};
yViewDesc.aspect = wgpu::TextureAspect::Plane0Only;
yViewDesc.format = wgpu::TextureFormat::R16Unorm;
yPlaneView = inputTexture.CreateView(&yViewDesc);

// UV plane: half resolution, two channels
wgpu::TextureViewDescriptor uvViewDesc{};
uvViewDesc.aspect = wgpu::TextureAspect::Plane1Only;
uvViewDesc.format = wgpu::TextureFormat::RG16Unorm;
uvPlaneView = inputTexture.CreateView(&uvViewDesc);

I initially spent time looking for a plane field on SharedTextureMemoryIOSurfaceDescriptor for per-plane import. It doesn't exist — the API handles planes at the view level, not the import level. One import, two aspect views. Dawn handles the half-resolution UV plane implicitly.

Finding all of this required reading Dawn's test files and WebGPU spec extension proposals. There's no "how to import a YUV IOSurface" tutorial anywhere.

The YUV→RGB Shader

The pipeline auto-inserts this as pass 0 when appleLog is true:

@group(0) @binding(0) var yPlaneTex: texture_2d<f32>;
@group(0) @binding(1) var uvPlaneTex: texture_2d<f32>;
@group(0) @binding(2) var outputTex: texture_storage_2d<rgba16float, write>;

@compute @workgroup_size(16, 16)
fn main(@builtin(global_invocation_id) id: vec3u) {
  let dims = textureDimensions(yPlaneTex);
  if (id.x >= dims.x || id.y >= dims.y) { return; }

  let yRaw = textureLoad(yPlaneTex, vec2i(id.xy), 0).r;
  let uvRaw = textureLoad(uvPlaneTex, vec2i(id.xy / 2u), 0).rg;

  // Video range expansion (10-bit: 64/1023, 940/1023, 960/1023)
  let y = (yRaw - 0.06256) / (0.91887 - 0.06256);
  let cb = (uvRaw.r - 0.06256) / (0.93842 - 0.06256) - 0.5;
  let cr = (uvRaw.g - 0.06256) / (0.93842 - 0.06256) - 0.5;

  // BT.2020 YCbCr → RGB
  let r = y + 1.4746 * cr;
  let g = y - 0.16455 * cb - 0.57135 * cr;
  let b = y + 1.8814 * cb;

  // Do NOT clamp — values outside [0,1] are valid HDR data
  textureStore(outputTex, vec2i(id.xy), vec4f(r, g, b, 1.0));
}

Three things matter: video range expansion (Apple Log uses 64–940, not 0–1023), BT.2020 matrix (not BT.709 — subtle color difference), and no clamp (the whole point of Apple Log is values > 1.0 for super-bright highlights).

RGBA16Float Everywhere

The original pipeline used RGBA8Unorm ping-pong textures — 8 bits per channel, [0, 1] range. Apple Log needs RGBA16Float to preserve HDR values > 1.0. But WebGPU requires the shader's texture_storage_2d<format> to exactly match the texture format. No implicit conversion.

Rather than maintaining two shader variants for every user shader, I switched the entire pipeline to RGBA16Float — both SDR and Apple Log modes. It costs 2x texture memory for SDR (~66MB vs ~33MB at 4K), but the GPU has 6-8 GB and the complexity savings are worth it. One format everywhere means every shader works in both modes without changes.

This decision touched every texture allocation, every MakeImageFromTexture call, every built-in shader, and every example shader. Lots of files, one conceptual change.

The LUT Shader

With Apple Log RGB in RGBA16Float, a .cube LUT maps it to a displayable color space. The shader uses the input RGB as 3D texture coordinates:

@group(0) @binding(3) var lutTex: texture_3d<f32>;
@group(0) @binding(4) var lutSampler: sampler;

let lutCoord = clamp(color.rgb, vec3f(0.0), vec3f(1.0));
let lutColor = textureSampleLevel(lutTex, lutSampler, lutCoord, 0.0);

textureSampleLevel, not textureSample — this tripped me up. textureSample requires implicit derivatives (fragment shader only). textureSampleLevel takes an explicit mip level and works in compute.

The clamp is intentional here: the LUT only covers [0, 1]. Values outside would sample garbage. The LUT's edge entries already handle extreme highlights and shadows.

The Resource System

Getting LUT data to the shader required a new abstraction — typed GPU resources uploaded once at setup and bound to shader passes:

const { currentFrame } = useGPUFrameProcessor(camera, {
  resources: {
    lut: GPUResource.texture3D(lutData, {
      width: 33, height: 33, depth: 33,
      format: 'rgba32float',
    }),
  },
  pipeline: (frame, res) => {
    'worklet';
    frame.runShader(LUT_WGSL, { inputs: { lut: res.lut } });
  },
});

The capture proxy assigns handle IDs during pipeline setup and records which resources each pass needs. The native side uploads data once and creates bind group entries. The same system handles 3D textures, 2D textures, and storage buffers — general-purpose GPU resource input.

Two Ways to Use the Output

I mentioned in Part 1 that the pipeline has two consumption paths for buffer data. The same principle applies to the entire processed frame, and it matters more with Apple Log because you're making a choice about what gets recorded.

Burn-in via onFrame: Drawing directly onto the GPU texture. Anything drawn here — LUT-graded video, histogram overlays, waveform monitors, focus peaking — becomes part of the frame. It shows up in recordings, screenshots, and exports.

const { currentFrame } = useGPUFrameProcessor(camera, {
  resources: { lut: lutResource },
  pipeline: (frame, res) => {
    'worklet';
    frame.runShader(LUT_WGSL, { inputs: { lut: res.lut } });
    return {};
  },
  onFrame: (frame) => {
    'worklet';
    // Draw exposure guide on the graded frame — burned into video
    frame.canvas.drawRect(Skia.XYWHRect(x, y, w, h), paint);
  },
});

Display-only via React: Buffer data and the current frame are exposed as Reanimated shared values. You can draw overlays in React that appear on screen but aren't part of the recorded video:

const { currentFrame, buffers } = useGPUFrameProcessor(camera, {
  resources: { lut: lutResource },
  pipeline: (frame, res) => {
    'worklet';
    frame.runShader(LUT_WGSL, { inputs: { lut: res.lut } });
    const hist = frame.runShader(HISTOGRAM_WGSL, { output: Uint32Array, count: 256 });
    return { hist };
  },
});

// Histogram overlay in React — visible on screen, NOT in recorded video
const histPicture = useDerivedValue(() => {
  const hist = buffers.value.hist as Uint32Array | null;
  if (!hist) return emptyPicture;
  return createPicture((canvas) => { /* draw bars */ });
});

return (
  <Canvas>
    <SkImage image={currentFrame} />
    <Picture picture={histPicture} />  {/* display-only overlay */}
  </Canvas>
);

This distinction matters for professional video work. A colorist wants to see scopes and guides while grading, but doesn't want them burned into the export. A filmmaker wants focus peaking during capture but clean footage in the final file. Same pipeline, same data, two output paths — the user chooses per element.

The canvas flicker fix from earlier (separate canvasTex + split mutex) is what makes the burn-in path reliable. Without it, onFrame draws would intermittently vanish when the compute pipeline overwrote the shared texture.

The Full Apple Log Pipeline

Camera (10-bit YUV, Apple Log)
  → IOSurface import as R10X6BG10X6Biplanar420Unorm
  → Y plane view (R16Unorm) + UV plane view (RG16Unorm)
  → Pass 0: YUV→RGB (BT.2020, video range, no clamp)
  → RGBA16Float ping-pong texture
  → Pass 1: LUT application (3D texture sample)
  → RGBA16Float output
  → Skia Graphite MakeImageFromTexture
  → SkImage → Canvas

Every arrow is a GPU-side operation or a metadata bind. Zero pixel copies from camera to display. From JS, enabling Apple Log is one line:

const camera = useCamera({
  device: 'back',
  colorSpace: 'appleLog',
});

What I Learned

The debugging stories have a common thread: the bug is never where you think it is.

The 33MB memcpy wasn't a GPU problem. The histogram flicker wasn't a shader problem. The empty SkImage wasn't a threading problem. In every case, the first theory was wrong, and the fix was simpler than the theory.

A few specific takeaways:

Profile the CPU before debugging the GPU. GPU work was fast. The pipeline was slow because of a dead memcpy in the camera delegate.

Never make two changes at once in a GPU pipeline. GPU errors are silent — no exceptions, just wrong pixels. When I changed SkImage creation timing and added an RGBA view format override simultaneously, I couldn't tell which one broke rendering. Isolate variables.

Keep a working reference. The single-pass pipeline that worked perfectly was the key to debugging the multi-pass rewrite. Every time something broke, I could diff against it.

Timing instrumentation pays for itself immediately. Adding per-step microsecond timing to the native pipeline (lock → import → bind → compute → buffers → makeImage) found the 33MB memcpy on the first run and has been useful for every optimization since.

What's Next

Performance profiling with Apple Log — how much does YUV→RGB + LUT add to GPU time?
ONNX Runtime WebGPU EP — ML inference on the same Dawn device, using Apple Log's wider dynamic range for better feature extraction
Android — AHardwareBuffer + SharedTextureMemory, same architecture, different platform primitives
Recording — render the LUT-graded output to an AVAssetWriter surface

The pipeline code is at react-native-webgpu-camera. Still spike phase — the API will change — but the architecture is solid.

This is Part 2 of an ongoing series. Part 1 covers the initial spike: zero-copy IOSurface import, multi-pass compute shaders, buffer readback, and the Skia Graphite setup gauntlet.

Zero-Copy GPU Compute on Camera Frames in React Native — What Actually Worked

Kim Brandwijk — Sat, 14 Mar 2026 23:15:05 +0000

I'm building a React Native camera library that runs WebGPU compute shaders on live camera frames. Not a tech demo — a real pipeline that needs to sustain 60fps+ on a phone without melting it.

This is the story of the spike that validated whether it's even possible. I hit 13 distinct build and integration failures along the way. Here's what I tried, what broke, and the architecture that actually works.

The Problem

If you've done real-time camera processing in React Native, you've probably used VisionCamera's frame processors. They work, but there's a fundamental bottleneck: pixel data has to cross the JS bridge. At 1080p BGRA, that's about 8MB per frame. At 30fps, you're copying 240MB/sec between native and JavaScript — and every byte has to be garbage collected on the other side.

I wanted something different: GPU compute shaders running directly on camera frames, with the results rendered by Skia — all without the pixels ever leaving the GPU. The stack: WebGPU (via Dawn) for compute, Skia Graphite for rendering, and a native pipeline that keeps everything on the metal.

The first question was whether React Native could even support this. I set out to find out.

The Naive Attempts

Attempt 1: Just Copy the Pixels

The most obvious approach: grab camera frame bytes in the native camera delegate, ship them to JS, create a Skia image.

// Don't do this
const pixels = WebGPUCameraModule.getCurrentFramePixels();
const data = Skia.Data.fromBytes(pixels); // 8MB allocation
const image = Skia.Image.MakeImage(info, data, bytesPerRow);

This works for about three seconds. Then the app crashes from memory pressure. Creating 8MB of Skia.Data thirty times a second overwhelms the garbage collector. Even with explicit .dispose() on previous images and a 30fps cap, it's too much. The GC can't keep up with the allocation rate.

Lesson: Never copy raw pixel data to JS for real-time video. If you're allocating megabytes per frame in JavaScript, you've already lost.

Attempt 2: WebGPU from JavaScript

Okay, so I need to keep pixels on the GPU. Skia Graphite bundles Dawn (Google's WebGPU implementation) and exposes navigator.gpu to JavaScript. You can create textures, compile shaders, dispatch compute — all from JS. The plan: use device.queue.writeTexture() to upload camera pixels to a GPU texture, run a compute shader, render the result.

I got all the way to a working compute pipeline. Shader compiled, bind groups created, compute dispatched. The Canvas even showed something — a solid green rectangle. The Apple GPU profiler overlay appeared for the first time, confirming actual GPU work. Progress.

Except the green was the shader's output for all-zero input. writeTexture() silently returned zeros.

I dug into the react-native-skia source. The Dawn JS bindings are partially implemented:

What works from JS:

navigator.gpu.requestAdapter() / requestDevice()
device.createShaderModule(), createComputePipeline(), createTexture()
encoder.beginComputePass() / dispatchWorkgroups() / submit()
copyBufferToBuffer() — verified with a [10,20,30,40] roundtrip

What doesn't work from JS:

device.queue.writeTexture() — implemented in C++ but returns zeros through JSI. Likely an ArrayBuffer bridging issue.
device.queue.copyBufferToTexture() — completely stubbed out (commented code, never implemented)
ImportSharedTextureMemory() — C++ only, not exposed to JS

Every path to getting pixel data into a texture from JavaScript was broken. I could create GPU resources and dispatch compute, but couldn't feed them data.

After changing the shader to output pink instead of green — and seeing pink — I had my confirmation: the compute pipeline worked perfectly. The input was just empty.

Lesson: If you see a solid color from a compute shader, your input texture is probably empty. Verify data made it to the GPU before debugging the shader.

The Architecture That Works

If JS can't write to textures, the native side has to do it. And if I'm already on the native side, I might as well skip the copy entirely.

iOS camera frames come as CVPixelBuffers backed by IOSurface — which is already GPU memory. Dawn's SharedTextureMemory API can import an IOSurface directly as a GPU texture, zero copies. The pipeline:

Camera (AVCaptureSession)
  → CVPixelBuffer (backed by IOSurface — already on GPU)
  → Dawn SharedTextureMemory.ImportSharedTextureMemory(ioSurface)
  → wgpu::Texture (input)
  → Compute Shader (Sobel edge detection)
  → wgpu::Texture (output)
  → Skia Graphite MakeImageFromTexture()
  → SkImage (texture-backed, no pixel copy)
  → Skia Canvas

Every arrow is either a GPU-side operation or a metadata bind. No pixel copies anywhere.

The native C++ pipeline handles the hot path — IOSurface import, compute dispatch, output texture management. The JS side doesn't poll or copy anything. Instead, it uses Reanimated's useFrameCallback — a worklet that runs on the UI thread every display frame — to grab the latest compute output.

The trick is that worklets run on Reanimated's UI runtime, which has a separate globalThis from the JS runtime. You can't just install a global function and call it from a worklet. Instead, you create a JSI host object on the JS thread and pass it into the worklet closure — Reanimated shares host objects across runtimes:

const stream = useSharedValue<CameraStream | null>(null);
const currentFrame = useSharedValue<SkImage | null>(null);

// Create the stream host object after pipeline setup
stream.value = globalThis.__webgpuCamera_createStream();

// useFrameCallback runs on UI thread — stream is shared across runtimes
useFrameCallback(() => {
  'worklet';
  const s = stream.value;
  if (!s) return;
  const img = s.nextImage();
  if (img) {
    currentFrame.value?.dispose();
    currentFrame.value = img;
  }
});

__webgpuCamera_createStream() returns a CameraStreamHostObject — a plain jsi::HostObject with a nextImage() method that returns a JsiSkImage. The shared value is passed directly to the Skia Canvas, which auto-redraws when the value changes. No React re-renders, no setState, no JS thread involvement at all.

This follows the exact pattern that react-native-skia uses for video playback (useVideo): a JSI host object with a nextImage() method, called from a UI thread worklet, feeding a shared value that drives the Canvas.

The key insight is what crosses the JS boundary: not pixels, not even texture data — just an opaque handle. The heavy work stays on the GPU, and the render loop stays on the UI thread.

Thread Safety

The native pipeline runs on the camera delegate thread. The UI thread reads the latest output via useFrameCallback. The sk_sp<SkImage> output is shared between them, protected by a mutex.

There's a subtler problem: the CameraStreamHostObject holds a raw pointer to the C++ pipeline. If the user stops the camera and the pipeline gets destroyed, the host object has a dangling pointer. I solve this with a shared liveness flag:

// Pipeline constructor
_alive = std::make_shared<std::atomic<bool>>(true);

// Pipeline destructor
_alive->store(false);

// JSI lambda captures the shared_ptr, not the raw pipeline pointer
[pipeline, alive](jsi::Runtime &rt, ...) -> jsi::Value {
  if (!alive->load()) return jsi::Value::null();
  // safe to use pipeline
}

The shared_ptr<atomic<bool>> is reference-counted and outlives both the pipeline and the lambda. Cheap to check, safe against destruction races.

The Skia Graphite Setup Gauntlet

Getting the compute pipeline right was the fun part. Getting Skia Graphite to compile was the other part.

@shopify/react-native-skia supports a Graphite backend (SK_GRAPHITE=1) that uses Dawn for GPU rendering. It also bundles Dawn's WebGPU implementation and exposes navigator.gpu. This is what makes the whole architecture possible — one shared GPU context for both Skia rendering and WebGPU compute.

Here's what "add dependency, build" actually looked like.

Header clashes (Issues 1-3)

I started with both react-native-wgpu and @shopify/react-native-skia installed. Immediate header conflict — Skia Graphite bundles its own Dawn, and react-native-wgpu has its own modified JSI headers. jsi2/EnumMapper.h file not found.

Fix: Remove react-native-wgpu. Skia Graphite is self-contained.

But now the Dawn headers that Skia's C++ code includes (webgpu/webgpu_cpp.h) were missing. Turns out they're shipped in a separate npm package (react-native-skia-graphite-headers) that you have to discover by reading the podspec source. And it also ships Skia private headers (src/gpu/graphite/ContextOptionsPriv.h) that are needed for compilation.

I wrote an Expo config plugin to copy both dawn/ and skia/ headers into the right location during prebuild.

Version mismatch (Issue 4)

Using "*" (latest) versions for the three react-native-skia-graphite-* npm packages. Headers came from one version, iOS binaries from another. Result: Undefined symbols: SkAndroidCodec::MakeFromData — an Android symbol referenced in an iOS build. Classic version skew.

I switched to a git submodule approach: own the Skia source, guarantee header/binary consistency.

The size problem (Issues 5-7)

The submodule brought 2.4 GB into the project. EAS Build (Expo's cloud build service) reported 1.1 GB upload.

Breakdown: Android libs (1.2 GB), macOS libs (410 MB), example apps (110 MB), Skia externals (175 MB). None needed for an iOS build.

Created .easignore to strip it down. Immediately broke codegen — React Native needs src/specs/ for Turbo Module bindings, and I'd excluded all of src/. Then broke pod install — the podspec's prepare_command checks if both iOS and macOS xcframeworks exist before skipping the npm download step. Exclude macOS libs → it tries to require.resolve('react-native-skia-graphite-apple-ios') → which I don't have.

Every exclusion had a hidden dependency.

The silent bug (Issue 8)

Ran bun run install:skia-graphite. Script reported success: "Dawn/WebGPU headers copied." Headers directory was empty.

The install script downloads a tar.gz and looks for dawn/include at the archive root. The archive actually nests it three levels deep at packages/skia/cpp/dawn/include/. The path mismatch means the copy silently finds zero files and reports success.

Manual fix: tar xzf headers.tar.gz --strip-components=3.

Podspec path resolution (Issues 12-13)

My native C++ pipeline needs headers from Skia's source tree. The podspec computes the path with Ruby's File.expand_path:

skia_pkg_root = File.expand_path('../../../../../../...', __dir__)

I counted wrong. Six levels of .. instead of five. Off by one directory — landed at the parent of my repo root instead of the repo root itself.

Cost: Two wasted EAS builds. Each takes ~15 minutes and fails near the very end of compilation.

Then, even with the right path, HEADER_SEARCH_PATHS didn't work. The recursive glob /** was outside the quotes:

# Broken — shell sees unquoted glob
"\"#{root}/cpp\"/**"

# Working — Xcode recursive search
"\"#{root}/cpp/\"/**"

One character difference. Two more builds to discover.

Lesson: Always verify File.expand_path output before committing to a cloud build:

ruby -e "d = '/path/to/ios'; puts File.expand_path('../../../../../target', d)"

Multi-Pass: Chaining Shaders

Once the single-pass pipeline worked, the next question was obvious: can we chain multiple shaders? Run Sobel edge detection first, then feed its output into a color-mapping pass, all in one frame.

The architecture already had the answer. The single-pass pipeline used a ping-pong texture pair (texA and texB) — but only used one bounce. Multi-pass generalizes it: pass 0 reads the camera input and writes to texA, pass 1 reads texA and writes to texB, pass 2 reads texB and writes to texA, and so on. The final output alternates based on the number of passes.

From the JS side, the API is just an array of shaders:

const { currentFrame } = useGPUFrameProcessor(camera, (frame) => {
  'worklet';
  frame.runShader(SOBEL_WGSL);        // pass 0: edge detection
  frame.runShader(SOBEL_COLOR_WGSL);  // pass 1: colorize edges
});

The hook captures the shader chain at setup time using a proxy object — the 'worklet' directive means the function body is declarative, not imperative. The proxy records which shaders were called and in what order, then sends the chain to the native pipeline as a single setup call. No shaders execute during capture.

The Multi-Pass Regression

Getting multi-pass to render was harder than getting it to compute. The first build produced a solid red screen — the pipeline was running (GPU profiler confirmed work), but nothing visible.

I'd made two changes simultaneously during the multi-pass rewrite:

Deferred SkImage creation — Based on a theory that Skia's Graphite recorder was thread-local, I added a dirty flag to defer MakeImageFromTexture from the camera thread to the UI thread. The theory was wrong. The working single-pass version had always created SkImage on the camera thread, and it rendered fine.
RGBA view format override — The camera provides BGRA textures. I added an explicit RGBA texture view with viewFormats on the BGRA input, thinking the compute shader needed RGBA. It didn't — the default BGRA view worked for texture sampling.

The fix was reverting both changes to match the working single-pass pattern. But I'd made both changes at once, so I couldn't tell which one caused the red screen. This violated a basic principle: change one thing at a time when debugging GPU pipelines. GPU errors are silent — no exceptions, no crashes, just wrong pixels. You need to isolate variables.

Lesson: Dawn doesn't throw when you create a texture view with a mismatched format. It just renders garbage. If your output is a solid color after a pipeline change, diff against the last working version and revert one change at a time.

Buffer Readback: Getting Data Out of the GPU

Shaders that produce textures are useful for visual effects. But many real use cases need data — histograms, feature detection, classification scores. This requires reading GPU buffer contents back to JavaScript.

The pipeline supports buffer outputs alongside texture outputs. A shader can declare a storage buffer binding, write to it with atomics, and the pipeline copies the result to a staging buffer for CPU readback:

const { currentFrame, buffers } = useGPUFrameProcessor(camera, (frame) => {
  'worklet';
  frame.runShader(HISTOGRAM_WGSL, { output: Uint32Array, count: 256 });
});

The WGSL shader computes a 256-bin luminance histogram using atomic operations, while simultaneously passing through the camera frame to the output texture:

@group(0) @binding(0) var inputTex: texture_2d<f32>;
@group(0) @binding(1) var outputTex: texture_storage_2d<rgba8unorm, write>;
@group(0) @binding(2) var<storage, read_write> histogram: array<atomic<u32>, 256>;

@compute @workgroup_size(16, 16)
fn main(@builtin(global_invocation_id) gid: vec3u) {
  let color = textureLoad(inputTex, vec2i(gid.xy), 0);
  textureStore(outputTex, gid.xy, color);  // passthrough

  let lum = dot(color.rgb, vec3f(0.2126, 0.7152, 0.0722));
  let bin = min(u32(lum * 256.0), 255u);
  atomicAdd(&histogram[bin], 1u);
}

The native side uses double-buffered staging buffers. While frame N's compute shader writes to GPU buffer A, the CPU reads frame N-1's result from staging buffer B. MapAsync fires a callback when the data is ready, and readBuffer() returns the most recently completed map. The data is always 1-2 frames stale — invisible at 120fps.

Two Ways to Use the Data

Buffer readback creates an interesting design question: where do you consume the data?

The onFrame callback runs inside the pipeline, drawing directly onto the GPU texture. Anything drawn here is burned into the frame — it shows up in recordings, screenshots, and the live preview:

const { currentFrame } = useGPUFrameProcessor(camera, {
  pipeline: (frame) => {
    'worklet';
    const hist = frame.runShader(HISTOGRAM_WGSL, { output: Uint32Array, count: 256 });
    return { hist };
  },
  onFrame: (frame, { hist }) => {
    'worklet';
    if (!hist) return;
    // Draw histogram bars on the GPU texture — becomes part of the video
    for (let i = 0; i < 256; i++) {
      const barH = (hist[i] / maxVal) * 200;
      frame.canvas.drawRect(Skia.XYWHRect(x + i * 2, y + 200 - barH, 2, barH), paint);
    }
  },
});

But buffers is also returned as a Reanimated shared value. You can consume it in React for UI-only overlays that don't burn into the video:

const { currentFrame, buffers } = useGPUFrameProcessor(camera, (frame) => {
  'worklet';
  frame.runShader(HISTOGRAM_WGSL, { output: Uint32Array, count: 256 });
});

// Draw histogram in React — UI overlay only, not in the recorded video
const histPicture = useDerivedValue(() => {
  const hist = buffers.value.__buf_0 as Uint32Array | null;
  if (!hist) return emptyPicture;
  return createPicture((canvas) => { /* draw histogram bars */ });
});

return (
  <Canvas>
    <SkImage image={currentFrame} />
    <Picture picture={histPicture} />  {/* overlay — not recorded */}
  </Canvas>
);

Same data, two consumption points. Recording a tutorial with bounding boxes? Use onFrame. Live preview with debug overlays? Use buffers in React. The user gets both options for free.

ML Inference on the GPU: The ONNX Runtime Spike

With compute shaders and buffer readback working, the next frontier is ML model inference. Running a depth estimation or object detection model on the same Dawn GPU device that processes camera frames — no CPU roundtrip, no data copies between GPU contexts.

ONNX Runtime already has a WebGPU execution provider that accepts an external Dawn device. The C++ API takes string-encoded pointers for WGPUDevice, WGPUInstance, and DawnProcTable via ConfigOptions:

std::unordered_map<std::string, std::string> options;
options["webgpuDevice"] = std::to_string(reinterpret_cast<size_t>(dawnDevice));
options["webgpuInstance"] = std::to_string(reinterpret_cast<size_t>(dawnInstance));
options["dawnProcTable"] = std::to_string(reinterpret_cast<size_t>(&dawnProcs));
sessionOptions.AppendExecutionProvider("WebGPU", options);

This means we can pass Skia Graphite's Dawn device directly to ONNX Runtime — same GPU context, zero-copy tensor input from compute shader output buffers.

The React Native ONNX package (onnxruntime-react-native) uses native C++ via JSI and supports CPU, XNNPACK, CoreML, and NNAPI execution providers. It doesn't support WebGPU — yet. The spike adds "webgpu" to the supported backends and wires the device/instance/proc-table through the JSI layer. The model runs as a C++ execution provider on the same GPU — one GPU context, one set of resources, no synchronization overhead.

The Dawn Build Problem

Building ONNX Runtime with --use_webgpu --use_external_dawn sounds like it should use an existing Dawn installation. It doesn't. "External Dawn" means "Dawn is not bundled in ONNX Runtime's source tree" — it still fetches and compiles Dawn from source during the build. This adds 30+ minutes to the build for something we already have compiled inside Skia Graphite.

The right fix is a --dawn_home flag that points at prebuilt Dawn headers and libraries. That doesn't exist yet — but it would be a nice upstream contribution alongside the React Native WebGPU EP itself.

What I Learned

The spike validated the core question: yes, you can run WebGPU compute shaders on live camera frames in React Native, at real-time frame rates, with zero pixel copies. And then it kept going — multi-pass shader chains, GPU buffer readback, and the beginnings of ML inference on the same device.

The architecture works: IOSurface import → Dawn compute → Skia Graphite rendering. Everything stays on the GPU. JavaScript gets opaque handles, not pixels.

The cost of getting here was disproportionate to the complexity of the actual pipeline code. The C++ compute pipeline is ~400 lines. The build system integration was 13+ issues across multiple days.

What's actually running

Native C++ compute pipeline imports camera frames via IOSurface (zero-copy)
Multi-pass WGSL shader chains — edge detection, colorization, histogram — on the shared Dawn device
GPU buffer readback with double-buffered staging for compute data (histograms, future: bounding boxes)
Output wraps as SkImage via Skia Graphite's texture interop
useGPUFrameProcessor hook with two consumption paths: onFrame for burn-in, buffers for UI overlays
UI thread worklet grabs the latest SkImage via JSI and feeds it to Skia Canvas — no JS thread, no React re-renders
All on a physical iPhone 16 Pro, via EAS Build

The numbers

From Apple's GPU profiler overlay on an iPhone 16 Pro:

Metric	Single-pass	Multi-pass (2 shaders)
GPU frame time	2.88ms	4ms
Display FPS	120fps	120fps
Camera FPS	120fps	120fps
Resolution	3840x2160 (4K)	3840x2160 (4K)
Sustained FPS	120fps, 0% drops	120fps, 0% drops
Thermal	nominal	nominal

Even with two compute passes, the GPU has massive headroom — 4ms of work in an 8.3ms frame budget at 120Hz.

Getting the render loop right took three iterations. The first version used setTimeout — 20fps, even though the GPU overlay showed 40fps. Switching to requestAnimationFrame doubled throughput to 30fps. But each frame still called setState to update the Skia Canvas, which meant a full React re-render per camera frame. At 4K this bottleneck was brutal — 21.7fps even though the GPU overlay showed 41fps.

The final version eliminates JavaScript from the render loop entirely. A CameraStreamHostObject (JSI host object) is created on the JS thread and stored in a Reanimated shared value. useFrameCallback runs on the UI thread, calls stream.nextImage() on the shared host object, and writes the result to another shared value that drives the Skia Canvas. No setState, no React reconciliation, no JS thread involvement at all. This is the same pattern react-native-skia uses for video playback (useVideo), and it's the one that unlocked 120fps.

What's next

ONNX Runtime WebGPU EP for React Native — run ML models on the same Dawn device, PR back upstream
frame.runModel() API — async side-channel that produces buffer data, not textures
Android support — AHardwareBuffer is the IOSurface equivalent, same SharedTextureMemory API
Recording pipeline — render the composited output to an AVAssetWriter surface
Upstream contributions — several issues filed, a few more to file

The bigger picture

React Native's GPU story is still early. The pieces exist — Dawn, Skia Graphite, JSI — but they weren't designed to work together for this use case. Half the WebGPU JS bindings are stubbed out. The install tooling has silent bugs. Header management is a puzzle.

But the foundation is real. Dawn's SharedTextureMemory is the key primitive — it lets you import platform GPU resources (IOSurface, AHardwareBuffer) directly as WebGPU textures. Skia Graphite shares the Dawn device, so compute output becomes Skia input without a copy. And JSI lets you hand GPU resource handles to JavaScript without serialization.

The fact that ONNX Runtime already accepts an external Dawn device via string-encoded pointers is a signal that this convergence was anticipated. One shared GPU context for rendering, compute, and ML inference — that's the end state. We're just wiring it together.

If you're building something that needs real-time GPU compute in React Native — camera effects, AR overlays, video processing, on-device ML — this path works. It's just not paved yet.

This is an ongoing project. Follow along or check out the code: react-native-webgpu-camera

From 10,000 to 18: Drastically Improving Luau Bytecode Decompilation Quality

Kim Brandwijk — Mon, 26 Jan 2026 14:58:22 +0000

A deep dive into the techniques that reduced unnamed temporaries by 98.8%

Introduction

When reverse-engineering compiled Luau bytecode (the VM used by Roblox and various game engines), decompilers face a fundamental challenge: the compiler has optimized away much of the structure that made the original code readable. Variable names become register numbers, control flow becomes jumps, and elegant expressions become sequences of stack operations.

I spent several weeks improving medal, an open-source Luau decompiler, contributing 7 pull requests that transformed output like this:

-- BEFORE: Unreadable mess of temporaries
local v1_ = treesData.growingTrees
local v2_ = #v1_
local v3_ = 1
while v3_ <= v2_ do
    local v4_ = v1_[v3_]
    local v5_ = v4_.treeTypeIndex
    local v6_ = self.pooledAnimals:getOrCreateNext()
    v6_:setPosition(v4_.x, v4_.y, v4_.z)
    v3_ = v3_ + 1
end

Into this:

-- AFTER: Clean, readable code with original names
local growingTrees = treesData.growingTrees
local numGrowingTrees = #growingTrees
local growingTreeIndex = 1
while growingTreeIndex <= numGrowingTrees do
    local tree = growingTrees[growingTreeIndex]
    local treeTypeIndex = tree.treeTypeIndex
    local animal = self.pooledAnimals:getOrCreateNext()
    animal:setPosition(tree.x, tree.y, tree.z)
    growingTreeIndex = growingTreeIndex + 1
end

This article walks through each optimization technique, showing the bytecode patterns that cause problems and the AST transformations that fix them.

How Decompilation Works

Before diving into specific optimizations, let's understand the overall decompilation pipeline and the key data structures involved.

The Decompilation Pipeline

┌─────────────┐    ┌─────────────┐    ┌─────────────┐    ┌─────────────┐    ┌─────────────┐
│  Bytecode   │───▶│     CFG     │───▶│     SSA     │───▶│     AST     │───▶│   Source    │
│   (Input)   │    │Construction │    │   Form      │    │  Building   │    │   (Output)  │
└─────────────┘    └─────────────┘    └─────────────┘    └─────────────┘    └─────────────┘
     Parse            Lift              Transform          Structure          Format

Stage 1: Bytecode Parsing
Read the binary bytecode format, extracting:

Instructions (opcodes + operands)
Constant pool (strings, numbers, tables)
Debug information (variable names, line numbers)
Function prototypes (nested functions, upvalue references)

Stage 2: CFG Construction
Build a Control Flow Graph - a directed graph where:

Nodes are "basic blocks" (sequences of instructions with no jumps in the middle)
Edges represent possible execution paths (jumps, branches, fall-through)

Stage 3: SSA Transformation
Convert to Static Single Assignment form for analysis (explained below).

Stage 4: AST Building
Reconstruct high-level Abstract Syntax Tree structures:

Convert CFG patterns back to if/while/for statements
Collapse temporaries into expressions
Apply pattern-matching optimizations

Stage 5: Code Generation
Format the AST as readable Lua source code.

What is a CFG?

A Control Flow Graph represents all possible execution paths through a program. Each node is a "basic block" - a straight-line sequence of instructions with:

One entry point (execution always starts at the first instruction)
One exit point (execution always ends at the last instruction)

Original bytecode:           CFG representation:

0: LOADN R0, 0              ┌──────────────────┐
1: LOADN R1, 10             │ Block 0 (entry)  │
2: JUMPIFLT R0, R1, +5  ───▶│ R0 = 0           │
                            │ R1 = 10          │
                            │ if R0 < R1 goto 1│
                            └────────┬─────────┘
                                     │
                         ┌───────────┴───────────┐
                         ▼                       ▼
                ┌─────────────────┐    ┌─────────────────┐
                │    Block 1      │    │    Block 2      │
                │ (loop body)     │    │ (after loop)    │
                │ R0 = R0 + 1     │    │ return R0       │
                │ jump to Block 0 │    └─────────────────┘
                └────────┬────────┘
                         │
                         └──────────▶ (back to Block 0)

The CFG makes control flow explicit, which is essential for:

Detecting loops (back-edges in the graph)
Finding unreachable code
Understanding variable lifetimes

What is SSA?

Static Single Assignment is an intermediate representation where every variable is assigned exactly once. This simplifies analysis because you always know where a value came from.

-- Original code
x = 1
x = x + 1
y = x * 2

-- SSA form
x₁ = 1
x₂ = x₁ + 1
y₁ = x₂ * 2

Each assignment creates a new "version" of the variable (subscripts). But what happens when control flow merges?

-- Original
if condition then
    x = 1
else
    x = 2
end
print(x)  -- Which x?

SSA uses phi functions (φ) at merge points:

-- SSA form
if condition then
    x₁ = 1
else
    x₂ = 2
end
x₃ = φ(x₁, x₂)  -- "x₃ is x₁ if we came from then-branch, x₂ if from else-branch"
print(x₃)

Why SSA matters for decompilation:

Copy propagation: If x₂ = x₁, we can replace all uses of x₂ with x₁
Dead code elimination: If a variable version is never used, remove its assignment
Name preservation: When merging versions, we can prefer the one with a debug name

What is an AST?

An Abstract Syntax Tree represents the hierarchical structure of source code. Unlike bytecode (a flat list of instructions), an AST captures nesting and relationships.

-- Source code
if x > 0 then
    y = x * 2
end

-- AST representation
IfStatement
├── condition: BinaryOp(>)
│   ├── left: Local("x")
│   └── right: Number(0)
├── then_block: Block
│   └── AssignStatement
│       ├── left: Local("y")
│       └── right: BinaryOp(*)
│           ├── left: Local("x")
│           └── right: Number(2)
└── else_block: Block (empty)

Key AST node types:

Statements: Assign, If, While, For, Return, Call (as statement)
Expressions (RValues): Local, Number, String, Binary, Unary, Call, Index, Closure
LValues (assignable): Local, Index (table field), Global

The decompiler's job is to reconstruct this tree structure from flat bytecode, then format it as readable source.

Where Optimizations Happen

Different optimizations happen at different stages:

Stage	Optimizations
CFG	Loop detection, unreachable code removal
SSA	Copy propagation, dead code elimination, name preservation
AST	Pattern matching (and-chains, ternaries), expression inlining
Format	Method notation (`:` vs `.`), spacing, indentation

Most of the improvements in this article happen at the SSA and AST stages, where we have enough structure to recognize patterns but haven't yet committed to final output.

The Anatomy of Luau Bytecode

Before diving into optimizations, let's understand what we're working with. Luau bytecode is a register-based VM instruction set. Here's what a simple function looks like:

-- Original source
function Color.new(r, g, b, a)
    local self = setmetatable({}, Color_mt)
    self.r = r or 0
    self.g = g or 0
    self.b = b or 0
    self.a = a or 1
    return self
end

Bytecode:

=== Function 2 (new) ===
Parameters: 4, Stack: 7, Upvalues: 1

     0: LOP_NEWTABLE    R5, 3, 0          -- Create empty table
     2: LOP_GETUPVAL    R6, 0             -- Get Color_mt upvalue
     3: LOP_FASTCALL2   setmetatable, R5  -- setmetatable({}, Color_mt)
     5: LOP_GETIMPORT   R4, setmetatable
     7: LOP_CALL        R4, 3, 2          -- Result in R4
     8: LOP_ORK         R5, R0, K2        -- r or 0
     9: LOP_SETTABLEKS  R5, R4, "r"       -- self.r = result
    11: LOP_ORK         R5, R1, K2        -- g or 0
    12: LOP_SETTABLEKS  R5, R4, "g"       -- self.g = result
    ...
    20: LOP_RETURN      R4, 2             -- return self

Local Debug Info:
  R0: "r"    (PC 0-21)
  R1: "g"    (PC 0-21)
  R2: "b"    (PC 0-21)
  R3: "a"    (PC 0-21)
  R4: "self" (PC 8-21)

Key observations:

Registers are reused - R5 is used as a scratch register multiple times
Debug info exists but is partial - Parameters and some locals have names, but temporaries don't
Structure is flattened - The compiler doesn't preserve expression boundaries

Preserving Debug Names Through SSA

The Problem

Luau bytecode includes debug information mapping registers to variable names. But during SSA (Static Single Assignment) transformation, these names were being lost:

-- Raw decompilation output
local v1_ = tonumber(x)
local v2_ = tonumber(y)
local v3_ = tonumber(z)

The bytecode actually had debug names:

Local Debug Info:
  R0: "x" (PC 0-15)
  R1: "y" (PC 0-15)
  R2: "z" (PC 0-15)

The Solution

The fix involved three changes:

Parse debug info from bytecode (luau-lifter/src/deserializer/function.rs)
Preserve names during lifting (luau-lifter/src/lifter.rs)
Prefer named locals during SSA destruction (cfg/src/ssa/destruct.rs)

The SSA destruction phase was the critical fix. When merging SSA versions, we now check which version has a debug name:

// In destruct.rs - prefer named locals when selecting representatives
fn select_representative(&self, versions: &[RcLocal]) -> RcLocal {
    // Prefer version with a name
    for version in versions {
        if version.name().is_some() {
            return version.clone();
        }
    }
    versions[0].clone()
}

Result

-- After: Original names preserved
x = tonumber(x)
y = tonumber(y)
z = tonumber(z)

Fixing Loop Variable Inlining

The Problem

While loop conditions were having their initial values incorrectly substituted:

-- Broken output
while 1 <= numGrowingTrees do
    -- uses growingTreeIndex internally
end

The Bytecode Pattern

     0: LOP_LOADN      R2, 1           -- growingTreeIndex = 1
     5: LOP_JUMPIFNOTLT R2, R1, +20    -- while growingTreeIndex <= numGrowingTrees
    10: LOP_GETTABLE   R4, R0, R2      -- trees[growingTreeIndex]
    ...
    18: LOP_ADDK       R2, R2, K1      -- growingTreeIndex = growingTreeIndex + 1
    20: LOP_JUMPBACK   -15             -- loop back

The SSA inlining pass saw R2 = 1 and tried to substitute 1 everywhere R2 was used - including the loop condition.

The Solution

Detect loop headers via back-edge analysis and protect loop phi parameters:

// In inline.rs
fn is_loop_header(&self, block_id: BlockId) -> bool {
    // A block is a loop header if any predecessor has a higher block ID
    // (indicating a back-edge from within the loop)
    self.function.predecessors(block_id)
        .any(|pred| pred > block_id)
}

fn should_inline(&self, local: &RcLocal, block_id: BlockId) -> bool {
    if self.is_loop_header(block_id) {
        // Don't inline phi parameters in loop headers
        if self.is_phi_parameter(local, block_id) {
            return false;
        }
    }
    true
}

Propagating Upvalue Names

The Problem

When a local is captured as an upvalue by a nested closure, the closure's bytecode might not have the debug name even though the parent function does:

-- Broken output
function ConfigurationManager:configurationKeyIterator()
    return function()
        -- upvalues: (ref) v_u_3_, (copy) numElements
        if numElements <= v_u_3_ then
            return nil
        end
        v_u_3_ = v_u_3_ + 1

The Solution

During upvalue linking, propagate names from parent locals:

// In link_upvalues()
fn link_upvalues(&mut self, parent_func: &Function, child_func: &mut Function) {
    for (upval_idx, parent_local) in child_func.upvalue_sources.iter().enumerate() {
        let child_upval = &mut child_func.upvalues[upval_idx];

        // If parent has a name and child doesn't, propagate it
        if let Some(name) = parent_local.name() {
            child_upval.set_name_if_unnamed(name);
        }
    }
}

Result

-- After: Upvalue inherits parent's name
function ConfigurationManager:configurationKeyIterator()
    return function()
        -- upvalues: (ref) currentIndex, (copy) numElements
        if numElements <= currentIndex then
            return nil
        end
        currentIndex = currentIndex + 1

Self Parameter to Method Notation

The Problem

Lua supports two calling conventions for methods:

obj.method(obj, args) - explicit self
obj:method(args) - implicit self (syntactic sugar)

The decompiler was always outputting the explicit form:

function BaseMission.initialize(self)
function BaseMission.delete(self)
function BaseMission.update(self, dt)

The Solution

Detect functions defined on tables with self as first parameter:

fn should_convert_to_method(func: &Function) -> bool {
    // Must be assigned to a table field (Class.method pattern)
    let is_table_method = matches!(&func.definition_site,
        Some(LValue::Index(idx)) if idx.table.is_global());

    // First parameter must be named "self"
    let has_self_param = func.parameters.first()
        .and_then(|p| p.name())
        .map_or(false, |n| n == "self");

    is_table_method && has_self_param
}

When formatting, convert to colon notation and remove the self parameter:

impl Display for NamedFunction {
    fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result {
        if self.use_method_notation {
            // Output: function Table:method(other_params)
            write!(f, "function {}:{}({})",
                self.table_name,
                self.method_name,
                self.params_without_self())?;
        } else {
            // Output: function Table.method(self, other_params)
            write!(f, "function {}.{}({})", ...)?;
        }
    }
}

Result

-- After: Idiomatic Lua
function BaseMission:initialize()
function BaseMission:delete()
function BaseMission:update(dt)

Boolean Ternary Simplification

The Problem

The Luau compiler transforms boolean expressions into control flow:

-- Source
MobileHUD.new(g_server ~= nil, ...)

-- Compiled to bytecode, then decompiled
local v3_
if g_server == nil then
    v3_ = false
else
    v3_ = true
end
MobileHUD.new(v3_, ...)

The Bytecode Pattern

     0: LOP_GETGLOBAL  R5, "g_server"
     2: LOP_JUMPIFNIL  R5, +4           -- if g_server == nil
     4: LOP_LOADB      R3, 1, +2        -- v3_ = true, skip next
     6: LOP_LOADB      R3, 0            -- v3_ = false
     8: ...                             -- continue

The Solution

Pattern match on the AST after initial decompilation:

fn simplify_boolean_ternaries(block: &mut Block) {
    let mut i = 0;
    while i < block.len() {
        // Look for: if cond then x = true else x = false end
        if let Statement::If(if_stat) = &block[i] {
            if let (Some(then_assign), Some(else_assign)) =
                (single_bool_assign(&if_stat.then_block),
                 single_bool_assign(&if_stat.else_block))
            {
                // Both branches assign to same variable
                if then_assign.target == else_assign.target {
                    let (then_val, else_val) =
                        (then_assign.value, else_assign.value);

                    if then_val == true && else_val == false {
                        // x = cond
                        block[i] = assign(then_assign.target,
                            if_stat.condition.clone());
                    } else if then_val == false && else_val == true {
                        // x = not cond
                        block[i] = assign(then_assign.target,
                            negate(if_stat.condition.clone()));
                    }
                }
            }
        }
        i += 1;
    }
}

For conditions like x == nil, transform to x ~= nil for the boolean context.

Result

-- After: Direct boolean expression
MobileHUD.new(g_server ~= nil, ...)

Numeric For-Loop Inlining

The Problem

For-loop bounds were extracted into temporaries:

local v2_ = select("#", ...)
for i = 1, v2_ do
    -- body
end

The Bytecode Pattern

     0: LOP_GETIMPORT  R2, select
     2: LOP_LOADK      R3, "#"
     3: LOP_GETVARARGS R4, -1
     4: LOP_CALL       R2, -1, 2        -- v2_ = select("#", ...)
     5: LOP_LOADN      R3, 1            -- initial = 1
     6: LOP_MOVE       R4, R2           -- limit = v2_
     7: LOP_LOADN      R5, 1            -- step = 1
     8: LOP_FORNPREP   R3, +10          -- for i = R3, R4, R5

The Solution

Search backwards from for-loops to find assignments that can be inlined:

fn inline_into_numeric_for_loops(&self, block: &mut Block) {
    for for_idx in 0..block.len() {
        let Statement::NumericFor(nf) = &block[for_idx] else { continue };

        // Check if limit is a local reference
        let RValue::Local(limit_local) = &nf.limit else { continue };

        // Search backwards for assignment to this local
        for assign_idx in (0..for_idx).rev() {
            if let Statement::Assign(assign) = &block[assign_idx] {
                if assign.targets_local(limit_local) {
                    // Inline the RHS directly into the for-loop
                    let new_limit = assign.right[0].clone();
                    block[for_idx].as_numeric_for_mut().limit = new_limit;
                    block.remove(assign_idx);
                    break;
                }
            }
        }
    }
}

Result

-- After: Expression inlined
for i = 1, select("#", ...) do
    -- body
end

And-Chain and Multi-Return Collapse

And-Chain Simplification

Pattern: local v = a; if v then v = b end

-- Before
local v = self.isValid
if v then
    v = self.isReady
end
return v

-- After
local v = self.isValid and self.isReady
return v

The transformation detects when:

A local is assigned a value
The next statement is if local then local = other_value end
The local is the condition AND is reassigned in the then-branch

Multi-Return Collapse

Pattern: Multiple returns captured then immediately assigned to fields

-- Before
local v1_, v2_, v3_ = getWorldTranslation(node)
self.x = v1_
self.y = v2_
self.z = v3_

-- After
self.x, self.y, self.z = getWorldTranslation(node)

Implementation searches for where each temporary is used:

fn collapse_multi_return_assignments(block: &mut Block) {
    // Find: local temps = call()
    let assign_idx = find_multi_return_assign(block)?;
    let temps = &block[assign_idx].as_assign().left;

    // For each temp, find where it's used in subsequent statements
    let mut targets = Vec::new();
    for (i, temp) in temps.iter().enumerate() {
        if temp.is_underscore() {
            targets.push(LValue::Local(underscore()));
            continue;
        }

        // Search for: target = temp
        let target = find_temp_usage(block, assign_idx + 1, temp)?;
        targets.push(target);
    }

    // Rewrite to: targets = call()
    let call = block[assign_idx].as_assign().right[0].clone();
    block[assign_idx] = Assign::new(targets, vec![call]).into();

    // Remove the now-dead individual assignments
    remove_temp_assignments(block, assign_idx + 1, temps);
}

Side-Effect Lookahead and SSA Name Preservation

This set of optimizations together achieved a 98.5% reduction in unnamed temporaries.

Preserving Named Variables in SSA Copy Propagation

The most impactful single change. During SSA copy propagation, when eliminating copies like named = temp, prefer keeping named:

// In construct.rs - propagate_copies()
if assign.is_copy() {
    let (from, to) = (assign.right, assign.left);

    // Prefer keeping the variable with a debug name
    let from_has_name = from.name().is_some();
    let to_has_name = to.name().is_some();

    if from_has_name && !to_has_name {
        // from has name, to doesn't -> keep from, map to → from
        self.local_map.insert(to.clone(), from.clone());
    } else {
        // Keep to (original behavior)
        self.local_map.insert(from.clone(), to.clone());
    }
}

This single change reduced temporaries from ~2176 to ~611 (72% reduction).

Side-Effect Inlining with Lookahead

Previously, temporaries with side effects (like function calls) could only be inlined into the immediately following statement. This failed for:

local v4_ = tostring(x)
local v5_ = y              -- intervening statement
local v6_ = tostring(v5_)
foo(v4_ .. v6_)            -- v4_ used here, but not adjacent

The fix: look ahead to find where the temporary is used, verifying no intervening statement modifies the values it depends on:

fn find_safe_inline_target(&self, local: &RcLocal, rvalue: &RValue,
                            block: &Block, def_idx: usize) -> Option<usize> {
    // What locals does this RValue read?
    let reads: HashSet<_> = rvalue.values_read().into_iter().collect();

    // Scan forward looking for where local is used
    for i in (def_idx + 1)..block.len() {
        let stmt = &block[i];

        // Check if this statement writes to any local we need to read
        let writes = stmt.values_written();
        if writes.iter().any(|w| reads.contains(w)) {
            return None;  // Can't inline past this
        }

        // Check if this statement uses our local
        if stmt.values_read().contains(local) {
            return Some(i);
        }
    }
    None
}

Or-Chain Simplification

Pattern: local v = x; if not v then v = y end → local v = x or y

-- Before
local activeElement = self:getActiveElement()
if not activeElement then
    activeElement = self:getDefaultElement()
end

-- After
local activeElement = self:getActiveElement() or self:getDefaultElement()

Write-Only Local Simplification

Pattern: Locals declared then only written (never read) become _:

-- Before
local v28_, v29_
v28_, rotY, v29_ = getWorldRotation(self.node)

-- After
_, rotY, _ = getWorldRotation(self.node)

Dead Local Removal

Locals that are assigned but never read are eliminated:

-- Before: v8_ is never used
local v8_ = numKeys
for i = 2, numKeys do
    -- uses numKeys directly
end

-- After
for i = 2, numKeys do
    -- v8_ eliminated
end

`__set_list` Pattern Simplification

The Luau compiler generates SETLIST for array literals, which decompiles to:

-- Before
local v35_ = {}
__set_list(v35_, 1, {a, b, c})
return v35_

-- After
return {a, b, c}

Inlining Into Nested Blocks

The Problem

Side-effect-free values defined at parent scope couldn't be inlined into nested if/while/for blocks:

local v14_ = true
if direction ~= nil then
    pressedAccept = v14_   -- v14_ not inlined
end

The inliner was clearing pending inlines before recursing into nested blocks.

The Solution

Pass deferred inlines down into nested blocks:

fn apply_pending_deferred_to_nested(&mut self, statement: &mut Statement) {
    if self.deferred_inlines.is_empty() {
        return;
    }

    match statement {
        Statement::If(if_stat) => {
            self.apply_deferred_to_block(&mut if_stat.then_block);
            self.apply_deferred_to_block(&mut if_stat.else_block);
        }
        Statement::While(while_stat) => {
            self.apply_deferred_to_block(&mut while_stat.block);
        }
        // ... other control structures
    }
}

fn apply_deferred_to_block(&mut self, block: &mut Block) {
    for stmt in block.iter_mut() {
        // Apply any pending inlines to this statement
        for local in stmt.values_read() {
            if let Some(rvalue) = self.deferred_inlines.remove(&local) {
                self.replace_local_with_rvalue(stmt, &local, rvalue);
            }
        }
        // Recurse into nested blocks
        self.apply_pending_deferred_to_nested(stmt);
    }
}

Result

-- After: v14_ inlined
if direction ~= nil then
    pressedAccept = true
end

Numeric For-Loop Shadowing Fix

The Problem

The for-loop inlining was working backwards - it was replacing good variable names with temporaries instead of the other way around:

-- Before (broken)
local v27_ = minX
for minX = v27_, maxX, step do  -- Wrong direction!

The Bytecode Pattern

     0: LOP_GETUPVAL   R3, 0             -- minX from upvalue
     1: LOP_MOVE       R4, R3            -- Copy to R4 (for-loop init)
     2: LOP_GETUPVAL   R5, 1             -- maxX
     3: LOP_LOADN      R6, 1             -- step = 1
     4: LOP_FORNPREP   R4, +10           -- for minX = R4, R5, R6

Local Debug Info:
  R3: "minX" (PC 0-20)    -- Original variable
  R4: "minX" (PC 4-15)    -- For-loop counter shadows it

The issue: register R3 has the debug name "minX", but R4 (the for-loop counter) also has the name "minX". When we see R4 = R3, we were inlining the wrong direction - replacing the named R3 with temp, instead of eliminating the temp.

The Solution

Only inline when the ASSIGNED local (LHS) is an unnamed temporary:

fn inline_into_numeric_for_loops(block: &mut Block) {
    // ...
    // Only inline if the ASSIGNED local is an unnamed temporary
    let assigned_local = assign.left[0].as_local()?;
    let is_unnamed_temp = assigned_local.name()
        .map_or(true, |n| n.starts_with('v') && n.ends_with('_'));

    if !is_unnamed_temp {
        continue;  // Don't inline away named variables
    }
    // ... proceed with inlining
}

Result

-- After (correct)
for minX = minX, maxX, step do

Intervening Statements in SetList Collapse

The Problem

The __set_list simplification only worked when the table creation was immediately followed by the SetList call. Real code often has intervening statements:

local v3_ = {}
local i = self.componentJoints[spec.frontAxisJoint].rotLimit  -- intervening!
__set_list(v3_, 1, {unpack(i)})
spec.rotLimit = v3_

The Bytecode Pattern

     0: LOP_NEWTABLE   R3, 0, 1          -- v3_ = {}
     2: LOP_GETTABLEKS R4, R0, "componentJoints"
     4: LOP_GETTABLEKS R5, R1, "frontAxisJoint"
     6: LOP_GETTABLE   R4, R4, R5
     7: LOP_GETTABLEKS R4, R4, "rotLimit"  -- i = self.componentJoints[...].rotLimit
     9: LOP_DUPTABLE   R5, K0             -- Prepare array for SETLIST
    11: LOP_GETIMPORT  R6, unpack
    13: LOP_MOVE       R7, R4
    14: LOP_CALL       R6, 2, -1          -- unpack(i)
    15: LOP_SETLIST    R5, R6, -1         -- __set_list call
    17: LOP_MOVE       R5, R3             -- Copy table
    18: LOP_SETTABLEKS R5, R0, "rotLimit" -- spec.rotLimit = v3_

The NEWTABLE (line 0) and SETLIST (line 15) have unrelated statements between them. The decompiler needs to recognize these can still be collapsed.

The Solution

Allow safe intervening statements - those that don't read or write the table local:

fn find_set_list_with_intervening(block: &Block, table_local: &RcLocal,
                                   start_idx: usize) -> Option<(usize, Vec<usize>)> {
    let mut intervening_indices = Vec::new();

    for i in start_idx..block.len() {
        let stmt = &block[i];

        // Found the SetList call?
        if is_set_list_for(stmt, table_local) {
            return Some((i, intervening_indices));
        }

        // Check if this statement touches the table local
        if let Statement::Assign(assign) = stmt {
            let reads = assign.values_read();
            let writes = assign.values_written();

            if reads.contains(table_local) || writes.contains(table_local) {
                return None;  // Can't skip this statement
            }

            intervening_indices.push(i);
        } else {
            return None;  // Non-assignment statement, stop
        }
    }
    None
}

Result

-- After: intervening statement preserved, table eliminated
local i = self.componentJoints[spec.frontAxisJoint].rotLimit
spec.rotLimit = {unpack(i)}

Post-Simplification Inlining Pass

The Problem

After simplify_and_chains collapses split and-chain assignments, some temporaries become single-def-single-use but aren't inlined because the inlining pass already ran:

-- After simplify_and_chains:
local v69_ = A and B and C == D  -- now single def!
hotspot:setVisible(v69_)         -- single use!

The Solution

Add a new pass inline_single_use_unnamed_temps that runs after and-chain simplification:

pub fn inline_single_use_unnamed_temps(block: &mut Block) {
    loop {
        let mut changed = false;
        let mut i = 0;

        while i < block.0.len().saturating_sub(1) {
            if let Statement::Assign(assign) = &block.0[i] {
                // Check: single LHS that's an unnamed temp
                if assign.left.len() != 1 { i += 1; continue; }
                let Some(local) = assign.left[0].as_local() else { i += 1; continue; };

                if !is_unnamed_temp(local) { i += 1; continue; }

                // Check: exactly one use in the next statement
                let next_stmt = &block.0[i + 1];
                let uses = count_local_uses(next_stmt, local);

                if uses == 1 {
                    // Inline!
                    let rvalue = assign.right[0].clone();
                    replace_local_in_statement(&mut block.0[i + 1], local, &rvalue);
                    block.0.remove(i);
                    changed = true;
                    continue;
                }
            }
            i += 1;
        }

        if !changed { break; }
    }
}

Result

-- After: temp inlined
hotspot:setVisible(A and B and C == D)

SetList in Call Arguments

The Problem

Tables were being created for use as function arguments:

local v81_ = {}
__set_list(v81_, 1, {translation[1], translation[2], translation[3]})
local j = j(node, get, set, v81_, time)

The Bytecode Pattern

     0: LOP_NEWTABLE   R10, 0, 3         -- v81_ = {}
     2: LOP_DUPTABLE   R11, K0           -- Prepare array
     4: LOP_GETTABLE   R12, R5, K1       -- translation[1]
     6: LOP_GETTABLE   R13, R5, K2       -- translation[2]
     8: LOP_GETTABLE   R14, R5, K3       -- translation[3]
    10: LOP_SETLIST    R11, R12, 3       -- __set_list(v81_, 1, {...})
    12: LOP_GETLOCAL   R11, 0            -- j (function)
    14: LOP_MOVE       R12, R1           -- node
    15: LOP_MOVE       R13, R2           -- get
    16: LOP_MOVE       R14, R3           -- set
    17: LOP_MOVE       R15, R10          -- v81_ as argument
    18: LOP_MOVE       R16, R4           -- time
    19: LOP_CALL       R11, 6, 2         -- j(node, get, set, v81_, time)

The table in R10 is used as an argument to the call at line 17. Previous patterns only handled tables used in direct assignments or returns, not as call arguments.

The Solution

Pattern 4: Detect when the table local is used as a call argument:

// Check if table local is used in a call on the RHS of an assignment
if let Statement::Assign(next_assign) = &block[usage_idx] {
    for rvalue in &next_assign.right {
        // Handle both direct calls and Select-wrapped calls
        let call_args = match rvalue {
            RValue::Call(call) => Some(&call.arguments),
            RValue::MethodCall(mc) => Some(&mc.arguments),
            RValue::Select(sel) => {
                if let RValue::Call(call) = &*sel.value {
                    Some(&call.arguments)
                } else { None }
            }
            _ => None,
        };

        if let Some(args) = call_args {
            // Find and replace the table local in arguments
            for arg in args {
                if arg.as_local() == Some(&table_local) {
                    // Replace with the array literal
                    *arg = array_literal.clone();
                    // Remove table creation and __set_list
                }
            }
        }
    }
}

Result

-- After: inline array literal in call
local j = j(node, get, set, {translation[1], translation[2], translation[3]}, time)

Nested Block Recursion for For-Loops

The Problem

Numeric for-loop inlining only processed the top-level block, missing patterns inside if/while/for bodies:

for x = x, z, cellSize do
    local v81_ = _              -- nested!
    for _ = v81_, maxY, cellSize do

The Bytecode Pattern

     0: LOP_MOVE       R3, R0            -- x (outer loop init)
     1: LOP_MOVE       R4, R2            -- z (outer loop limit)
     2: LOP_MOVE       R5, R6            -- cellSize (outer loop step)
     3: LOP_FORNPREP   R3, +20           -- for x = x, z, cellSize
     -- Inside outer loop body:
     5: LOP_LOADNIL    R7                -- _ (the variable named underscore)
     6: LOP_MOVE       R8, R7            -- v81_ = _ (copy for inner loop)
     7: LOP_GETUPVAL   R9, 0             -- maxY
     8: LOP_MOVE       R10, R6           -- cellSize
     9: LOP_FORNPREP   R8, +10           -- for _ = v81_, maxY, cellSize

The pattern v81_ = _ followed by for _ = v81_ appears inside the outer for-loop's body. The inlining pass wasn't recursing into nested blocks, so this remained unoptimized.

The Solution

Recurse into nested blocks:

fn inline_into_numeric_for_loops(block: &mut Block) {
    // Process this block's statements
    for statement in &mut block.0 {
        // ... handle numeric for at this level ...

        // Recurse into nested blocks
        match statement {
            Statement::If(if_stat) => {
                inline_into_numeric_for_loops(&mut if_stat.then_block.lock());
                inline_into_numeric_for_loops(&mut if_stat.else_block.lock());
            }
            Statement::While(while_stat) => {
                inline_into_numeric_for_loops(&mut while_stat.block.lock());
            }
            Statement::NumericFor(nf) => {
                inline_into_numeric_for_loops(&mut nf.block.lock());
            }
            Statement::GenericFor(gf) => {
                inline_into_numeric_for_loops(&mut gf.block.lock());
            }
            Statement::Repeat(rep) => {
                inline_into_numeric_for_loops(&mut rep.block.lock());
            }
            _ => {}
        }
    }
}

Result

-- After: nested for-loop also simplified
for x = x, z, cellSize do
    for _ = _, maxY, cellSize do

Boolean Condition Simplification

The Problem

Patterns where a temporary captures a boolean expression, then is used as both condition and value:

local v23_ = isActive and isEnabled
if v23_ then
    self.shouldProcess = v23_
end

The Bytecode Pattern

     0: LOP_GETUPVAL   R3, 0             -- isActive
     1: LOP_JUMPIFNOT  R3, +4            -- if not isActive, skip
     3: LOP_GETUPVAL   R3, 1             -- isEnabled (short-circuit)
     5: LOP_JUMPIFNOT  R3, +6            -- if v23_ then
     7: LOP_GETTABLEKS R4, R0, "shouldProcess"
     9: LOP_MOVE       R5, R3            -- self.shouldProcess = v23_
    10: LOP_SETTABLEKS R5, R0, "shouldProcess"

The and-expression is evaluated into R3, then R3 is tested. If true, R3 is assigned to the field. But inside the then-branch, we know R3 is truthy - so we can replace it with true.

The Solution

Inside the then-branch, we know v23_ evaluated to true. Replace it:

fn simplify_boolean_condition_assignments(block: &mut Block) {
    let mut i = 0;
    while i + 1 < block.0.len() {
        // Pattern: local v = <bool>; if v then ... v ... end
        let Statement::Assign(assign) = &block.0[i] else { i += 1; continue };
        let Statement::If(if_stat) = &block.0[i + 1] else { i += 1; continue };

        // Must be single assignment to unnamed temp
        let local = assign.left[0].as_local()?;
        if !is_boolean_expression(&assign.right[0]) { i += 1; continue; }

        // Condition must be exactly this local
        let cond_local = if_stat.condition.as_local()?;
        if cond_local != local { i += 1; continue; }

        // Replace uses of local with `true` in then-branch
        replace_local_with_true_in_block(&mut if_stat.then_block.lock(), local);

        // Inline the boolean expression into the condition
        if_stat.condition = assign.right[0].clone();
        block.0.remove(i);  // Remove the assignment
    }
}

Result

-- After: temp eliminated, true substituted
if isActive and isEnabled then
    self.shouldProcess = true
end

Function Call Boolean Patterns

The Problem

The boolean condition simplification only worked for boolean expressions. But function calls returning booleans (or truthy values) have the same pattern:

local v46_ = superFunc(self)
if v46_ then
    turnOn = v46_
end

Also needed to handle the negated case:

local v48_ = superFunc(self)
if not v48_ then
    turnOff = v48_
end

The Bytecode Pattern

     0: LOP_GETUPVAL   R5, 0             -- superFunc
     1: LOP_MOVE       R6, R0            -- self
     2: LOP_CALL       R5, 2, 2          -- v46_ = superFunc(self)
     3: LOP_JUMPIFNOT  R5, +5            -- if v46_ then
     5: LOP_MOVE       R6, R5            -- turnOn = v46_
     6: LOP_SETUPVAL   R6, 1

The call result in R5 is tested, then assigned. For the negated case:

     3: LOP_JUMPIF     R5, +5            -- if not v48_ (note: JUMPIF vs JUMPIFNOT)
     5: LOP_MOVE       R6, R5            -- turnOff = v48_

Inside the then-branch after if v we know v is truthy; after if not v we know v is falsy.

The Solution

Extend the pattern to handle Call, MethodCall, and Select (for multi-return), plus detect negation:

// Extend RHS check
let rhs = &assign.right[0];
let is_boolean_expr = is_boolean_expression(rhs);
let is_call_like = matches!(rhs,
    RValue::Call(_) | RValue::MethodCall(_) | RValue::Select(_));

if !is_boolean_expr && !is_call_like {
    continue;
}

// Handle negated conditions: if not v then x = v end
let (cond_local, is_negated) = if let Some(l) = if_stat.condition.as_local() {
    (l, false)
} else if let Some(unary) = if_stat.condition.as_unary() {
    if matches!(unary.operation, UnaryOperation::Not) {
        if let Some(l) = unary.value.as_local() {
            (l, true)
        } else { continue; }
    } else { continue; }
} else { continue; };

// Replace with appropriate boolean
if is_negated {
    // In "if not v then", v is false in the then-branch
    replace_local_with_false_in_block(&mut if_stat.then_block.lock(), &local);
} else {
    // In "if v then", v is true in the then-branch
    replace_local_with_true_in_block(&mut if_stat.then_block.lock(), &local);
}

Result

-- After: function call inlined, temp replaced with boolean
if superFunc(self) then
    turnOn = true
end

if not superFunc(self) then
    turnOff = false
end

Recursive Use Counting Fix

The Problem

The use-counting for determining if a temp can be inlined wasn't recursing into nested blocks:

local v23_ = getValue()
if v23_ then           -- use 1
    process(v23_)      -- use 2 (in nested block - was missed!)
end

This led to incorrect inlining that removed the temp even though it was used twice.

The Solution

Count uses recursively across all nested blocks:

fn count_local_uses_recursive(block: &Block, local: &RcLocal) -> usize {
    let mut count = 0;

    for statement in &block.0 {
        // Count direct uses
        count += statement.values_read().iter()
            .filter(|l| *l == local)
            .count();

        // Recurse into nested blocks
        match statement {
            Statement::If(if_stat) => {
                count += count_local_uses_recursive(&if_stat.then_block.lock(), local);
                count += count_local_uses_recursive(&if_stat.else_block.lock(), local);
            }
            Statement::While(w) => {
                count += count_local_uses_recursive(&w.block.lock(), local);
            }
            Statement::NumericFor(nf) => {
                count += count_local_uses_recursive(&nf.block.lock(), local);
            }
            Statement::GenericFor(gf) => {
                count += count_local_uses_recursive(&gf.block.lock(), local);
            }
            Statement::Repeat(r) => {
                count += count_local_uses_recursive(&r.block.lock(), local);
            }
            _ => {}
        }
    }

    count
}

The Numbers

Metric	Before	After	Reduction
Unnamed temporaries	~10,000	18	99.8%
`v_u_` upvalue temps	79	0	100%
Files with temps	500+	11	97.8%

Remaining Patterns (The Hard Ones)

The 123 remaining temporaries fall into categories that are difficult or impossible to optimize:

For-loop bound preservation - Semantically required:

   local v27_ = minX
   for minX = v27_, maxX, step do  -- Can't inline; minX is shadowed

Boolean short-circuit chains - Used multiple times:

   local v = condition1 and condition2
   if v then
       v = condition3   -- v is reassigned
   end

Complex control flow results - Values computed across branches that can't be simplified to expressions

Lessons Learned

1. SSA is Your Friend (and Enemy)

SSA form makes many optimizations easy (copy propagation, dead code elimination), but it can also destroy information. The key insight: preserve names during SSA transformations, not just after.

2. Order Matters

The order of optimization passes is critical:

Parse debug info early
SSA construction with name preservation
SSA-based optimizations
SSA destruction (prefer named representatives)
AST-level pattern matching
Final cleanup passes

Running pattern matching before SSA destruction would miss opportunities; running it too early interferes with other passes.

3. Test on Real Code

A real-world codebase (~500 Luau files, ~200K lines) was invaluable. Synthetic tests miss edge cases; real code has all the weird patterns.

4. Bytecode Debug Info is Gold

Even partial debug information dramatically improves output quality. Always parse it, always preserve it.

Conclusion

Decompilation is fundamentally about reconstructing intent from implementation. The compiler has thrown away variable names, flattened expressions, and transformed elegant source into efficient bytecode. Our job is to reverse that transformation.

The techniques here - SSA name preservation, pattern matching, lookahead analysis - are applicable to any decompiler. The key is understanding what the compiler does and doing the opposite.

All code is available in the medal repository. PRs welcome.

This work was done to improve reverse engineering of Luau bytecode for modding purposes.