DEV Community: Saish

Unreal Engine GPU Stack: Chapter 2 Scene View Extensions

Saish — Sun, 25 Oct 2026 20:43:54 +0000

How to inject a custom rendering pass into Unreal's frame without modifying the engine.

In Chapter 1 we did all the required world building... three threads, the RHI, and the Render Dependency Graph. The next practical question is how your own GPU work gets into Unreal's frame.

There are a few common answers from Youtube tutorials, and none of them are perfect. A post-process material can't run a compute shader or write to an render target. A Scene Capture Component renders the whole scene again from another view just to hand you pixels you already had.

Imagine overlying your active digital twin data over real world CCTV footage. Or building a cam overlay over Unreal Automotive HMD or just compositing real world background to generate synthetic data. All can be done efficiently using GPUCompositionGraphicsPipeline or SynthPass. More about this in this chapter.

1. What's happening in GPUCompositionGraphicsPipeline

This chapter uses one plugin as its running example: GPUCompositionGraphicsPipeline. It takes a live video feed (anything that plays into a UMediaTexture) and composites the rendered frame over it. All of it runs on the GPU, in the middle of Unreal's own post-processing.

It has three parts:

UCompGraphicsSubsystem, an engine subsystem that runs on the Game thread. This is the Blueprints facing side. It creates and owns the extension, holds the media texture and the output render target, and tags the actors that should appear in the composite with custom stencil value 20.
FCoreSVE, the Scene View Extension. It subscribes to the post-processing stage right after FXAA and runs the compositing callback on the Render thread.
Two passes inside that callback. A pixel shader (FCustomDepthStencilMaskCreatorPS) turns the custom stencil buffer into a mask. Then a compute shader (FCompositorCS) blends scene color, the mask and the media texture. The result is drawn to the viewport and copied into a render target, so it can be used outside the viewport too.

So in one frame, the Game thread only sets values. On the Render thread, the renderer asks FCoreSVE where it wants to run. After FXAA it adds the mask pass, then the compositing pass, then the copy and draw. Every snippet below is from this plugin unless it's marked as engine source.

Engine code below is from UE 5.6

The interface lives in the Engine module (Engine/Public/SceneViewExtension.h), but anything useful you do inside it needs the rendering modules. The plugin's Build.cs:

// GPUCompositionGraphicsPipeline.Build.cs
PublicDependencyModuleNames.AddRange(new string[] {
    "Core", "CoreUObject", "Engine", // ISceneViewExtension, FSceneViewExtensionBase
    "RenderCore",                    // FRDGBuilder, global shaders
    "Renderer",                      // FScreenPassTexture, FPostProcessMaterialInputs
    "RHI",                           // textures, formats, feature levels
    "Projects",                      // IPluginManager, for the shader directory mapping
    "MediaAssets",                   // UMediaTexture, the video input
});

Source in SynthPass: GPUCompositionGraphicsPipeline.Build.cs L12–L21

The headers you'll typically include:

One header to be careful with: FPostProcessingInputs, the argument to PrePostProcessPass_RenderThread, is defined in Renderer/Internal/PostProcess/PostProcessInputs.h. You can override the hook with only a forward declaration. But if you want to read its members, you have to add Renderer/Internal to your include paths, and nothing in Internal or Private is a stable API. Stay in Public wherever you can.

The plugin does take that trade-off. It includes PostProcess/PostProcessInputs.h and a few other renderer headers, so it adds both paths. It builds on 5.6, but an engine upgrade can break it:

string EngineDir = Path.GetFullPath(Target.RelativeEnginePath);

PrivateIncludePaths.AddRange(new string[] {
    Path.Combine(EngineDir, "Source/Runtime/Renderer/Private"),
    Path.Combine(EngineDir, "Source/Runtime/Renderer/Internal")
});

Source in SynthPass: GPUCompositionGraphicsPipeline.Build.cs L36–L45

The shaders have to be found before they compile. If the extension uses global shaders from your plugin, the Shaders directory mapping from Chapter 1 must exist before the engine compiles global shaders, which happens early in startup. Thus, such modules use "LoadingPhase": "PostConfigInit" in the .uplugin. With Default, the engine can't resolve your virtual shader path. The plugin's .uplugin sets PostConfigInit, and StartupModule does only the mapping:

// GPUCompositionGraphicsPipeline.cpp
void FGPUCompositionGraphicsPipelineModule::StartupModule()
{
    FString PluginShaderDir = FPaths::Combine(
        IPluginManager::Get().FindPlugin(TEXT("GPUCompositionGraphicsPipeline"))->GetBaseDir(), TEXT("Shaders"));
    AddShaderSourceDirectoryMapping(TEXT("/Plugins/GPUCompositionGraphicsPipeline"), PluginShaderDir);
}

Source in SynthPass: GPUCompositionGraphicsPipeline.cpp L7–L12

There was a second problem I didn't expect here: mapping the shader directory and creating the extension can't happen at the same point in startup.

// Engine source: Engine/Private/SceneViewExtension.cpp
void FSceneViewExtensions::RegisterExtension(const FSceneViewExtensionRef& RegisterMe)
{
    if (ensure(GEngine))
    {
        auto& KnownExtensions = GEngine->ViewExtensions->KnownExtensions;
        // ... compact dead entries ...
        KnownExtensions.AddUnique(RegisterMe);
    }
}

A module loading at PostConfigInit starts long before GEngine exists. So map the shader directory in StartupModule, but create the extension later... in a subsystem's Initialize, from FCoreDelegates::OnPostEngineInit, or when gameplay asks for it. The plugin uses the last option: nothing is created until a Blueprint calls InitilizeCompositor on the subsystem.

2. Creating and owning an extension

The plugin's extension, trimmed to what it uses:

// CoreGraphicsPipeline.h
class FCoreSVE : public FSceneViewExtensionBase
{
public:
    FCoreSVE(const FAutoRegister& AutoRegister);

    virtual void SubscribeToPostProcessingPass(EPostProcessingPass PassId, const FSceneView& View,
        FAfterPassCallbackDelegateArray& InOutPassCallbacks, bool bIsPassEnabled);

    FScreenPassTexture GraphicsCompositing(FRDGBuilder& GraphBuilder, const FSceneView& View,
        const FPostProcessMaterialInputs& Inputs);
};

// CoreGraphicsPipeline.cpp
FCoreSVE::FCoreSVE(const FAutoRegister& AutoRegister) : FSceneViewExtensionBase(AutoRegister) {}

Source in SynthPass: CoreGraphicsPipeline.h L10–L29, CoreGraphicsPipeline.cpp L27–L31

FAfterPassCallbackDelegateArray is the older name. In 5.6 it's an alias for FPostProcessingPassDelegateArray, so both compile.

And creating it, from the subsystem:

// CompGraphicsSubsystem.h
TSharedPtr<class FCoreSVE, ESPMode::ThreadSafe> CoreSVE;

// CompGraphicsSubsystem.cpp
void UCompGraphicsSubsystem::InitilizeCompositor()
{
    // Custom depth with stencil, which the mask pass reads
    UKismetSystemLibrary::ExecuteConsoleCommand(GetWorld(), FString("r.CustomDepth 3"));

    CoreSVE = FSceneViewExtensions::NewExtension<FCoreSVE>();
}

Source in SynthPass: CompGraphicsSubsystem.h L87, CompGraphicsSubsystem.cpp L26–L34

FAutoRegister has a private constructor, and only FSceneViewExtensions can create one. So new FCoreSVE(...) doesn't compile. The only way in is NewExtension<T>(), which constructs the extension, wraps it in a thread-safe shared pointer, and registers it in one step. The first constructor argument must be FAutoRegister. Any extra arguments you pass to NewExtension are forwarded after it.

Ownership is yours. The engine's registry only holds weak pointers. If you drop your shared pointer, the extension is gone from the next frame on, and that's also how you unregister it:

void UCompGraphicsSubsystem::DeinitializeCompositor()
{
    if (CoreSVE.IsValid())
    {
        CoreSVE->IsActiveThisFrameFunctions.Empty();
    }
    CoreSVE.Reset();
    // ... release the render target and show-only actors ...
}

Source in SynthPass: CompGraphicsSubsystem.cpp L77–L95

The frame already in flight is safe (the frame that is being rendered in gpu). Each view family copies the active extensions into ViewFamily.ViewExtensions as strong references, so the Render thread keeps the extension alive until that frame finishes. This is the same idea as the scene proxies in Chapter 1: the Game thread can let go while the Render thread finishes using it.

ESPMode::ThreadSafe isn't optional. The pointer is created on the Game thread and used on the Render thread, so its reference count must be atomic.

Who should hold the pointer? Something that lives as long as the effect should: an engine subsystem for effects that persist across levels, a world subsystem for per-level effects, or an actor or component when the effect belongs to one object. The plugin uses an engine subsystem, so the compositor survives level changes.

3. Every injection point in the frame

The hooks follow the frame from Game-thread setup to the end of rendering. Let's see which thread calls them.

3.1 Game thread hooks: shaping the MVP

These hooks run while the engine builds the view family for the next frame. They don't take an FRDGBuilder, so you can't add passes here. Their job is to change what gets rendered, not how.

Let's take a closer look at BeginRenderViewFamily . It's the last point where you're on the Game thread and know that a frame is about to be rendered. It's the correct place to copy settings, UObject data and texture resource pointers into a plain struct that the Render-thread hooks will read. Reading UObjects directly from Render-thread hooks is the kind of data race Chapter 1 warned about. It usually works, until the Game thread changes the object in the middle of a frame. You also need to consider the copy cost for data from one thread to another. It a tradeoff between performance and unsafe code (Just like rust providing memory safety at compile times but programmers need to write unsafe() code at some places to extract every drop of perf)

3.2 Render thread: the scene renderer

These hooks receive an FRDGBuilder&. As Chapter 1 explained, nothing runs when you call RDG. You're adding passes to a graph that executes later, and the graph orders them after whatever came before your hook.

You can see that the per-view hooks (PreRenderView, PrePostProcessPass, PostRenderView) run once for each view. Split screen, stereo rendering and scene captures with multiple views all call them more than once per frame, so don't assume one call means one frame.

4. Injection points inside post-processing

SubscribeToPostProcessingPass works differently from the other hooks. It doesn't render anything. At the start of post-processing, the renderer calls it once per view for each stage and asks "do you want to run something here?". You answer by adding a delegate. The plugin answers only for the FXAA stage, and only while the compositor is running:

// CoreGraphicsPipeline.cpp
void FCoreSVE::SubscribeToPostProcessingPass(EPostProcessingPass PassId, const FSceneView& View,
    FAfterPassCallbackDelegateArray& InOutPassCallbacks, bool bIsPassEnabled)
{
    UCompGraphicsSubsystem* CompGraphicsSubsystem = GEngine->GetEngineSubsystem<UCompGraphicsSubsystem>();
    if (!CompGraphicsSubsystem) { return; }
    if (!CompGraphicsSubsystem->GetShouldCompositorRun()) { return; }

    if (PassId == EPostProcessingPass::FXAA)
    {
        InOutPassCallbacks.Add(FAfterPassCallbackDelegate::CreateRaw(this, &FCoreSVE::GraphicsCompositing));
    }
}

Source in SynthPass: CoreGraphicsPipeline.cpp L39–L51

This hook runs on the Render thread, and it reads a UObject (the subsystem) directly. That's the race described above. It works because a bool flag is rarely changed mid-frame. The safer version copies the flag in BeginRenderViewFamily.

These are the stages in 5.6, in the order they run in the frame. The engine header notes that each one sits immediately before a post-process material blendable location, so if you know the material blend locations, you already know these:

The plugin's stage, FXAA, comes after tonemapping. So the composite blends display-ready colors with the video, which is the space the footage is already in. Here's the FXAA case from PostProcessing.cpp, because it surprises people:

// Engine source: Renderer/Private/PostProcess/PostProcessing.cpp
if (PassSequence.IsEnabled(EPass::FXAA))
{
    SceneColor = AddFXAAPass(GraphBuilder, View, PassInputs);
}
SceneColor = AddAfterPass(EPass::FXAA, SceneColor);   // your callbacks, unconditionally

That's the reason why the plugin's callback still runs when the project uses TSR and FXAA is off. The bIsPassEnabled argument tells you whether the pass itself is running this frame. In the engine's recommendation, subscribe only when you'll actually produce a GPU resource.

4.1 The contract for a post-processing callback

Every stage uses the same delegate signature. The plugin's callback:

FScreenPassTexture GraphicsCompositing(FRDGBuilder& GraphBuilder, const FSceneView& View, const FPostProcessMaterialInputs& Inputs);

Source in SynthPass: CoreGraphicsPipeline.h L24, CoreGraphicsPipeline.cpp L53–L113

Three rules to keep in mind while setting a callback:

You receive scene color and must return scene color.
Multiple subscribers to the same stage run in priority order, and each one gets the previous one's output.
OverrideOutput is only valid if you're last. For the later stages (from ReplacingTonemapper onwards), your callback may be the final pass that writes to the back buffer. In that case Inputs.OverrideOutput is valid and you must write into it. Otherwise it's invalid and you create your own output (Creates a black screen on viewport).

GraphicsCompositing adds the mask pass and then hands off to FCompositorCS::ImplementCompShader, which handles the output. Trimmed to the parts that follow the contract:

// CompositorCS.cpp
FScreenPassTexture FCompositorCS::ImplementCompShader(FRDGBuilder& GraphBuilder, const FSceneView& SceneView,
    const FPostProcessMaterialInputs& Inputs, /* ...mask, media texture, blend values... */)
{
    // Rule 1: take scene color in
    const FScreenPassTexture& SceneColor = FScreenPassTexture::CopyFromSlice(
        GraphBuilder, Inputs.GetInput(EPostProcessMaterialInput::SceneColor));

    // Rule 3: write into OverrideOutput if we're last, otherwise make our own target
    FScreenPassRenderTarget Output = Inputs.OverrideOutput;
    if (!Output.IsValid())
    {
        Output = FScreenPassRenderTarget::CreateFromInput(
            GraphBuilder, SceneColor, ERenderTargetLoadAction::ENoAction, TEXT("CompositorViewportOutput"));
    }

    // Compositing compute pass writes OutputTexture, then it's copied into the output render target.
    // (Chapter 3 covers writing the shader and adding the pass.)

    AddDrawTexturePass(GraphBuilder, SceneView, FScreenPassTexture(OutputTexture), Output);

    // Rule 1: hand scene color back
    if (bShallCompOutputOnViewport)
    {
        return FScreenPassTexture(Output);
    }
    return SceneColor;
}

Source in SynthPass: CompositorCS.cpp L14–L140

5. What's next

This chapter covered where your work can go in the frame. It skipped what that work is: the shader, its parameter struct, and how a pass is added to the graph. Chapter 3 covers that with compute shaders: declaring a global shader in a plugin, binding its parameters, dispatching it through RDG from one of the hooks above, and what happens between FComputeShaderUtils::AddPass and threads running on an SM.

Unreal Engine GPU Stack: Chapter 1 World Building and Context Setup

Saish — Sun, 18 Oct 2026 20:56:23 +0000

In the prologue, moving a per-pixel workload from Tick into a compute shader took a scene from 7 FPS to 65 FPS. That result worked. What it didn't give me was an explanation. It was a Blackbox for me. I knew the shader ran, but I couldn't say which thread dispatched it, when it actually executed on the GPU.

Before writing more GPU code, I needed some kind of a world context building. What happens between "I changed a value on an actor" and "a pixel changed on screen"? What's happening inside at a lower level?

This chapter builds that world along with fundamentals. There isn't much shader code here. The goal is to have a mental model solid enough that when something goes wrong in later chapters, you can guess where it went wrong.

1. Bit of high school level of boring history for better context

Initially, the CPU did all the math (SISD) needed to draw graphics. It worked, but became too slow as graphics got more complex.

People realized that graphics requires doing the "same calculations many times with different data on different threads" (SIMT) ( moving vertices, multiplying matrices, calculating pixels, and so on). So they built specialized hardware that could do many of these calculations "at the same time".

That specialized hardware eventually became what we know today as "GPUs".

1.1 The narrative SISD vs SIMD vs SIMT

SISD (Single Instruction, Single Data): One instruction works on one piece of data at a time. This is the classic scalar CPU model... add two numbers, then the next two, then the next.

SIMD (Single Instruction, Multiple Data): One instruction works on a small fixed-width vector of data at once, such as 4, 8 or 16 floats with SIMD units on a CPU. You (or the compiler) pack the data into vector registers explicitly. (registers are the most smallest, fastest and closest data storage to core or hardware that does compute)

SIMT (Single Instruction, Multiple Threads): The GPU's model. Threads aren't scheduled individually. They're bundled into groups of 32 (a warp on NVIDIA) or 32/64 (a wavefront on AMD), and every thread in the group executes the same instruction at the same time, each on its own data. You write the shader as if it were one thread handling one pixel, and the hardware runs it 32 or 64 at a time.

2. Character building: Understanding the hardware and its workings technically, CPU, GPU and PCIe bridge

Before looking at Unreal's graphics system, it helps to know what they are ultimately feeding. A CPU and a GPU are both processors that fetch instructions and do math on data, but they were built for different purposes. The CPU is built to finish one task as fast as possible. The GPU is built to finish as many tasks as possible in a given time.

2.1 The CPU: optimized for latency

A modern desktop CPU has somewhere between 8 and 24 physical cores, running at roughly 4 to 6 GHz. Each core is a very complex machine. It predicts which way branches will go before they're evaluated, reorders and schedules instructions so independent work runs while it waits on slow work, and runs several instructions per clock cycle in a single core through multiple execution sub units.

Most of that complex stuff above exists to hide one problem: memory is slow. Reading from main memory takes around 70 to 100 nanoseconds, which is several hundred clock cycles staying idle. So CPUs has dedicated hardware for caches.

Data moves between these levels in 64-byte cache lines, not individual values (like tetris but all blocks are horizontal lines). Read one float and its fifteen neighbors come along for free. Code that reads memory in order gets that benefit, while code that jumps around (pointer chasing through randomly spawned actors and components, for example) keeps paying the full trip to DRAM.

The CPU is great at branchy, unpredictable, serial logic i.e. gameplay, AI, physics queries, the scene management the Render thread does. But it doesn't have width. Eight cores with 8-wide SIMD compute unit gives you roughly 64 floats being processed per cycle at best. A 4k image has just over eight million pixels. That's the workload from the prologue, and that's why it hit a wall.

2.2 The GPU: optimised for throughput

A GPU does things differently. It runs at lower clocks, roughly 1.5 to 2.8 GHz, and each individual core is simple... no branch prediction like CPU, small caches per core. Instead what it has is a huge number of small parallel computing units. Modern highend GPUs have thousands of them, grouped into larger units such as Streaming Multiprocessors (NVIDIA) or Compute Units (AMD). This lets them run tens of thousands of threads at the same time.

A modern GPU is more than just thousands of compute cores. It contains different types of hardware, each designed for a specific kind of work. CUDA/Shader cores handle general calculations and graphics, Tensor Cores accelerate AI and matrix operations, RT Cores speed up ray tracing, while dedicated hardware handles tasks like video encoding, decoding, and data transfers. Together, these specialised parts make GPUs efficient at handling many different workloads in parallel.

All of this just to achieve SIMT!

Few questions that clears the fog about GPU, streaming multiprocessors and warps/wavefont.

is a streaming multiprocessor actual hardware on GPU? Yes. A Streaming Multiprocessor (SM) is an actual physical hardware block inside an NVIDIA GPU. it is structured like this.
is warp just a driver level abstraction?
No. A warp is basically a GPU hardware/architecture concept, exposed to programmers through programming. There is a specific warp scheduler hardware (orange in above pic) that handles thread allocation. If a SM has 1024 threads then it can 32 warps.
can a warp contain threads from two different SM?
No. A warp stays entirely on one SM.
does rendering and compute shaders run on same SM's?
Yes. Graphics shaders and compute shaders generally use the same SMs/execution resources on a GPU. Although there is dedicated hardware to perform some fixed function operations.

There are more nuances to this, we will address them in detail when we cover compute shaders in future chapters.

2.3 The bridge: PCIe

On a desktop PC, the CPU and a discrete GPU are separate devices with separate memory, connected by PCI Express. This is the narrowest pipe in the whole system.

A 4K render target in a 16-bit float format is about 66 MB. Copying it across PCIe 4.0 takes around 2.6 ms, roughly 16% of a 60 FPS frame budget, before the GPU has done any work on it. The same texture read from VRAM by a shader is a lot cheaper. The lesson is simple... data should cross the bus as rarely as possible. Upload once, keep it on the GPU, and only bring results back when something on the CPU truly needs them.

Bandwidth is not the only cost. Every transfer and every dispatch also has fixed overhead... driver work, command submission, synchronization. For small workloads that overhead dominates, which is why a tiny compute shader can end up slower than doing the same work on the CPU.

It's also why the two processors don't talk to each other directly. The CPU writes commands into a buffer, and the GPU consumes that buffer later, usually a frame or more behind. Asking the GPU for an answer "right now" means the CPU has to stop and wait for the GPU to catch up, and the GPU sits idle until the CPU gives it more work. That's the same stall that shows up later as FlushRenderingCommands() and as naive readback, and the whole thread architecture in the next section exists to avoid it.

Not every platform works like this. Consoles, Apple Silicon and most mobile chips use unified memory, where CPU and GPU share the same physical RAM. There's no PCIe copy, but they compete for the same bandwidth (most likely unified memories have lower speeds than dedicated vram and HBMs). Mobile GPUs also usually use tile-based rendering, which processes the screen in small tiles held in on-chip memory. That changes which techniques are cheap and which are expensive. This series targets desktop hardware, but it's worth remembering that "the GPU" is not one architecture.

3. Three threads and a few transfers

The first thing that changed how I read Unreal's rendering code was realising that a frame isn't produced by one thread. There are literally three threads involved.

The Game thread runs gameplay... actors tick, components update, Blueprints execute (something similar to updating UBO info in Vulkan terms). The Render thread takes the resulting scene state and decides what to draw and in what order (Holds and manages vertex proxies or host(cpu) native/ staging VBO in vulkan terms). The RHI thread translates those decisions into actual D3D12, Vulkan or Metal commands and submits them to the driver. The GPU then executes them on its own schedule.

These threads are pipelined, not synchronised step by step. While the Render thread is building frame N, the Game thread is usually already working on frame N+1, and the GPU may still be finishing frame N-1. That overlap is where most of the engine's throughput comes from, and it's also where most of the confusing bugs come from.

The consequence is that the Game thread can't simply reach into the renderer and change things. Any data the renderer needs has to be handed over, and the Game thread has to keep going without waiting. Unreal does this in two ways.

For components that draw something, the transfer is a scene proxy. A UPrimitiveComponent creates a render-thread-owned copy of the state needed to draw it, generally a subclass of FPrimitiveSceneProxy. When the component's transform or render state changes, the engine pushes an update to that proxy instead of letting the renderer read the component directly. The renderer only ever sees proxies. That separation is why a component can be garbage-collected on the Game thread while its last frame is still being drawn.

For everything else, there's ENQUEUE_RENDER_COMMAND, which queues a lambda to run on the Render thread at some later point.

const FLinearColor Tint = ComputeTintOnGameThread();

ENQUEUE_RENDER_COMMAND(UpdateMyTint)(
    [Proxy = MyProxy, Tint](FRHICommandListImmediate& RHICmdList)
    {
        // Runs on the Render thread, possibly a frame or more later.
        Proxy->SetTint_RenderThread(Tint);
    });

The important detail is the capture list. The lambda runs after the calling function has returned, so anything captured by reference is likely gone by then. Capturing by value, or moving data into the lambda, is what actually carries the state across the thread boundary. Sometimes if you are sure that a really large object wont be garbage collected or killed or accessed by the game thread, you can just pass it as reference inside the lambda. It's unsafe but we cut overhead of creating a copy of the object.

This is also where the problem from the prologue came from. FlushRenderingCommands() makes the Game thread wait until the Render thread has processed everything queued so far. It's a correct way to get a consistent view of render state, which is why AI generated code reaches for it. But it collapses the pipeline... for as long as the flush takes, the Game thread and the Render thread are no longer overlapping. That was the 7.5ms. It wasn't a bug in terms of output. It was a design flaw that was correct and slow at the same time.

4. Unreal is already a graphics library

Unreal allows us to declare and do GPU calls which is similar to any low level graphics library but is abstracted by two layers. This is very similar to WGPU as Unreal's graphics library translates calls into native low-level GPU drivers like Vulkan, Metal, DirectX 12, or WEB GPU (some people have already done it!).

Once the data reaches the Render thread, it goes through two abstraction layers before it becomes GPU work.

The lower one is the RHI, the Render Hardware Interface. It's Unreal's portable graphics API... textures, buffers, shaders, pipeline state, draw and dispatch calls, expressed once and implemented per backend. Code talks to the RHI mostly through command lists (FRHICommandList), which record commands that the RHI thread later translates into native API calls. Each RHI also reports a feature level, such as SM5, SM6 or ES3_1 for mobile, which decides what the hardware can be asked to do. That matters a lot once compute shaders enter the picture.

Working directly at the RHI level feels familiar if you come from Vulkan or D3D12, and it comes with the same responsibilities. You create the resources, you track their lifetimes, and you issue the transitions that tell the GPU a texture is moving from "being written" to "being read."

The layer above it is the Render Dependency Graph (RDG), and this is where most modern rendering code in Unreal actually lives. Instead of issuing commands immediately, you describe a frame's worth of work as a graph of passes. Each pass declares, through its parameter struct, which resources it reads and which it writes.

So the stack looks like this

This is the realisation I mentioned in the prologue. The question isn't whether to use a graphics library in Unreal. The RHI and RDG already are one. The work is learning its rules.

5. Why all this matters?

This chapter didn't add a single line of shader code, but it changed how I and you will read the problems that came later. "My shader doesn't run" stopped being one problem and became several... the shader wasn't found, the pass was culled or the resource was never transitioned, or maybe the result was read before the GPU produced it. Each of those lives in a different layer, and knowing the layers is what makes it possible to tell them apart.

It also changed how I use AI-generated rendering code. The model can write a pass quickly. What I have is is the ability to ask which thread that pass runs on, what it waits for, and whether the profile agrees with what the ai generated code claims to do.

The next chapter uses this world to read an existing plugin... GPUCompositionGraphicsPipeline, which composites a media texture into the rendered frame. Where does it enter the frame, which thread does each part run on, and why is the work declared as passes instead of issued as draw calls?

Unreal Engine GPU Stack: Prologue

Saish — Sun, 11 Oct 2026 20:57:48 +0000

1. First contact: The frame that fell apart

The problem that started this for me was simple. A couple of years ago, I was required to process a lot of data every frame. There was a specific algorithm that I wanted to run on every pixel of each video frame the system received, within a certain timeframe, let's say 20ms.

The obvious approach was the one every Unreal Engine developer reaches for first: loop over the frame's pixel array in Tick, do the calculations, move on. At a few thousand pixels, it was fine. At a few hundred thousand, it was noticeable. Pushing toward 8.3 million pixels, the frame time fell apart. Multithreading could only help so much.

Running stat unit made the problem clear enough. The Game thread was the one blowing the budget, while the GPU was comfortably idle. Unreal Insights confirmed it.. the time was spent in my update loop, doing the same small piece of math for every pixel, one after another.

That's a workload the GPU is built for. So I moved the update into a compute shader, kept the data on the GPU, and profiled again.

The result is the kind of before-and-after (7 FPS to 65 FPS on an RTX 3080) that makes GPU programming look like magic. It isn't. Getting there raised more questions than it answered. Why exactly was it faster? What was Unreal doing with my shader between dispatch and screen? Searching for answers to these questions led me down a deeper rabbit hole into Unreal Engine's GPU side.

That rabbit hole meant hours of deep dives into Unreal Engine source code, since content on this topic is very scarce on the web. I had to go on many side quests, digging into much lower-level graphics APIs like DirectX, OpenGL and sometimes Vulkan. Many concepts from CUDA translated very well while writing compute shaders. Random Chinese blog posts proved to be really good reference points in this odyssey.

Even now, there is still no single good source of information that answers all the questions I had a few years ago.

This series is my attempt to answer those questions properly.

2. Why the GPU stack and not graphics programming?

Graphics programming is about producing images. The GPU stack covers everything that runs on the GPU: compute, data movement, interop and inference. Generating synthetic sensor data is a good example. It needs physics calculations on the GPU that read vertex buffers already living there, plus uploads and readbacks that stay in sync with the frame.

3. Graphics and AI are converging

The GPU in a modern game isn't only rendering triangles anymore. Neural rendering is already shipping: DLSS 5, XeSS and newer versions of FSR use neural networks for upscaling, frame generation reconstructs whole frames, and ray reconstruction and neural denoising replace hand-tuned filters. Behind those, research is moving quickly toward neural materials, neural texture compression, Gaussian splatting, learned radiance caches, world models, and ML-driven animation and physics.

What surprised me was how much of this ends up running on the same GPU, using the same memory and fighting for the same frame budget as the renderer. Once you look past the terminology, the overlap is large. A tensor is a buffer. Running inference is dispatching compute work. Moving data between CPU and GPU costs the same whether it's vertex positions or model weights. The engineer who understands how a frame moves through the GPU is in a good position for both real-time graphics and on-device AI, which is why this series includes a chapter on inference inside the engine rather than treating it as a separate topic.

4. Who is this for?

The same problem looks different depending on where you come from.

Who hits this wall, and how, varies a lot.

If you're a gameplay programmer, it usually shows up as a system that stops scaling. Crowds, projectiles, destruction or simulation work fine until the object count grows, and then Tick and Blueprint loops become the bottleneck. The hard part isn't writing a shader. It's moving the system to the GPU without breaking the gameplay logic that depends on it.

If you're a general software engineer, you might be looking to build a non-gaming software extension or layer with Unreal. The GPU is best understood as a massively parallel processor. The ideas are the same as in CUDA or an ML framework.. batch the work, keep data on the device, avoid round trips. What's different in a game engine is the strict frame budget and the fact that you share the GPU with a renderer that has its own opinions.

If you're a technical artist, the wall is the edge of the material editor. You can see why an effect is expensive but not what's happening beneath the nodes, and some ideas simply can't be expressed as a graph. You might find answers to questions like: Why does adding a Scene Capture Component to the level eat up so much FPS? How can I optimize my work? The short answer is that a scene capture renders the scene again from another view, so all the rendering pipeline calculations and dependencies have to be redone.

And if you're a graphics programmer coming from Vulkan, DirectX 12 or OpenGL, you already understand the GPU. The challenge is mapping that knowledge onto Unreal's abstractions, which don't always look like the APIs you know.

This series tries to serve all four, starting from the problem rather than from the API.

5. Vibe coding without understanding internals

AI assistants can produce a compute shader, a render pass or a Scene View Extension in seconds. That's genuinely useful, and I started to use them recently after a bit of reluctance. But GPU code in Unreal is an especially easy place for generated code to go wrong. Recently, while building a GPU-accelerated data processing pipeline, I prompted Claude Fable with a good amount of context to build the system. It wrote the code for me, the code compiled, and everything worked fine. But when I profiled the system, I found that a specific part of it was taking 7.5ms by blocking the game thread. After searching through a few hundred lines of code, I found a particular line where FlushRenderingCommands() was used, which was unnecessary for this design. FRHIGPUFence would have worked better (I'll explain why in the following chapters). With a simple prompt to rework the system to use FRHIGPUFence properly instead of FlushRenderingCommands(), I was able to cut the processing time on the game thread for that system from 7.5ms to 2.3ms.

Part of the reason is specific to Unreal.. its rendering APIs change between 5.x releases. Functions get renamed, deprecated or moved, and the model's training data contains all of those versions at once. The generated code often mixes an older pattern with a newer one, and the result may or may not compile.

The more dangerous case is when it does compile. Code that compiles is not the same as code that works. A render pass can build cleanly and output a black screen. A readback can work perfectly and quietly stall the frame. Each of those looks like one symptom but has a different root cause, and without a mental model of the layers underneath, you can't tell which one you're looking at. Worse, you can't tell a correct answer from a confident wrong one.

6. The OpenGL vs Unreal beginner's dilemma

Experts often suggest going with OpenGL as your first graphics API to learn the concepts. While cross-referencing a lot of rendering content from OpenGL, I had a big decision to make. As a beginner, I found that OpenGL had easily available learning resources, while Unreal's were scarce and the learning curve was steep. As a person with a toxic habit of doing things the difficult way, I chose Unreal, and after a few months, I had a realization.

The realization that changed my approach was that Unreal already is a graphics library. The RHI (Render Hardware Interface) is an abstraction over D3D12, Vulkan and Metal. The Render Dependency Graph (RDG) sits on top of it and manages resources and passes. The question isn't whether to use a graphics library. It's whether to learn the one already shipping in the engine.

If you already know Unreal Engine, go with it as your graphics library using this series as learning material, since it's easier to build games and projects when the rest of the tooling is already available. If you're starting from the beginning without any game engine experience, I'd probably start with OpenGL.

7. What this series covers

Each chapter starts from a problem and builds a solution on the GPU. The chapters roughly follow the order in which I had to learn things, from the fundamentals of how Unreal renders a frame, to running neural networks inside it.

GPU world building and context setup. The foundations the rest of the series builds on: rendering fundamentals, the Game, Render and RHI threads, and what RHI and RDG actually do.
Scene View Extensions. How to inject your own rendering pass into Unreal's frame without modifying the engine.
Compute shaders. Writing a shader in a plugin, dispatching it through the Render Dependency Graph, and running general-purpose work on the GPU.
CPU-GPU data upload and readback. Getting data onto the GPU and results back, and why a readback that works can still stall the frame.
Combining compute, vertex and pixel shaders for particle effects. Simulating on the GPU with compute and drawing the results without ever copying them back to the CPU.
GPU mesh and a custom mesh pipeline. How Unreal actually draws a mesh, through scene proxies, vertex factories and mesh batches, and how to build your own.
GPU vertex animation. Animating thousands of meshes by doing the work in the vertex pipeline.
GPU physics. Running simulation on the GPU, and the problems that appear when gameplay needs to know the results.
GPU-accelerated cross-API interoperability. Sharing GPU memory between Unreal and other APIs, using video streaming as the example.
GPU-accelerated AI inference. Running neural networks inside the engine, on the same GPU and within the same frame budget as the renderer.
Profiling and debugging. Unreal Insights, GPU profiling, RenderDoc and RDG's debugging tools, and how to tell whether something is actually faster and why, or why it isn't running at all. It's listed last, but it's worth reading whenever the numbers stop making sense.

Epilogue: What's next. Where this leads, what's possible, and projects and examples worth studying.

If you have questions along the way, send them. They'll shape the FAQ at the end of future chapters.