Abusing Steam Overlay To Create Arbitrary Detours

As we know, in order for the steam overlay to work it needs to hook the game's rendering pipeline.

The first step I took was to determine which module is responsible for hooking said pipeline. Upon a quick glance in Process Hacker it is 'GameOverlayRenderer.dll'.

List of modules that are active in the game. Note: Throughout this article, the game we are targeting for this POC is 'Half-Life 2: Deathmatch' with -insecure.

Next, I popped the module into IDA and began searching for relevant strings and very quickly found interesting results.

Search result for 'hook' in GameOverlayRenderer.

After generating the psuedocode, we are lucky enough to have warnings that tell us what the first two arguments are.

The first two arguments, as annotated by the warnings.

If we follow x-refs to this function, we only find one, which is a wrapper for this function with one less argument. This function is what the rest of the process calls to process hooks so we'll focus on using it.

Aforementioned wrapper. From now on this will be referred to as HookFuncWrapper.

Once we inspect how valve uses this function, the arguments become clear.

How valve hooks D3D9 using this function.

Storing and using this function is pretty straightforward.

auto hook_func_wrapper = reinterpret_cast< char( __cdecl* )( unsigned __int8*, int, LPCVOID*, int ) >( util::pattern::find( "GameOverlayRenderer.dll", { 0x55, 0x8B, 0xEC, 0x51, 0x8B, 0x45, 0xAE, 0xC7 } ) );