Abusing c_variadic in Rust

What is c_variadic?

c_variadic is a recently stabilized feature in Rust that lets you send variadics through C/C++ FFI interop.

The best example is with int printf(const char* format, ...): The ... are your variadics and allows you to pass any number of parameters.

You can utilize it like this in Rust:

unsafe extern "C" {
    fn printf(format: *const i8, v_params: ...) -> i32;
}

pub fn main() {
    unsafe {
        printf(c"My magic number is: %d".as_ptr(), 1234);
    }
}

Backstory

When I was developing a cheat for a video game I eventually grew sick of users requesting new features; I'm not a machine - I chose to implement the Rune scripting language into it.

It worked great at the start, did its job and it opened up the potential for custom community-driven sub-cheat menus.

Although as the project itself shifted to being a cheat framework, this didn't meet my own demands, which were:

1. Injecting into unprotected games via LoadLibraryA - ✅
2. Hooking ImGui based on the game renderer (using hudhook, great project) - ✅
3. Adding WinAPI Memory R/W functions, alongside with pattern scanning and such - ✅
4. Adding runtime ImGui UI building into Rune scripting - ✅
5. Adding function hooks and function pointer calling into Rune scripting - 💀

Yeah... 2 challenges from it:

1. How do I add function calling?
2. How do I hook functions and call back to Rune?

Implementation

I first tried to tackle the first problem, I could dynamically cast a Rune Value into its native type by doing loads of type-checking and conversions.

So I tried to just call a function with 1 parameter using it and casting the desired function pointer into a extern "C" fn(p_0: *const std::ffi::c_void) -> *const std::ffi::c_void

This worked and the return value was obtainable & sent back to the calling Rune function.

Multi-parameter functions

Not every function is a single parameter. In fact, almost none are in a game.

I can collect the parameters from a Rune function by having the user just pass them all into a large Vec<Value> and then doing type-conversion; that part is solved.

But how would I call functions with varying amount of parameters? I can't change the extern-syntax at runtime; Rust is typed.

Abusing c_variadic

With c_variadic available, I first tried to just force a random function as unsafe extern "C" fn(...) -> *const std::ffi::c_void - and prayed.

The result? It worked but with one caveat: It only works on 64-bit and that's mainly because 32-bit often use different calling-conventions, like thiscall, fastcall and whatnot.

Implementing it in Rune

Rune obviously doesn't have variadics like that, and shouldn't.

So... how should it be implemented?
Simple: match params_vec.len() { ... } - account for a n amount of parameters.

I opted for 15 parameters, making a large match-statement and getting the next parameter from the vector and passing it in.

The result? It worked - but with the downside that we have absolutely no idea how many parameters a function pointer can actually take, so a user could pass in 10 into a function that only takes 2.

The solution? Let them blow their foot off.

Function Hooks

The first problem was solved, now this... The library I was using for hooking was retour in case you were wondering btw.

Where do hooks go?

This was probably the biggest problem of it all: Where do they get sent off to? - Rune as mentioned, is a VM-based language and we can't just redirect it to a VM pointer.

The (ugly) solution

I opted for creating a struct which held data, here's a 1:1 representation of it:

/// Holds the information about a Rune detour.
#[derive(Default)]
pub struct RDetour {
    /// The ID of the detour.
    detour_id: u8,

    /// The pointer of which function will be treated as target, and will be redirected to a
    /// determined detour holder from `determine_detour_holder()`.
    /// If `None`, this detour isn't ready to be used and is free to be acquired.
    from_ptr: Option<isize>,

    /// The `RawDetour` instance.
    /// If `None`, this detour isn't ready to be used and is free to be acquired.
    detour: Option<Box<RawDetour>>,

    /// Rune function to be called as a callback, should return a `isize` of the original functions
    /// return value as a pointer, or a modified value if needed.
    /// If `None`, this detour isn't ready to be used and is free to be acquired.
    rune_function: Option<SyncFunction>,

    /// Optional paramater to be passed into `rune_function` upon callback.
    /// Can be a structure for example, so that variables can be updated.
    opt_param: Option<ValueWrapper>,
}

You may notice one special type: ValueWrapper - and that's just a standard Rune Value but with force-implemented Send + Sync, and that's for one ugly reason: Every RDetour struct instance was stored globally.

It was behind an Atomic RefCell which helped a ton, but yeah not the code you'd typically wanna be writing.

To actually have a real, native function that the hook can call back to, I opted for creating a macro that generated 35 "detour holder" functions that all registered themselves linearly, which then created a new RDetour struct instance and saved it globally.

Installing new hooks from Rune

To install new hooks from within Rune, I ran through these steps:

1. Find the first-available detour out of the 35 total ones
2. Switch out rune_function to an user-defined Rune-level callback that the hook would call
3. Switch out from_ptr to what the user specified
4. Assign opt_param if the user has defined a custom Rune Value to access within rune_function
5. Create the hook

If a step failed, then it would log the error visually.

Processing the incoming data

Since we have native hook/detour holders, all data goes to native stub-like functions.

Once data was received, the stub had generated code that would instruct it to look up the hook that was set on its ID.

That helper function (call_rune_function_on_id) would then get the Rune callback function and call it with these parameters:

1. Trampoline pointer the user should call themselves to continue control-flow. Skip it and the function will just be a glorified return;
2. A 10-param-sized tuple, the closest to a variadic in Rune. The parameters were - you guessed it - obtained by having every stub-function have a c_vardiadic body
3. The optional Rune Value so the user can utilize it, if desired

Once the user utilized the newly-added fn_ptr_call function on the trampoline pointer and the right parameters, it would continue on as expected.

The users were often instructed to see how many parameters the function actually wanted, and not just blow off the whole hill with 10.

They often didn't listen (neither did I, I'm lazy) - and it worked for 99% of all cases.

Conclusion

While Rust is a safe language by design, it very much is capable of the same things as C/C++ if you know what you are doing.

Unsafe Rust shouldn't be frowned upon, it should be taught and studied, there's tons of neat tricks that you can utilize.