DEV Community: Aviram Hassan

Fun with macOS's SIP

Aviram Hassan — Thu, 22 Aug 2024 13:49:10 +0000

While developing mirrord, which heavily relies on injecting itself into other people's binaries, we ran into some challenges posed by macOS’s SIP (System Integrity Protection). This post details how we ultimately overcame these challenges, and we hope it can be of help to other people hoping to learn about SIP, as we've learned the hard way that there's very little written about this subject on the internet.

What is mirrord?

mirrord lets you run a local process in the context of a cloud service, which means we can test our code on our staging cluster without actually deploying it there. This leads to shorter feedback loops (you don’t have to wait on long CI processes to test your code in staging conditions) and a more stable staging environment (since untested services aren’t being deployed there). There is a detailed overview of mirrord in this blog post.

What is SIP and why does mirrord care about it?

Apple introduced SIP in 2015 to prevent tampering with system binaries because those usually have high permissions and entitlements.
mirrord works by injecting its library (.dylib or .so) into the local process, the one you want to “mirror” into the cloud. In order to inject itself into binaries, it uses LD_PRELOAD on Linux and DYLD_INSERT_LIBRARIES on macOS.
One of the features of SIP is that it disallows the use of DYLD_INSERT_LIBRARIES with protected binaries. Bummer - we can’t use mirrord to locally run e.g. bash or ls in a cloud context. we can’t run mirrord exec bash or mirrord exec ls. We’ll have to find a way to get around SIP!

Detecting if a binary is SIP-protected

In order to start bypassing SIP, we needed to find a way to check if a binary is even SIP protected to begin with. We initially used pretty coarse heuristic, assuming that a binary is SIP-protected if it’s in one of these locations:

/System
/bin
/usr
/sbin
/var

However, we found that the “stat” function can return a flag called RESTRICTED which we read could be related, and decided to use that instead¹. The code is quite simple:

       let metadata = std::fs::metadata(&complete_path)?;
       if (metadata.st_flags() & SF_RESTRICTED) > 0 {
           return Ok(SipStatus::SomeSIP(complete_path, None));
       }

Shebang!

Detection was simple enough when mirrord was run directly on a SIP-protected binary. However, we soon ran into a less trivial (but common) scenario - an interpreter script that starts with a shebang pointing to a SIP-protected binary, for example yarn/npm/pyenv. If we look at npm for instance, it points to a file which starts with the following code:

#!/usr/bin/env node
require('../lib/cli.js')(process)

In this example it will execute env, which will execute node with the next line.
The problem? /usr/bin/env is SIP protected, meaning it will strip our DYLD_INSERT_LIBRARIES then run node without mirrord. Thanks for nothing SIP!
So we also needed to check whether the file is a “shebang” file (i.e starts with #!), what file the shebang points to, and whether that file is a SIP-protected binary.

Bypassing SIP

Now that we found a way to detect whether we’re being run on a SIP-protected binary, we need to figure out how to bypass SIP and let mirrord load into the binary with DYLD_INSERT_LIBRARIES. When googling around, we found people saying you can bypass SIP by copying the binary to another directory and re-signing it. We found that to be partially true.
Why partially? Because if you tried to do it on Apple Silicon (arm), it wouldn’t work. This is because beginning with M1, macOS ships with arm64e binaries. The e indicates an arm64 extension that adds pointer authentication. It’s another security measurement added by Apple (kudos to Apple for having great security, too bad it affects us).
We won’t go into details about what pointer authentication does, but you can read more about it here.
So why was this a problem? First, mirrord is written in Rust, which doesn’t support compiling arm64e binaries. The other problem is that only Apple-signed binaries can run with arm64e architecture.
This is what happens if we try the “old” trick:
➜ /tmp cp /usr/bin/env /tmp/env
➜ /tmp codesign -f -s - /tmp/env
/tmp/env: replacing existing signature
➜ /tmp /tmp/env
[1] 20114 killed /tmp/env

Killed! And it was so young. :(
Recording using Console (macOS’s built in log viewer) while running the binary reveals the reason:

From Apple’s point of view, arm64e is preview only, i.e the ABI can change and they don’t want a third party building on top of it, as it’s not guaranteed to work. You can enable running third party executables with arm64e ABI only if you boot into recovery mode and change the settings, which is not something we want to ask our users to do.

Handling arm64e

Initially we tried to convert the arm64e ABI into arm64 on the fly. Yes, people familiar with how this ABI works probably think we’re insane, but we were optimistic.. and it actually worked! for example, if you take /usr/bin/env and just change the file headers to say it’s arm64, you’d be able to re-sign it and run it normally! We actually do it for our binaries to be able to load to arm64e binaries:

# from our release.yaml https://github.com/metalbear-co/mirrord/blob/main/.github/workflows/release.yaml
    - name: build mirrord-layer macOS arm/arm64e
      # Editing the arm64 binary, since arm64e can be loaded into both arm64 & arm64e
      # >> target/debug/libmirrord_layer.dylib: Mach-O 64-bit dynamically linked shared library arm64
      # >> magic bits: 0000000 facf feed 000c 0100 0000 0000 0006 0000
      # >> After editing using dd -
      # >> magic bits: 0000000 facf feed 000c 0100 0002 0000 0006 0000
      # >> target/debug/libmirrord_layer.dylib: Mach-O 64-bit dynamically linked shared library arm64e
      run: |
        cargo +nightly build --release -p mirrord-layer --target=aarch64-apple-darwin
        cp target/aarch64-apple-darwin/release/libmirrord_layer.dylib target/aarch64-apple-darwin/release/libmirrord_layer_arm64e.dylib
        printf '\x02' | dd of=target/aarch64-apple-darwin/release/libmirrord_layer_arm64e.dylib bs=1 seek=8 count=1 conv=notrunc

It didn’t work for all binaries though (ls for example) and when we started digging we found out that there are a lot of new features being used in arm64e, for example specific relocations that contain pointer authentication stuff. We decided to give up on ABI conversion for the time being.
Luckily, Apple ships fat binaries on both architecture machines. Fat binaries are Apple’s name for Mach-O files containing two different binaries, each built for a different architecture, so the runtime can decide which one it will use. By default, it will choose arm64e, but we can do something nice with the x64 binary.

```file /bin/ls
/bin/ls: Mach-O universal binary with 2 architectures: [x86_64:Mach-O 64-bit executable x86_64] [arm64e:Mach-O 64-bit executable arm64e]
/bin/ls (for architecture x86_64): Mach-O 64-bit executable x86_64
/bin/ls (for architecture arm64e): Mach-O 64-bit executable arm64e



The idea is that we take the binary we want to load ourself into, extract only the x64 binary (on arm), re-sign it, and run it. The only downside here is that we require Rosetta[^2] to be installed on the system and there’s a performance impact - but usually system binaries are used for simple operations like `env` or `bash` (see the shebang case).

## Putting it all together

1. Detect SIP (Shebang/Restricted)
2. Patch

    a. Extract x64 binary into a new file

    b. chmod +x it

    c. Sign it

3. Execute



```rs
   /// Read the contents (or just the x86_64 section in case of a fat file) from the SIP binary at
   /// `path`, write it into `output`, give it the same permissions, and sign the new binary.
   fn patch_binary<P: AsRef<Path>, K: AsRef<Path>>(path: P, output: K) -> Result<()> {
       let data = std::fs::read(path.as_ref())?;
       let binary_info = BinaryInfo::from_object_bytes(&data)?;


       let x64_binary = &data[binary_info.offset..binary_info.offset + binary_info.size];
       std::fs::write(output.as_ref(), x64_binary)?;
       // Give the new file the same permissions as the old file.
       std::fs::set_permissions(
           output.as_ref(),
           std::fs::metadata(path.as_ref())?.permissions(),
       )?;
       codesign::sign(output)
   }

Integration into mirrord took two steps:

When using mirrord directly on a SIP-protected binary, do the patch
When using mirrord on a process, and that process executes a SIP-protected binary, replace it on the fly. This was done by having mirrord hook execve in the process

execve hook:


/// Hook for `libc::execve`.
///
/// Patch file if it is SIPed, use new path if patched.
/// If any args in argv are paths to mirrord's temp directory, strip the temp dir part.
/// So if argv[1] is "/var/folders/1337/mirrord-bin/opt/homebrew/bin/npx"
/// Switch it to "/opt/homebrew/bin/npx"
/// then call normal execve with the possibly updated path and argv and the original envp.
#[hook_guard_fn]
pub(crate) unsafe extern "C" fn execve_detour(
   path: *const c_char,
   argv: *const *const c_char,
   envp: *const *const c_char,
) -> c_int {
   // Do unsafe part of path conversion here.
   let rawish_path = (!path.is_null()).then(|| CStr::from_ptr(path));
   let mut patched_path = CString::default();
   let final_path = patch_if_sip(rawish_path)
       .and_then(|s| match CString::new(s) {
           Ok(c_string) => {
               patched_path = c_string;
               Success(patched_path.as_ptr())
           }
           Err(err) => Error(Null(err)),
       })
       .unwrap_or(path); // Continue even if there were errors - just run without patching.


   let argv_arr = Nul::new_unchecked(argv);


   // If we intercept args, we create a new array.
   // execve takes a null terminated array of char pointers.
   // ptr_vec will own the vector that will be passed to execve as an array.
   let mut ptr_vec: Vec<*const c_char> = Vec::new();
   let final_argv = intercept_tmp_dir(argv_arr)
       .map(|new_vec| {
           ptr_vec = new_vec; // Make sure vector still lives when we pass the pointer to execve.
           ptr_vec.as_ptr()
       })
       .unwrap_or(argv);

// Call execve's default implementation
   FN_EXECVE(final_path, final_argv, envp)
}

Bonus content: Why is this possible?

You might be asking yourself, “If this is a security feature by Apple, why is it possible to just bypass it that way?” The answer is that we do not bypass the security feature, just the problem it created for us. Apple operating systems have the concept of “entitlements”, which are definitions of which special operations an executable is allowed to perform, and which special resources it should have access to. Before released applications can have entitlements, they need to go through some approval process with Apple. Shared libraries get the entitlements of the host executable, so if we could load any library to any process, a non-Apple-approved library could enjoy entitlements it shouldn’t have by loading into an entitled process. That would be a pretty straight forward privilege escalation of that library’s code. SIP and the hardened runtime prevent that from happening.

When we, in our bypassing mechanism, copy an executable and resign it, it loses its entitlements. So it is still guaranteed that our dynamic library could not run with any ungranted entitlements. The integrity of granted entitlements is preserved.
The loss of entitlements is not a problem for mirrord, because we do not expect to ever execute with mirrord any application that requires Apple entitlements. So we give up the mirrored application’s entitlements (which do not exist or are not needed), in order to be able to load our library into that application.

Afterword

You’re welcome to check out the full implementation in our GitHub repository, Join our Backend Engineers Discord community, subscribe to our newsletter and of course use mirrord!

Originally posted in MetalBear's Blog in January 24, 2023.

But in fact there might be other locations that are SIP protected, and if someone has a good and reliable way to detect if a binary is SIP, we’d love to hear it. ↩

Hooking Go from Rust - Hitchhiker’s Guide to the Go-laxy

Aviram Hassan — Wed, 12 Oct 2022 11:53:59 +0000

Most mainstream programming languages strive to fit into a few common standards, to increase interoperability and decrease adoption friction. Golang isn’t one of those (there are several articles on the subject). In this blog post we’ll demonstrate how to overcome Go’s isolationist design and integrate with it from another language (in our case Rust).

Why do we need to interop with Go? mirrord works by hooking system calls to the operating systems and applying logic that decides whether to execute locally or remotely. To do that, mirrord side-loads (using LD_PRELOAD) into the process, then hooks relevant functions.
To cover most common scenarios, mirrord hooks libc functions and this works for most common languages (Python, Go on macOS, Rust, Node to name a few) as they all rely on libc.

Mostly Harmless

Golang doesn’t use libc on Linux, and instead calls syscalls directly. This is mostly harmless for the common developer - they don’t care about the assembly, syscalls, linkage, etc - they just want their binary to work. Therefore, being self-contained provides a very good user experience, as Go applications aren’t dependent on the local machine’s libc.

It’s pretty harmful for us, though. Since we explicitly override libc functions, our software simply doesn’t function when run with Go apps (or any other process that doesn’t call libc). Therefore, we must hook Golang functions!

Almost, but not quite, entirely unlike tea

Luckily for us, Go applications are not entirely unlike other software. Golang has to work with the operating system, so it has to use syscalls. Since libc doesn’t add much logic on top of the syscalls it wraps, we can still use all our existing code - we just have to override a different function with it.

How do we hook Golang functions? Same way we do libc functions - with Frida. The problem is that writing Rust code that can work from a Go routine call state isn’t trivial. Go has its own ABI¹, which doesn’t conform to any common ABI. This nonconformance is relatively common, though. For example, Rust also has an unstable internal ABI. If we could recompile the Go binary before side-loading into it, we could use cgo to have standard C ABI accessible, but in our use case we can’t. This means we have to implement a trampoline².

The trampoline will be written in Assembly and its purpose is to translate a Go function call into a Rust function call, then return the result as the caller of the original Go function expected it to return.

Looking at the backtrace of our Go binary and dependencies of the net/http package, it was obvious that it involved the use of the syscall package. By reverse engineering the Go binary using Ghidra, we mapped out the relevant flows (socket, listen, accept, etc.) to three different functions that we need to hook:

syscall.Syscall6.abi0 - syscalls with 6 parameters letting the runtime know we switched to a blocking operation so it can schedule on another thread/goroutine.
syscall.Syscall.abi0 - same as syscall.Syscall6.abi0 but with three parameters.
syscall.RawSyscall.abi0 - same as the above but without notifying the runtime.

Don’t Panic

The Big Jump

Let’s start with a very basic trampoline, hooking syscall.RawSyscall.abi0 (a routine that calls a syscall with 3 parameters, also used by socket in the syscall package). Below is the disassembly of this function:

We will implement this trampoline by moving arguments from the stack to registers as Rust expects in a C ABI, then return the result on the stack as Go expects.

From stack to registers

mov rsi, QWORD PTR [rsp+0x10]
mov rdx, QWORD PTR [rsp+0x18]
mov rcx, QWORD PTR [rsp+0x20]
mov rdi, QWORD PTR [rsp+0x8]

Golang has its own ABI(as mentioned before), precisely ABI0 and ABIInternal. Go keeps backward compatibility with a stack based calling convention along with the recently introduced register based calling convention. Turns out ABI0 functions follow a stack based convention, which is why we move values from the stack rather than registers.

Calling the handler

call c_abi_syscall_handler

Following the stack based convention from Go, we move the arguments to registers. But what registers exactly and why? Since we’re hooking a function that directly makes the syscall, we would require a handler to manage the syscalls for us. Our handler will be called using the C ABI calling convention, it will match on syscalls and redirect them based on their type to their specific detours and return the result in the specific register conforming to the C ABI.

#[no_mangle]
unsafe extern "C" fn c_abi_syscall_handler(
    syscall: i64,
    param1: i64,
    param2: i64,
    param3: i64,
) -> i32 {        
    let res = match syscall {        
        libc::SYS_socket => {
            let sock = libc::socket(param1 as i32, param2 as i32, param3 as i32);            
            sock
        }
        _ => libc::syscall(syscall, param1, param2, param3) as i32,
    };
    return res;
}

Putting it back on the stack for Go

asm_linux_amd64.s:

// func Syscall(trap int64, a1, a2, a3 uintptr) (r1, r2, err uintptr);
// Trap # in AX, args in DI SI DX R10 R8 R9, return in AX DX
// Note that this differs from "standard" ABI convention, which
// would pass 4th arg in CX, not R10.

As mentioned above and as we saw in the disassembly, we will move the result returned by the handler back to the stack like so:


mov  QWORD PTR [rsp+0x28],rax
mov  QWORD PTR [rsp+0x30],rdx
mov  QWORD PTR [rsp+0x38],0x0
ret

Summing it up

#[cfg(target_os = "linux")]
#[cfg(target_arch = "x86_64")]
#[naked]
unsafe extern "C" fn go_raw_syscall_detour() {
    asm!(
        "mov rsi, QWORD PTR [rsp+0x10]",
        "mov rdx, QWORD PTR [rsp+0x18]",
        "mov rcx, QWORD PTR [rsp+0x20]",
        "mov rdi, QWORD PTR [rsp+0x8]",
        "call c_abi_syscall_handler",
        "mov  QWORD PTR [rsp+0x28],rax",
        "mov  QWORD PTR [rsp+0x30],rdx",
        "mov  QWORD PTR [rsp+0x38],0x0",
        "ret",
        options(noreturn),
    );
}

Note the usage of the Naked function feature. Naked functions give us full control over the generated assembly(as needed in our use case) since Rust doesn’t generate an epilogue/prologue for them.

Let’s do a sample run and see if everything works:

Great! It works just as we expected. However, the actual detours in mirrord contain logs and do a lot of book-keeping. Let’s start by adding a simple debug statement and see where things go.

#[no_mangle]
unsafe extern "C" fn c_abi_syscall_handler(
    syscall: i64,
    param1: i64,
    param2: i64,
    param3: i64,
) -> i32 {    
    debug!("c_abi_sycall_handler received syscall: {syscall:?}");
    let res = match syscall {        
        libc::SYS_socket => {
            let sock = libc::socket(param1 as i32, param2 as i32, param3 as i32);            
            sock
        }
        _ => libc::syscall(syscall, param1, param2, param3) as i32,
    };
    return res;
}

And action!

mehula@mehul-machine:~/golang-e2e/server$ LD_PRELOAD=../target/debug/libmirrord.so ./server
2022-08-15T17:15:36.497241Z DEBUG mirrord: LD_PRELOAD SET
2022-08-15T17:15:36.498403Z DEBUG mirrord: "syscall.RawSyscall.abi0" hooked
Server listening on port 8080
2022-08-15T17:15:36.505606Z DEBUG mirrord: c_abi_sycall_handler received syscall: 41
2022-08-15T17:15:36.505689Z DEBUG mirrord: c_abi_sycall_handler received syscall: 41
2022-08-15T17:15:36.505738Z DEBUG mirrord: c_abi_sycall_handler received syscall: 41
unexpected fault address 0x0
fatal error: fault
[signal SIGSEGV: segmentation violation code=0x80 addr=0x0 pc=0x7fa45a87f6b2]

goroutine 1 [running]:
runtime.throw({0x7e0b21?, 0x46?})
        /usr/local/go/src/runtime/panic.go:992 +0x71 fp=0xc0002372a8 sp=0xc000237278 pc=0x4354b1
runtime: unexpected return pc for runtime.sigpanic called from 0x7fa45a87f6b2
stack: frame={sp:0xc0002372a8, fp:0xc0002372f8} stack=[0xc000218000,0xc000238000)
0x000000c0002371a8:  0x000000000045551b <runtime.write+0x000000000000003b>  0x0000000000000002 
0x000000c0002371b8:  0x000000c0002371f0  0x0000000000436bae <runtime.recordForPanic+0x000000000000004e> 
0x000000c0002371c8:  0x000000000045551b <runtime.write+0x000000000000003b>  0x0000000000000002 
0x000000c0002371d8:  0x00000000008833ec  0x0000000000000001 
0x000000c0002371e8:  0x0000000000000001  0x000000c000237228 
0x000000c0002371f8:  0x0000000000436eb2 <runtime.gwrite+0x00000000000000f2>  0x00000000008833ec 
0x000000c000237208:  0x0000000000000001  0x0000000000000001 
0x000000c000237218:  0x000000c000237295  0x0000000000000003 
0x000000c000237228:  0x000000c000237278  0x000000000046274e <runtime.systemstack+0x000000000000002e> 
0x000000c000237238:  0x00000000004356f0 <runtime.fatalthrow+0x0000000000000050>  0x000000c000237248 
0x000000c000237248:  0x0000000000435720 <runtime.fatalthrow.func1+0x0000000000000000>  0x000000c0000021a0 
0x000000c000237258:  0x00000000004354b1 <runtime.throw+0x0000000000000071>  0x000000c000237278 
0x000000c000237268:  0x000000c000237298  0x00000000004354b1 <runtime.throw+0x0000000000000071> 
0x000000c000237278:  0x000000c000237280  0x00000000004354e0 <runtime.throw.func1+0x0000000000000000> 
0x000000c000237288:  0x00000000007e0b21  0x0000000000000005 
0x000000c000237298:  0x000000c0002372e8  0x000000000044a8c5 <runtime.sigpanic+0x0000000000000305> 
0x000000c0002372a8: <0x00000000007e0b21  0x0000000000000046 
0x000000c0002372b8:  0x00007fa45a0f27c0  0x00007fa45a89e8e9 
0x000000c0002372c8:  0x0000000000000000  0x0000000000000000 
0x000000c0002372d8:  0x00007fa45a0f27c0  0x0000000000000000 
0x000000c0002372e8:  0x000000c000237ab0 !0x00007fa45a87f6b2 
0x000000c0002372f8: >0x000000c000237320  0x000000c0002373e8 
0x000000c000237308:  0x00007fa45a0f27c0  0x0000000000000000 
0x000000c000237318:  0x00007fa45a0f27c0  0x00007fa45a0f27c0 
0x000000c000237328:  0x00007fa45a0f27c0  0x0000000000000000 
0x000000c000237338:  0x0000000000000000  0x0000000000000000 
0x000000c000237348:  0x00007fa45a895d84  0x0000000000000000 
0x000000c000237358:  0x000000000237c328  0x00007fa45a0f2840 
0x000000c000237368:  0x0000000000000000  0x00007fa45b698f50 
0x000000c000237378:  0x0000000000000000  0xffffffffffffffff 
0x000000c000237388:  0x00007fa45a0f2840  0x0000000000000000 
0x000000c000237398:  0x00007fa45a0f2840  0xffffffffffffffff 
0x000000c0002373a8:  0x0000000000000000  0x00007fa45a0f27c0 
0x000000c0002373b8:  0x00007fa45a0f27c0  0x00007fa45a0f27c0 
0x000000c0002373c8:  0x00007fa45a0f27c0  0x00007fa45a87e8e7 
0x000000c0002373d8:  0x0000000000000000  0x00007fa45a895d44 
0x000000c0002373e8:  0x000000c000237420  0x000000000237c340 
runtime.sigpanic()
        /usr/local/go/src/runtime/signal_unix.go:825 +0x305 fp=0xc0002372f8 sp=0xc0002372a8 pc=0x44a8c5

Why did this goroutine panic with our hook?

Go’s runtime scheduler follows a very peculiar yet smart way of scheduling goroutines. The scheduler mainly works on four important objects:

G - The goroutine
N - Number of goroutines
M - OS thread (N is mapped to M)
P - Represents the notion of a processor i.e. resource provider for M when its running a goroutine.

As described in the design document for the Go runtime scheduler,

“When a new G is created or an existing G becomes runnable, it is pushed onto a list of runnable goroutines of current P. When P finishes executing G, it first tries to pop a G from own list of runnable goroutines; if the list is empty, P chooses a random victim (another P) and tries to steal a half of runnable goroutines from it.”

In summary, every G runs on an M assigned to a P.

Now that we know a bit about how Go schedules goroutines, by looking at this source file we can see Golang doesn’t work with the “system stack” (on Linux in most cases, the pthread stack) but with its own goroutine stack implementation with a minimum size of 2048 bytes.

Goroutine stack is dynamic, i.e. it is constantly expanding/shrinking depending on the current needs. This means any common code that runs in system stack assumes it can grow as it wishes (until it exceeds max stack size) while actually, it can’t unless using Go APIs for expanding. Our Rust code isn’t aware of it, so it uses parts of the stack that aren't actually usable and causes stack overflow.

We lack some steps. One might consider using runtime.morestack, but that's probably not ideal for us because that involves managing the stack manually per our needs. Luckily, we aren’t the first ones to do FFI³ in Go, so we looked into what cgo does when calling foreign functions:

Referring runtime/cgocall.go:

// Cgo call and callback support.1
//
// To call into the C function f from Go, the cgo-generated code calls
// runtime.cgocall(_cgo_Cfunc_f, frame), where _cgo_Cfunc_f is a
// gcc-compiled function written by cgo.
//
// runtime.cgocall (below) calls entersyscall so as not to block
// other goroutines or the garbage collector, and then calls
// runtime.asmcgocall(_cgo_Cfunc_f, frame).
//
// runtime.asmcgocall (in asm_$GOARCH.s) switches to the m->g0 stack
// (assumed to be an operating system-allocated stack, so safe to run
// gcc-compiled code on) and calls _cgo_Cfunc_f(frame).
//
// _cgo_Cfunc_f invokes the actual C function f with arguments
// taken from the frame structure, records the results in the frame,
// and returns to runtime.asmcgocall.
//
// After it regains control, runtime.asmcgocall switches back to the
// original g (m->curg)'s stack and returns to runtime.cgocall.
//
// After it regains control, runtime.cgocall calls exitsyscall, which blocks
// until this m can run Go code without violating the $GOMAXPROCS limit,
// and then unlocks g from m.
//

We will skip the non-blocking part i.e. calling runtime.entersyscall/runtime.exitsyscall for letting the scheduler beware of the “blocking” call so that the scheduler can yield its time to another goroutine as seen in the case of Syscall.Syscall6.abi0 and Syscall.Syscall.abi0. Therefore, we just replace the stack from the goroutine to the system stack using the implementation ofruntime.asmcgocall.abi0.

mov rbx, QWORD PTR [rsp+0x10]
mov r10, QWORD PTR [rsp+0x18]
mov rcx, QWORD PTR [rsp+0x20]
mov rax, QWORD PTR [rsp+0x8]
mov    rdx, rsp
mov    rdi, QWORD PTR fs:[0xfffffff8]
cmp    rdi, 0x0
je     2f
mov    r8, QWORD PTR [rdi+0x30]
mov    rsi, QWORD PTR [r8+0x50]
cmp    rdi, rsi
je     2f
mov    rsi, QWORD PTR [r8]
cmp    rdi, rsi
je     2f
call   go_systemstack_switch
mov    QWORD PTR fs:[0xfffffff8], rsi
mov    rsp, QWORD PTR [rsi+0x38]
sub    rsp, 0x40
and    rsp, 0xfffffffffffffff0
mov    QWORD PTR [rsp+0x30], rdi
mov    rdi, QWORD PTR [rdi+0x8]
sub    rdi, rdx
mov    QWORD PTR [rsp+0x28],rdi
mov    rsi, rbx
mov    rdx, r10
mov    rdi, rax
call   c_abi_syscall_handler

After saving the arguments in some untouched registers we call the handler on the system stack, and shuffle the registers/stack data to match Go’s expectations, mainly returning parameters to be in a specific place in the stack.

mov    QWORD PTR [rsp+0x28], -0x1
mov    QWORD PTR [rsp+0x30], 0x0
neg    rax
mov    QWORD PTR [rsp+0x38], rax
xorps  xmm15, xmm15
mov    r14, QWORD PTR FS:[0xfffffff8]
ret
3:
mov    QWORD PTR [rsp+0x28], rax
mov    QWORD PTR [rsp+0x30], 0x0
mov    QWORD PTR [rsp+0x38], 0x0
xorps  xmm15, xmm15
mov    r14, QWORD PTR FS:[0xfffffff8]
ret

After stitching together all the ABI0 syscall detours with mirrord, let’s look if things work as expected -

Success! 🥂

Complete implementation of all hooks is available here.

We decided not to handle the non-blocking changes that Go makes, primarily because it doesn’t really matter for our use-case (having “a bit of delay” isn’t critical to the value we try to provide with mirrord). We are planning to address it later on, though.

So Long, and Thanks for All the Fish

One of the ideas we had while working on this was to write a framework that will provide APIs to hook Go functions, i.e make trampolines from Rust using proc macros. It felt like too big of a project, and what we ended up doing suits our current needs, but if anyone is up for working on such a framework, we’d be happy to sponsor it! We’d love to hear your feedback and thoughts in our Backend Engineers community on Discord.

Feel free to checkout mirrord, send corrections/issues with the blog post on our website’s repository or just reach us at hi@metalbear.co.

Originally posted in MetalBear's blog.

Application binary interface - how binary programs interact, how do they use the stack, registers, etc. ↩
The cool Gopher was made using https://gopherize.me/. ↩
Foreign Function Interface - How to interact with external functions provided by other frameworks/dependencies. ↩

Carcinisation of mirrord (or: why we use Rust)

Aviram Hassan — Tue, 16 Aug 2022 06:59:53 +0000

Carcinisation (or carcinization) is an example of convergent evolution in which a crustacean evolves into a crab-like form from a non-crab-like form. (source: Wikipedia).

A classic example of carcinisation is MetalBear's mirrord project, where several different components converged on Rust as their main language. In this post, we'll detail their different evolutionary paths and explain why they ended up being written in Rust.

First of all, what is mirrord?

mirrord is an open-source tool that lets developers run local processes in the context of their cloud environment. It is meant to provide the benefits of running your service in a cloud environment (e.g. staging) without going through the hassle of actually deploying it there, and without disrupting the environment by deploying untested code.¹

mirrord has four main components:

Agent - Runs in the cloud and acts as proxy for the users.
Layer - Shared library that runs inside the local service, hooking IO operations (sockets, filesystem) and proxying those to the agent.
CLI - Wrapper to inject/load the Layer into local processes.
VS Code Extension - Same as CLI, but for VSCode.

Why Rust?

Agent

This component is shipped as a container image. The layer creates a job with this image, providing it with elevated permissions on the same node as the impersonated pod. The job then enters the impersonated pod's namespaces in order to be able to access its file system and network interfaces.

Switching Namespaces

Ideally, we would like to switch namespaces only for the necessary code flows, so we can have minimal impact on the impersonated pod. Linux lets you control namespace based on threads, so you can have different functionalities running in different threads and on different namespaces. Controlling threads this way can be hard in some frameworks/languages. For example, in Go, threads are actually abstracted so you need to do some tinkering to ensure correctness. On the other hand, Rust doesn't really abstract anything, and offers very fine tuned control over threads and namespaces.

Performance

One of mirrord's goals is to let multiple developers work on the same environment without impacting each other. To achieve that, our agent has have a very small footprint. Rust lets us have a fixed-size memory layout, without many allocations and without the overhead of a garbage collector.

Thread Safety

The Agent has many functionalities running at the same time, requiring the ability to move data between threads safely.

Rust provides great primitives and safety around concurrency and task management. It doesn't let us send data types that are thread-bound unknowingly thanks to Send and it warns us when we hold references to non-atomic data types across threads with Sync.

Layer

This component is shipped as a dylib/so (shared library) and loaded into the local process that's being plugged into the cloud. Once loaded, it hooks many libc functions (and some other frameworks, such us libuv) to create a smart management layer that decides what operation happens locally and what is being relayed to run remotely.

Low Level

It was obvious to us that the layer had to be written in a low level language. Yes, we could create a bridge layer² (JS actually has one built in with Frida) but that would add complexity and security concerns. Whereas in other parts of the solution we run in a self-contained context (i.e. our own process), this time we're being loaded into a process that isn't aware of our existence and we can't introduce bugs into it. Rust + Frida let us hook low level functions such as accept, bind, socket, open, etc in a relatively safe manner leveraging great abstractions such as Arc to manage our internal data structures and sync primitives such as mpsc to build communication between different parts of our code (in the layer, it's mainly between the "main loop thread" and hooks).

Performance

Like the Agent, the Layer also needs to have low overhead in order to provide great developer experience.

CLI

The CLI is responsible for injecting the layer into the target process. Right now, our main (and only) load mechanism is using DYLD_INSERT_LIBRARIES (on macOS) and LD_PRELOAD on Linux.

Implementing this in another language could've been easy but we decided to go with Rust for several reasons.

Future-proof

The load method is fairly simple, but if in the future we'd want to introduce a more sophisticated injection method like using "ptrace" or other methods. Rust would let us implement it and use those functions and layouts at more comfort than other languages.

Standalone

Rust generates standalone binaries (apart from libc). On top of that, using Cargo's new bindeps feature we could embed mirrord-layer into the CLI, creating a smooth and transparent experience instead of having to ship two files or downloading dependencies at runtime.

VS Code Extension

The extension can't really be written in Rust due to VS Code support for JS only. We did consider using WASM, but the bridging logic would "cost" too much to be worth any value Rust might provide (which, for the extension, would mostly be consistency with our other components)..

Company

Bonus section! I had a feeling having our codebase be mainly in Rust would make hiring engineers a lot easier. As a Rust enthusiast, I would have loved to work somewhere where I'd get to work with Rust regularly, and I suspected that many others felt the same. I even posted a poll in/r/rust to see how others feel. The results were encouraging - around 40% of the people who voted thought the market for hiring Rust engineer is employer driven. I suspect that in any other ecosystem, the results would have been much more one sided in favor of the market being employee driven. When we finally started hiring, we posted in the "Who's Hiring" mega thread in /r/rust, and received applications from some great candidates, making building the team both fun and fast.

Afterword

Rust can be used for many use cases, software and applications. We believe that every task requires different tools, but in our case our project's whole ecosystem fit right into Rust. You always need to do your research before choosing a language and stack, but generally speaking Rust is powerful and versatile and can be an amazing choice for a lot of different solutions.

Do you have any questions/corrections? Our website is completely open-source, so feel free to submit it as an issue or PR to our repo.

Want to help mirrord? Have a look at our open issues in the GitHub issue tracker and feel free to contribute.

Originally posted in MetalBear's blog.

https://mirrord.dev/docs/overview/introduction/ ↩
A software layer that connects the low level C run time with the JS runtime (translating the hooks into JS function calls) ↩