DEV Community

Sofia Winters
Sofia Winters

Posted on • Updated on

Does Rust's no_mangle function conflict with OS APIs?

I faced an odd behavior today. I wrote a Rust function and a test like below and ran them on Windows.

// the crate socket2 is needed
use socket2::{Domain, Socket, Type};
use std::net::{IpAddr, Ipv4Addr, SocketAddr};

#[no_mangle]
extern "C" fn send() {
    let socket = Socket::new(Domain::ipv4(), Type::dgram(), None).unwrap();
    let local = SocketAddr::new(IpAddr::V4(Ipv4Addr::LOCALHOST), 0);
    let remote = SocketAddr::new(IpAddr::V4(Ipv4Addr::LOCALHOST), 5678);

    socket.bind(&local.into()).unwrap();
    socket.connect(&remote.into()).unwrap();

    let buf = vec![0u8; 0];
    socket.send(&buf).unwrap();
}

fn main() {}

#[cfg(test)]
mod tests {
    #[test]
    fn it_works() {
        super::send();
    }
}
Enter fullscreen mode Exit fullscreen mode

The result was like this:

thread 'tests::it_works' has overflowed its stack
error: test failed, to rerun pass '-p function --bin function'

Caused by:
  process didn't exit successfully: `ab583070df1ffc3d.exe tests::it_works --exact -Z unstable-options --format=json --show-output` (exit code: 0xc00000fd, STATUS_STACK_OVERFLOW)

Process finished with exit code -1073741571 (0xC00000FD)
Enter fullscreen mode Exit fullscreen mode

Why did the stack overflow!? I couldn't understand what happened but found 2 solutions.

  1. Change the name of the function from send to the others like send2.
  2. Remove #[no_mangle].

Then, I noticed the name of the function conflicted with OS's send because the name wasn't mangled. In other codes, I've also seen access violation, heap corruption errors, so this will cause undefined behavior.

Then, I've made the smallest reproduction code!

mod api {
    extern "system" {
        pub fn ntohl(netlong: u32) -> u32;
    }
}

#[no_mangle]
extern "C" fn ntohl() {
    unsafe { api::ntohl(0) };
}

fn main() {
    ntohl();
}
Enter fullscreen mode Exit fullscreen mode

I'm not bright about C or system calls, so this will make sense for those who understand them. However, I wonder if the rust compiler can detect this conflict because this is extremely unsafe.

Discussion (0)