Building a secure/sandboxed environment for executing untrusted code

What is a Sandbox?

Let's try to understand the meaning of the term sandbox first, before learning how to build one. Sandbox is a like a container that isolates the environment in which the software is run. In other words, a sandbox provides a secure environment which restricts the software inside the sandbox from accessing the resources of the host, the resources can be a file-system, network, some set of kernel system-calls etc. The application of sandbox is obvious, you can execute the code which you don't trust without worrying about security much. Here are some examples that will help you understanding sandbox better:

1. Sandboxing is built into modern browsers, this restricts malicious websites from stealing your sensitive data or damaging the client machine, because the code runs inside a sandbox and the sandbox restricts the code from calling any host-level functions.
2. Most of the online coding tutorials that allow remote code execution are powered by sandboxing tools, these tools provide a separate isolated environment for you, thus restricting you from accessing the server resources or files belonging to other users.

In this tutorial, we will be building a simple sandbox solution. Note that this is not the perfect one (Many companies have worked years together on security), but still it manages to give you an idea of building your own sandbox.

What will be the consequences if you don't use a sandbox?

Imagine a scenario where your browser did not provide any security/isolation, in such case anyone good at JavaScript/C++ can somehow manage to access your file-system from the browser itself without even notifying you, this will allow attackers to steal your private content and use them for blackmailing you. Imagine an online code-editing tool without a sandbox, you as the user can write a code that deletes the files on their server, rendering the server useless, you can also use server's resources to mine bitcoins or attack others, you can also delete the files of other users using the tool. These are very critical consequences and any developer who thinks about building a browser, code-editing tool or any software that allows remote-code-execution should consider building a sandbox and securing his/her system from attackers.

What sandbox must provide:

1. The sandbox should hide the host file-system from the untrusted code which runs inside the sandbox.
2. It should block the code inside the sandbox from making system-calls directly on the host-kernel which can be dangerous.

Prerequisites of this tutorial:

1. Basic knowledge of containers and docker.
2. Basic knowledge of creating and deploying software as containers.
3. Basic knowledge of Linux and System-Calls.

Solution-1 : Using Containers with Docker

If you are into software development you must be aware of containers. Containers provide an isolated environment where the software can run with all its dependencies. Containers provide their own file-system, so the app that runs inside the container cannot access the host's original file-system. So let's build a simple program to demonstrate this. I will be using C, you can use any language of your choice.

This program just lists all the files from the root / and exits. (list_files.c)

#include <stdlib.h>

int main(int argc, char **argv) {
   system("ls /");
}

This is a simple program and should work without any issues if you are any linux machine. Let's compile this, we will compile this a static binary so that we don't need libc support in our container. But wait, we will first run on the host directly.

gcc list_files.c -static -static-libgcc -static-libstdc++ -o list_files

If compilation is successful, this should produce a binary by name list_files, we will make sure this is static executable and does not depend on libc.

ldd list_files

This will output something similar if it is a static binary.

not a dynamic executable

Now, let's run this, note that we are running directly on the host.

./list_files

This will output all the files under root /

bin    dev   initrd.img      lib64   mnt   root  snap      sys  var
boot   etc   initrd.img.old  lost+found  opt   run   srv       tmp  vmlinuz
cdrom  home  lib         media   proc  sbin  swapfile  usr  vmlinuz.old

Since we ran the code directly on the host machine, we are able to see the contents of the root file-system. The code has exclusive access to our file-system and other resources. So it is not recommended to run untrusted code directly. Now let's containerize it and see how we can provide basic restrictions. We will use busybox a minimal container image. We don't have to install anything specific, remember why we built static executable. This is how our Dockerfile looks.

FROM busybox

COPY ./list_files /list_files
WORKDIR /
ENTRYPOINT ["/list_files"]

So, that's it, we will copy the list_files binary and run it on the start of container. Let's build and run.

docker build . -t sandbox_test
docker run --rm -ti sandbox_test

Now, it should produce the following output:

bin         etc         list_files  root        tmp         var
dev         home        proc        sys         usr

You can see that the / has changed. It is the file-system of busybox our application is seeing, not our host file-system. Containers also provide process-isolation, network isolation etc so our application is somehow isolated. Hurray! We built a simple sandbox (This is not the final solution, read till the end)

Let's refine our work further, till now we manually packaged every application we built as a container image, which is lot of work. So, let's build a generic sandbox container that can run any binary inside the container environment without packing them explicitly. To build this, we follow the steps as below:

1. We create a small C program that starts inside the container.
2. The program reads from stdin and writes the contents of stdin to a file inside the container.
3. It then executes the file it created.
4. Returns the output back to the host as a string.(writes to stdout)

Let's build this! I am using C, you can use GO, Rust or any language that runs on native metal (rather than interpreted like Java/Python etc). Create a file called sandbox.c and let's write a function called write_stdin_to_file which reads from stdin and writes to a file called binary.

Includes and some definitions:

#include <unistd.h>
#include <stdlib.h>
#include <stdio.h>
#include <fcntl.h>
#include <stdbool.h>

#define BUFFER_SIZE 4096
#define OUTPUT_BUFFER 1024
typedef unsigned char uchar;

write_stdin_to_file function:

void write_stdin_to_file(int * size) {

    uchar buffer[BUFFER_SIZE];
    int read_bytes = 0, itrs = 0;
    * size = 0;

    int fd = open("./binary", O_RDWR | O_CREAT, 0777);
    FILE * fp = fdopen(fd, "wb");

    if (fp == NULL) {
        fprintf(stdout, "Failed to open the file for writing\n");
        exit(-1);
    }

    while (true) {
        read_bytes = read(0, buffer, BUFFER_SIZE);
        if (read_bytes < 0) {
            fprintf(stdout, "Failed to read binary data, exiting");
            fclose(fp);
            exit(-1);
        }

        if (read_bytes == 0) {
            //EOF
            break;
        }

        //write data to the file
        *size = *size + read_bytes;
        fwrite(buffer, sizeof(uchar), read_bytes, fp);
    } 

    //wrote the file, close it.
    fclose(fp);
}

And now the main function, main calls write_stdin_to_file and then executes it.

int main(int argc, char **argv) {
    int size = 0, fread_bytes = 0;

    char output_buffer[OUTPUT_BUFFER];
    write_stdin_to_file(&size);

    if (size == 0) {
        fprintf(stdout, "Empty binary file, discarding\n");
        exit(0);
    }

    FILE * process_fd = popen("./binary", "r");
    if (process_fd == NULL) {
        fprintf(stdout, "Failed to execute the binary\n");
        exit(-1);
    }

    fprintf(stdout, "Executing binary inside the sandbox\n");

    //read the data as buffers and stream it to stdout
    while (true) {
        fread_bytes = fread(output_buffer, sizeof(uchar), sizeof(uchar) * OUTPUT_BUFFER, process_fd);

        if (fread_bytes == 0) {
            //EOF
            fclose(process_fd);
            exit(0);
        }

        if (fread_bytes < 0) {
            //Error 
            fprintf(stdout, "Failed to read the output");
            exit(-1);
        }

        output_buffer[fread_bytes] = '\0';

        fprintf(stdout, "%s", output_buffer);
    }

    return 0;
}

Let's compile this as static executable:

gcc src/sandbox.c -static -static-libgcc -static-libstdc++ -o sandbox

Now, lets containerize it using docker.

FROM busybox

COPY ./sandbox /sandbox
WORKDIR /
ENTRYPOINT ["/sandbox"]

Let's build it:

docker build . -t sandbox:latest

After the build is complete, we can run it. But before running, keep this in mind, the entrypoint is sandbox binary, this binary listens of stdin, so we need to pass the binary we need to execute as stdin so that the sandbox binary can execute it inside the container environment.

cat list_files | docker run --rm -i sandbox

This is a simple command, we are using cat to read the binary from host system, after reading, it pipes the output to sandbox program which runs inside the docker. The sandbox executes it inside the container environment and then emits the output as stdout, since stdout is the terminal, the output will be printed on the screen, we can see that below:

Executing binary inside the sandbox
bin
binary
dev
etc
home
proc
root
sandbox
sys
tmp
usr
var

That's it! We have created a container image called sandbox which we can use it to run any binary inside the container environment. But is this the end?? Of course, not!! This approach is still unprotected. Let's see why and how we can address this issue.

Problems with using containers alone as sandbox

We somehow managed to provide network and file-system isolation, but still we are not safe. That's because of the nature of containers. Containers do not provide kernel-level isolation, in other words, eventhough the containers are isolated, they still use host-kernel for their functionality, i.e any system call made by the application inside the container will execute some host-kernel function which is not secure, that means, any clever programmer can write an application that combines multiple system-calls and escape the container isolation to get into the host environment, then he can do anything. But we can solve this as well. Using Userspace kernels.

Userspace Kernel with gVisor:

gVisor by Google is a userspace application kernel written in Go. Userspace kernel is a software that runs completely in user-mode and has less privilege (since it runs in user-mode). It also acts as a kernel emulation layer, that means, it can act as a fake kernel and can receive and process system-calls, thus hiding the host kernel. gVisor is compatible with OCI and provides a OCI runtime called runsc that can be used by container management tool like docker as the underlying runtime. (docker uses runc as the default runtime).

You can install gVisor by following the guide here. Once installed, make sure you have registered runsc as one of the possible runtimes for docker. Check the file /etc/docker/daemon.json, it should contain an entry like this:

{
    "runtimes": {
        "runsc": {
            "path": "/usr/bin/runsc"
        }
    }
}

You can then restart the docker daemon to ensure the changes are applied.

sudo systemctl restart docker

That's it! We can now run our binaries inside a sandbox with complete isolation. Here is how we can include gVisor in our execution command.

cat list_files | docker run --runtime=runsc --rm -i sandbox

You can see --runtime=runsc added, which means we are telling the docker to use gVisor (runsc) instead of runc.

Why I wrote this?

Recently I was building a clone of Go playground as a weekend project, a tool for running go programs online. Folks who developed Go playground were aware of security and they used NaCL sandbox initially. You can read the blog on this here. After the deprecation of NaCL, the developers shifted to this approach to provide security. While I was scratching my head over security for my hobby project, I got a chance to read original playground's source code and found a solution very similar to this. I thought of adding this to my project and also write a blog on this. To learn more, you can follow these repositories:

1. Original Playground.
2. My clone of Playground.

You can also consider using MircoVM projects as an alternative to gVisor, like :

1. AWS Firecracker
2. Kata Containers

If you do, please let me know in the comments.
Thanks for spending your precious time in reading this post. Do let me know your opinions and better options in comments.

Comments

[Manan Chawla]

Wonderfully written tutorial not much articles are there on this topic. I was myself working on making a code hosting website like repl.it myself. Thanks for telling about gvisor didnt know that earlier. Also i had a questions can we provide root access inside the container but with security assured so that the person doesnt somehow escalates itself to host server.

[Narasimha Prasanna HN]

Yes, since the isolation is at kernel level, you can provide root access inside the container. Also gVisor has many configuration options that might help you. For more configuration options, you can read the Documentation .

[Manan Chawla]

Man this is awesome thanks alot once again. Also what if we wanna make windows sandboxes rather than linux. Any open source alternative for that?

[Narasimha Prasanna HN]

Hey, tbh, for windows there are not much sandboxing technologies that are free. Most of the products are paid ones. I would recommend you to use Linux distros or get a Linux setup running on a VM you can use VMWare basic emulator for that

[kritikgarg]

💡 👏 Excellent read, Narasimha! Your insights on building a secure/sandboxed environment for executing untrusted code are very informative and helpful. 🔐 It's crucial for developers to understand how to create a safe environment for executing untrusted code.I appreciate the thoroughness of your approach and the practical examples you provided.🙌

Alongside that, I also read the Salesforce Sandboxes article, which provides comprehensive information on Salesforce sandboxes, including pricing, types, and features. This article is a great resource for those looking to learn about Salesforce sandboxes.

[okyanusoz]

Awesome tutorial, thank you!