Running Untrusted Code Safely with AWS Lambda (and keeping my VPS out of it)

I run a small learning platform where teachers give coding assignments and students write Python in the browser. I wanted two things that sound simple and are not:

1. A "Run" button that executes the student's code and shows the output.
2. An autograder that runs each submission against hidden test cases and scores it.

The hard part isn't running Python. The hard part is running someone else's Python — code I didn't write, from a teenager who might paste anything — without letting it mine crypto, exfiltrate data, read my secrets, or take down the box that also serves my live classes and API.

Here's the architecture I landed on, why each piece exists, and every security decision behind it. The executor is AWS Lambda; my app server (a cheap VPS) never runs a line of student code.

Why not "just run it on the server"?

The obvious approach — shell out to python3 on my VPS — is a bad idea for a few reasons:

• Isolation. A subprocess on my app server shares the kernel, filesystem, network, and environment with my database credentials, API, and live-class WebSocket hub. One os.environ read or one requests.get and I have a problem.
• Blast radius. A while True: fork() or a memory bomb takes down everything, not just the code runner.
• Scaling. 50 students clicking Run at 9:01am is a thundering herd my single box shouldn't absorb.

So I split execution off the server entirely. Lambda is a great fit: it's ephemeral (fresh microVM per burst), isolated (Firecracker), scales to zero, and I pay per invocation. Student code runs there and nowhere else.

Design rule: my VPS orchestrates (auth, rate-limit, record), Lambda executes. The VPS never touches student code.

---

The architecture

Browser (Monaco editor)
   │  POST /code/run { source, stdin }   (JWT)
   ▼
Go API on a VPS  ── auth, size caps, per-user throttle (Redis)
   │  lambda:InvokeFunction  (IAM user, SigV4, HTTPS)
   ▼
AWS Lambda "python-runner"   (private, no function URL)
   • VPC private subnet — NO NAT, NO Internet Gateway
   • Security group — NO inbound, NO outbound
   • Firecracker microVM per execution
   └── subprocess: python3 -I, scrubbed env, rlimits, wall-clock kill

Two separate identities do two separate jobs, and this separation is the backbone of the whole thing:

• The execution role is what the Lambda assumes while it runs. It has almost nothing: CloudWatch Logs + the VPC networking permission. Nothing else.
• The invoker is an IAM user whose keys live on my VPS. Its entire power is lambda:InvokeFunction on one function ARN.

If either credential leaks, it's nearly useless — that's the point.

What happens when a student clicks "Run"

1. Browser — Monaco holds the code. Run fires POST /code/run { source, stdin, assignmentId } with the student's JWT.
2. Middleware — a global rate limiter, then JWT auth (plus a Redis token-blocklist check).
3. Handler — validates and enforces size caps: source ≤ 64 KB, stdin ≤ 16 KB. (A payload bomb shouldn't even reach Lambda.)
4. Throttle — a per-user SetNX key in Redis with a 2-second TTL. Spamming Run just gets a 429.
5. Invoke — the backend marshals { mode:"run", source, stdin, timeoutMs:8000 } and calls lambda.Invoke (RequestResponse) with the invoker user's keys. The function is private — there is no public URL to attack.
6. Lambda — AWS runs the request in a Firecracker microVM and calls the handler.
7. Sandbox — the handler writes the code to /tmp, spawns a locked-down python3 -I subprocess, feeds it stdin, and captures the result under a hard timeout.
8. Return — stdout/stderr (truncated), exit code, timedOut, durationMs come back as JSON; the browser renders it in a console.

Autograding on submit is the same pipeline with mode:"grade" and a list of test inputs. The Lambda runs one fresh subprocess per test and returns each stdout; my backend compares those to the hidden expected outputs (which never leave the server) and computes the score.

The sandbox (the important 40 lines)

This is the Lambda handler. Every line is a defense.

import os, sys, time, resource, subprocess, tempfile

MAX_TIMEOUT_S = 8
CPU_SECONDS   = 5
MEM_BYTES     = 256 * 1024 * 1024
MAX_PROCS     = 64            # blocks fork bombs
MAX_FILE_BYTES= 8 * 1024 * 1024
OUTPUT_CAP    = 64 * 1024     # bytes per stream
PY = sys.executable

def _apply_limits():
    resource.setrlimit(resource.RLIMIT_CPU,   (CPU_SECONDS, CPU_SECONDS + 1))
    resource.setrlimit(resource.RLIMIT_AS,    (MEM_BYTES, MEM_BYTES))
    resource.setrlimit(resource.RLIMIT_NPROC, (MAX_PROCS, MAX_PROCS))
    resource.setrlimit(resource.RLIMIT_FSIZE, (MAX_FILE_BYTES, MAX_FILE_BYTES))
    os.setsid()  # own process group, so a timeout kills the whole tree

def run_once(source, stdin, timeout_s):
    with tempfile.TemporaryDirectory() as workdir:
        path = os.path.join(workdir, "main.py")
        open(path, "w").write(source)
        env = {  # minimal env — NO AWS_* credentials leak through
            "PATH": os.path.dirname(PY) + ":/usr/bin:/bin",
            "HOME": workdir, "TMPDIR": workdir, "LANG": "C.UTF-8",
        }
        try:
            p = subprocess.run(
                [PY, "-I", path],                 # -I: isolated mode
                input=stdin.encode(),
                stdout=subprocess.PIPE, stderr=subprocess.PIPE,
                cwd=workdir, env=env, timeout=timeout_s,
                preexec_fn=_apply_limits, check=False,
            )
            return {"stdout": p.stdout[:OUTPUT_CAP].decode(errors="replace"),
                    "stderr": p.stderr[:OUTPUT_CAP].decode(errors="replace"),
                    "exitCode": p.returncode, "timedOut": False}
        except subprocess.TimeoutExpired:
            return {"stdout": "", "stderr": "[time limit exceeded]",
                    "exitCode": -1, "timedOut": True}

Security considerations (the part I actually care about)

I treat student code as actively hostile. Defense in depth means several independent barriers, so that any single mistake isn't fatal.

1. The credential-leak trap nobody warns you about

AWS Lambda injects the execution role's temporary credentials into the function as environment variables (AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, AWS_SESSION_TOKEN). A student's Python can just do:

import os; print(dict(os.environ))

…and read them. Two mitigations, and I do both:

• Least-privilege role. The execution role can do nothing but write logs and manage its VPC network interface. Leaked creds are worthless.
• Scrubbed subprocess env. I never inherit the Lambda env into the child. The subprocess gets a hand-built env with only PATH, HOME, TMPDIR, LANG. The AWS_* variables simply aren't there.

2. No network — two independent barriers

Student code should never open a socket. I block it at the infrastructure layer, twice:

• The Lambda's VPC subnet has no NAT gateway and no Internet Gateway — there's no route to the internet.
• The security group has no outbound rules — even intra-VPC traffic is denied.

requests.get("https://evil.com") fails at the network layer. Removing NAT also removes any path to the instance metadata endpoint. Two separate barriers, so one misconfiguration doesn't open the door.

3. Resource limits — CPU, memory, processes, files, time

Inside the sandbox I set OS rlimits before exec:

• RLIMIT_CPU — CPU-seconds cap (kills busy loops).
• RLIMIT_AS — address space cap (kills memory bombs).
• RLIMIT_NPROC — process cap (kills fork bombs).
• RLIMIT_FSIZE — file-size cap (only /tmp is writable anyway).

On top of that, a wall-clock timeout kills the whole process group, and Lambda's own timeout/memory ceiling is the outermost fence. python -I (isolated mode) ignores PYTHON* env vars and user site-packages, so nothing sneaks in through configuration.

4. Output caps

A while True: print("x") shouldn't return a gigabyte. Each stream is truncated to 64 KB before it leaves the function.

5. Cost and blast-radius control

Untrusted code plus pay-per-invoke means "abuse" and "bill" are the same axis. I bound it:

• Reserved concurrency on the function (e.g. 20). A spam attack can't fan out to thousands of paid invocations.
• Per-user throttling in the API (Redis SetNX, ~1 run / 2s).
• Input size caps at the edge, before anything is invoked.

6. Least-privilege everywhere

• The invoker (the VPS's IAM user) can only call lambda:InvokeFunction on one ARN. It can't read S3, can't touch other functions, can't do anything else.
• The function is private — invoked via the SDK/SigV4, not a public Function URL. There's no internet-facing endpoint to probe.

7. Hidden test cases stay hidden

For autograding, the Lambda receives only test inputs and returns actual stdout. The expected outputs never leave my server — the comparison happens in my Go backend. Even if a student could somehow read their own invocation event, there's no answer key in it.

8. The microVM is the outer wall

Everything above is defense in depth inside the function. The real hard boundary is that each execution runs in its own Firecracker microVM, isolated from AWS's infrastructure and from my other invocations. The application-level hardening exists so that a bug there still doesn't hand an attacker anything useful.

What this deliberately is not

• Not an interactive terminal. Lambda is request/response — it can't pause mid-run to ask for input. Input is supplied up front. (For a true interactive REPL you'd run Python in the browser via WebAssembly/Pyodide, or a long-lived container — both were overkill for "basic Python assignments.")
• Not for heavy or dependency-laden code. Standard library only, single file. That keeps the function a plain zip with fast cold starts. Libraries would mean a container image and slower starts.

Constraints are a feature here: a small, boring, stdlib-only sandbox is easy to reason about and hard to abuse.

Cost

A run is ~128–256 MB for 1–3 seconds — a fraction of a cent. Fifty students doing dozens of runs a day is single-digit dollars a month, and reserved concurrency + throttling cap the worst case. The VPS, meanwhile, does nothing expensive: it authorizes, rate-limits, and forwards a small JSON payload.

Takeaways

• Separate orchestration from execution. Your app server should never run untrusted code. Push it to something ephemeral and isolated.
• Assume the code is hostile. Scrub the environment, cut the network twice, cap every resource, truncate output.
• Two identities, both minimal. The thing that runs code and the thing that invokes it should each be able to do almost nothing.
• Least privilege turns leaks into non-events. If stolen credentials can't do anything, a leak isn't a breach.
• Defense in depth. No single control is trusted to be perfect — the microVM, the network isolation, the rlimits, the scrubbed env, and the least-privilege roles each independently limit what can go wrong.

The result: students get an instant "Run" button and autograded assignments, and I sleep fine knowing their code runs in a disposable box with no network, no secrets, and no way to reach anything I care about.