Building a multi-tenant WebAssembly runtime in Rust

Every SaaS product eventually gets the same request: "Can we run our own logic when X happens?" The usual answer is a webhook, or a pull request and a redeploy. The nicer answer is what Cloudflare Workers, Fastly Compute and Fermyon Spin do: let customers upload sandboxed code and run it for them, with hard limits.

I built a small version of that in Rust, called warpline, and published 0.1.0 to crates.io this week. It's sized for one box, not a global edge network, but the isolation problems are the same ones the big platforms have. This post walks through how it works and, more usefully, the bugs I found along the way. Most of them are easy to write and hard to notice.

• Code: https://github.com/Bunty9/warpline
• Crates: warpline-core, warpline-host, warpline-control

The shape of it

There are two services and one shared volume:

• warpline-control accepts uploads. It compiles each WebAssembly component once with Cranelift, checks its imports against what the host provides, and stores it content-addressed on disk. It also issues per-tenant API keys.
• warpline-host serves POST /tenants/{tenant}/functions/{func}/invoke. It looks up the tenant's function, loads the precompiled module (usually from an in-memory LRU), runs it with that tenant's limits and writes a metering row.

Postgres holds tenants, hashed API keys, per-tenant limits and the metering ledger. Prometheus metrics sit on a separate loopback-only listener, because a scrape endpoint lists every tenant and function name.

The guest contract

Guests are Component Model components. The whole capability surface is one WIT file:

package warpline:host@0.1.0;

interface kv {
    get: func(key: string) -> option<list<u8>>;
    put: func(key: string, value: list<u8>);
}

interface log { emit: func(level: string, msg: string); }

interface http-out {
    record request { url: string, method: string, body: list<u8> }
    record response { status: u16, body: list<u8> }
    fetch: func(req: request) -> result<response, string>;
}

world handler {
    import kv;
    import log;
    import http-out;
    export handle: func(input: list<u8>) -> list<u8>;
}

A Rust guest is a few lines with wit-bindgen. The wasm32-wasip2 target emits components directly, with no adapter step:

wit_bindgen::generate!({ world: "handler", path: "wit" });

struct MyGuest;
impl Guest for MyGuest {
    fn handle(input: Vec<u8>) -> Vec<u8> { input } // echo
}
export!(MyGuest);

On the host side, wasmtime::component::bindgen! generates a trait per interface, and I implement them on the per-call context. That context carries the tenant ID, a shared KV store, the tenant's HTTP allowlist and a resource limiter. WASI is deny-by-default: no preopened directories, no environment, no arguments, no sockets. A guest gets only clocks, randomness and a stdout sink, which is what Rust's std needs to link.

Bug 1: the runtime itself was the vulnerability

The project started on wasmtime 27. When I ran cargo deny check, it reported about 25 RustSec advisories against wasmtime and wasmtime-wasi 27. Several were sandbox escapes: a Cranelift miscompile on aarch64, component-model string transcoding out-of-bounds writes, and WASI filesystem permission bypasses.

The lesson isn't subtle, but it's easy to skip: for a sandbox, the sandbox's version is your security boundary. No amount of careful host code matters if the guest can walk out of linear memory. I upgraded to wasmtime 49 before writing anything else, and cargo deny now runs in CI so the next advisory fails the build.

Bug 2: a memory cap that counted the wrong thing

wasmtime lets you install a ResourceLimiter on each Store. My first version was the obvious one:

fn memory_growing(&mut self, current: usize, desired: usize, _max: Option<usize>)
    -> wasmtime::Result<bool>
{
    Ok(desired <= self.mem_cap_bytes)
}

The catch: desired is the new size of one linear memory. A component can contain several core instances, each with its own memory, and wasmtime asks the limiter about each memory separately. With a 64 MiB cap, a component with four memories could use 256 MiB.

The fix is to account for the total across the store, and to cap how many memories and instances a store can create at all:

fn memory_growing(&mut self, current: usize, desired: usize, max: Option<usize>)
    -> wasmtime::Result<bool>
{
    // wasmtime consults the limiter before the memory's own declared max;
    // reject those here so a doomed grow isn't charged.
    if max.is_some_and(|m| desired > m) {
        return Ok(false);
    }
    let new_total = self.total_bytes + desired.saturating_sub(current);
    if new_total <= self.mem_cap_bytes {
        self.total_bytes = new_total;
        self.peak_bytes = self.peak_bytes.max(self.total_bytes);
        Ok(true)
    } else {
        self.cap_hit = true;
        Ok(false)
    }
}

The max check at the top came from a second pass. wasmtime calls the limiter before checking the memory's own declared maximum. Without that early return, a grow the limiter approved but wasmtime then rejected would still be charged, and a guest could inflate its own accounting until it hit a spurious limit. That only hurts the guest itself, but it's the kind of thing that turns into a confusing support ticket.

The limiter also records peak memory, which goes into the metering row.

Bug 3: a CPU timer that punished everyone

wasmtime offers two ways to cap CPU time:

• Fuel: every instruction costs fuel. It's precise, but you pay a counter decrement in the hot path.
• Epoch interruption: generated code checks a global epoch counter at function entries and loop back-edges, and traps when a store's deadline passes. The check is one load and compare, and something else advances the counter.

For short handlers, epochs are the better trade. My first implementation spawned a tokio task per call that slept for the budget and then called engine.increment_epoch().

The epoch is engine-wide. Every call's timer advanced the counter for every store on the engine. With enough concurrent calls, a well-behaved tenant's 100 ms budget could expire after a few milliseconds because other tenants' timers kept firing.

The fix has three parts:

1. One ticker thread per engine advances the epoch every 1 ms.
2. Each store counts its own ticks, using a deadline callback instead of one absolute deadline.
3. The callback yields back to tokio on every tick. A guest in a tight loop no longer pins an executor worker for its whole budget.

store.epoch_deadline_callback(move |_store| {
    ticks_elapsed += 1;
    if ticks_elapsed >= budget_ticks {
        Err(wasmtime::Error::new(CpuBudgetExceededMarker))
    } else {
        Ok(wasmtime::UpdateDeadline::Yield(1))
    }
});
store.set_epoch_deadline(1);

The marker error lets the host tell "CPU budget exceeded" (HTTP 408) apart from every other trap. A tokio::time::timeout wraps the whole call as a wall-clock backstop, in case a guest is stuck waiting on a slow host call where epoch ticks can't reach it.

I added a test that runs two infinite-loop guests on a single-threaded tokio runtime alongside a third task that must finish first. Without the yield, that test hangs.

Bug 4: the timer that drifted 12%

With the budget now counted in ticks, the benchmarks showed this:

Budget	Actual time to trap
10 ms	+0.45 ms
50 ms	+3.18 ms
100 ms	+12.11 ms

The error grew with the budget, which pointed at the ticker rather than the per-call path. The loop was:

loop {
    std::thread::sleep(Duration::from_millis(1));
    engine.increment_epoch();
}

sleep promises at least the duration. Each iteration overshot by about 0.12 ms, and because the budget is a tick count, the overshoot added up: 100 ticks took about 112 ms.

The fix is to sleep until an absolute deadline, so an overshoot on one tick shortens the next one:

let tick = Duration::from_millis(EPOCH_TICK_MS);
let mut next = Instant::now();
while !stop.load(Ordering::Relaxed) {
    next += tick;
    let now = Instant::now();
    if next > now {
        std::thread::sleep(next - now);
    } else if now - next > tick * 10 {
        next = now; // badly behind (suspend, starvation): resync, don't burst
    }
    engine.increment_epoch();
}

After the change, all three budgets landed within ±0.7 ms of target. The resync branch stops a suspended laptop or a starved thread from coming back and firing a burst of ticks that would kill every running guest at once.

Bug 5: an allowlist that followed redirects

Guests can make outbound HTTP calls, but only to hosts their tenant has explicitly allowed. The default allowlist is empty. The first version parsed the URL, checked the host against the list and sent the request with reqwest.

reqwest follows redirects by default. So allowed.example.com could answer with 302 Location: http://169.254.169.254/latest/meta-data/, the cloud metadata endpoint, and the host would fetch it for the guest. The allowlist only ever saw the first hop.

Closing that properly took more than turning off redirects:

• No redirects (redirect::Policy::none()). The guest gets the 302 and decides what to do with it.
• No proxy (.no_proxy()). With HTTPS_PROXY set, the proxy resolves the hostname, so any DNS-level check on our side never runs.
• Private addresses blocked at DNS resolution. An allowlisted domain could simply resolve to 10.0.0.5, or be rebound to it later. A custom reqwest::dns::Resolve implementation filters out loopback, RFC 1918, link-local, CGNAT, unique-local and multicast answers before reqwest connects.
• IP literals checked separately. reqwest never calls your resolver when the URL host is already an IP address, so those are checked directly.
• Embedded IPv4 addresses. ::ffff:10.0.0.1, ::10.0.0.1, NAT64's 64:ff9b::a00:1 and 6to4's 2002:a00:1:: all reach 10.0.0.1 on the right network. The classifier extracts the embedded address and checks that.
• A 5 s timeout and a 1 MiB response cap, read incrementally so a huge response is never fully buffered.

The IP classifier is a pure function with a table-driven test, which made adding each new case cheap.

Bug 6: string concatenation is not isolation

Tenant KV keys were scoped by prefixing them: t/{tenant}/{key}. That looks isolated until you notice:

• tenant a writing key b/x produces t/a/b/x
• tenant a/b writing key x produces t/a/b/x

And axum percent-decodes path segments, so a tenant named a/b was reachable via /tenants/a%2Fb/....

I fixed it twice, on purpose:

1. Structurally. The store is keyed by a (tenant, key) tuple, so there's no string to collide.
2. At the boundary. Tenant and function names must match [a-z0-9][a-z0-9_-]{0,62}, checked before any lookup or filesystem path is built.

The same review found that the KV store had no total limit. Values were capped at 1 MiB each, but a guest could put unique keys forever. It now has per-call caps and a per-tenant quota. The quota counts key bytes plus a per-entry overhead, not just value bytes. Without that, a guest could write millions of empty values under 512-byte keys and use gigabytes while its quota read zero.

Bug 7: a cache that stranded every function on upgrade

Compiling a component with Cranelift takes milliseconds to seconds. Loading a precompiled .cwasm takes 0.26 ms. So the control plane compiles once and the host deserializes.

Component::deserialize is unsafe, because wasmtime can't fully verify that a blob came from a compatible engine. My first cache named files by the SHA-256 of the source. After the wasmtime upgrade, every cached file failed to deserialize, the cache said "hit", nothing recompiled, and every function returned 500 until each tenant re-uploaded.

Now:

• Files are named {digest}-{compat}.cwasm, where compat comes from Engine::precompile_compatibility_hash(). An engine change is a clean cache miss.
• The source .wasm stays on disk, so a miss recompiles instead of failing.
• Digests must be exactly 64 lowercase hex characters before they're used in a path. A digest read from a pointer file must never become ../../somewhere handed to an unsafe deserializer.
• Writes go to a unique temp file, get fsynced, then are atomically renamed. The first version used {digest}.{pid}.tmp, which collided when one process handled two uploads of the same module concurrently.

The rest of the hardening list

The smaller fixes still matter in a multi-tenant system:

• Uploads are compiled with bounded concurrency (a semaphore at half the cores) and type-checked before anything is written to disk. Otherwise one tenant can saturate the control plane, or fill the disk with modules that will never run.
• The per-tenant function quota takes a Postgres advisory lock on the tenant. My first lock was per (tenant, function), which let two concurrent uploads of different functions both see 99 and both pass.
• Metering goes through a bounded channel to a single batching writer. It's drained on shutdown and counts dropped rows, instead of spawning a task per call that piles up when the database is slow.
• API-key lookups use a short TTL cache that stores both hits and misses, plus a 2 s pool acquire timeout. Without it, a flood of junk tokens could exhaust the connection pool for every tenant.
• Logs are capped per call (100 lines, 64 KiB). A guest shouldn't be able to flood your log pipeline.

Numbers

These were measured on an 8-core i5-9300H laptop against a trivial echo guest. They show runtime overhead, not real workloads.

Metric	Target	Result
Load a precompiled module	< 1 ms	0.26 ms
Warm call p99	< 5 ms	0.12 ms (p50 0.05 ms)
Calls per second, one core	> 1,000	~16,000
CPU cap accuracy (10/50/100 ms)	± 5 ms	within ± 0.7 ms
Memory cap (16 MiB)	holds	peak 15.8 MiB, trapped

Every call gets a fresh Store. The only thing reused between calls is the loaded component and its pre-linked, type-checked instance template (HandlerPre). Instance pooling would push these numbers further, and it's on the list.

What I'd tell someone building one of these

1. Use the runtime's limits, but read what they count. "Per memory" versus "per store" is a factor of N.
2. Every global in the runtime is shared across tenants. The epoch counter is the obvious one; the connection pool and the log pipeline are the less obvious ones.
3. An egress allowlist is a stack of checks, not one. Hostname, redirects, proxies, DNS answers, IP literals and embedded IPv4 all need covering.
4. Isolate structurally, then validate at the boundary. Either one alone left a gap.
5. Measure your caps. The 12% timer drift was invisible until a benchmark compared actual trap time to budget.

What's next

Instance pooling, S3-backed module storage, time-window upload rate limits, and deduplicating concurrent cold loads of the same module. If you're building something similar, or you find a hole in the isolation model, issues and PRs are very welcome.

• GitHub: https://github.com/Bunty9/warpline
• cargo install warpline-host warpline-control