DEV Community

Cover image for 10 agents, 30 GB of RAM and one queue: how they share without clashing
José Antonio Garrido
José Antonio Garrido

Posted on Originally published at josegarrido.dev

10 agents, 30 GB of RAM and one queue: how they share without clashing

For the two weeks after I published Sluice 0.8.0, I ran it every day on my two machines, both with 30 GiB of RAM, with up to 10 AI agents working at once on one of them. Sluice puts the tests, builds and heavy commands of all the AI agents on a Linux machine into one queue, with a shared CPU and memory budget, and runs each check once per git tree; I told how it started in the first article. Even with that queue, my agents' test suites could still run a machine out of memory. How they share it now, in short: the queue decides who runs, the kernel's ceilings keep each job within its share, and the desktop guard keeps the desktop's share free. This is what I learned, and each lesson became a version, from 0.9.0 to 0.12.2:

  • A retry is still the same suite (0.9.0). Memory admission carries a gate's estimate to its variants and waits after every memory kill.
  • When the machine thrashes, only the kernel arrives in time (0.10.0). Every gate job (tests, linters, type checks) runs under a memory ceiling the kernel enforces, even while everything else is stalled.
  • A browser doesn't stay where you start it (0.11.0). Chromium, and so every Playwright browser, moves itself out of its job; Sluice brings it back, so its memory counts, is limited and dies with the job.
  • The desktop needs a say (0.11.0). Each job gets a swap ceiling too, and no job starts while your desktop session is stalling on memory.
  • The environment is part of what you test (0.11.0). Variables such as NODE_OPTIONS go into the cache key.
  • A daemon shouldn't inherit whoever starts it (0.12.0). One agent session's environment, token included, no longer reaches everyone's jobs.

A retry is still the same suite

A week after publishing 0.8.0 I went through what had happened on my second machine: 12 threads, 30 GiB of RAM, 8 GiB of swap, and between 4 and 10 agents working at once. In 37 hours there were 11 global out-of-memory kills, 7 kills by systemd-oomd, 16 test jobs killed for memory and one agent session killed along with them. Swap was full every time.

There were four causes:

  • After a kill, the agents retried with --filter or --test-concurrency=1. That was a new command for Sluice, with no history, so it guessed 1.3–1.7 GiB for a suite that needs 11–14 GiB.
  • The safety margin after a kill compounded up to 1.5×.
  • Memory outside the queue (agents, builds, browsers) grew while a suite ran.
  • The agents' systemd-oomd window was 5 s, and it ended an agent 6 s after a test job was killed.

Version 0.9.0 changed the memory admission:

  • A variant of a command with other arguments inherits the estimate of its gate instead of a guess.
  • Only a guess can start as a probe.
  • The margin after a kill is 1.25×.
  • After any memory kill, nothing starts for 10 s, and not until the pressure goes down.
  • The memory reserve grows to what the rest of the machine grew while heavy jobs ran (the 90th percentile, at most a quarter of the RAM).
  • The agents' systemd-oomd window is 15 s and the jobs' is 8 s, so the job always goes first.

sluice status now says where the reserve comes from, and flags a job that started alone beyond its estimate. No config change needed.

When the machine thrashes, only the kernel arrives in time

With 0.9.0 the same machine hit the next wall. Between 30 October and 1 November one test suite ran 61 times, every time alone on the machine and beyond its estimate: 46 passed, 9 failed their tests and 6 were killed for memory. Each of those six kills was a global OOM, and before each one the system journal fell silent for 22, 30, 37, 67, 72 and about 150 seconds. The whole machine was thrashing, and systemd-oomd and Sluice's own guard were stalled with everything else: anything that runs in userspace arrives too late.

Version 0.10.0 hands that job to the kernel. Every second the daemon writes the memory.max of each job's cgroup, so that all jobs together can never take the machine below a floor. The kernel enforces it even while the daemon is frozen. It is on by default with full containment:

# ~/.config/sluice/config.toml
[admission]
memory_reserve = "auto"   # a tenth of the RAM: 3 GiB on 30 GiB
memory_ceiling = true
memory_floor = "auto"     # half the reserve: 1.5 GiB on 30 GiB
Enter fullscreen mode Exit fullscreen mode

sluice status shows each running job's ceiling. From 2 November, the next 387 jobs on that machine ran with no OOM and no memory kill, and swap never went above 36 %.

The trade-off is deliberate: a run that used to pass by pushing the rest of the machine into swap is now killed inside its own job. The machine stays usable, and the message tells the agent to retry once memory is free, or to tell you that the gate doesn't fit.

A browser doesn't stay where you start it

On 5 November it was my development machine's turn: 20 threads, 30 GiB of RAM, 8 GiB of swap, GNOME on Wayland, Sluice 0.10.0. Between 18:40 and 19:20 a frontend test suite of about 14.5–15 GiB started at three times its estimate, Playwright jobs took 7.7–9.4 GiB each, and a production build ran outside Sluice at the same time. The machine reached 21 GiB used (66 %), swap went from 3.6 to 5.2 GiB, swap-out peaked at 1,759 pages per second, load hit 32, and GNOME Shell's input lag reached 61–80 ms. There was no OOM and nothing was killed: the 0.10.0 ceiling never engaged. The shell ended up not responding, and at 20:37 I rebooted with SysRq.

I still don't know what hung the shell: it stayed hung after the memory recovered. But looking into it I found something else. That day systemd had created 385 Chromium scopes on the machine, and that is where the browsers started by the test jobs had gone: outside their jobs.

This is what happens. At startup, every Chromium browser process, headless included, prepares the connection to the XDG desktop portal. The portal needs an app id, and Chromium gets one by asking the systemd user manager, over the session bus, for a new scope with its own pid: StartTransientUnit. The process ends up in app.slice/app-org.chromium.Chromium-<pid>.scope. There is no flag to turn it off.

For a job runner that means the browser's memory leaves the job's accounting, its learned peak, its memory.max and its kill. And Playwright starts browsers in a process group of their own, so a process-group kill misses them too.

I checked it on 6 November with a probe: a headless shell running for 6–8 seconds, about 200 MB. The browser process moved to its own scope; the renderers it had already forked stayed in the job.

The cheap fixes didn't hold up:

  • Unsetting DBUS_SESSION_BUS_ADDRESS only works for Chromium, and breaks the tests that need the session bus.
  • The FLATPAK_SANDBOX_DIR trick, which skips the move, stopped working in Chromium on 14 October.
  • A cgroup namespace doesn't help: systemd makes the move from the root namespace.

What works is a property of cgroup v2. To move a process, the kernel only asks for write access to the cgroup.procs of the common ancestor of both cgroups, and that ancestor is user@UID.service, which belongs to you. So since 0.11.0, every second the daemon walks each job's process tree and writes any descendant it finds outside back into the job's cgroup. This is a job on 0.12.1 that starts Playwright's headless Chromium (156) directly. The browser leaves its cgroup 265 ms after the job starts, and it's back in the same tick:

Sluice's daemon log on 0.12.1: chrome-headless leaves the job's cgroup for its own Chromium scope and is moved back in the same tick

INFO sluice_gate_daemon::server: moved a process of the job back into its cgroup job=1794597479890-504544-15a79e pid=643155 comm=chrome-headless from=/user.slice/user-$UID.slice/user@$UID.service/app.slice/app-org.chromium.Chromium-643155.scope
Enter fullscreen mode Exit fullscreen mode

When the job ends, whatever is left of it is killed, inside its cgroup or outside it: escapees through a pidfd, after checking the process start time, so a reused pid is never signalled. An integration test reproduces the escape with the same StartTransientUnit call.

It is on by default ([containment] reclaim_escaped = true), and it applies to every job: the setting lives in your user config, not in each gate. If your tests deliberately start something with systemd-run --user --scope, or daemonise a process that must outlive the job, the daemon will pull those back too; for that case, set it to false and Sluice only reports escapes. What Sluice itself places, in its agents scope or in any slice below its own (as when a test runs Sluice inside a job), is not an escape. Anything else, Chromium's scopes included, is moved back.

The desktop needs a say

The same version, 0.11.0, adds two more limits:

[admission]
swap_ceiling = true
swap_floor = "auto"     # a quarter of the swap
desktop_guard = true
Enter fullscreen mode Exit fullscreen mode
  • Swap ceiling. Every second the daemon also writes each job's memory.swap.max, so that the jobs together can't take the free swap below the floor. A job at its memory ceiling dies in its own cgroup instead of swapping the machine out.
  • Desktop guard. Sluice reads the memory pressure of your desktop session (session.slice of the systemd user manager). While it is at or above 10 % (some avg10, until it drops under 5 %), no job starts, not even alone on the machine, and the ceilings keep the whole reserve free.

Your jobs will wait while the desktop is under pressure; that's the point. sluice status shows each running job's ceiling and the session's pressure on its memory: line. Here with a single light job on an idle machine, so swap, pressure and escapes, which it only prints when they are not zero, don't show up:

sluice status on 0.12.1 with one light job running: its 22.1 GiB ceiling and the memory: line with the desktop's pressure at 0.0 %

The environment is part of what you test

Also in 0.11.0: a variable such as NODE_OPTIONS or RUSTFLAGS changes what a test checks without changing the tree. With key_env its value goes into the cache key, and requests with other values never share a pass. sluice init writes them per toolchain, together with the file the package manager rewrites on every install. On a throwaway npm project:

[gates.test]
cmd = "npm test"
weight = "auto"
timeout = "30m"
hooks = ["pre-push"]
key_inputs = ["node --version", "npm --version"]
key_env = ["NODE_OPTIONS", "NODE_ENV"]
key_files = ["node_modules/.package-lock.json"]
env = { CI = "1" }
Enter fullscreen mode Exit fullscreen mode

sluice doctor warns when a gate leaves them out:

[warn] key-env        gate 'test': key_env leaves out RUSTFLAGS, RUSTDOCFLAGS, CARGO_BUILD_TARGET (a change to them reuses a pass made with other values); needs sluice >= 0.11.0 to read key_env
Enter fullscreen mode Exit fullscreen mode

Adding key_env changes each gate's key once, so each gate runs once more.

A daemon shouldn't inherit whoever starts it

Every job descends from the daemon, and an agent session can start the daemon on its own. Since 0.12.0 the daemon doesn't keep the environment of whoever started it, so one agent session's id, API address and bearer token never reach anyone else's jobs. A check that asks "am I under an agent?" by walking up the process tree no longer mistakes your own runs for an agent's, and merge trains, which run their gates, their prepare and their key_inputs in the daemon's environment, never see the token.

The daemon keeps only an allowlist (PATH, HOME, the locale, XDG_*, the session bus, SLUICE_* and a few more), and a sluice daemon started by hand re-executes itself clean. The log names the dropped variables that look like credentials, without showing their values. Gates started by run, verify or a hook still get the environment of whoever asked for them. The API keys your shell profile exports no longer reach the daemon: merge trains get them from your login shell (the default tool_env), and with [queue] tool_env = "daemon" you have to name the ones they need, such as a proxy, in pass_env.

# ~/.config/sluice/config.toml
[daemon]
pass_env = ["HTTPS_PROXY"]
Enter fullscreen mode Exit fullscreen mode

And a gate can drop variables from its own environment with env_remove, for markers that env can't unset.

Update to 0.12.2

sluice update
Enter fullscreen mode Exit fullscreen mode

It verifies the checksum, and it only stops the daemon if it is idle: one running gates keeps going until its next restart. If you use the Claude Code plugin, update it too, and then restart your agent sessions, which keep the old sluice mcp until they reconnect:

claude plugin marketplace update sluice
claude plugin update sluice@sluice
Enter fullscreen mode Exit fullscreen mode

Then check it:

sluice doctor   # daemon-version and plugin both say 0.12.2, plus any key_env warnings
sluice status   # the daemon version, and the memory: line with the desktop's pressure
Enter fullscreen mode Exit fullscreen mode

0.12.1 fixes the escape detection of 0.11.0 and 0.12.0. If you're on either and can't update yet, turn off the escape handling in your user config:

# ~/.config/sluice/config.toml
[containment]
reclaim_escaped = false
Enter fullscreen mode Exit fullscreen mode

and restart the daemon while it is idle with sluice daemon stop (it drops queued jobs). With false Sluice neither moves processes back nor kills leftovers at the end of a job; the detection still runs, so you may see a burst of WARN lines. Once updated, remove the line to get the containment back, unless your own tests need it.

None of these versions breaks an existing config. The new keys are one-way: a shared .sluice.toml that uses key_env needs every machine on 0.11.0 or later, and one that uses env_remove, on 0.12.0.

Try it

You need Linux. Containment needs cgroup v2 and a systemd user manager, and the memory policy needs systemd-oomd with systemd 257 or later; in a container or on cgroup v1 Sluice runs without containment, and sluice doctor tells you why. The memory work in this article comes from Ubuntu 26.04 machines with kernel 7.0.

If you're new to it: on a 4-CPU GitHub runner, with 0.9.0 (median of 3 rounds), Sluice cut the time until every agent had committed from 103 to 72 s with 4 agents (−30 %) and from 213 to 119 s with 8 (−44 %).

curl -fsSL https://raw.githubusercontent.com/josegarridodigio/sluice/main/scripts/install.sh | sh
sluice doctor
sluice init --dry-run
sluice init
Enter fullscreen mode Exit fullscreen mode

The code, the CHANGELOG and the decision behind each change (ADR-0026 to ADR-0030) are at github.com/josegarridodigio/sluice. If it's useful to you, a star there helps other people find it. And if you want the next version when it ships, with what I learned running it, the newsletter is just below.

Top comments (7)

Collapse
 
aiapikey profile image
HAOHAOnovai •

The distinction worth making explicit: per-job kernel ceilings bound each job, but thrash is an aggregate event. Ten jobs each comfortably under their own ceiling can still fill the machine if their peaks align - and agent suites do align, because they are triggered by the same commits.

The invariant that prevents it is admission on the sum: admit a job only if the running total of predicted peaks stays under roughly 0.8x RAM, and predict from the high-water mark of that exact (tree, command) pair rather than the command alone. Same command on a different tree is a different peak.

With that in place the per-job ceiling becomes a backstop instead of the primary control, which is where it belongs.

Collapse
 
yeyo11 profile image
José Antonio Garrido •

Agreed that thrash is an aggregate event, and that's how both layers already work. Admission is on the sum: a job starts only if free memory minus the reserve minus the remaining headroom of every running job covers its expected peak. The ceilings aren't independent per job either: every second the daemon splits the headroom above a floor across the running jobs, so together they can't push the machine below it, even if their peaks line up.

Where I'd push back is keying by tree. Every commit changes the tree, so a (tree, command) high-water mark is almost always a cold start, and cold starts are exactly what hurt in 0.8.0: a retry with different flags got a 1.5 GiB guess for an 11–14 GiB suite. Peaks drift slowly from commit to commit. The real danger is a fresh guess, so Sluice keys by gate and command and gives an unseen variant its gate's largest known peak.

Collapse
 
aiapikey profile image
HAOHAOnovai •

Fair - tree-keyed marks go cold on every commit, so they would mostly record cold starts. The version I would actually keep keys on the command shape and the binary hash, and uses the tree only as a bucket for the margin, not for the value: a commit invalidates the margin estimate but not the measured peak.

That matches what you are doing with the per-tick split, incidentally. Your ceilings already recompute from MemAvailable every second, so the peak is a property of the binary and the input, while the margin is a property of the machine state right now. Keying one of those on a commit hash would be the mistake.

Collapse
 
ademisler profile image
Adem İşler • • Edited

the 265 ms chromium cgroup escape is a useful finding. with reclaim_escaped running once a second, a browser can allocate outside the job's memory.max before it's pulled back. have you stress-tested that window with a fast allocator, or measured the worst transient RSS outside the gate? the pidfd cleanup is a good separate safeguard.

Collapse
 
yeyo11 profile image
José Antonio Garrido •

Thanks. One clarification on the number: 265 ms is how long after the job started the browser left, not how long it stayed out. Detection and the move back happen in the same sampler tick, so the time outside is at most one tick, about a second. In that run it was a headless shell at around 200 MB, so the overshoot was small.

No, I haven't stress-tested that window with a fast allocator, and you're right that it's the open edge. Two things limit it. The ceilings are recomputed every tick from MemAvailable, so memory allocated outside the job still shrinks every job's ceiling on the next tick: it's missing from the job's accounting, not from the machine's. And Chromium's move is a one-off at startup, so anything it forks after we pull it back stays in the job. The daemon already logs every escape, so my next step is to measure the worst case outside the job from real runs. I'd do a synthetic stress test in a VM, not on a desktop.

Collapse
 
aiapikey profile image
HAOHAOnovai •

Thanks for the correction - one tick is a much smaller hole than 265 ms, and it makes the sampler interval the real budget. That is the number I would write next to the reserve, because it is the only window in which an out-of-band allocation is invisible to the ceiling.

The child-process test is cheap: allocate hard for exactly one tick and see whether the ceiling recompute or the OOM killer wins. If the killer wins, the reserve has to cover one tick of a fast allocator rather than one tick of a headless shell at 200 MB - and that is a much larger number.

Collapse
 
onelifevibe profile image
OneLifeVibe •

The Chromium escape is the part I had not thought about before. A queue can be perfect on paper and still lose accounting when the browser quietly moves itself into its own scope, so the one second reclaim loop feels like the right kind of fix, cheap and aimed at the actual leak. The desktop guard is the other detail that stands out. Holding jobs while the session is under pressure is a small courtesy that keeps the machine usable, and it is easy to see why the kernel ceiling alone was not enough once everything started thrashing.