DEV Community

Mukesh
Mukesh

Posted on

Scale-to-Zero Background Workers on Vultr: A Graceful Drain Pattern for Solo Developers

#vultr #autoscaling #backgroundjobs #devops

The problem with "just turn it off when idle"

If you run background workers on Vultr — a queue consumer, a scraper, a batch job runner, an autonomous agent that wakes up to do work — you've probably hit the same tradeoff. Leave the instance running 24/7 and you're paying full price for capacity that's idle 90% of the time. Write a cron job that powers off the instance when the queue is empty, and eventually you'll kill a worker mid-job, lose the work, and spend an evening explaining to yourself why a customer's export never finished.

The naive version of "scale to zero" looks like this:

# DON'T do this
if [ "$(queue_depth)" -eq 0 ]; then
  vultr-cli instance delete $INSTANCE_ID
fi
Enter fullscreen mode Exit fullscreen mode

It works fine in testing, because in testing the queue is always empty when you check it. In production, a job can arrive in the gap between your depth check and the delete call, or a worker can already be three minutes into a nine-minute job when the scaler decides the queue "looks idle." You need the scale-down decision to be aware of in-flight work, not just queued work, and you need the worker itself to cooperate with shutdown instead of being yanked out from under a job.

This is the pattern I use to run a fleet of Vultr workers that scales from zero to several instances and back, without losing jobs, and without needing Kubernetes, a service mesh, or anything beyond the Vultr API, systemd, and a Postgres table.

Step 1: track in-flight work with leases, not just a queue

A queue length of zero doesn't mean nothing is happening — it means nothing is waiting. The signal your scaler actually needs is "how many workers currently hold a lease on a job." A simple leases table does this:

CREATE TABLE job_leases (
  job_id      UUID PRIMARY KEY,
  worker_id   TEXT NOT NULL,
  leased_at   TIMESTAMPTZ NOT NULL DEFAULT now(),
  expires_at  TIMESTAMPTZ NOT NULL
);
Enter fullscreen mode Exit fullscreen mode

A worker inserts a row (with a short TTL, renewed via heartbeat) when it picks up a job and deletes it on completion. Your scaler's "is it safe to remove a worker" check becomes:

SELECT count(*) FROM job_leases WHERE expires_at > now();
Enter fullscreen mode Exit fullscreen mode

If that count is zero and the queue is empty and it's stayed that way for a cool-down window (I use 90 seconds), only then is it safe to scale down. This single table eliminates the race condition — a worker that just grabbed a job is visible to the scaler before it's visible as "busy" in any process-level metric.

Step 2: make the worker drain instead of die

Even with accurate lease data, you still want the worker to shut down cleanly rather than being hard-killed, because there's an unavoidable window between the scaler's check and the actual instance deletion. Handle SIGTERM in the worker to finish the current job, release its lease, and only then exit:

import signal, sys, time

shutting_down = False

def handle_sigterm(signum, frame):
    global shutting_down
    shutting_down = True

signal.signal(signal.SIGTERM, handle_sigterm)

while True:
    if shutting_down:
        release_lease(current_job_id)
        sys.exit(0)
    job = claim_next_job()
    if job is None:
        time.sleep(2)
        continue
    lease_job(job.id, ttl_seconds=120)
    run_job(job)          # heartbeat renews the lease every 30s
    release_lease(job.id)
Enter fullscreen mode Exit fullscreen mode

Pair this with a systemd unit that gives the process real time to exit instead of the 90-second default:

[Service]
ExecStart=/usr/bin/python3 /opt/worker/worker.py
KillSignal=SIGTERM
TimeoutStopSec=300
Enter fullscreen mode Exit fullscreen mode

With this in place, deleting the instance sends SIGTERM on shutdown, the worker finishes whatever job it's mid-way through (up to 5 minutes), and only then does the instance actually go away. Vultr's own instance deletion doesn't wait on your process, so the scaler is the one responsible for calling a graceful stop before it calls delete — which is the next piece.

Step 3: scale down through the Vultr API, not a power switch

Don't power off instances you plan to leave stopped — you're still billed for the attached block storage and, depending on plan, sometimes the instance itself. For true scale-to-zero, delete the instance and recreate it from a snapshot when load returns. Build the snapshot once, after your worker image is configured:

curl -s -X POST "https://api.vultr.com/v2/snapshots" \
  -H "Authorization: Bearer $VULTR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"instance_id": "'"$BASE_INSTANCE_ID"'", "description": "worker-base"}'
Enter fullscreen mode Exit fullscreen mode

Scaling up spins a new instance from that snapshot with a cloud-init script that starts the worker service on boot:

curl -s -X POST "https://api.vultr.com/v2/instances" \
  -H "Authorization: Bearer $VULTR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "region": "ewr",
    "plan": "vc2-1c-2gb",
    "snapshot_id": "'"$SNAPSHOT_ID"'",
    "label": "worker-'"$(date +%s)"'",
    "tag": "autoscale-worker"
  }'
Enter fullscreen mode Exit fullscreen mode

Scaling down sends SIGTERM over SSH first, polls for the process to exit (or the timeout to elapse), then deletes:

ssh worker@$IP 'sudo systemctl stop worker'   # triggers graceful drain
# poll: systemctl is-active worker == inactive, or wait up to 300s
curl -s -X DELETE "https://api.vultr.com/v2/instances/$INSTANCE_ID" \
  -H "Authorization: Bearer $VULTR_API_KEY"
Enter fullscreen mode Exit fullscreen mode

Boot-to-ready from a pre-baked snapshot on Vultr is typically well under a minute, which matters — if scale-up is slow, you'll either over-provision out of caution or make callers wait.

Step 4: guardrails that keep the scaler boring

The scaler itself is a small polling loop (cron every 30s, or a long-running process) with three deliberately conservative rules:

  • A floor of at least one warm worker during business hours if job latency matters more than the marginal cost of one instance — scale-to-zero doesn't have to mean zero when a job arrives.
  • A cool-down of 2-3x your job's max expected runtime before either scaling up or down again, so you don't oscillate on a bursty queue.
  • A hard cap on concurrent instances, tied to a monthly cost ceiling, checked before every scale-up call — this is the line that stops a bug in your producer from spinning up fifty $10/month instances overnight. 
def should_scale_down(leases, queue_depth, last_scale_event):
    idle_for = now() - last_scale_event
    return leases == 0 and queue_depth == 0 and idle_for > timedelta(seconds=90)
Enter fullscreen mode Exit fullscreen mode

What this actually saves

On a workload with a bursty daytime queue and near-zero overnight traffic, running a single always-on vc2-1c-2gb worker costs the same whether it processes one job or a thousand. Scaling to zero overnight and during quiet weekend stretches cut real compute hours by roughly 60% in my own fleet, with zero dropped jobs across several months — the leases table and the drain handler are what make that safe to claim, not just the deletion API call.

The pattern generalizes past Vultr: any provider with a snapshot-and-recreate API and per-second or per-minute billing supports the same shape. The two pieces that actually do the work — lease-tracked in-flight jobs and a worker that drains instead of dying — are provider-agnostic. Vultr's API just makes the create/delete/snapshot loop cheap and fast enough that scaling to true zero is worth doing instead of settling for a permanently-on box "just in case."

Top comments (0)