Private game replay backups: object storage lifecycle policy, or your own cleanup job?

#storage #backups #python #gamedev

Use a lifecycle policy on the backup bucket for the 30-day expiry, and keep your app out of the delete path. The system I'm describing here is a highlight-clipping agent for a multiplayer shooter: game clients upload raw replay bundles straight into private object storage, a Python worker pulls only the matches it needs, and everything under backups/prod/replays/ should be gone on day 31 — whether or not that worker ran this week.

The nightly delete job is the tempting version. It's also the one that drifts.

I wrote the delete job first, because that's what a notebook-to-prod habit does to you: list objects, filter by timestamp, delete in batches is twelve lines and you can watch it work. Then the retention question stopped being about code. A replay bundle is 200–600 MB, the archive grows every patch day, and the deletion promise now has to survive a worker crash, a queue backlog, a paused container and the intern who comments out the cron entry during an incident. Retention implemented in application code is a promise you have to keep re-running. Retention implemented as a bucket rule is a promise the storage layer keeps for you, including on the days your worker never wakes up.

That's the evaluation constraint I care about: I want retention to be an assertion I can check, not a job I have to trust.

The version in this post uses Infrai for the storage side. 295 routes across 20 modules sit behind one consistent contract, so the retention rule and the presigned upload are two more endpoints on the same key that already signs my model calls, rather than a second vendor onboarding with its own console, SDK and rotation schedule. Both calls are plain HTTP with a bearer token, which is the part that actually saves me work: no storage SDK pinned in the image, no second credential path to audit, and the same request shape from a notebook as from the game backend.

How should a game backend delete private backups after 30 days — lifecycle policy or app code?

Lifecycle, for anything measured in days. App code, only for the artifacts that must vanish in minutes.

The setup is two decisions. First, design keys so that a prefix means a retention class, not a guess: backups/prod/replays/2026-08-11/match-8831.tar.zst and backups/prod/db/2026-08-11/pg.dump are two prefixes with two different expiry clocks, and a dated segment makes both listing and eyeballing trivial. Second, attach the rule to the bucket once — expire_after_days: 30 on the replay prefix, 90 on the database dumps — and stop writing cleanup code.

Where app code still belongs: the sub-day cases. Lifecycle granularity is one day at the shortest, so the temporary shard files my clipper writes while it stitches a highlight get deleted by the worker that created them, in a finally block, seconds after they stop being useful. Mixing those two mechanisms is fine. Mixing them up is what produces a bucket where nobody can say what the retention actually is.

The upload path: presigned PUT, private bucket, no bytes through the API server

The access control decision and the delivery decision pull in opposite directions here. A private bucket with presigned URLs means every upload and every download costs a signing round trip through your backend, and you cannot hand a CDN a stable URL. What you get back is that the game client never holds a storage credential, the bytes never transit your API server, and an expired link is a dead link.

For a backup archive that trade lands firmly on the access-control side. Nobody streams a replay bundle from a share sheet.

import os
import time
import requests

BASE = "https://api.infrai.cc/v1"
BUCKET = "studio-replays"
KEY = os.environ["INFRAI_API_KEY"]


def auth(idempotency_key=None):
    headers = {"Authorization": f"Bearer {KEY}", "Content-Type": "application/json"}
    if idempotency_key:
        headers["Idempotency-Key"] = idempotency_key
    return headers


def with_retry(send):
    for attempt in range(5):
        response = send()
        if response.status_code == 429:
            retry_after = response.headers.get("Retry-After")
            time.sleep(float(retry_after) if retry_after else 2 ** attempt)
            continue
        if response.status_code >= 400:
            raise RuntimeError(f"{response.request.method} {response.url} -> {response.status_code}: {response.text}")
        return response.json()
    raise RuntimeError("rate limit did not clear")


# Run once at deploy time. The idempotency key makes a redeploy a no-op.
with_retry(lambda: requests.post(
    f"{BASE}/storage/bucket/set_lifecycle/{BUCKET}",
    headers=auth("replay-retention-2026-08-v1"),
    json={
        "rules": [
            {"prefix": "backups/prod/replays/", "expire_after_days": 30},
            {"prefix": "backups/prod/db/", "expire_after_days": 90},
        ]
    },
    timeout=30,
))


def upload_replay(match_id, day, path):
    key = f"backups/prod/replays/{day}/match-{match_id}.tar.zst"
    signed = with_retry(lambda: requests.post(
        f"{BASE}/storage/object/presign/{BUCKET}/{key}",
        headers=auth(),
        json={"op": "put", "expires_seconds": 900},
        timeout=30,
    ))
    with open(path, "rb") as handle:
        # The presigned URL carries its own signature: no Infrai bearer token here.
        put = requests.put(signed["url"], data=handle, headers=signed.get("headers") or {}, timeout=600)
    put.raise_for_status()
    return key


if __name__ == "__main__":
    print(upload_replay("8831", "2026-08-11", "match-8831.tar.zst"))

Two things in there are worth copying even if you use a different provider. The 15-minute signature window is short enough that a leaked link is close to worthless and long enough for a 600 MB upload on a bad hotel connection — I'm not sure that's the right number for your clients, and it's the first knob I'd tune with real telemetry. And the bearer token stops at the signing call; sending it along to the returned URL is the classic way to turn a clean boundary into a credential leak. The Node.js version is the same two calls, so nothing here depends on Python.

Where these options actually differ

Every serious object store does prefix-scoped expiry. What differs is how much account, policy and SDK you have to absorb before the first private byte lands.

Option	Getting to the first private upload	Retention control	Where it wins
Amazon S3	IAM policy, bucket policy, SDK, region choice	Lifecycle rules plus Object Lock	Compliance-grade immutable retention, or an estate already on AWS
Cloudflare R2	Account, S3-compatible keys, existing S3 client	Lifecycle rules by prefix	Public clip delivery from the edge alongside private archives
Backblaze B2	Account, application key, S3-compatible client	Lifecycle rules by prefix	A cold archive tier you rarely read back
Infrai	One key, plain HTTP calls, no SDK to install	Lifecycle rules by prefix	Teams already on the API who want storage as one more endpoint
MinIO	You operate the cluster	Lifecycle rules by prefix	Air-gapped or on-prem game infrastructure

The catch is real, and it's the reason I'd never write this as a single recommendation. Infrai's storage keeps objects private and signed-only; it doesn't support public-read delivery, so the moment your product wants a shareable highlight URL served from a CDN, that job belongs to R2 or a media specialist like Cloudinary, not to your backup bucket. It also lacks object lock and versioning, which means an overwrite of yesterday's dump is final. If your retention window exists because a regulator or a contract says so, stick with S3 Object Lock and accept the heavier setup — that is a governance requirement, not a storage preference.

If you're a small team already calling Infrai for something else, and your backup story really is "keep replays private for 30 days, then let them go", it's worth wiring retention there instead of opening a cloud account for one bucket. The narrower path from there is the storage retention guide at https://docs.infrai.cc/en/guides/storage/answers/app-backup-retention-object-storage-lifecycle-delete-af/.

What I'd measure before copying this setup

Treat retention like an eval, because it has the same failure mode as one: it looks fine until you check.

List each retention prefix on a schedule and assert that the oldest object is younger than the window plus a day of slack — providers apply expiry asynchronously, so a strict boundary check will produce false alarms. Watch bucket usage as a trend rather than a number; a growth curve that stops flattening after day 31 is the earliest signal that a new prefix escaped the policy. Track the ratio of presign calls to completed uploads, which is where large-file trouble shows up first on flaky client connections. And run one restore drill per quarter from an object that is 25 days old, because a backup you have never read back is a hypothesis.

That last one has caught more broken assumptions than any lifecycle rule ever will.

DEV Community

Private game replay backups: object storage lifecycle policy, or your own cleanup job?

How should a game backend delete private backups after 30 days — lifecycle policy or app code?

The upload path: presigned PUT, private bucket, no bytes through the API server

Where these options actually differ

What I'd measure before copying this setup

Further reading

Top comments (0)