DEV Community

IversonBlake8417
IversonBlake8417

Posted on

Four Metrics That Prove Daily Image Backup Retention on Compatible Object Storage

Use one private bucket, a few well-named prefixes, and three lifecycle rules — then spend the rest of your effort on the four metrics that tell you those rules actually ran. The deciding constraint here isn't price per terabyte. It's that a backup job fails loudly and a retention rule fails silently, so the cheap storage decision and the observability decision are really the same decision.

Here's the system I'm describing, because it changes which trade-offs matter. A plugin marketplace stores product images for every vendor on the platform — screenshots, icons, banners. Two things have to work every day: a nightly backup of app data and images, and an on-demand export of one tenant's images when that vendor asks for their data. Retention is 30 days for the daily copies and 90 days for the monthly ones.

Nobody gets paged when a rule stops matching.

That's the failure I've watched teams eat. The nightly job stays green for months, the dashboard says "backup succeeded," and the bucket quietly keeps every object ever written because someone renamed a prefix from backup/ to backups/ and the lifecycle filter still pointed at the old one. You discover it from the invoice, or from a vendor asking why their screenshots from 14 months ago are still retrievable.

Before and after: who owns deletion

Before, deletion lives in your application. A cron job lists old keys, deletes them in batches, and writes a log line. It works until the job's credentials rotate, or the list call starts paginating past your page limit, or the container gets OOM-killed halfway through — and the only evidence is an absence of log lines nobody alerts on.

After, the bucket owns deletion and your app owns proof. The storage layer expires objects by rule; your code's job shrinks to one question asked once a day: does what's actually in the bucket match what the rules promised?

The flow reads like this: nightly job → backups/daily/2026-09-12/ → expiry rule at 30 days → next morning's listing or inventory pass → three gauges → one alert. Deletion and verification are deliberately not the same code path. If the same job that writes backups is also the job that proves retention, a single bad deploy blinds you to both.

What should I measure to trust daily backup retention and lifecycle rules?

Four signals. Three of them come out of a single listing pass, which is why this is cheap enough to run every morning.

Metric Where it comes from What it catches Page when
backup_objects_past_retention listing or inventory report, compared to the rule window a rule that silently stopped matching your keys above 0 for two consecutive days
backup_newest_object_age_seconds max LastModified per prefix the upload job died while its dashboard stayed green above 26 hours
backup_bytes_total same pass, summed per prefix growth that never plateaus, or a cliff that means the filter matched far more than you meant slope change sustained 3 days
export_denied_total access logs on the download path clock skew, rotated keys, callers replaying a dead link any sustained increase

The second one matters more than it looks. A count of successful backup runs tells you a process exited zero; the age of the newest object tells you bytes actually landed. Those are different claims, and only one of them survives a misconfigured endpoint.

Give the first metric a grace day. Expiry is asynchronous — objects can stay listable for a while after their expiration date, and the docs are explicit that storage charges stop at the expiration date even when the delete hasn't landed yet. Alerting at > 0 with no tolerance gets you a pager that cries every morning, and a team that mutes it by week two.

The rule I apply when delivery simplicity fights access control

Tenant-scoped data goes in a private bucket, always, and is delivered through short-lived signed URLs. Non-tenant assets — the marketplace's own logos, public plugin banners, anything already visible to anonymous visitors — live in a separate public bucket behind a CDN. One bucket per trust boundary, not one bucket per application.

Splitting them is the whole point. A public bucket plus a CDN is the simplest delivery story that exists: no signing, no expiry, cache hit ratios near 1.0, and no per-request credential math. But "simple delivery" and "tenant isolation" can't coexist in one bucket, because the moment a single key in it must be private, every object needs an authorization decision, and you've paid the complexity cost for the marketing images too. So pay it once, in the bucket that holds tenant data, and keep the boring public bucket boring.

A signed URL is a bearer token. That framing has saved me more design arguments than any diagram: it carries the signer's permission to whoever holds the string, it can't outlive the credentials that produced it, and a SigV4 presigned URL caps out at seven days regardless of what you pass. For a vendor downloading their own export, 10 minutes is plenty. For an email link a human might open tomorrow, sign on demand behind an authenticated endpoint instead of mailing the URL itself.

The tenant export, end to end

Here's the lifecycle configuration first, with the third rule being the one people forget.

import { S3Client, PutBucketLifecycleConfigurationCommand } from "@aws-sdk/client-s3";

export const store = new S3Client({
  region: process.env.STORE_REGION ?? "us-east-1",
  endpoint: process.env.STORE_ENDPOINT,   // any compatible endpoint, EU or US
  forcePathStyle: true,
  credentials: {
    accessKeyId: process.env.STORE_KEY!,
    secretAccessKey: process.env.STORE_SECRET!,
  },
});

await store.send(new PutBucketLifecycleConfigurationCommand({
  Bucket: "marketplace-private",
  LifecycleConfiguration: {
    Rules: [
      {
        ID: "daily-30",
        Status: "Enabled",
        Filter: { Prefix: "backups/daily/" },
        Expiration: { Days: 30 },
        NoncurrentVersionExpiration: { NoncurrentDays: 30 },
        AbortIncompleteMultipartUpload: { DaysAfterInitiation: 7 },
      },
      {
        ID: "monthly-90",
        Status: "Enabled",
        Filter: { Prefix: "backups/monthly/" },
        Expiration: { Days: 90 },
        NoncurrentVersionExpiration: { NoncurrentDays: 90 },
      },
      {
        ID: "tenant-exports-7",
        Status: "Enabled",
        Filter: { Prefix: "exports/" },
        Expiration: { Days: 7 },
      },
    ],
  },
}));
Enter fullscreen mode Exit fullscreen mode

Exports are the sneaky part. They're derived copies of tenant data that land outside the prefix your backup rules cover, which makes them the longest-lived copy of that data in your account unless a rule names them. So the export gets its own short window, and the object key is built to land inside it.

import { GetObjectCommand, ListObjectsV2Command, PutObjectCommand } from "@aws-sdk/client-s3";
import { getSignedUrl } from "@aws-sdk/s3-request-presigner";

type Entry = { key: string; size: number; etag: string; lastModified: number };

export async function listPrefix(bucket: string, prefix: string): Promise<Entry[]> {
  const out: Entry[] = [];
  let token: string | undefined;
  do {
    const page = await store.send(new ListObjectsV2Command({
      Bucket: bucket, Prefix: prefix, ContinuationToken: token,
    }));
    for (const o of page.Contents ?? []) {
      out.push({
        key: o.Key!,
        size: o.Size ?? 0,
        etag: (o.ETag ?? "").replaceAll('"', ""),
        lastModified: o.LastModified?.getTime() ?? 0,
      });
    }
    token = page.IsTruncated ? page.NextContinuationToken : undefined;
  } while (token);
  return out;
}

export async function tenantExport(bucket: string, tenantId: string) {
  const files = await listPrefix(bucket, `tenants/${tenantId}/images/`);
  const manifest = {
    tenant: tenantId,
    generated_at: new Date().toISOString(),
    object_count: files.length,
    total_bytes: files.reduce((n, f) => n + f.size, 0),
    files,
  };

  const key = `exports/${tenantId}/${manifest.generated_at}.json`;  // inside tenant-exports-7
  await store.send(new PutObjectCommand({
    Bucket: bucket, Key: key,
    ContentType: "application/json",
    Body: JSON.stringify(manifest),
  }));

  const url = await getSignedUrl(store, new GetObjectCommand({
    Bucket: bucket, Key: key,
    ResponseContentDisposition: `attachment; filename="${tenantId}-images.json"`,
  }), { expiresIn: 600 });

  log.info("export.signed", { tenant: tenantId, key, objects: files.length, ttl_s: 600 });
  return url;
}
Enter fullscreen mode Exit fullscreen mode

The manifest carries size and ETag per object on purpose. A tenant export that lists 412 images with their checksums is a thing you can diff against a restore; a zip file is a thing you can only hope about.

Then the morning audit, reusing the same listing helper:

const WINDOWS: Record<string, number> = { "backups/daily/": 30, "backups/monthly/": 90 };
const DAY_MS = 86_400_000;

export async function auditRetention(bucket: string) {
  for (const [prefix, days] of Object.entries(WINDOWS)) {
    const files = await listPrefix(bucket, prefix);
    const cutoff = Date.now() - (days + 1) * DAY_MS;   // +1 day: expiry runs asynchronously
    const overdue = files.filter((f) => f.lastModified > 0 && f.lastModified < cutoff);
    const newest = files.reduce((m, f) => Math.max(m, f.lastModified), 0);

    metrics.gauge("backup_objects_past_retention", overdue.length, { prefix });
    metrics.gauge("backup_bytes_total", files.reduce((n, f) => n + f.size, 0), { prefix });
    metrics.gauge("backup_newest_object_age_seconds",
      newest === 0 ? Number.POSITIVE_INFINITY : (Date.now() - newest) / 1000, { prefix });

    if (overdue.length) {
      log.warn("retention.overdue", { prefix, days, count: overdue.length, sample: overdue[0].key });
    }
  }
}
Enter fullscreen mode Exit fullscreen mode

An empty prefix reporting infinite age is intentional. A gauge of zero overdue objects looks identical whether the rule is working perfectly or the prefix is empty because your writer broke — and those deserve different alerts.

Two objections worth answering

"Can't I set the rule once and trust it?" Not quite, and the reasons are mechanical rather than mysterious. Rule filters match on prefix, tags and object size, so any key-naming change can orphan a rule without erroring. On a versioned bucket, expiring the current version only writes a delete marker — the old versions keep billing until a noncurrent-version rule removes them, which is the single most common surprise on a bucket that looks "emptied." Lifecycle support across compatible providers is also a subset game in 2026: the operation name is shared, the semantics aren't identical, and at least one major provider models retention natively as days-from-upload-to-hidden plus days-from-hidden-to-deleted rather than markers plus noncurrent expiry. Read the provider's own retention page before assuming your config ported cleanly.

One cost caveat belongs here too, since the reader asking this question is optimizing for cheap. Check whether your tier bills a minimum storage duration. If it charges a 90-day floor, a 30-day expiry rule deletes the object and you still pay to day 90 — the rule is correct and the savings are imaginary. Egress is the other half: providers differ enormously on whether pulling your backups back out is free, metered, or capped at a multiple of stored bytes, and a restore test is exactly the moment you find out.

"Why instrument the export path at all — it's one object?" Because it's the path where access control actually gets decided, at request time, by your code. Log every signing event with tenant id, key, requester and TTL, and count denials separately from successes. When someone pastes a link into a shared channel, that log is the difference between a five-minute scope question and a week of guessing.

Honestly, your mileage may vary on the alert thresholds — 26 hours and two days of tolerance fit a nightly job with a daily audit, and you should shift them if your window is tighter. What shouldn't move is the split: the bucket deletes, your code proves, and the export is treated as data with its own retention rather than a convenience file.

References

Top comments (0)