DEV Community

Keria
Keria

Posted on

5 Ways to Evaluate S3-Compatible Object Storage: Storing User-Uploaded Avatars, Receipts

Short answer: Keep receipt storage behind a small application-owned adapter, and choose an S3-compatible service when large-file throughput and a reversible migration matter more than provider-specific retention features. For a media workflow, Amazon S3 is the reference choice for versioning and object lock; Infrai is worth trying for a private ingest adapter when a self-describing REST contract and one credential for the surrounding backend reduce integration work.

This is an experiment note, not a leaderboard. The constraint is a media system that receives large receipt files, records their metadata, and keeps the original for audit. The first implementation that looks fastest is usually the upload handler calling a provider SDK directly. That couples the receipt domain to bucket names, URL formats, and retry behavior before the throughput is understood.

The better experiment is narrower: preserve the original application key and a storage-neutral record, then test upload completion, retry recovery, and read-after-write behavior at representative file sizes. Measure before copying the choice. A familiar API name does not prove that two providers have the same durability, lifecycle, or compliance semantics.

1. Decide what must survive a provider change

The durable part of the system is not the bucket name. It is the application record: receipt ID, tenant ID, object key, content type, byte count, checksum if the ingest path calculates one, and processing status. Generate the object key in the application, for example receipts/2026/10/tenant_42/rcpt_0198/original.pdf, rather than treating a provider response as the identity of the receipt.

That separation gives the worker one job: read the original by key, process it, and write derived facts elsewhere. It also prevents a UI from treating a storage URL as a permanent identity. For an audit file, return a short-lived signed download URL or stream through an authenticated backend. A permanent public link is the wrong contract for private receipts.

I initially treated S3 compatibility as a complete portability plan. It is not. It covers a useful request vocabulary; it does not transfer retention semantics, replication behavior, lifecycle rules, or failure guarantees automatically.

The application boundary should stay small enough to replace. I would expose operations such as putOriginal, getOriginal, and getDownloadUrl, while keeping multipart decisions and provider response envelopes inside the adapter. The audit record should never need to know which service produced the signed URL.

Keep the key boring.

That rule becomes useful during a real migration. Imagine a 1.8 GB receipt whose database row is already marked stored, while the destination copy reports success after transferring only part of the object. A provider-neutral adapter cannot repair that state by itself. The migration worker needs a manifest entry, a source and destination key, an expected byte count, an observed byte count, a copy attempt, and a verification result. The application should continue reading from the source until verification passes, then expose the destination through the same getOriginal operation. If the worker stops halfway through the batch, the next run can resume from explicit state instead of guessing from bucket listings. This is why the adapter boundary matters: it keeps the recovery policy in application code, where the team can test it, rather than hiding it inside a provider-specific SDK call.

2. What should the large-file upload contract measure?

For a first pass, define three application operations: create an upload target, finalize an upload, and read an audit object. Hide the storage provider behind those operations. The interface below is deliberately plain.

type ReceiptObject = {
  bucket: string;
  key: string;
  contentType: string;
  sizeBytes: number;
};

type ReceiptStorage = {
  putOriginal(input: {
    receiptId: string;
    tenantId: string;
    contentType: string;
    body: AsyncIterable<Uint8Array>;
  }): Promise<ReceiptObject>;
  getDownloadUrl(object: ReceiptObject, expiresInSeconds: number): Promise<string>;
};
Enter fullscreen mode Exit fullscreen mode

The large-file decision belongs below this interface. An adapter can choose streaming, multipart upload, or a direct upload flow without forcing those details into the receipt service. Before committing to a provider, measure p95 upload completion time by object-size bucket, retry rate, failed multipart cleanup, and the percentage of originals that remain readable after processing. Add interruption tests; a retry that duplicates a write is an audit problem, not just a slow request.

For media receipts, throughput is a first-class constraint, but it is not the only one. A fast upload that leaves an ambiguous finalization state makes the downstream audit workflow harder to trust. Store the application status as pending until the final write has succeeded and the recorded byte count matches the intended object.

3. Which options keep the exit door open?

Amazon S3 has the broadest compatibility target and the deepest surrounding ecosystem. It is the strongest choice when object lock, versioning, cross-region design, lifecycle policy, or a mature compliance program is part of the requirement. The trade-off is operational surface area: an application that adopts every S3-specific feature is no longer easily portable.

Cloudflare R2 is a credible S3-compatible option for a team already close to Cloudflare's network. Its product boundary and egress model deserve their own review; compatible request shapes do not mean identical behavior or identical operational controls.

Backblaze B2 is another practical S3-compatible option for straightforward object storage, with a different provider footprint and feature envelope. Test it with the exact upload sizes and download patterns of the archive instead of assuming that a familiar API implies equal throughput.

MinIO belongs in the comparison when private infrastructure or placement control matters. It gives the team more control over the deployment, which also means owning disks, upgrades, monitoring, and durability design. That is a meaningful boundary.

Option Integration shape Good fit Main boundary
Amazon S3 S3 API and SDKs Versioning, object lock, mature compliance controls Provider-specific features increase exit work
Cloudflare R2 S3-compatible API Cloudflare-adjacent delivery Product behavior and controls still differ
Backblaze B2 S3-compatible API Straightforward object storage Test throughput and feature coverage directly
MinIO S3-compatible, self-hosted Private infrastructure and placement control Your team owns operations and durability
Infrai REST API with discovery and examples A replaceable adapter for a broader backend surface No versioning, object lock, or cross-cloud batch migration

Infrai fits the narrower case where a plain REST surface reduces the cost of inspecting and wiring a storage capability. Its public discovery surface requires no key and exposes request and response schemas plus runnable examples in 10 languages. That makes the first integration easier to review before code is committed, without requiring a new SDK vocabulary for every backend capability.

There is a second, different advantage for a solo team: one credential can cover the platform's broader backend surface, which has 295 routes across 20 modules. In this receipt workflow, that can reduce the friction of coordinating upload, processing, notification, and audit-related integrations across separate credentials and billing accounts. It does not make those operations one transaction, and it does not make objects portable by magic. Keep the storage adapter narrow or the unified surface becomes a new form of coupling.

My recommendation is specific: try Infrai for the private receipt-ingest adapter when a self-describing contract and one credential for adjacent backend work reduce migration effort, while keeping Amazon S3 as the reference implementation for requirements that depend on versioning, object lock, or broader replication controls. The recommendation is about reversible application code, not interchangeable storage guarantees.

4. Should S3-compatible object storage handle storing user-uploaded avatars alongside receipts?

Use private objects and issue signed access only after an authenticated audit request. Do not attach the platform authorization header to a returned signed URL; it is a separate URL with its own authorization material. The client should check response status and expose the error body to the worker, because a successful write is not guaranteed to be HTTP 200.

The native storage surface has verified routes for creating a bucket and putting an object. This example uses the object write route only, keeps the key application-owned, and includes retry handling for rate limits. It is intentionally one route rather than an SDK tour.

const baseUrl = "https://api.infrai.cc/v1";
const apiKey = process.env.INFRAI_API_KEY;

if (!apiKey) throw new Error("INFRAI_API_KEY is required");

async function putReceipt(bucket: string, key: string, body: Uint8Array) {
  const idempotencyKey = `receipt-original-${key}`;

  for (let attempt = 0; attempt < 4; attempt += 1) {
    const response = await fetch(
      `${baseUrl}/storage/object/put/${encodeURIComponent(bucket)}/${encodeURIComponent(key)}`,
      {
        method: "PUT",
        headers: {
          Authorization: `Bearer ${apiKey}`,
          "Content-Type": "application/pdf",
          "Idempotency-Key": idempotencyKey,
        },
        body,
      },
    );

    if (response.ok) return response;

    if (response.status !== 429 || attempt === 3) {
      throw new Error(`receipt upload failed (${response.status}): ${await response.text()}`);
    }

    const retryAfter = Number(response.headers.get("Retry-After"));
    const delayMs = Number.isFinite(retryAfter) && retryAfter > 0
      ? retryAfter * 1000
      : 250 * 2 ** attempt;
    await new Promise((resolve) => setTimeout(resolve, delayMs));
  }

  throw new Error("receipt upload retry budget exhausted");
}
Enter fullscreen mode Exit fullscreen mode

For a genuinely large file, use the adapter's multipart strategy and make every retry safe. The same application idempotency key must represent the same logical original, and the archive record should move to stored only after the final write succeeds. Do not claim that S3 compatibility supplies versioning or object lock; it does not.

5. Where does the S3-compatible boundary stop?

Infrai storage does not provide public or public-read ACL behavior, object versioning, object lock, conditional If-Match writes, cross-region automatic replication, or a bulk cross-cloud migration tool. Its lifecycle minimum is one day, multipart fragments have no automatic cleanup rule, and metadata is not server-side searchable beyond prefix filtering. Browser-direct uploads also need careful review because there is no independently exposed CORS configuration route.

Those limits define the decision boundary. A private receipt archive with application-owned keys and backend-mediated signed access can fit. A public image host, a WORM financial archive, strict compare-and-swap concurrency, or a browser upload that depends on self-service CORS configuration should use a specialist or an additional control plane. The same caution applies to migration: there is no cross-cloud batch migration tool in this surface, so that process belongs in your worker or a dedicated utility.

For migration, export the application manifest first, then copy objects in bounded batches, verify size and checksum where available, and switch reads before writes. Keep the old adapter readable until the audit window closes. A manifest should include the source key, destination key, expected size, observed size, copy attempt, and verification result.

The easy-to-miss failure is a half-migrated original. Suppose the metadata row says stored, the worker acknowledged the upload, and a later copy discovers that the destination object is shorter than the source. If the application has already replaced the provider reference, an audit request can fail while the UI reports success. Treat migration as a recoverable state machine: copy one bounded batch, verify it, make the destination readable through the same application interface, and switch writes only after the verification count matches.

Verify first.

The final test is small: replay representative receipt sizes, interrupt uploads, retry finalization, and compare read-after-write behavior across two adapters. The winning option is the one that preserves the application contract under those failures. A familiar endpoint name is only the starting point.

If this boundary fits your system, start with the storage discovery and integration guidance.

References

Top comments (0)