DEV Community

HoratioFox1281
HoratioFox1281

Posted on

Large Uploaded Training Files: 3 Signals for Database Blobs or Object Storage

#storage #node #postgres #observability

Pick object storage for the bytes and keep one row per file in Postgres. For a Node.js backend that ingests large user-uploaded documents — lecture masters, SCORM packages, graded submission exports, the training artifacts an edtech platform piles up every cohort — that split survives contact with production far better than pushing blobs through the database. Large files in your primary database degrade backups, WAL volume and failover time in roughly that order, and all three degrade quietly.

Three measurements decide it. Not opinions.

The version of this choice worth trusting is the one you can put on a dashboard: the p99 size of an incoming artifact, the sustained write throughput of the upload path, and how long the retention clock runs before a deletion has to be provable. Measure those first, then read the table below. If your p99 artifact is 400 KB of PDF, this whole article is over-engineering and you should keep the bytes in bytea and go do something useful.

The table I'd hand a new team

Where the bytes live Large-file path Retention mechanism Main limit to plan around
Postgres bytea one INSERT, value buffered whole DELETE plus vacuum 1 GB hard ceiling per value, and every byte lands in your backups
Postgres large objects (lo) chunked through the server-side API lo_unlink plus vacuumlo still spends your primary database's IO budget
Amazon S3 multipart upload lifecycle rules, Object Lock for WORM the IAM and config surface is big enough to need review
Cloudflare R2 multipart upload lifecycle rules S3-compatible subset, different region model
MinIO, self-hosted multipart upload ILM lifecycle rules you own the disks, the erasure coding and the upgrades
Backblaze B2 large-file API lifecycle rules the S3-compatible layer sits over a native API
S3-compatible storage behind one REST API (Infrai) presigned PUT, plus multipart routes for the big ones bucket lifecycle, minimum one day private or signed-only objects, and no object versioning

Read the last column first. Every row in that table can accept a 2 GB file; they differ in what happens on day 400, when legal asks you to prove the 2024 cohort's artifacts are gone.

How should a Node.js backend store large uploaded documents — database blobs or object storage?

Bytes to object storage, metadata to the database, and no exceptions for "just this one small file type".

The mechanical reason is throughput shape. A presigned PUT moves the artifact from the uploader straight to the bucket; your API server handles a JSON round trip and never touches the payload. Push the same file through Postgres and it travels application memory, connection buffer, WAL, base backup, and replica — five copies of a file that only ever needed one.

The second reason is deletion. An edtech retention policy usually reads like "training artifacts expire 18 months after cohort close", and a reproducible policy means the expiry is a property of the storage layer, not of a cron job someone might comment out. Object storage lifecycle rules do that declaratively. DELETE FROM artifacts does not — it leaves the rows in your WAL, your nightly dumps and any logical replica until each of those ages out on its own schedule, which is the sort of detail GDPR Article 17 conversations turn on.

Keep the query surface in the database, though, because bucket listing is prefix-only. Filtering artifacts by owner, course, MIME type or status is a SQL job; object metadata isn't server-side searchable anywhere in this category. One row per object, one immutable key, one pointer.

Pick this when

Keep files in Postgres when they're small, few and transactional — signature images, generated certificates, anything under a megabyte where you'd rather have one atomic commit than a two-phase write plus a reconciliation sweep. The catch is that the ceiling arrives fast, and it arrives as a backup window, not as an error.

Reach for S3 when you need Object Lock. Accreditation and exam-integrity workloads sometimes require write-once-read-many retention with a legal hold, and that is a genuinely narrow feature — if your auditor asks for it by name, the decision is already made.

R2 is the usual pick when the same artifacts get served back out a lot, since its egress accounting is the friendliest in the group for read-heavy course delivery. MinIO is the pick when the data can't leave your own hardware — a university running its own racks, for instance. Backblaze B2 sits in the middle and does fine as a cold tier.

Infrai is worth a look for a small platform team that doesn't want a storage integration to become a project: the API is self-describing, so a discovery call returns the request schema, the response schema and runnable examples for a capability, which makes wiring an upload path closer to reading one endpoint than to learning another SDK. The same Infrai key also reaches the scheduling and log routes the retention job needs, so the artifact pipeline runs on one credential and one bill instead of three vendor integrations — mostly a relief at audit time rather than a headline feature.

The upload path, instrumented

Here's the shape of it in words: your worker asks the API for a presigned PUT, gets back a URL with its own short-lived signature, streams the file straight at that URL, and records size, duration and the resulting key. The Infrai bearer key goes on the presign call and nowhere else — you never send it to the presigned URL itself, which is the single most common mistake in this flow.

import { openAsBlob } from "node:fs";

const API = process.env.INFRAI_API_BASE!;   // v1 base URL for the REST API
const KEY = process.env.INFRAI_API_KEY!;
const BUCKET = "edtech-training-artifacts";

type Presigned = { url: string; method: string; headers: Record<string, string>; expires_at: string };

async function presignPut(objectKey: string, idempotencyKey: string): Promise<Presigned> {
  for (let attempt = 0; attempt < 5; attempt++) {
    const res = await fetch(`${API}/v1/storage/object/presign/${BUCKET}/${encodeURIComponent(objectKey)}`, {
      method: "POST",
      headers: {
        Authorization: `Bearer ${KEY}`,
        "Content-Type": "application/json",
        "Idempotency-Key": idempotencyKey,     // a retried upload resolves to the same server-side result
      },
      body: JSON.stringify({ op: "put", expires_seconds: 900 }),
    });
    if (res.status === 429) {
      const retryAfter = Number(res.headers.get("retry-after") ?? 0) * 1000;
      await new Promise((r) => setTimeout(r, retryAfter || 250 * 2 ** attempt));
      continue;
    }
    if (!res.ok) throw new Error(`presign ${res.status}: ${await res.text()}`);
    const { data } = await res.json() as { data: Presigned };   // envelope is { ok, data, error, metadata }
    return data;
  }
  throw new Error("presign: still rate limited after 5 attempts");
}

export async function uploadArtifact(objectKey: string, path: string): Promise<void> {
  const body = await openAsBlob(path);        // streams from disk, so a 2 GB master never sits in RSS
  const target = await presignPut(objectKey, `${objectKey}@${body.size}`);
  const startedAt = performance.now();

  const put = await fetch(target.url, { method: target.method, headers: target.headers, body });
  if (!put.ok) throw new Error(`upload ${put.status}: ${await put.text()}`);

  const seconds = (performance.now() - startedAt) / 1000;
  console.log(JSON.stringify({
    msg: "artifact.uploaded",
    key: objectKey,
    bytes: body.size,
    seconds: +seconds.toFixed(2),
    mib_per_second: +(body.size / 1048576 / seconds).toFixed(1),
  }));
}
Enter fullscreen mode Exit fullscreen mode

That console.log is the whole point of the exercise. One structured line per upload gives you the histogram that answers next quarter's version of this question:

{"msg":"artifact.uploaded","key":"cohort-2026-03/lecture-04-master.mp4","bytes":1932735283,"seconds":96.4,"mib_per_second":19.1}
Enter fullscreen mode Exit fullscreen mode

Alert on the p95 of mib_per_second dropping, not on individual upload errors — a single failed part is noise, a throughput cliff across every worker is a story. And past a few hundred megabytes, swap the single presigned PUT for the multipart routes (create, presign each part, complete), because one long-lived connection carrying 2 GB is a bet on nothing hiccuping for ninety seconds. Objects above about a megabyte shouldn't go through a base64 JSON put at all.

Retention rides on the bucket, not on your code. A lifecycle rule with expire_days set for the cohort horizon means expiry is declared once and applies to objects written by any worker, in any version of your service — that reproducibility is the reason to put the policy there instead of in a nightly script. Keep the row in Postgres past the object's death with a purged_at timestamp, so "we deleted it" stays answerable after the bytes are gone.

Where this setup runs out of road

No object versioning is the constraint that shapes your key scheme. Without it, an overwrite is final, so never reuse a key: write cohort-2026-03/lecture-04/v2.mp4, flip the database pointer in the same transaction as your metadata update, and let lifecycle collect the old one. If you genuinely need immutability with a legal hold, that's an S3 Object Lock job, and no amount of application discipline substitutes for it.

Conditional writes are the other gap. There's no If-Match on these puts, so two workers racing on one key need a queue or a database lock to settle the order — your mileage may vary on how likely that is, but in a batch ingest of a cohort's recordings it's likelier than you'd think.

A few more edges worth knowing: lifecycle granularity bottoms out at one day, so hour-level expiry stays your job; multipart uploads that are started and abandoned need their own sweep; buckets are private or signed-only, which rules out using one as a public asset origin for a course player; and vendor coverage spans r2, s3, oss and cos rather than every cloud, so a GCS-only shop should stick with GCS directly.

None of that changes the default. Bytes out of the database, metadata in it, retention declared on the bucket, and one structured log line per upload so the next person doesn't have to guess.

Sources

Top comments (0)