DEV Community

Cover image for Resumable Multipart Uploads in Lioran S3
Swaraj Puppalwar
Swaraj Puppalwar

Posted on

Resumable Multipart Uploads in Lioran S3

#typescript #storage #performance #node

Resumable Multipart Uploads in Lioran S3

Uploading a 5 KB JSON file and uploading a 40 GB archive should not use the same application strategy.

For large objects, Lioran S3 V1 Pre-Alpha provides multipart uploads with chunking, bounded concurrency, retries, progress reporting, resumability, and final assembly.

Lioran S3 is developed by Lioran Developer Solutions (LDS) under Lioran Group, led by Founder & CTO Swaraj Puppalwar.

Install the driver 

npm install @liorans3/driver@prealpha
Enter fullscreen mode Exit fullscreen mode

Connect 

import {
  BastionClient,
} from "@liorans3/driver";

const client = new BastionClient(
  process.env.LIORAN_S3_URI!
);

const bucket = client.bucket("backups");
Enter fullscreen mode Exit fullscreen mode

High-level multipart upload

The simplest API accepts a file path and handles the multipart session for you.

const result =
  await bucket.uploadMultipart(
    "nightly/database.tar.gz",
    "./database.tar.gz",
    {
      concurrency: 4,

      onProgress(progress) {
        console.log({
          percent: progress.percent,
          uploaded: progress.bytesUploaded,
          total: progress.totalBytes,
          part: progress.partNumber,
          parts: progress.totalParts,
        });
      },
    }
  );

console.log(result.key);
Enter fullscreen mode Exit fullscreen mode

Why multipart exists

Large uploads fail in real networks.

Connections reset.

Processes restart.

Reverse proxies time out.

Users close laptops.

A single request containing the entire file forces failure recovery to restart too much work.

Multipart turns one large transfer into independently uploadable parts.

Conceptually:

large-file
  ├── part 1
  ├── part 2
  ├── part 3
  ├── part 4
  └── ...
Enter fullscreen mode Exit fullscreen mode

Parts can be uploaded concurrently and committed only when the multipart session is completed.

Manual multipart control

For applications that need explicit resumability, use the lower-level session API.

Create:

const upload =
  await bucket.createMultipartUpload(
    "images/archive.iso",
    {
      contentType:
        "application/x-iso9660-image",
    }
  );
Enter fullscreen mode Exit fullscreen mode

Upload a part:

await upload.uploadPart(
  1,
  partBuffer
);
Enter fullscreen mode Exit fullscreen mode

Inspect status:

const status =
  await upload.status();

console.log(status);
Enter fullscreen mode Exit fullscreen mode

Complete:

const object =
  await upload.complete();

console.log(object);
Enter fullscreen mode Exit fullscreen mode

Abort:

await upload.abort();
Enter fullscreen mode Exit fullscreen mode

Concurrency is not free speed

It is tempting to set concurrency to 128 because 128 looks faster than 4.

That is benchmark astrology.

The useful concurrency value depends on:

  • network bandwidth
  • round-trip latency
  • server CPU
  • storage throughput
  • filesystem behavior
  • reverse proxy settings
  • part size
  • memory limits
  • competing workloads

Start modestly:

{
  concurrency: 4
}
Enter fullscreen mode Exit fullscreen mode

Then measure.

Bounded-memory behavior

Multipart is not an excuse to buffer the entire file.

The storage engine is designed around bounded chunks so object size should not directly imply equivalent heap usage.

Your application should follow the same rule.

Avoid:

const giantFile =
  await fs.promises.readFile(
    "./100gb-file.bin"
  );
Enter fullscreen mode Exit fullscreen mode

Prefer file streams or the driver's file-path multipart API.

Recovery model

Incomplete multipart state remains distinct from committed object state.

That distinction matters.

A failed upload should not cause GET or LIST to report a partially assembled object as successfully committed.

Only the completed assembly becomes the visible object.

CLI equivalent

The CLI exposes the same workflow.

Automatic:

liorans3 multipart upload   media   archives/backup.tar.gz   ./backup.tar.gz   --part-size-mb 16   --concurrency 4
Enter fullscreen mode Exit fullscreen mode

Manual session:

liorans3 multipart create   media   datasets/large.csv
Enter fullscreen mode Exit fullscreen mode

Status:

liorans3 multipart status   <upload-id>   --bucket media   --key datasets/large.csv
Enter fullscreen mode Exit fullscreen mode

Resume:

liorans3 multipart resume   <upload-id>   ./large.csv   --bucket media   --key datasets/large.csv
Enter fullscreen mode Exit fullscreen mode

Abort:

liorans3 multipart abort   <upload-id>   --bucket media   --key datasets/large.csv
Enter fullscreen mode Exit fullscreen mode

Application pattern

A backend can store the upload ID in its own database:

type UploadRecord = {
  uploadId: string;
  bucket: string;
  key: string;
  localPath: string;
  createdAt: Date;
};
Enter fullscreen mode Exit fullscreen mode

After a restart, restore the record, query multipart status, and continue instead of starting from zero.

Correctness before benchmark numbers

The priority order for a storage system should be:

  1. correct object assembly
  2. crash-safe behavior
  3. integrity
  4. bounded memory
  5. resumability
  6. throughput

Throughput without the first five is just an enthusiastic corruption generator.

Docs: https://docs.liorans3.sbs

Top comments (0)