DEV Community

Surhid Amatya
Surhid Amatya

Posted on

Multipart Image Upload, Part 2: Chunking the Image Locally

Multipart Image Upload: A Practical Guide for Web, Mobile and Backend
Part 1: Understanding Chunked Multipart Upload
Part 2: Chunking the Image Locally (this post)
Part 3: Building the Multipart Upload API
Part 4: Making Uploads Reliable and Resumable
Part 5: Production Architecture and Cross-Platform Implementation

Before any HTTP request goes out, the client has to turn one file into a list of chunks. This is the most code-heavy part of the series, and the idea is small: one formula, and one operation, reading a range of bytes.

The example

A 5.5 MB image, with a chunk size of 1 MB (1,048,576 bytes):

Part 1  =  1.0 MB   (bytes 0 to 1 MB)
Part 2  =  1.0 MB
Part 3  =  1.0 MB
Part 4  =  1.0 MB
Part 5  =  1.0 MB
Part 6  =  0.5 MB   (the remainder)
Enter fullscreen mode Exit fullscreen mode

The part count comes from:

totalParts = ceil(fileSize / chunkSize)
Enter fullscreen mode Exit fullscreen mode

Here that's ceil(5.5 / 1) = 6. The last part is smaller whenever the file size isn't an exact multiple of the chunk size. That's normal, and the server has to accept it.

The byte range for any part follows from the part number (1-based):

start = (partNumber - 1) * chunkSize
end   = min(start + chunkSize, fileSize)      // exclusive
Enter fullscreen mode Exit fullscreen mode

For part 6, start is 5 MB and end is min(6 MB, 5.5 MB), so the chunk is 0.5 MB. Every implementation below is this arithmetic plus a file read.

Bytes, not smaller images

Here's the mistake to avoid.

Wrong:

image → resize → image 1, image 2, image 3
Enter fullscreen mode Exit fullscreen mode

Resizing or cropping gives you separate, smaller, valid images. You can't join three small JPEGs back into the original, because you changed the content.

Correct:

image bytes → byte ranges → part 1, part 2, part 3
Enter fullscreen mode Exit fullscreen mode

A file is a sequence of bytes. Chunking cuts that sequence into consecutive ranges without looking at what's inside. Part 3 isn't a picture of anything, and an image viewer would refuse to open it. Join parts 1 to 6 in order and you get the identical file back.

It also means the same code splits a JPEG, a PDF or a video.

TypeScript (web)

In the browser a File is a Blob, and Blob.slice() does what we need. It doesn't copy anything into memory. It returns a reference to a range, and the bytes are only read when the request is sent.

const CHUNK_SIZE = 1024 * 1024; // 1 MB

function getChunk(file: File, partNumber: number): Blob {
  const start = (partNumber - 1) * CHUNK_SIZE;
  const end = Math.min(start + CHUNK_SIZE, file.size);
  return file.slice(start, end);
}

const totalParts = Math.ceil(file.size / CHUNK_SIZE);

for (let partNumber = 1; partNumber <= totalParts; partNumber++) {
  const chunk = getChunk(file, partNumber);
  // build FormData and send it (Part 3 shows the server side)
}
Enter fullscreen mode Exit fullscreen mode

Since slice is cheap, slice one chunk at a time as you send it. Building an array of every chunk up front is a habit that hurts on large files.

Flutter (Dart)

Memory is tighter on mobile, so read one range at a time from a RandomAccessFile instead of loading the file.

import 'dart:io';
import 'dart:math';
import 'dart:typed_data';

const chunkSize = 1024 * 1024; // 1 MB

Future<Uint8List> readChunk(
  RandomAccessFile file,
  int fileSize,
  int partNumber,
) async {
  final start = (partNumber - 1) * chunkSize;
  final end = min(start + chunkSize, fileSize);

  await file.setPosition(start);
  return file.read(end - start);
}

Future<void> upload(File image) async {
  final fileSize = await image.length();
  final totalParts = (fileSize / chunkSize).ceil();
  final raf = await image.open();

  try {
    for (var part = 1; part <= totalParts; part++) {
      final chunk = await readChunk(raf, fileSize, part);
      // send chunk
    }
  } finally {
    await raf.close();
  }
}
Enter fullscreen mode Exit fullscreen mode

setPosition moves the read cursor and read(n) returns up to n bytes from there. Close the handle in finally, or a failed upload leaks it.

Python

import math

CHUNK_SIZE = 1024 * 1024  # 1 MB

def read_chunk(file, file_size: int, part_number: int) -> bytes:
    start = (part_number - 1) * CHUNK_SIZE
    file.seek(start)
    return file.read(min(CHUNK_SIZE, file_size - start))

def upload(file_path: str, file_size: int) -> None:
    total_parts = math.ceil(file_size / CHUNK_SIZE)

    with open(file_path, "rb") as file:
        for part_number in range(1, total_parts + 1):
            chunk = read_chunk(file, file_size, part_number)
            # send chunk
Enter fullscreen mode Exit fullscreen mode

Open in binary mode ("rb"). Text mode can change line endings and corrupt the bytes.

.NET (C#)

const int ChunkSize = 1024 * 1024; // 1 MB

static async Task<byte[]> ReadChunkAsync(
    FileStream fileStream,
    int partNumber)
{
    long start = (long)(partNumber - 1) * ChunkSize;
    long remaining = fileStream.Length - start;
    var buffer = new byte[Math.Min(ChunkSize, remaining)];

    fileStream.Seek(start, SeekOrigin.Begin);

    int offset = 0;
    while (offset < buffer.Length)
    {
        int read = await fileStream.ReadAsync(
            buffer.AsMemory(offset, buffer.Length - offset));
        if (read == 0) break;
        offset += read;
    }

    return buffer;
}
Enter fullscreen mode Exit fullscreen mode

The loop is there because one ReadAsync call may return fewer bytes than you asked for, so you keep reading until the buffer is full. The (long) cast stops 32-bit overflow on files over 2 GB.

Where chunking goes wrong

Most corrupt uploads I'd expect to see come from a short list:

  • Mixing 0-based and 1-based part numbers between client and server, which shifts every chunk by one.
  • Treating end as inclusive in one place and exclusive in another. Above, it's exclusive, same as slice.
  • Assuming a read returns everything you asked for. Check the byte count.
  • 32-bit offsets on big files.
  • Loading the whole file into memory, which defeats the point on a phone.

A cheap test catches most of these: the chunk lengths must add up to fileSize, and joining the chunks must reproduce the original file's hash.

State of play

For each part the client now has a partNumber, the totalParts and a block of bytes. The file on disk was only read, never changed, and nothing has touched the network yet.

Part 3 builds the other end: a Spring Boot endpoint that takes each part as multipart/form-data, works out which part it is from the metadata, and stores it so the file can be rebuilt.

Top comments (0)