DEV Community

Code Green
Code Green

Posted on

1

System design of REST API that facilitates the uploading of large files using chunking

This post outlines the design of a REST API that facilitates the uploading of large files using chunking, asynchronous processing, and tracking mechanisms. The API will allow clients to upload files in manageable chunks, freeing them from blocking operations during the upload process.

Asynchronous Processing

To unblock the client during file uploads, the API can implement asynchronous processing using a message queue or background worker. Here’s how it works:

  1. Client Initiates Upload: The client sends a request to initiate an upload and receives an upload ID.
  2. Client Uploads Chunks: The client uploads each chunk asynchronously.
  3. Background Processing: Each uploaded chunk is processed in the background by a worker service that assembles the chunks into a complete file.
  4. Client Checks Status: The client can check the status of the upload at any time using the status endpoint.

Chunking Strategy

  • Chunk Size: Define a maximum chunk size (e.g., 5 MB) to ensure efficient uploads.
  • Total Chunks Calculation: Clients should calculate the total number of chunks based on the file size and chunk size before initiating the upload.

API Overview

Base URL

POST /api/v1/uploads
Enter fullscreen mode Exit fullscreen mode

API Endpoints

1. Initiate Upload

Endpoint:

POST /api/v1/uploads/initiate
Enter fullscreen mode Exit fullscreen mode

Description: Initiates a file upload session and generates an upload ID for tracking.

Request Body:

    {
      "fileName": "example.txt",
      "fileSize": 104857600,  // Size in bytes (100 MB)
      "fileType": "text/plain"
    }
Enter fullscreen mode Exit fullscreen mode

Response:

    {
      "uploadId": "1234567890abcdef",
      "status": "initiated",
      "message": "Upload session initiated successfully."
    }
Enter fullscreen mode Exit fullscreen mode

2. Upload Chunk

Endpoint:

POST /api/v1/uploads/{uploadId}/chunks
Enter fullscreen mode Exit fullscreen mode

Description: Uploads a chunk of the file.

Path Parameters:

  • uploadId: The unique identifier for the upload session.

Request Body:

    {
      "chunkIndex": 0,
      "chunkData": "",
      "totalChunks": 10
    }
Enter fullscreen mode Exit fullscreen mode

Response:

    {
      "status": "chunk received",
      "message": "Chunk uploaded successfully.",
      "nextChunkIndex": 1
    }
Enter fullscreen mode Exit fullscreen mode

3. Complete Upload

Endpoint:

POST /api/v1/uploads/{uploadId}/complete
Enter fullscreen mode Exit fullscreen mode

Description: Finalizes the upload session after all chunks have been uploaded.

Path Parameters:

  • uploadId: The unique identifier for the upload session.

Response:

    {
      "status": "completed",
      "message": "File uploaded successfully.",
      "fileLocation": "/uploads/example.txt"
    }
Enter fullscreen mode Exit fullscreen mode

4. Check Upload Status

Endpoint:

GET /api/v1/uploads/{uploadId}/status
Enter fullscreen mode Exit fullscreen mode

Description: Checks the status of the ongoing upload session.

Path Parameters:

  • uploadId: The unique identifier for the upload session.

Response:

    {
      "uploadId": "1234567890abcdef",
      "status": "in progress", // or 'completed', 'failed'
      "uploadedChunks": 5,
      "totalChunks": 10
    }
Enter fullscreen mode Exit fullscreen mode

Conclusion

This REST API design provides a comprehensive solution for uploading large files efficiently. By utilizing chunking, asynchronous processing, and tracking mechanisms via upload IDs, clients can seamlessly upload large files without blocking their operations, enhancing user experience and system performance.

Image of Datadog

The Future of AI, LLMs, and Observability on Google Cloud

Datadog sat down with Google’s Director of AI to discuss the current and future states of AI, ML, and LLMs on Google Cloud. Discover 7 key insights for technical leaders, covering everything from upskilling teams to observability best practices

Learn More

Top comments (0)

Postmark Image

Speedy emails, satisfied customers

Are delayed transactional emails costing you user satisfaction? Postmark delivers your emails almost instantly, keeping your customers happy and connected.

Sign up