Upload Large Files in Chunks with Spring Boot

1. Introduction to Chunked File Uploads

Uploading files in chunks refers to the process of dividing large files into smaller parts (chunks) and uploading them sequentially or in parallel to the server. Once all parts are uploaded, the server merges them to recreate the original file.

1.1 Why Use Chunked File Uploads?

Chunked file uploads help overcome common problems with uploading large files, such as:

• Network Instability : Uploading large files can fail if the connection drops. With chunking, only the missing chunks need to be re-uploaded.
• Server Memory Limitations : Uploading a large file in one go can consume significant memory. Chunking reduces the load.
• User Experience : Users can pause and resume uploads, improving their experience.

1.2 How Does Chunked Uploading Work?

The process involves:

• Splitting a file into smaller chunks on the client side.
• Uploading each chunk individually to the server.
• Reconstructing the file from these chunks once all have been uploaded.

Now, let's dive into the practical implementation of this concept in Spring Boot.

2. Implementing Chunked File Upload in Spring Boot

2.1 Setting Up the Project

First, you’ll need to set up a Spring Boot project. Include the necessary dependencies in your pom.xml file:

<dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-web</artifactId>
</dependency>

2.2 Client-Side File Splitting

To upload a file in chunks, you need a front-end that splits the file into smaller parts. Here's an example in JavaScript using the File API.

function uploadFileInChunks(file) {
    const chunkSize = 1024 * 1024; // 1MB per chunk
    let start = 0;
    let chunkIndex = 0;

    while (start < file.size) {
        const chunk = file.slice(start, start + chunkSize);
        uploadChunk(chunk, chunkIndex, file.name);
        start += chunkSize;
        chunkIndex++;
    }
}

function uploadChunk(chunk, chunkIndex, fileName) {
    const formData = new FormData();
    formData.append('chunk', chunk);
    formData.append('chunkIndex', chunkIndex);
    formData.append('fileName', fileName);

    fetch('/upload-chunk', {
        method: 'POST',
        body: formData
    }).then(response => response.json())
      .then(data => console.log(data))
      .catch(error => console.error('Error uploading chunk:', error));
}

2.3 Spring Boot API for Chunk Upload

Now, let's create the API that handles each chunk on the server side. We'll create a ChunkController class that accepts file chunks and saves them to a temporary directory.

@RestController
public class ChunkController {

    private static final String TEMP_DIR = "temp-uploads/";

    @PostMapping("/upload-chunk")
    public ResponseEntity<String> uploadChunk(@RequestParam("chunk") MultipartFile chunk,
                                              @RequestParam("chunkIndex") int chunkIndex,
                                              @RequestParam("fileName") String fileName) throws IOException {

        File dir = new File(TEMP_DIR + fileName);
        if (!dir.exists()) {
            dir.mkdirs();
        }

        File chunkFile = new File(dir, "chunk_" + chunkIndex);
        try (OutputStream os = new FileOutputStream(chunkFile)) {
            os.write(chunk.getBytes());
        }

        return ResponseEntity.ok("Chunk " + chunkIndex + " uploaded successfully.");
    }
}

Each chunk is saved with a unique name, chunk_ , in a temporary directory that corresponds to the original file's name. Once all chunks are uploaded, you can reconstruct the original file by merging the chunks.

2.4 Merging the Chunks

Once all chunks are uploaded, we need an API that merges them into a single file. Here's how to handle that:

@PostMapping("/merge-chunks")
public ResponseEntity<String> mergeChunks(@RequestParam("fileName") String fileName) throws IOException {
    File dir = new File(TEMP_DIR + fileName);
    File mergedFile = new File("uploads/" + fileName);

    try (OutputStream os = new FileOutputStream(mergedFile)) {
        for (int i = 0; i < dir.listFiles().length; i++) {
            File chunkFile = new File(dir, "chunk_" + i);
            Files.copy(chunkFile.toPath(), os);
            chunkFile.delete(); // Delete chunk after merging
        }
    }

    // Delete temporary directory
    dir.delete();

    return ResponseEntity.ok("File merged successfully.");
}

This method takes the uploaded chunks, merges them into a single file, and deletes the temporary files once the process is complete.

3. Demonstrating the Chunk Upload

Let’s demonstrate the entire chunk upload process with a sample:

Step 1: Splitting the File

When a user selects a file on the client side, the uploadFileInChunks() function is triggered. The fiStep 2: Uploading Each Chunkle is split into 1MB chunks and each chunk is sent to the server.

Step 2: Uploading Each Chunk

The Spring Boot ChunkController receives the chunks and saves them to the temporary directory.

Step 3: Merging the Chunks

Once all chunks are uploaded, the client can send a request to merge the chunks. The server will combine the chunks into a single file, storing it in the final destination folder.

Successful chunk uploads display a message like: "Chunk 1 uploaded successfully."

After merging, a final message will be shown: "File merged successfully."

4. Conclusion

In this article, we explored a step-by-step approach to handle large file uploads by dividing them into smaller chunks in Spring Boot. This method is efficient for large files, improving reliability and user experience by handling network interruptions and reducing server load.

Have questions or need further clarification? Drop a comment below and let’s discuss!

Read posts more at : Upload Large Files in Chunks with Spring Boot