Optimizing File Processing in React with Multipart Uploads and Downloads

Preface

Front-end developers now need to be able to process file streams efficiently and reliably. Whether uploading, downloading, reading, displaying, or performing other file-processing operations, binary data must be processed efficiently and reliably. With increasing file sizes and the limitations of network transmission, file uploading and downloading in slices has gradually become a necessary technology to improve performance and user experience.

File multi-part upload and download makes file transmission more reliable and efficient by splitting large files into multiple small fragments and taking advantage of resuming upload at breakpoints. In this process, front-end developers should be familiar with Blob objects and ArrayBuffer. This familiarity can help you process and manipulate binary data. Using the React framework can make it more convenient to manage and operate file objects and quickly implement multi-part upload and download functions for files.

This article will delve into how to use React to implement file uploading and downloading in slices and introduce related basic concepts and technologies. I will focus on how to handle the binary data of files efficiently and how to leverage file stream operations to optimize file processing tasks in front-end development. By studying this article, you can master a set of methods for efficiently processing file stream operations and provide better solutions for your front-end development work. Let's start exploring together!

Introduction

File transfer is a fundamental requirement for many front-end applications. However, traditional methods for uploading and downloading large files can lead to performance and user experience problems.

Fortunately, front-end technology provides several efficient solutions, including file stream operations and slice download and upload. This article will delve into these technologies and explain how to use them to optimize file transfer efficiency and improve user experience.

Front-end File Flow Operation

In front-end development, file stream operations allow you to process files through a data flow, performing operations such as reading, writing, and deleting files. This article will introduce several basic concepts and technologies of front-end file stream operations in detail.

Basic Concepts of Data Flow and File Processing

In front-end development, files can be processed as data streams. A data stream is a sequence of data that is transmitted from one source to another destination.

Blob objects and ArrayBuffer: Processing Binary Data

When processing files on the front end, you often need to process binary data. Blob (Binary Large Object) objects are a type of object in JavaScript that can store large amounts of binary data. Blob objects can be created through the constructor or generated through other APIs, such as the FormData object. ArrayBuffers are another type of object in JavaScript that can store binary data. They are typically used for lower-level operations, such as manipulating and processing binary data directly.

Look at the example below:

import React, { useState } from "react";

function FileInput() {
  const [fileContent, setFileContent] = useState("");

  // Read file content to ArrayBuffer
  function readFileToArrayBuffer(file) {
    return new Promise((resolve, reject) => {
      const reader = new FileReader();

      // Register callback function when file reading is complete
      reader.onload = function(event) {
        const arrayBuffer = event.target.result;
        resolve(arrayBuffer);
      };

      // Read file content to ArrayBuffer
      reader.readAsArrayBuffer(file);
    });
  }

  // Convert ArrayBuffer to hexadecimal string
  function arrayBufferToHexString(arrayBuffer) {
    const uint8Array = new Uint8Array(arrayBuffer);
    let hexString = "";
    for (let i = 0; i < uint8Array.length; i++) {
      const hex = uint8Array[i].toString(16).padStart(2, "0");
      hexString += hex;
    }
    return hexString;
  }

  // Handle file select event
  function handleFileChange(event) {
    const file = event.target.files[0]; // Get selected file

    if (file) {
      readFileToArrayBuffer(file)
        .then(arrayBuffer => {
          const hexString = arrayBufferToHexString(arrayBuffer);
          setFileContent(hexString);
        })
        .catch(error => {
          console.error("File read failed:", error);
        });
    } else {
      setFileContent("Please select a file");
    }
  }

  return (
    <div>
      <input type="file" onChange={handleFileChange} />
      <div>
        <h4>File content:</h4>
        <pre>{fileContent}</pre>
      </div>
    </div>
  );
}

export default FileInput;

In the above code, I created a functional component called FileInput. This component contains a file selection box and a preformatted text element for displaying the file contents. When the user selects a file, the file content is read to an ArrayBuffer using FileReader. The ArrayBuffer is then converted to a hexadecimal string and the result is displayed on the page.

Use FileReader to Reading Files

FileReader is a front-end browser API that allows you to read file contents asynchronously and convert them into usable data formats, such as text or binary data. It provides methods such as readAsText() and readAsArrayBuffer(), which you can choose based on your needs.

Display the File Stream

Once you have successfully read the contents of the file, you can display the file stream on the front-end page. The specific display method depends on the file type. For example, you can display text files directly in a text box or area, picture files using an img tag, and audio and video files using audio or video tags. By displaying the file stream on the front-end page, can preview and view the file content online.

File Slice Download

File slice download is the topic of today's article. Let's first take a look at the main process.

graph LR
Start (A) --> Select file {B}
B -- User selects file --> Slice file into multiple slices [C]
C --> Upload slices {D}
D -- Upload complete --> Merge slices into a complete file [E]
E -- File merge complete --> Upload success (F)
D -- Upload interrupted --> Save upload progress [G]
G -- Upload resumed --> {D}
G -- Upload canceled --> Upload canceled (H)

Performance Issues With Traditional File Downloads

File slice downloading is a technology that improves file download efficiency by splitting large files into smaller fragments (slices) and downloading them concurrently. This speeds up the overall download speed, especially for users with unreliable or slow internet connections.

Large file downloads can be slow and inefficient using traditional methods. The server must send the entire file to the client, which can lead to the following:

Long wait times: Users may have to wait a long time to start using large files.
Network congestion: Other users may experience slow download speeds if the network bandwidth is used up by large file downloads.
Difficulty resuming downloads: If a network failure or user interruption occurs, the entire file must be downloaded again.

Use File Slicing to Improve Download Efficiency

File slicing downloads split files into small fragments, typically a few hundred KB to several MB in size. This allows clients to:

Start downloading quickly: Only the first slice needs to be downloaded, so clients can start using the file sooner.
Download concurrently: Multiple concurrent requests can be used to download slices, which fully utilizes bandwidth and increases overall download speeds.
Resume downloads: If a download is interrupted, only the unfinished slices need to be redownloaded, not the entire file.

Slice upload code example:

const [selectedFile, setSelectedFile] = useState(null);
const [progress, setProgress] = useState(0);

// Handle file selection event
function handleFileChange(event) {
  setSelectedFile(event.target.files[0]);
}

// Handle file upload event
function handleFileUpload() {
  if (selectedFile) {
    // Calculate the number of chunks and the size of each chunk
    const fileSize = selectedFile.size;
    const chunkSize = 1024 * 1024; // Set chunk size to 1MB
    const totalChunks = Math.ceil(fileSize / chunkSize);

    // Create a FormData object and add file information
    const formData = new FormData();
    formData.append('file', selectedFile);
    formData.append('totalChunks', totalChunks);

    // Loop through and upload chunks
    for (let chunkNumber = 0; chunkNumber < totalChunks; chunkNumber++) {
      const start = chunkNumber * chunkSize;
      const end = Math.min(start + chunkSize, fileSize);
      const chunk = selectedFile.slice(start, end);
      formData.append(`chunk-${chunkNumber}`, chunk, selectedFile.name);
    }

    // Make a file upload request
    axios.post('/upload', formData, {
      onUploadProgress: progressEvent => {
        const progress = Math.round((progressEvent.loaded / progressEvent.total) * 100);
        setProgress(progress);
      }
    })
      .then(response => {
        console.log('File upload successful:', response.data);
      })
      .catch(error => {
        console.error('File upload failed:', error);
      });
  }
}

Slice uploading and downloading is typically implemented using the file processing functions provided by the front-end library or framework. The back-end service implementation is responsible for handling the uploaded or downloaded chunks and assembling them into the complete file.

In the above example, we implemented slice uploading using the following steps:

1. The handleFileChange() function is called when the user selects a file. This function saves the selected file to the selectedFile state variable.

2. The handleFileUpload() function is called when the user clicks the upload button. This function calculates the number of slices and the size of each slice and then creates a FormData object to store the file information and slice data.

3. The FormData object is then used to make a file upload request to the back-end service.

4. The back-end service handles the uploaded chunks and assembles them into the complete file.

Solution to Implement Client-side Slice Downloading

The basic solution to implement client-side slice downloading is as follows:

1. The server side cuts the large file into multiple slices and generates a unique identifier for each slice.

2. The client sends a request to obtain the slice list and starts downloading the first slice.

3. During the download process, the client initiates concurrent requests to download other slices based on the slice list, and gradually splices and merges the downloaded data.

4. When all slices are downloaded, the client merges the downloaded data into a complete file.

Here is an example

function downloadable() {

  // Make a file download request
  fetch('/download', {
    method: 'GET',
    headers: {
      'Content-Type': 'application/json',
    },
  })
    .then(response => response.json())
    .then(data => {
      const totalSize = data.totalSize;
      const totalChunks = data.totalChunks;

      // Initialize variables
      let downloadedChunks = 0;
      let chunks = [];

      // Download each chunk
      for (let chunkNumber = 0; chunkNumber < totalChunks; chunkNumber++) {
        fetch(`/download/${chunkNumber}`, {
          method: 'GET',
        })
          .then(response => response.blob())
          .then(chunk => {
            downloadedChunks++;
            chunks.push(chunk);

            // When all chunks are downloaded
            if (downloadedChunks === totalChunks) {
              // Merge chunks
              const mergedBlob = new Blob(chunks);

              // Create an object URL to generate a download link
              const downloadUrl = window.URL.createObjectURL(mergedBlob);

              // Create an <a> element and set attributes
              const link = document.createElement('a');
              link.href = downloadUrl;
              link.setAttribute('download', 'file.txt');

              // Simulate a click to download
              link.click();

              // Release resources
              window.URL.revokeObjectURL(downloadUrl);
            }
          });
      }
    })
    .catch(error => {
      console.error('File download failed:', error);
    });
}

Let's take a look at the code. First, use the Blob object to create a total object URL, which is used to generate a download connection. Then create a label and set the href attribute to the object URL just created. Continue to set the attribute of the label to download the file name, so that the file can be automatically downloaded when clicked.

Show Download Progress and Completion Status

To display the download progress and completion status on the client, you can implement the following functions:

Progress bar: The client can calculate the overall download progress by tracking the download progress of each slice. This progress can then be displayed in a progress bar to give the user a visual indication of how long the download will take.

Completion status: When all slices have been downloaded, the client can display a completion status, such as a completion icon or text. This lets the user know that the download is complete and that they can now access the file.

Here we can continue the slice upload code example to write:

// Handle file download event

function handleFileDownload() {
  axios.get('/download', {
    responseType: 'blob',
    onDownloadProgress: progressEvent => {
      const progress = Math.round((progressEvent.loaded / progressEvent.total) * 100);
      setProgress(progress);
    }
  })
    .then(response => {
      // Create a temporary URL object for download

      const url = window.URL.createObjectURL(new Blob([response.data]));
      const link = document.createElement('a');
      link.href = url;
      link.setAttribute('download', 'file.txt');
      document.body.appendChild(link);
      link.click();
      document.body.removeChild(link);
    })
    .catch(error => {
      console.error('File download failed:', error);
    });
}

When the user clicks the download button, the handleFileDownload() function is called to handle the file download event.

The handleFileDownload() function uses the axios library to initiate a file download request with the responseType set to blob to return binary data.

The function then listens for the onDownloadProgress event to track the download progress and update the display of the progress bar.

Once the download is complete, the function creates a temporary URL object for downloading and dynamically creates an <a> element to simulate a click and download the file.

Problems and Solutions For Large File Uploads

Large file uploads can be slow, inefficient, and unreliable, but some solutions can improve the performance and stability of the process.

Problems With Traditional File Upload Methods

• Large files take a long time to upload and can easily cause timeouts.

• They occupy server and network bandwidth resources, which can affect the access speed of other users.

• If the upload is interrupted, the entire file needs to be re-uploaded, which is inefficient.

• It is difficult to display and control the upload progress.

Advantages of Front-end File Slicing Upload

• Splits large files into smaller chunks for faster and more reliable uploads.

• Monitors and displays the upload progress to improve the user experience.

• Makes full use of the browser's concurrent upload capabilities to reduce the load on the server.

• Implements a breakpoint resumption function to avoid re-uploading uploaded chunks.

Method to Implement Front-end Slice Upload

Use the File API in JavaScript to get the file object, and use the Blob.prototype.slice() method to cut the file into multiple slices.

• To upload a large file in slices, you can use the FormData object to send the slices to the server via AJAX or the Fetch API.
• The backend server will receive the slices and save them temporarily. Once all the slices have been received, the server will merge them into the complete file.
• The client can monitor the upload progress event and display the progress in a progress bar or prompt.

Here is an example:

const [file, setFile] = useState(null); // To store the file that I uploaded locally

const chunkSize = 1024 * 1024; // The size of each slice is 1MB

const upload = () => {
  if (!file) {
    alert("Please select a file to upload!");
    return;
  }

  const start = 0;
  const end = Math.min(chunkSize, file.size);

  while (start < file.size) {
    const chunk = file.slice(start, end);

    // Create a FormData object
    const formData = new FormData();
    formData.append("file", chunk);

    // Send the slice to the server
    fetch("upload-api", {
      method: "POST",
      body: formData
    })
      .then(response => response.json())
      .then(data => {
        console.log(data); // Handle the response result
      })
      .catch(error => {
        console.error(error); // Handle the error
      });

    start = end;
    end = Math.min(start + chunkSize, file.size);
  }
};

return (
  <div>
    <input type="file" onChange={handleFileChange} />
    <button onClick={upload}>Upload</button>
  </div>
);

The Upload function component uses React's useState hook to manage the selected file. It monitors changes in the file input box through onChange events and updates the file state in the handleFileChange() function.

When the "Upload" button is clicked, the upload() function is called. This function cuts the file into multiple equal-sized slices and uses FormData objects and the fetch() function to send the slice data to the server

How to Implement Breakpoint Resumption: Record and Restore Upload Status

• On the front end, you can use localStorage or sessionStorage to store uploaded slice information, including uploaded slice index and slice size.

• Before each upload, check whether the uploaded slice information exists in the local storage. If it exists, continue uploading from the breakpoint.

• On the backend, a temporary folder or database can be used to record received tile information, including uploaded tile index and tile size.

• Before the upload completes, save the upload status so that upload progress can be resumed if the upload is interrupted.
  

import React, { useState, useRef, useEffect } from "react";

function Upload() {
  const [file, setFile] = useState(null); // The file that I uploaded locally
  const [uploadedChunks, setUploadedChunks] = useState([]); // The list of chunks that have been uploaded
  const [uploading, setUploading] = useState(false); // Whether the upload is in progress
  const uploadRequestRef = useRef(null); // A reference to the current upload request

  const handleFileChange = (event) => {
    const selectedFile = event.target.files[0];
    setFile(selectedFile);
  };

  const uploadChunk = async (chunk) => {
    // Create a FormData object
    const formData = new FormData();
    formData.append("file", chunk);

    // Send the slice to the server
    return await fetch("your-upload-url", {
      method: "POST",
      body: formData
    })
      .then(response => response.json())
      .then(data => {
        console.log(data); // Handle the response result
        return data;
      });
  };

  const upload = async () => {
    if (!file) {
      alert("Please select a file to upload!");
      return;
    }

    const chunkSize = 1024 * 1024; // 1MB
    const totalChunks = Math.ceil(file.size / chunkSize);

    let start = 0;
    let end = Math.min(chunkSize, file.size);

    setUploading(true);

    for (let i = 0; i < totalChunks; i++) {
      const chunk = file.slice(start, end);
      const uploadedChunkIndex = uploadedChunks.indexOf(i);

      if (uploadedChunkIndex === -1) {
        try {
          const response = await uploadChunk(chunk);
          setUploadedChunks((prevChunks) => [...prevChunks, i]);

          // Save the list of uploaded chunks to local storage
          localStorage.setItem("uploadedChunks", JSON.stringify(uploadedChunks));
        } catch (error) {
          console.error(error); // Handle the error
        }
      }

      start = end;
      end = Math.min(start + chunkSize, file.size);
    }

    setUploading(false);

    // Upload is complete, clear the list of uploaded chunks from local storage
    localStorage.removeItem("uploadedChunks");
  };

  useEffect(() => {
    const storedUploadedChunks = localStorage.getItem("uploadedChunks");

    if (storedUploadedChunks) {
      setUploadedChunks(JSON.parse(storedUploadedChunks));
    }
  }, []);

  return (
    <div>
      <input type="file" onChange={handleFileChange} />
      <button onClick={upload} disabled={uploading}>
        {uploading ? "Uploading..." : "Upload"}
      </button>
    </div>
  );
}

The Upload function component uses React's useState hook to create a uploadedChunks state to hold the uploaded slice index array, and the useRef hook to create an uploadRequestRef reference to store the current upload request.

When the user selects a file to upload, the handleFileChange() function updates the file state.

The uploadChunk() function sends a slice to the server and returns a Promise object to handle the response.

The upload() function resumes the download at a breakpoint by getting the total number of slices and setting the uploading state to true to disable the upload button. It then iterates through all the slices and checks if the slice index is already contained in the uploadedChunks array. If it is not, the function uploads the slice and adds the uploaded slice index to the uploadedChunks array. The function then uses localStorage to save the uploaded slice information. Finally, after the upload is complete, the function sets the uploading state to false and clears the locally stored slice information.

When uploading large files, it is important to consider the server's processing power and storage space, as well as security issues. It is also important to avoid uploading the same files concurrently to ensure the accuracy of resumed uploads. You can use unique file identifiers or user session identifiers to distinguish them.

Optimize User Experience: Application Scenarios of Slice Upload and Download

Slice upload and download can optimize user experience in the following application scenarios:

File Download and Upload in the Background Management System:

File download: Users may need to download large files, such as reports, log files, and database backups. Slicing files for download can improve download speed and stability, and allow users to interrupt downloads and resume where they left off.

File upload: Users may need to upload large files, such as data imports and file backups. Slice upload can improve upload efficiency, upload file slices in batches, and display the upload status progress.

Image/Video Upload and Preview:

Image Upload and Preview: Speed up image uploads and display progress in real-time with slice upload. After the upload is complete, provide a preview function so users can immediately view the uploaded images.

Video upload and preview: Slice upload ensures reliable and efficient upload of large video files, with real-time progress status display. After the upload is complete, slice download technology allows users to watch the video smoothly without waiting for the entire file to be downloaded.

File Operations on Cloud Storage and Cloud Disk Applications:

File upload: Cloud storage and cloud disk applications can use chunking to improve the upload speed and stability of large files. This allows users to upload large files more quickly and reliably, even if their internet connection is unstable.

File download: When users need to download large files from cloud storage or cloud disks, they can use chunking to speed up the download process. This is especially useful for users with slow internet connections.

File preview and online editing: By slicing files and previewing and editing them online, users can get a better user experience. For example, users can preview and edit large video files without having to wait for the entire file to download.

Conclusion

Multipart file upload and download is an effective solution for optimizing front-end file streaming operations. By slicing large files into multiple small fragments and taking advantage of the breakpoint resume feature, you can improve the reliability and efficiency of file transfers.

The React framework can easily manage and operate file objects, providing a more convenient programming method for multipart file upload and download. Familiarity with tools such as Blob objects and ArrayBuffer is also key to processing and manipulating binary data.

In this article, you have gained an in-depth understanding of the basic concepts and technologies of file stream operations, and how to use React to implement file upload and download in slices. You have also mastered how to handle binary data and leverage streaming and resumable technology to optimize file transfer performance and user experience.

The implementation of file uploading and downloading in slices provides an efficient file processing method for front-end development. It can also cope with the challenges of network limitations and large file transmission. Mastering this knowledge and technologies will give you a powerful toolbox for handling file stream operations in front-end development.

Comments

[Beverly Keti]

Quick question pertaining to the file slice download method! I've once struggled with downloading architectural rendering softwares off the internet. These are large files which take forever to download... and sometimes the download session gets interrupted midway. Can the file slice download method be implemented in such situations, and does this also present itself as a solution to the challenges faced in installing a "large file" software into a PC?

[Tai Kedzierski]

The OP should still answer from the JavaScript client point of view but...

Your question sounds like you're talkting about the general case of downloading files from arbitrary websites (i.e. not your own servers)

Downloads with interruptions can be commonplace here, and whether or not resuming a download is possible will depend on whether the server itself supports handling download requests starting at an offset.

If the server is just using a filesystem-backed Apache/nginx server then probably yes, but if the file is generated on the fly on the backend (which I assume is not uncommon), then it may be that the HTTP serving component itself cannot resume from a known file, needing to generate a new one per-request and thus.... cannot guarantee an identical file to start at offset from...

If there's a way around that I'm a taker! But as corollary, if you design a file download system, this is a use-case to consider 😉