Optimizing File Uploads in Full-Stack Applications with Amazon S3: A Case Study

Introduction

File uploads are a common feature in modern web applications, but they often come with challenges, especially when working with cloud storage solutions like Amazon S3. Recently, I faced a few roadblocks while integrating file uploads into a quote request form. This blog covers the lessons learned from tackling these issues and best practices for handling file uploads in full-stack applications.

The Problem: File Uploads and Form Handling

Our project required a form where users could upload files (such as images or PDFs) along with a few input fields (first name, last name, email, etc.). These files would then be stored in an S3 bucket, and the form submission would store file metadata in the database for later retrieval.

However, while implementing this, we encountered several issues:

1. Form data wasn’t being submitted correctly from the frontend.
2. Files uploaded were not showing up when fetched from the database.
3. Differences in how the frontend and backend were handling multipart/form-data.

The Solution: Keep It Simple

After a few rounds of debugging and testing, especially using Postman for backend verification, we realized the frontend was over-complicating the process. Instead of manually appending files to FormData, reverting to a simpler solution solved the problem.

Here’s how we approached the final solution:

1. Form Handling in Frontend Instead of manually appending the form fields and files into FormData, we used the following approach:

const handleSubmit = async (formData) => {
  await QuoteRequestService.create(formData);
  router.push("/quote-requests");
};

This allowed the backend to interpret the form data correctly without the need for complex file handling on the frontend.

2. Backend Handling of File Uploads In the backend, we ensured that the files were correctly handled by Multer (a Node.js middleware) and passed to a service responsible for uploading files to S3. Here’s an outline of how the backend processed this:

const fileUploadResponse = await FileService.upload({
  file, // This contains the uploaded file from req.files
  relation_id: newQuoteRequest.id,
  relation_type: "quote_request",
  field: "uploaded_files",
  folder: "quote_requests",
  acl: "public-read",
  compress_while_uploading: true,
});

3. Eager Loading and File Retrieval One of the challenges was retrieving uploaded files from the database after they were successfully uploaded to S3. By using eager loading, we ensured that the related files were fetched correctly:

const files = await FileRelated.find({
  where: { related_id: quoteRequest.id, related_type: "quote_request" },
  relations: ["file"],
});

Key Takeaways

• Simplify Form Handling: Don’t over-complicate things on the frontend. Ensure that the backend can handle file uploads efficiently, and only send what’s necessary.
• Testing with Postman: Tools like Postman can help isolate whether the issue lies with the frontend or backend.
• Amazon S3 Integration: Proper S3 configuration is crucial for handling file uploads and making files publicly accessible. Ensure you have the right access control settings in place (e.g., public-read).
• Error Handling and Debugging: Always log critical points of your code to understand where things might be breaking (e.g., logging file metadata and S3 keys).

Conclusion

Handling file uploads in full-stack applications can be tricky, but with a clear understanding of how the frontend and backend interact, you can avoid common pitfalls. The key is to keep things simple and well-tested. By focusing on backend handling and leveraging the capabilities of cloud storage solutions like S3, you can ensure a seamless experience for both developers and users.