Upload to S3 without losing you sanity

TL;DR scroll to The Actual Code header

The story

if you're like me, and you hate, and I mean HATE, working with user assets, then congratulations. You're a sane person!

I've spent days trying to find a safe, lean, intelligent, and clean (or slic (I just made that acronym up)) way to handle uploads, specifically to S3.

Now, I know there are a ton of ways. I've heard them all. I've combed through 100s of articles and most of the aws docs.
Maybe I'm dumb, or lack respect for the old ways, from the offered solutions there are hacking tools like multer and multiparty into your app, even if it doesn't actually support it like a nextjs application doesn't. Or generating hidden elements in a form sourced from some dark magic like it's 1999. Or the worst one, trying to wrangle a file to a readable steam and then to a byte array and pray that the next person reading your code was bored enough to memorize file-related api or is a huge js nerd.

Now, lets differentiate between uploading a file and uploading a file.

This isn't about getting a File inside an <input /> in a fancy way. For that there are many amazing solutions like uppy and react-dropzone that I really suggest you use over making some home-brew solution.

We're talking about what do to with the damn file once you have it. Let take the simpler use case of uploading a single file because I'm not your savior when it comes to galleries, Sorry 🤷‍♂️

So lets understand the problem. Why can't you just upload the asset directly to S3? It's technically possible, they even document how to do and and all sorts of neat stuff! Well sure, if you don't mind me snatching your credentials and uploading whatever I want, go ahead! Theres an exposure risk involved.

Usually, you'd just swallow the bitter pill and just pray that someone somewhere was motivated enough to write a sane solution recently enough as to not make it too outdated or even just broken.

But we don't want to do that.
So how do we get the file safely to where they should without getting it through our server? Well thats simple, you create a very temporary safe portal for it to go directly to your S3 bucket!

How? Well, if you go snooping around, you'll see that sometimes around 53BC, the aws-sdk package had an S3 module with a createPresignedPost method. What's that method? It creates a url and fields combo, that if used together in the alloted time, could be used to upload assets without going through your server. Wow! 😃
Too bad we don't have that any more. 🤦‍♂️

Or do we? 🤔
Snooping a bit more, we find that aws-sdk split itself into clients. Ok, that gets us @aws-sdk/client-s3. Looking even further, there's a @aws-sdk/s3-presigned-post. Ok back on track! 🥳

This is enough talk. I just wasted so much time on this I wanted to express myself and my journey.

The Actual Code

You need to first create yourself a server, lets say you already have one. It should have an endpoint to return the url and fields we need.
Add the required packages:

$ yarn add @aws-sdk/client-s3 @aws-sdk/s3-presigned-post

Supply the aws credentials however you prefer, I'll use an environment variable

AWS_ACCESS_KEY_ID=<ACCESS_KEY_ID>
AWS_SECRET_ACCESS_KEY=<SECRET_ACCESS_KEY>

and create an endpoint

// controllers/createUploadUrl.ts

import { createPresignedPost } from "@aws-sdk/s3-presigned-post"
import { S3Client } from "@aws-sdk/client-s3"

// fill your region and anything else you care about
const s3Client = new S3Client({ region: 'eu-central-1' })

// idk what framework you're using, adapt!
const handler = async (req: Request, res: Response) => {
  const { url, fields } = await createPresignedPost(s3Client, {
    Bucket: 'my-own-unique-bucket',
    Key: `my-app/assets/${req.body['fileName']}`,
    Fields: { acl: 'public-read' },
    Conditions: [{ acl: "public-read" }, { bucket: "my-own-unique-bucket" }, ["starts-with", "$key", "my-app/assets/"]

 return { url, fields }
}

Note in particular the Conditions field. That field will make sure that no surprising changes will be made to the request once you return the url and fields back to the frontend. Add that together with a self-destruct timer and not using your hidden credentials, and you have yourself a pretty darn secure portal for your clients files to fly through!

Now back in the frontend, it's just a matter of

async function sendAsset(file: File) {
  // use fetch or axios or whatever to reach your endpoint
  const { url, fields } = await createUploadUrl({
    name: `logo.${file.name.split(".").pop()}`,
  })
  // FormData accepts only a form element in the constructor, so we gotta build it ourselves
  const formData = new FormData()
  Object.entries(fields).forEach(([field, value]) => {
    formData.append(field, value)
  })
  // now the important part!
  formData.append('file', file) // Note, it has. to be called file!

  // This is also very important! Undocumented too 😢
  const headers = new Headers({'Content-Length': `${file.size + 5000}`})

  await fetch(url, {
    method: "POST",
    headers,
    body: formData
  })
}

bada bing bada boom. No hacks around it.
I hope this helps someone, I've struggling with this for too long

Tell me in the comments how you handle asset uploading 🥰