Event Based System with Localstack (Elixir Edition): Uploading files to S3 with PresignedURL's

Introduction

If you are reading this maybe you know what its Phoenix, but let me explain a few things first:

• Localstack: Its a local version of AWS, yes, S3, SQS, SNS, DynamoDB and more all available from in your local machine

We will create an a platform that uses a ReactJS Frontend that calls the api in Phoenix and Uploads files to S3 and then notifies the file upload to a SQS Queue and then we process the messages and notifies using PhoenixChannels to all the connected frontends

Complex? sounds like but its not a big deal, lets start it!

The real work

First steps

You need to create your base, an a Frontend and a Backend in Phoenix, maybe you have it so lets move to the important part

You need Docker and Compose to use this examples, strongly recommend because makes easy with almost 0 config on the Localstack side, so all the code must assume that you have your project dockerized and configured with compose

Adding Localstack

This is the main config on the docker-compose.yml file

  localstack:
    image: localstack/localstack:latest
    ports:
      - 4566:4566
    environment:
      # own env vars
      BUCKET_NAME: files
      DEFAULT_REGION: us-west-2
      AWS_ACCESS_KEY_ID: test
      AWS_SECRET_ACCESS_KEY: test
      # service env vars
      SERVICES: s3
      DISABLE_CORS_CHECKS: 1
      PROVIDER_OVERRIDE_S3: asf
      S3_SKIP_SIGNATURE_VALIDATION: 1
    volumes:
      - ./.localstack:/var/lib/localstack
      - ./init_localstack.sh:/etc/localstack/init/ready.d/init_localstack.sh

And this is the init_localstack.sh file content, a unique thing about localstack its that you can move all strings like an aws-cli tool, also the container deletes all the content and config once the container stops, so the script file must create all the resources that you need from Localstack

Basically the init file creates the bucket and add cors to ensure the calls must be allowed

#!/bin/bash

echo "########### Create S3 bucket ###########"
awslocal s3api create-bucket\
    --region $DEFAULT_REGION\
    --bucket $BUCKET_NAME\
    --create-bucket-configuration LocationConstraint=$DEFAULT_REGION

echo "########### Adding cors bucket ###########"
awslocal s3api put-bucket-cors\
    --bucket $BUCKET_NAME\
    --cors-configuration '{
      "CORSRules": [
        {
          "AllowedHeaders": ["*"],
          "AllowedMethods" : ["HEAD", "GET", "POST", "PUT", "DELETE"],
          "AllowedOrigins" : [
            "http://localhost:4000",
            "http://localhost:3000"
          ],
          "ExposeHeaders": []
        }
      ]
    }'

echo "########### List S3 bucket ###########"
awslocal s3api list-buckets

In this examples we will use a feature of aws named presigned url's that its a url with signatures that will allow to the read/write from any caller, to ensure your browser can reach the Localstack container, must add this line to your /etc/hosts file

127.0.0.1 localstack

Adding AWS to Phoenix

We will use the library ex_aws to manage all the AWS resources

Add this to your mix.exs

{:ex_aws, "~> 2.1"},
{:ex_aws_s3, "~> 2.0"}

And add this to your config files (adjust in dev.exs and runtime.exs)

config :ex_aws, :s3,
  scheme: "http://",
  region: System.get_env("AWS_S3_REGION"),
  host: System.get_env("AWS_S3_URL"),
  port: System.get_env("AWS_S3_PORT"),
  bucket: System.get_env("AWS_S3_BUCKET")

• Generating the Presigned URL

This function will generate the Presigned URL

  def presigned_url_upload(object_key, opts \\ []) do
    get_presigned_url(:put, object_key, opts)
  end

  defp get_presigned_url(http_method, object_key, opts) do
    # this its for a configuration issue with localhost
    opts =
      if is_prod() do
        opts
      else
        opts
        |> Keyword.put(:virtual_host, is_prod())
        |> Keyword.put(:bucket_as_host, true)
      end

    ExAws.S3.presigned_url(
      config(),
      http_method,
      s3_bucket(),
      object_key,
      opts
    )
  end

Full code here

and sharing the file

  # on some controller

  # 30 mins
  @presigned_upload_url_max_age 60 * 30
  # 10 MB
  @presigned_upload_url_max_file_size 10 * 1_000_000

  @presigned_url_opts [
    ["starts-with", "$Content-Type", ""],
    {:expires_in, @presigned_upload_url_max_age},
    {:content_length_range, [1, @presigned_upload_url_max_file_size]}
  ]

  def get_presigned_url(conn, _) do
    {:ok, url} = S3Client.presigned_url_upload("some_file_name", @presigned_url_opts)

    conn
    |> put_status(200)
    |> json(%{url: url})
  end

Uploading files from Frontend

First lets add the form component, something simple

        <input
          type="file"
          onChange={handleFileChange}
        />
        {uploaded &&
        <span className="uploaded">
          Uploaded
        </span>
        }

And upload it when a file its selected, this function example must check the if its a valid file, then get the presigned url and upload to this presigned url using fetch on a put method

  const [uploaded, setUploaded] = useState(false);

  const handleFileChange = async (e) => {
    if (!e.target.files) {
      return;
    }

    // get the presigned url
    let response = await api.getPresignedUrl();

    if (response.status < 200 && response.status > 300) {
      // show error getting presigned url
      return;
    }

    const file = e.target.files[0];
    const {url} = response.body;

    // uploading the file with put request to the presigned url
    response = await fetch(url, {
      body: file,
      method: "PUT"
    });

    if (response.ok && response.status >= 200 && response.status < 300) {
      setUploaded(true);
      setTimeout(setUploaded.bind(this, false), 5000); // will remove the 
    }
  };

And its all, you are pushing files to s3, the config its more extense than the real code but, a lot of times integrate some library or platforms to our code its more config than realworld code

The full code of this project its here and also includes a project structure front/back in same repo with docker compose config, just check the Makefile to see the available commands

In the next post we will:

• Add SQS Queues and add a Worker per Queue using Broadway
• Configure S3 Notification to SQS Queue When a file its uploaded
• Post Messages to a Queue using Elixir

Part 2 here!

See you in the next posts