DEV Community: Alvise Susmel

AWS S3 in Elixir with ExAws

Alvise Susmel — Wed, 04 Dec 2019 16:00:00 +0000

In this article we see how to store and retrieve files on AWS S3 using Elixir and with the help of ExAws.

We start by setting up an AWS account and credentials, configure an Elixir application and see the basic upload and download operations with small files.

Then, we see how to deal with large files, making multipart uploads and using presigned urls to create a download stream, processing data on the fly.

Create an IAM user, configure permissions and credentials

If you don't have an Amazon Web Services account yet, you can create it on https://aws.amazon.com/ and use the free tier for the first 12 months where you have up to 5GB of free S3 storage.

Be sure you check all the limits of the free tier before you start using the service. Always take a look at the billing page to keep track of the usage.

To access to AWS S3 resources, first we need to create an AWS IAM (Identity and Access Management) user with limited permissions.

Once logged into the AWS console, go to the users section of the security credentials page and click on Add user.

Menu on the top-right side of the AWS console

Create a new IAM user

When creating a user, we need to set a username and most importantly enable the Programmatic access: this means the user can programmatically access to the AWS resources via API.

Username and Programmatic access

Then we set the permissions, attaching the AmazonS3FullAccess policy and limiting the user to just the S3 service.

AmazonS3FullAccess policy

Now, this policy is fine for this demo, but it's still too broad: a user, or an app, can access to all the buckets, files and settings of S3.

By creating a custom policy, we can limit the user permissions to only the needed S3 actions and buckets. More on this at AWS User Policy Examples

Once the user is created, we can download the Access Key Id and the Secret Access Key. You must keep these keys secret because whoever has them can access to your AWS S3 resources.

IAM user Access key ID and Secret access key

To create an S3 bucket using the AWS console, go to the S3 section and click on Create bucket, set a bucket name (I've used poeticoding-aws-elixir) and be sure to block all the public access.

Bucket name and region

Block all public access

Configure `ex_aws` and environment variables

Let's create a new Elixir application and add the dependencies to make ex_aws and ex_aws_s3 work

# mix.exs
def deps do
  [
    {:ex_aws, "~> 2.1"},
    {:ex_aws_s3, "~> 2.0"},
    {:hackney, "~> 1.15"},
    {:sweet_xml, "~> 0.6"},
    {:jason, "~> 1.1"},
  ]
end

ExAws, by default, uses hackney HTTP Client to make requests to AWS.

We create the config/config.exs configuration file, where we set access id and secret access keys

# config/config.exs

import Config

config :ex_aws,
  json_codec: Jason,
  access_key_id: {:system, "AWS_ACCESS_KEY_ID"},
  secret_access_key: {:system, "AWS_SECRET_ACCESS_KEY"}

The default ExAws JSON codec is Poison. If we want to use another library, like Jason, we need to explicitly set the jason_codec property.

We don't want to write our keys in the configuration file. First, because who has access to the code can see them, second because we want to make them easy to change.

We can use environment variables: by passing {:system, "AWS_ACCESS_KEY_ID"} and {:system, "AWS_SECRET_ACCESS_KEY"} tuples the application gets the keys from the AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY environment variables.

In case you are on a Unix/Unix-like system (like MacOs or Linux), you can set these environment variables in a script

# .env file
export AWS_ACCESS_KEY_ID="your access key"
export AWS_SECRET_ACCESS_KEY="your secret access key"

and load them with source

$ source .env
$ iex -S mix

Keep this script secret. If you are using git remember to put this script into .gitignore to avoid to commit this file.

If you don't want to keep these keys in a script, you can always pass them when launching the application or iex

$ AWS_ACCESS_KEY_ID="..." \
  AWS_SECRET_ACCESS_KEY="..." \
  iex -S mix

In case you're on a Windows machine, you can set the environment variables using the command prompt or the PowerShell

# Windows CMD
set AWS_ACCESS_KEY_ID="..."

# Windows PowerShell
$env:AWS_ACCESS_KEY_ID="..."

Listing the buckets

Now we have everything ready: credentials, application dependencies and ex_aws configured with environment variables. So let's try the first request.

# load the environment variables
$ source .env

# run iex
$ iex -S mix

iex> ExAws.S3.list_buckets()
%ExAws.Operation.S3{
  http_method: :get,
  parser: &ExAws.S3.Parsers.parse_all_my_buckets_result/1,
  path: "/",
  service: :s3,
  ...,
}

The ExAws.S3.list_buckets() function doesn't send the request itself, it returns an ExAws.Operation.S3 struct. To make a request we use ExAws.request or ExAws.request!

iex> ExAws.S3.list_buckets() |> ExAws.request!()

%{
  body: %{
    buckets: [
      %{
        creation_date: "2019-11-25T17:48:16.000Z",
        name: "poeticoding-aws-elixir"
      }
    ],
    owner: %{ ... }
  },
  headers: [
    ...
    {"Content-Type", "application/xml"},
    {"Transfer-Encoding", "chunked"},
    {"Server", "AmazonS3"},
    ...
  ],
  status_code: 200
}

ExAws.request! returns a map with the HTTP response from S3. With get_in/2 we can get just the bucket list

ExAws.S3.list_buckets()
|> ExAws.request!()
|> get_in([:body, :buckets])

[%{creation_date: "2019-11-25T17:48:16.000Z", name: "poeticoding-aws-elixir"}]

put, list, get and delete

With ExAws, the easiest way to upload a file to S3 is with ExAws.S3.put_object/4

iex> local_image = File.read!("elixir_logo.png")
<<137, 80, 78, 71, 13, 10, 26, 10, 0, 0, ...>>

iex> ExAws.S3.put_object("poeticoding-aws-elixir", "images/elixir_logo.png", local_image) \
...> |> ExAws.request!()

%{
  body: "",
  headers: [...],
  status_code: 200
}

The first argument is the bucket name, then we pass the object key (the path) and the third is the file's content, local_image. As a fourth argument we can pass a list of options like storage class, meta, encryption etc.

Using the AWS management console, on the S3 bucket's page, we can see the file we've just uploaded.

Uploaded file visible on AWS Management console

We list the bucket's objects with ExAws.S3.list_objects

iex> ExAws.S3.list_objects("poeticoding-aws-elixir") \
...> |> ExAws.request!() \
...> |> get_in([:body, :contents]) \

[
  %{
    e_tag: "\"...\"",
    key: "images/elixir_logo.png",
    last_modified: "2019-11-26T14:40:34.000Z",
    owner: %{ ... }
    size: "29169",
    storage_class: "STANDARD"
  }
]

Passing the bucket name and object key to ExAws.S3.get_object/2, we get the file's content.

iex> resp = ExAws.S3.get_object("poeticoding-aws-elixir", "images/elixir_logo.png") \
...> |> ExAws.request!()

%{
  body: <<137, 80, 78, 71, 13, 10, 26, ...>>,
  headers: [
    {"Last-Modified", "Tue, 26 Nov 2019 14:40:34 GMT"},
    {"Content-Type", "application/octet-stream"},
    {"Content-Length", "29169"},
    ...
  ],
  status_code: 200
}

The request returns a response map with the whole file's content in :body.

iex> File.read!("elixir_logo.png") == resp.body
true

We can delete the object with ExAws.S3.delete_object/2.

iex> ExAws.S3.delete_object("poeticoding-aws-elixir", "images/elixir_logo.png") \
...> |> ExAws.request!()

%{
  body: "",
  headers: [
    {"Date", "Tue, 26 Nov 2019 15:04:35 GMT"},
    ...
  ],
  status_code: 204
}

After listing again the objects we see, as expected, that the bucket is now empty.

iex> ExAws.S3.list_objects("poeticoding-aws-elixir") 
...> |> ExAws.request!() 
...> |> get_in([:body, :contents])

[]

Multipart upload and large files

The image in the example above is just ~30 KB and we can simply use put_object and get_object to upload and download it, but there are some limits:

with these two functions the file is fully kept in memory, for both upload and download.
put_object uploads the file in a single operation and we can upload only objects up to 5 GB in size.

S3 and ExAws client support multipart uploads. It means that a file is divided in parts (5 MB parts by default) which are sent separately and in parallel to S3! In case the part's upload fails, ExAws retries the upload of that 5 MB part only.

With multipart uploads we can upload objects from 5 MB to 5 TB - ExAws uses file streams, avoiding to keep the whole file in memory.

Let's consider numbers.txt, a relatively large txt file we've already seen in another article - Elixir Stream and large HTTP responses: processing text ( you can download from this url https://www.poeticoding.com/downloads/httpstream/numbers.txt).

numbers.txt size is 125 MB, much smaller than the 5GB limit imposed by the single PUT operation. But to me this file is large enough to benefit from a multipart upload!

iex> ExAws.S3.Upload.stream_file("numbers.txt") \
...> |> ExAws.S3.upload("poeticoding-aws-elixir", "numbers.txt") \
...> |> ExAws.request!()

# returned response
%{
  body: "<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n\n<CompleteMultipartUploadResult>...",
  headers: [
    {"Date", "Tue, 26 Nov 2019 16:34:08 GMT"},
    {"Content-Type", "application/xml"},
    {"Transfer-Encoding", "chunked"},
  ],
  status_code: 200
}

First we create a file stream with ExAws.S3.Upload.stream_file/2
The stream is passed to ExAws.S3.upload/4, along with bucket name and object key
ExAws.request! initialize the multipart upload and uploads the parts

To have an idea of what ExAws is doing, we can enable the debug option in the ex_aws configuration

# config/config.exs

config :ex_aws,
 debug_requests: true,
 json_codec: Jason,
 access_key_id: {:system, "AWS_ACCESS_KEY_ID"},
 secret_access_key: {:system, "AWS_SECRET_ACCESS_KEY"}

We should see multiple parts being sent at the same time

17:11:24.586 [debug] ExAws: Request URL: "...?partNumber=2&uploadId=..." ATTEMPT: 1
17:11:24.589 [debug] ExAws: Request URL: "...?partNumber=1&uploadId=..." ATTEMPT: 1

Multipart upload timeout

When the file is large, the upload could take time. To upload the parts concurrently, ExAws uses Elixir Tasks - the default timeout for part's upload is set to 30 seconds, which could not be enough with a slow connection.

** (exit) exited in: Task.Supervised.stream(30000)
    ** (EXIT) time out

We can change the timeout by passing a new :timeout to ExAws.S3.upload/4, 120 seconds in this example.

ExAws.S3.Upload.stream_file("numbers.txt")
|> ExAws.S3.upload(
  "poeticoding-aws-elixir", "numbers.txt", 
  [timeout: 120_000]) 
|> ExAws.request!()

Download a large file

To download a large file it's better to avoid get_object, which holds the whole file content in memory. With ExAws.S3.download_file/4 instead, we can download the data in chunks saving them directly into a file.

ExAws.S3.download_file(
  "poeticoding-aws-elixir", 
  "numbers.txt", "local_file.txt"
) 
|> ExAws.request!()

presigned urls and download streams - process a file on the fly

Unfortunately we can't use ExAws.S3.download_file/4 to get a download stream and process the file on the fly.

However, we can generate a presigned url to get a unique and temporary URL and then download the file with a library like mint or HTTPoison.

iex> ExAws.Config.new(:s3) \
...> |> ExAws.S3.presigned_url(:get, "poeticoding-aws-elixir", "numbers.txt")

{:ok, "https://...?X-Amz-Credential=...&X-Amz-Expires=3600"}

By default, the URL expires after one hour - with the :expires_in option we can set a different expiration time (in seconds).

iex> ExAws.Config.new(:s3) \
...> |> ExAws.S3.presigned_url(:get, "poeticoding-aws-elixir", 
   "numbers.txt", [expires_in: 300]) # 300 seconds

{:ok, "https://...?X-Amz-Credential=...&X-Amz-Expires=300"}

Now that we have the URL, we can use Elixir Streams to process the data on the fly and calculate the sum of all the lines numbers.txt. In this article you find the HTTPStream's code and how it works.

# generate the presigned URL
ExAws.Config.new(:s3) 
|> ExAws.S3.presigned_url(:get, "poeticoding-aws-elixir", "numbers.txt")

# returning just the URL string to the next step
|> case do
  {:ok, url} -> url
end

# using HTTPStream to download the file in chunks
# getting a Stream of lines
|> HTTPStream.get()
|> HTTPStream.lines()

## converting each line to an integer
|> Stream.map(fn line-> 
  case Integer.parse(line) do
    {num, _} -> num
    :error -> 0
  end
end)

## sum the numbers
|> Enum.sum()

|> IO.inspect(label: "result")

In the first two lines we generate a presigned url. Then, with HTTPStream.get we create a stream that lazily downloads the file chunk by chunk, transforming chunks into lines with HTTPStream.lines, mapping the lines into integers and summing all the numbers. The result should be 12468816.

Understanding LiveView: build a Gallery app

Alvise Susmel — Wed, 20 Nov 2019 17:50:13 +0000

Understanding Phoenix LiveView

Setup
The Primitives
Building a Gallery app - (this article)

Subscribe to get FREE Tutorials by email. Start learning Elixir and Phoenix to build features and apps!

In the previous articles we've seen how to setup Phoenix LiveView and built a counter (which is like the "Hello World" code example in the LiveView world). We’ve also taken a look at the live-cycle, inspecting websocket messages, having a glimpse of how the things work under the hood.

It's now time to build our Gallery app! For simplicity we are not going to use any database. Our images will be just a fixed set of urls taken from Unsplash.

We are about to see different things, starting with a simple draft of the app and then going through a refactoring of the code, and building features like thumbnails and a slideshow option.

From a counter to a gallery

We start by using the counter's code we’ve seen in the previous article.

At the moment we aim to just get a working draft of a gallery - we’ll spend time later to refactor the code and add more functionalities.

# lib/gallery_web/live/gallery_live.ex

defmodule GalleryWeb.GalleryLive do
  use Phoenix.LiveView

  def mount(_session, socket) do
    {:ok, assign(socket, :counter, 0)}
  end

  def render(assigns) do
    ~L"""
    <label>Counter: <%= @counter %></label>
    <button phx-click="incr">+</button>
    """
  end

  def handle_event("incr", _, socket) do
    {:noreply, update(socket, :counter, &(&1 + 1))}
  end
end

So we start by copying the counter code in the GalleryLive module, checking that the correct route is in place in lib/gallery_web/router.ex

# lib/gallery_web/router.ex

defmodule GalleryWeb.Router do
  use GalleryWeb, :router
    ...

  scope "/", GalleryWeb do
        ...
    live "/gallery", GalleryLive
  end

end

We have a fixed list of image urls taken from unsplash.com

[
  "https://images.unsplash.com/photo-1562971179-4ad6903a7ed6?h=500&fit=crop",
  "https://images.unsplash.com/photo-1552673597-e3cd6747a996?h=500&fit=crop",
  "https://images.unsplash.com/photo-1561133036-61a7ed56b424?h=500&fit=crop",
  "https://images.unsplash.com/photo-1530717449302-271006cdc1bf?h=500&fit=crop"
]

we need an index to go through these images, incrementing and decrementing it like we did for the counter

At the beginning the index is 0 and it points to the first image. By incrementing it, the index will then point to the second, third and fourth image. Once reached the end of the list, it will go back to the first one.

Let's rename :counter to :idx in mount/2. Then, we change the template in render/1, adding two new buttons: prev and next. They send two different "prev" and "next" events, which we handle separately.

defmodule GalleryWeb.GalleryLive do
    use Phoenix.LiveView

    def mount(_session, socket) do
      {:ok, assign(socket, :idx, 0)}
    end

    def render(assigns) do
      ~L"""
      <label>Image Index: <%= @idx %></label>
      <button phx-click="prev">Prev</button>
      <button phx-click="next">Next</button>
      """
    end

    def handle_event("prev", _event, socket) do
      {:noreply, update(socket, :idx, &(&1 - 1)}
    end

    def handle_event("next", _event, socket) do
      {:noreply, update(socket, :idx, &(&1 + 1)}
    end
end

handle_event("next", _event, socket) is identical to the "incr" counter's version, and it increments the :idx by 1.
handle_event("prev", _event, socket) instead decrements :idx by 1.

It's better to move these updates in two separate functions: assign_prev_idx/1 and assign_next_idx/1.

def handle_event("prev", _event, socket) do
    {:noreply, assign_prev_idx(socket)}
end

def handle_event("next", _event, socket) do
    {:noreply, assign_next_idx(socket)}
end

def assign_prev_idx(socket) do
    socket
    |> update(:idx, &(&1 - 1))
end

def assign_next_idx(socket) do
    socket
    |> update(:idx, &(&1 + 1))
end

They take a socket and update/3 the :idx returning a new socket. There are different reasons why I prefer to move the update part into a different function, outside handle_event. With just an update/3 the advantage of doing so maybe is not that obvious, but in this way we keep the handle_event/3 cleaner, the update's logic easier to test, and if we give a good name to the new function it's also clearer what our handle_event/3 callback does.

To show the images, we define a new function called image(idx), where the argument is the index, and it returns an image url.

def image(idx) do
    [
    "https://images.unsplash.com/photo-1562971179-4ad6903a7ed6?h=500&fit=crop",
    "https://images.unsplash.com/photo-1552673597-e3cd6747a996?h=500&fit=crop",
    "https://images.unsplash.com/photo-1561133036-61a7ed56b424?h=500&fit=crop",
    "https://images.unsplash.com/photo-1530717449302-271006cdc1bf?h=500&fit=crop"
    ]
    |> Enum.at(idx)
end

For the sake of simplicity, image/1 uses a small fixed list of image urls.
We use Enum.at/2 to get the image url at the index idx. Enum.at/2 works also with a negative index, going through the list in reverse order

iex> ["a", "b", "c"] |> Enum.at(-1)
"c"

iex> ["a", "b", "c"] |> Enum.at(10)
nil

When the index is out-of-bound, Enum.at/2 by default returns nil. When the index reaches the end of the list, it should then go back at the beginning pointing to the first element.

We can use rem/2, the reminder of an integer division. The first argument is our index (dividend) and the second is the length of the list (divisor)

iex> rem(2, 3)
2
iex> rem(3, 3)
0
iex> rem(4, 3)
1
iex> rem(-4, 3)
1

iex> ["a", "b", "c"] |> Enum.at(rem(2, 3))
"c"
iex> ["a", "b", "c"] |> Enum.at(rem(3, 3))
"a"
iex> ["a", "b", "c"] |> Enum.at(rem(4, 3))
"b"

So we can use it to loop over the list, while the index increments.

Since our image list is constant, we can define it as module attribute and calculate its length at compilation time

defmodule GalleryWeb.GalleryLive do
  @images [
    "https://images.unsplash.com/photo-1562971179-4ad6903a7ed6?h=500&fit=crop",
    "https://images.unsplash.com/photo-1552673597-e3cd6747a996?h=500&fit=crop",
    "https://images.unsplash.com/photo-1561133036-61a7ed56b424?h=500&fit=crop",
    "https://images.unsplash.com/photo-1530717449302-271006cdc1bf?h=500&fit=crop"
  ]
    @images_count Enum.count(@images)
  ...

    def image(idx) do
        idx = rem(idx, @images_count)
        Enum.at(@images, idx)
    end
end

Using Enum.at in this case is more than fine, but remember that lists in Elixir are linked lists: if the list is big, Enum.at/2 can become expensive since it has to go through the whole list to reach the elements at the end.

Please drop me a comment below if you'd like to have an episode on how lists work in Elixir.

We can define an img tag, in the view in render/1 function, and use the image/1 function to set the src attribute's value.

def render(assigns) do
  ~L"""
  <label>Image Index: <%= @idx %></label>
  <button phx-click="prev">Prev</button>
  <button phx-click="next">Next</button>

  <img src="<%= image(@idx) %>">
  """
end

Going back to the browser, after refreshing the page, now we should see a working first version of our gallery - by pressing prev and next buttons we go through the images. 🎉🥳

A bit of refactoring: `Gallery` module

The next step is to do a bit of refactoring, which will make easier to add thumbnails and a slideshow functionality.

At the moment we have everything in the GalleryWeb.GalleryLive module - it would be nice to uncouple the gallery logic from the LiveView part, moving it to a different module called Gallery, defined in lib/gallery.ex

# lib/gallery.ex
defmodule Gallery do
  # image_ids()
    # first_id()


  # prev_image_id(ids, id)
  # prev_index(ids, id)

    # next_image_id(ids, id)
  # next_index(ids, id)

  # thumb_url(id)
  # large_url(id)

    # image_url(image_id, params)
end

`image_url/1`

The first function we are going to write is image_url/1. If we take a closer look at one of the Unsplash's image URL, we see that it's made by different parts

"https://images.unsplash.com/" the Unsplash base url
"photo-1562971179-4ad6903a7ed6" the image id
"?h=500&fit=crop" and some query params. We can use the query params to request a different image size: tuning h and w params we are able to request a large image or a thumbnail

Instead of keeping the list of URLs of large images, we can switch to a list of images ids. With just the image id we can build a URL for both thumbnails and large images.

# lib/gallery.ex

defmodule Gallery do
    @unsplash_url "https://images.unsplash.com"

    @ids [
    "photo-1562971179-4ad6903a7ed6",
    "photo-1552673597-e3cd6747a996",
    "photo-1561133036-61a7ed56b424",
    "photo-1530717449302-271006cdc1bf"
  ]

    def image_ids, do: @ids

  def image_url(image_id, params) do
    URI.parse(@unsplash_url)
    |> URI.merge(image_id)
    |> Map.put(:query, URI.encode_query(params))
    |> URI.to_string()
  end

end

We use the URI module to compose and generate the final URL. URI.parse/1 parses the
Unsplash base url returning a URI struct, then URI.merge/2 sets the image_id as the path

%URI{
  scheme: "https",
  port: 443,
  host: "[images.unsplash.com](http://images.unsplash.com/)",
  path: "/photo-1561133036-61a7ed56b424",
  query: nil,
    ...
}

URI.encode_query/1 then encodes a params map to a query string. If we want a thumbnail we can set both w and h to 100px

iex> URI.encode_query(%{w: 100, h: 100, fit: "crop"})
"fit=crop&h=100&w=100"

URI.to_string/1 returns the final URL, converting the struct into a string.

iex> Gallery.image_url("photo-1562971179-4ad6903a7ed6", %{w: 100, h: 100})

"https://images.unsplash.com/photo-1562971179-4ad6903a7ed6?h=100&w=100"

Let's add two helpers which can be useful later

# lib/gallery.ex

defmodule Gallery do

    def thumb_url(id), 
        do: image_url(id, %{w: 100, h: 100, fit: "crop"})

    def large_url(id), 
        do: image_url(id, %{h: 500, fit: "crop"})

  ...

end

thumb_url/1 returns a 100x100 image url and large_url/2 a 500px height image.

`next_image_id/2` and `prev_image_id/2`

Instead of dealing with an index directly (incrementing and decrementing it), we define a next_image_id(ids, id) function that given an id, returns the next element in the ids list.

# lib/gallery.ex

defmodule Gallery do

    def first_id(ids \\ @ids), do: List.first(ids)


    def next_image_id(ids\\@ids, id) do
        Enum.at(ids, next_index(ids, id), first_id(ids))
    end


    defp next_index(ids, id) do
      ids
      |> Enum.find_index(& &1 == id)
      |> Kernel.+(1)
    end

  ...
end

next_image_id function has two different arity:

next_image_id/1: passing only the id argument, ids will be equal @ids.
next_image_id/2: with this function we pass the ids list ourself, which can be useful to unit test the function.

next_image_id/2 uses the private function next_index(ids,id), which finds the index of the id element in the ids list, incrementing it by 1.

If the id is the last element in the list, next_index/2 returns an index that is out-of-bound and Enum.at/2 (in next_image_id/2) would return nil.

We can pass first_id(ids) as Enum.at/3 third argument - instead of returning nil, Enum.at/3 will return the first element in ids.

Let's see next_image_id/2 in action on the terminal

iex> Gallery.next_image_id(["a", "b", "c"], "b")
"c"
iex> Gallery.next_image_id(["a", "b", "c"], "c")
"a"

prev_image_id/2 and prev_index/2 are really similar to the next_* functions

# lib/gallery.ex

defmodule Gallery do

    def prev_image_id(ids\\@ids, id) do
        Enum.at(ids, next_index(ids, id))
    end


    defp prev_index(ids, id) do
      ids
      |> Enum.find_index(& &1 == id)
      |> Kernel.-(1)
    end

  ...

end

In the prev_ case we don't need to set the Enum.at/3 default value, because prev_index/2 doesn't return an index out-of-bound. When id is the first element of ids, prev_index/2 passes the -1 index to Enum.at/2, which returns the ids last element.

Refactoring `GalleryLive`

Now it's time to make some changes in GalleryLive and use the functions we've just built in Gallery.

We don't need anymore the @images, @images_count module attributes and image/1 function.

In mount/2, instead of :idx, we now assign Gallery.first_id() to :current_id

# lib/gallery_web/live/gallery_live.ex

defmodule GalleryWeb.GalleryLive do
  ...

    def mount(_session, socket) do
      {:ok, assign(:current_id, Gallery.first_id())}
    end

  ...
end

In render/1 we use @current_id and instead of image(@idx) we adopt Gallery.large_url(@current_id)

# GalleryWeb.GalleryLive
# lib/gallery_web/live/gallery_live.ex

def render(assigns) do
    ~L"""
    <label>Image id: <%= @current_id %></label>
  <button phx-click="prev">Prev</button>
  <button phx-click="next">Next</button>

  <img src="<%= Gallery.large_url(@current_id) %>">
  """
end

Then we replace assign_prev_idx/1 and assign_next_idx/1 with assign_prev_id/1 and assign_next_id/1, updating handle_event("prev", _, socket) and handle_event("next", _, socket) accordingly

# GalleryWeb.GalleryLive
# lib/gallery_web/live/gallery_live.ex

def handle_event("prev", _, socket) do
  {:noreply, assign_prev_id(socket)}
end

def handle_event("next", _, socket) do
  {:noreply, assign_next_id(socket)}
end

def assign_prev_id(socket) do
  assign(socket, :current_id, 
    Gallery.prev_image_id(socket.assigns.current_id))
end

def assign_next_id(socket) do
  assign(socket, :current_id, 
    Gallery.next_image_id(socket.assigns.current_id))
end

Refreshing the page on the browser we should get a similar result of what we got before, but this time, instead of an index, we use image ids.

Thumbnails

It's now really easy to add the thumbnails, using a comprehension that maps ids returned by Gallery.image_ids() to <img> tags. We use Gallery.thumb_url/1 to get a thumbnail url

<center>
  <%= for id <- Gallery.image_ids() do %>
    <img src="<%= Gallery.thumb_url(id) %>">
  <% end %>
</center>

It would be nice to see which of the images in the thumbnails is shown below.

# GalleryWeb.GalleryLive
# lib/gallery_web/live/gallery_live.ex

defp thumb_css_class(thumb_id, current_id) do
  if thumb_id == current_id do
    "thumb-selected"
  else
    "thumb-unselected"
  end
end

We write the thumb_css_class(thumb_id, current_id) helper into GalleryLive and use it to render the css class of the thumbnails img tag.

<%= for id <- Gallery.image_ids() do %>
    <img src="<%= Gallery.thumb_url(id) %>"
  class="<%= thumb_css_class(id, @current_id) %>">
<% end %>

thumb_css_class/2 returns "thumb-selected" css class when id and @current_id are equal, "thumb-unselected" otherwise.

Then we add the two css classes in assets/css/app.css

/* assets/css/app.css */

.thumb-selected {
  border: 4px solid #0069d9;
}

.thumb-unselected {
  opacity: 0.5;
}

Slideshow

With a slideshow feature we want that GalleryLive automatically changes the current image at a regular interval.

Let's start by assigning a :slideshow value in mount/2, initially set to :stopped.

def mount(_session, socket) do
  socket = 
    socket
    |> assign(:current_id, Gallery.first_id())
    |> assign(:slideshow, :stopped)
  {:ok, socket}
end

Then we change the view in render/1 by removing the <label> tag and adding a third button:

when @slideshow is :stopped we show a Play button, which sends a "play_slideshow" event when clicked
otherwise we show a Stop button, which sends a "stop_slideshow" event

<center>
  <button phx-click="prev">Prev</button>
  <button phx-click="next">Next</button>

  <%= if @slideshow == :stopped do %>
    <button phx-click="play_slideshow">Play</button>
  <% else %>
    <button phx-click="stop_slideshow">Stop</button>
  <% end %>
</center>

We handle the first event which starts the slideshow

def handle_event("play_slideshow", _, socket) do
  {:ok, ref} = :timer.send_interval(1_000, self(), :slideshow_next)
  {:noreply, assign(socket, :slideshow, ref)}
end

def handle_info(:slideshow_next, socket) do
  {:noreply, assign_next_id(socket)}
end

:timer.send_interval(milliseconds, pid, message) starts the slideshow by sending every second a :slideshow_next message to self(), the Gallery LiveView process. It returns a ref reference, which we'll need later to stop the slideshow, and we assign it to :slideshow.

The process now receives a :slideshow_next message every second. This message is handled by handle_info(:slideshow_next, socket) which calls assign_next_id(socket) to assign :current_id to the next image id.

To stop the slideshow, we implement the handle_event("stop_slideshow", _, socket) function that cancels the timer and assigns :slideshow back to :stopped.

def handle_event("stop_slideshow", _, socket) do
  :timer.cancel(socket.assigns.slideshow)
  {:noreply, assign(socket, :slideshow, :stopped)}
end

We made it! 🎉👩‍💻👨‍💻🎉 We finally got a slideshow feature that shows us the images automatically!

What's next?

We've seen a lot! If you want to copy & paste the code, you find the full code at this gist link with the two Gallery and GalleryWeb.GalleryLive modules we've built during the article.

If you want the plug&play project's code (to try everything out without coding it yourself) please let me know in the comments below.

You can now take a step forward: you can use LiveView live_link and pushState support to bring the image id in the URL and update the URL when showing a different image. This gives you a great way to share a specific image to other users.

Subscribe to get FREE Tutorials by email. Start learning Elixir and Phoenix to build features and apps!

The Primitives of Phoenix LiveView

Alvise Susmel — Wed, 13 Nov 2019 15:32:07 +0000

Understanding Phoenix LiveView

Setup
The Primitives (this article)
Building a Gallery app

Subscribe to get FREE Tutorials by email. Start learning Elixir and Phoenix to build features and apps!

In the previous article we saw how to setup Phoenix LiveView in order to have everything we need to build a simple LiveView gallery app.

Now, before creating our app, I'd like to explore the Phoenix LiveView primitives, understanding the magic behind LiveView while learning how we can build a simple counter.

Once built some confidence and understood how LiveView and its life-cycle work, we'll see in the next article that the gallery app is just an easy evolution of this counter.

New `CounterLive` module and static HTML

So, let's now put temporarily aside the GalleryWeb.GalleryLive module we've defined during the setup, and create a new live view module GalleryWeb.CounterLive in lib/gallery_web/live/counter_live.ex, which we'll use in this article mainly as a playground.

# lib/gallery_web/live/counter_live.ex

defmodule GalleryWeb.CounterLive do
  use Phoenix.LiveView

  def mount(_session, socket) do
    {:ok, socket}
  end

  def render(assigns) do
    ~L"""
    <label>Counter: 0</label>
    <button>+</button>
    """
  end
end

and add the live route in the router in lib/gallery_web/router.ex

# lib/gallery_web/router.ex

defmodule GalleryWeb.Router do
  use GalleryWeb, :router
  ...
  scope "/", GalleryWeb do
    ...
    live "/counter", CounterLive  
  end
end

By starting the Phoenix server and visiting the /counter page, we should see on the browser just a label and a button.

When a client connects to the CounterLive view, mount/2 is the first callback to be invoked. It's used to setup the view and prepare the data needed for the first render.

Then, the render/1 callback is invoked. In this function we use the ~L sigil to define an inline LiveView template. For small templates I find really convenient to have everything in the same module, but in case of larger templates we can move it to a separate html.leex file.

LiveView templates (LiveEEx) are similar to Phoenix EEx templates, except that they track the changes of the dynamic parts minimizing the data sent to the client.

Great, we have a working live view... but at the moment there are no dynamic parts. The counter <label> has just a static 0 and the <button> does nothing when clicked.

Page request and WebSocket connection

Before going ahead making the view dynamic, let's take a moment to see in detail the first part of the LiveView's life-cycle.

Let's log a string, along with the process id, whenever mount/2 or render/1 are invoked.

# CounterLive
# lib/gallery_web/live/counter_live.ex

require Logger

def mount(_session, socket) do
    Logger.info("MOUNT #{inspect(self())}")
    {:ok, socket}
end

def render(assings) do
    Logger.info("RENDER #{inspect(self())}")    
  ~L"""
    ...
  """
end

After refreshing the page just one time, we see in the logs that the mount/2 and render/1 are called two times with different *pid*s.

What's happening here?

The browser, to get the http://localhost:4000/counter page, sends an HTTP GET request to the server. The server invokes then mount/2 and render/1 to render the view, sending back the full HTML page.

Inside the HTML, we can see our view which is embedded in a <div> container that has some special data attributes (like data-phx-session, data-phx-view...). These attributes will then be used by the LiveView JavaScript library to start a stateful view.

<div data-phx-session="..." data-phx-view="CounterLive" id="phx--1wl284P">
  <label>Counter: 0</label>
  <button>+</button>
</div>

By answering to a HTTP GET request with a fully rendered page, we can support clients that do not necessarily run JavaScript, which makes LiveView also great for SEO.

When the browser receives and loads the page, it also loads the JavaScript we wrote during the setup in assets/js/app.js

...
let liveSocket = new LiveSocket("/live", Socket)
liveSocket.connect()

and connects to the server with a websocket. The Counter LiveView process will then track the changes, pushing the updated dynamic values to the client.

Once the browser and the server are connected via websocket, mount/2 is invoked again to setup the data, and render/1 to re-render the view.
This second time, only the view's HTML is sent back to the browser

Make it dynamic with assign/3

Let's now make the counter value, in the <label> tag, dynamic. In the mount/2 callback, we assign/3 a counter value to the socket

def mount(_session, socket) do
    socket = assign(socket, :counter, 0)
    {:ok, socket}
end

and in the template, in render/1, we change the fixed number with <%= @counter %>.

def render(assigns) do
    ~L"""
    <label>Counter: <%= @counter %></label>
    <button>+</button>
    """
end

After refreshing the page we get obviously the same result, but by changing the :counter value we should see that this change is reflected on the browser as well

Phoenix bindings, `handle_event/3` and `update/3`

The easiest way to add some user interaction is with the help of Phoenix Bindings which are special attributes to add to HTML elements in the LiveView template. In this example, we use the click event binding, adding phx-click="event name" attribute to the button element.

<button phx-click="incr">+</button>

When we click the + button, the browser sends via the websocket a "incr" event to the CounterLive view process running on the server.

Let's try it on the browser, by refreshing and clicking the + button.

We see a red background because the CounterLive process is crashed. Looking at the logs on the terminal, is clear that we need to implement the handle_event/3 callback.

defmodule CounterLive do
    def handle_event("incr", _event, socket) do
        ...
    end
end

The first argument matches the "incr" string in phx-click, the second gives details about the event and the third is the socket.

In handle_event/3 we want to increment the counter, but how can we access to the current counter value?

Let's inspect the socket when the mount/2 callback is invoked

def mount(_session, socket) do
    socket =
      socket
      |> assign(:counter, 0)
      |> IO.inspect()
    {:ok, socket}
end

Refreshing the page, we see on the terminal the Phoenix.LiveView.Socket struct.

#Phoenix.LiveView.Socket<

  assigns: %{counter: 0},
  changed: %{counter: true},

  endpoint: GalleryWeb.Endpoint,
  id: "phx-rf-5Qdcj",
  parent_pid: nil,
  view: GalleryWeb.CounterLive,
  ...
>

The assign/3 function sets the :counter value in the assigns map and flags the value as changed in the changed map.

So, to increment the counter, we could assign a new value socket.assigns.counter + 1

socket = assign(socket, :counter, socket.assigns.counter + 1)

and it would work. But when updating a value, I prefer to use the update/3 where we pass a function &(&1 + 1) that increments the counter by 1.

def handle_event("incr", _event, socket) do
    socket = update(socket, :counter, &(&1 + 1))
    {:noreply, socket}
end

The handle_event/3 callback then returns the new socket in the tuple {:noreply, socket}.

This is everything we need to do - we don't need to update the front-end elements our self - all this is taken over by LiveView!

Let's refresh and try the counter! By clicking + we now see on the browser that the counter value increases 🎉

How LiveView updates the counter on the browser

It really feels like magic: just incrementing the counter in handle_event/3, LiveView takes care of all the rest, from propagating the changes to updating the <label> tag.

How does LiveView manage these updates under the hood?

Browser and server are connected via a websocket connection. When we click the + button an "incr" event is sent to the CounterLive process.

We can see, using the browser inspector under network, the websocket connections and the messages exchanged with the server.

The LiveView process receives the "incr" event and it invokes our handle_event/3 implementation, updating the counter.

Every time we update or assign a value, LiveView re-renders the view, sending back only the updated dynamic values.

What the server pushes down to the client is just the updated counter value, with some metadata. It means that if we have a much larger template, LiveView (thanks to LiveEEx) tracks the changes happened to the template's dynamic parts and sends just the updated values to the browser.

When the browser receives updates from the server, it uses the patrick-steele-idem/morphdom JavaScript library to patch the DOM.
Using the inspector we see that only the content in the <label> tag is updated.

What's next

In this article we've seen the mount/2, render/1, handle_event/3 callbacks and LiveView's life-cycle, inspecting the messages and understanding a bit of the magic behind LiveView.

We have now all the elements we need to build, in the next article, our gallery app! 👩‍💻👨‍💻

Subscribe to get FREE Tutorials by email. Start learning Elixir and Phoenix to build features and apps!

How to Setup Phoenix LiveView

Alvise Susmel — Tue, 29 Oct 2019 17:18:48 +0000

Understanding Phoenix LiveView

Setup (this article)
The Primitives
Building a Gallery app

Subscribe to get FREE Tutorials by email. Start learning Elixir and Phoenix to build features and apps!

Phoenix LiveView has been released on Hex! Although it's still not a stable release (at the time of writing 0.3.1), it works really well and it's packed of cool features.

Since the first version, I'm so impressed with how easy and fast I can develop a real-time application with LiveView, without using a complex JavaScript framework.

But to me, the most important LiveView's aspect is the productivity boost! Be able to focus on where data reside and are processed while being sure that changes are pushed to the front-end, updating the UI automatically. It really feels like magic!

In this article we see how to get started with Phoenix LiveView by creating a new Phoenix app and going through the LiveView setup.

In the next articles we’ll see how to use this setup to build a super-simple LiveView application called Gallery, which we’ll use to develop our first LiveView page and understand how LiveView works under the hood.

In this app the user can interact with the LiveView page and go through a gallery of images by clicking the interface buttons.

Create a new Phoenix project

Let's first create a new Phoenix project. We need the latest stable Phoenix Framework and Node.js installed.

$ mix phx.new gallery --no-ecto

Since we don't need any database, we use the --no-ecto option. Our gallery will just be a list of image urls.

Now, how can we setup LiveView?
At the moment there are no LiveView generators, so we need to manually integrate it, changing some files and macros.

`:phoenix_live_view` in `mix.exs`

Let's start by adding the phoenix_live_view library under the mix.exs dependencies. We can set the release version we find on Hex

defp deps do
[
  {:phoenix, "~> 1.4.10"},
    ...
  {:phoenix_live_view, "~> 0.3.1"}
]
end

or if you prefer to try the latest committed code, you can directly use the GitHub LiveView master branch

defp deps do
[
  {:phoenix, "~> 1.4.10"},
    ...
  {:phoenix_live_view, github: "phoenixframework/phoenix_live_view"}
]
end

In our case we set the 0.3.1 version and get the dependencies running mix deps.get in the terminal

$ mix deps.get
...
New:
  phoenix_live_view 0.3.1
* Getting phoenix_live_view (Hex package)

Signing salt in `config/config.exs`

Now we need to update the endpoint configuration in config/config.exs, adding the LiveView signing salt.

To generate a new salt we use the mix task

$ mix phx.gen.secret 32
LIlgBfJ9j7xLJ6Almy982/ZydK/9y0vd

# config/config.exs

config :gallery, GalleryWeb.Endpoint,
    ...
  live_view: [
    signing_salt: "LIlgBfJ9j7xLJ6Almy982/ZydK/9y0vd"
  ]

LiveView Flash plug

Then, we add the LiveView Flash plug to our :browser pipeline in lib/gallery_web/router.ex, just after :fetch_flash

# lib/gallery_web/router.ex
pipeline :browser do
    ...
    plug :fetch_flash
    plug Phoenix.LiveView.Flash
    ...
end

Imports in `lib/gallery_web.ex`

We now update the lib/gallery_web.ex web file, adding some imports to the controller, view and router functions

# lib/gallery_web.ex

def controller do
    quote do
        ... 
      import Phoenix.LiveView.Controller
    end
end

def view do
  quote do
    ...
    import Phoenix.LiveView,
      only: [live_render: 2, live_render: 3, live_link: 1, live_link: 2]
  end
end

def router do
  quote do
    ...
    import Phoenix.LiveView.Router
  end
end

Socket in `lib/gallery_web/endpoint.ex`

Next, we expose a new socket for LiveView updates in the endpoint module lib/gallery_web/endpoint.ex. This socket is used by LiveView to send updates and receive events.

defmodule GalleryWeb.Endpoint do
  use Phoenix.Endpoint, otp_app: :gallery

  socket "/live", Phoenix.LiveView.Socket
  ...
end

JavaScript and CSS

Now let's focus on the client side - we need to add the LiveView JavaScript library in assets/package.json.

{
  "dependencies": {
        ...
    "phoenix_live_view": "file:../deps/phoenix_live_view"
  }
}

This library will run on the browser, updating the DOM, managing user events while talking with the LiveView process on the server.

To install the new javascript dependency we've just added, run

$ npm install --prefix assets

We are almost ready, we just need to add four lines of javascript code in assets/js/app.js, which starts LiveView on the client

// assets/js/app.js
import {Socket} from "phoenix"
import LiveSocket from "phoenix_live_view"

let liveSocket = new LiveSocket("/live", Socket)
liveSocket.connect()

and import the default LiveView CSS in assets/css/app.css

/* assets/css/app.css */

@import "../../deps/phoenix_live_view/assets/css/live_view.css";

Done! Let's see if everything works

Great, the setup is finished and LiveView is ready! Let's see now if everything works by creating a super-simple LiveView module that renders some text.

After creating the lib/gallery_web/live directory, we open in it a new file called gallery_live.ex. In this new file we define the GalleryWeb.GalleryLive module (our LiveView page - just copy/paste the code below, we will dig into LiveView functionalities in next articles)

defmodule GalleryWeb.GalleryLive do
  use Phoenix.LiveView

  def mount(_session, socket) do
    {:ok, socket}
  end

  def render(assigns) do
    ~L"""
    <h1>LiveView is awesome!</h1>
    """
  end
end

Inside the render/1 function we have a LiveView template that will show on our browser the message LiveView is awesome!.

Almost there... we just need to add a live route in our router lib/gallery_web/router.ex

# lib/gallery_web/router.ex

defmodule GalleryWeb.Router do
  use GalleryWeb, :router

  ...

  scope "/", GalleryWeb do
    ...

      live "/gallery", GalleryLive  
  end
end

and after running the Phoenix server

$ mix phx.server

[info] Running GalleryWeb.Endpoint with cowboy 2.7.0 at 0.0.0.0:4000 (http)
[info] Access GalleryWeb.Endpoint at http://localhost:4000
...

opening the page http://localhost:4000/gallery we should see our LiveView page.

What's next

In the following article, the Primitives of Phoenix LiveView we learn the LiveView basics, exploring the Phoenix LiveView primitives, understanding the magic behind LiveView while learning how we can build a simple counter.

Take also a look at the Phoenix LiveView documentation. It's a great read which gives a solid understanding of how things work and which kind of features are available.

Another great resource is the chrismccord/phoenix_live_view_example GitHub repo, where you find easy code examples showing all the different LiveView's features.

Subscribe to get FREE Tutorials by email. Start learning Elixir and Phoenix to build features and apps!

Phoenix LiveView live_link

Alvise Susmel — Wed, 26 Jun 2019 14:45:03 +0000

Subscribe to get FREE Tutorials by email. Start learning Elixir and Phoenix to build features and apps!

In the previous article focused on live_redirect, we've seen how in Phoenix LiveView we can change the URL without changing the location or refreshing the page.

In this way we can keep the URL updated with the current page state, making easier for the user to bookmark or share it.

We are going to see how to refactor the code with live_link, making the code simpler and reducing the number of messages exchanged between our browser and LiveView.

Pictures example with `live_redirect/2`

Let's consider the LiveView Pictures page example, and see how, in this case, is convenient to use live_link/2 instead of live-redirect/2.

In this example we have a list of thumbnails. When we click on a thumbnail, LiveView changes the URL and updates the page showing the full-size image.


defmodule DemoWeb.PicturesLive do

  ...

  def render(assigns) do

    ~L"""

    ...



    <%= for {id, pic} <- pictures do %>



    <div class="column" 

         phx-click="show" 

         phx-value="<%= id %>">

      <%= pic.author %>

      <img src="<%= picture_url(pic.img, :thumb) %>">

    </div>



    <% end %>



    ...

    """

  end

end

Each thumbnail element has phx-click="show" and phx-value="<%= id %>" attributes. In this way when we click a thumbnail the "show" event is sent to the server along with the picture id.


def handle_event("show", id, socket) do

  {:noreply, live_redirect(socket, to: Routes.live_path(socket, DemoWeb.PicturesLive, id))}

end

The LiveView process handles this event with handle_event("show", id, socket) function, sending a live_redirect message back to the browser.

Inspect LiveView messages

Let's see better what happens, checking the messages exchanged between the browser and Phoenix.

I'm using the Chrome inspector, but with other browsers should be similar.

We open the inspector, we select the network tab and clicking WS we show only the WebSocket connections. Refreshing the page we now see the WebSocket connection used by Phoenix LiveView and clicking on it we can see the messages exchanged between the browser and LiveView.

Clicking on a thumbnail we see four messages


[...,"event",{"type": "click", "event": "show", "value":"XesILKdmkwM"}]



[...,"live_redirect",{"kind": "push", "to": "/pictures/XesILKdmkwM"}]

When clicking a thumbnail, the browser sends a "show" event message to LiveView.
Phoenix sends back a live_redirect message with the new URL and the LiveView front-end javascript library changes the URL to the new one using history.pushState().


[...,"link",{"url":"/pictures/XesILKdmkwM"}]



[...,"phx_reply",{"response":{"diff": {...}}}]

After changing the URL, the browser sends a message back to the server with the new URL
LiveView processes this message with handle_params/3, renders the new page and sends back a response with the differences to apply on the page.

This back and forth of messages is redundant, we can avoid the first two messages (click and live_redirect) with live_link.

live_link

By using live_link we don't need to handle the click event ourself. We can then remove the two attributes, phx-click and phx-values, along with the handle_event("show", id, socket) function.


defmodule DemoWeb.PicturesLive do



  def render(assigns) do

    ... 

    <%= for {id, pic} <- pictures do %>



      <%= live_link to: Routes.live_path(@socket, DemoWeb.PicturesLive, id) do %>



        <div class="column">

          <%= pic.author %>

          <img src="<%= picture_url(pic.img, :thumb) %>">

        </div>



      <% end %>

    <% end %>



        ...

    end

end

We wrap our thumbnail tag with live_link and use the same Routes.live_path/3 function as we did with live_redirect.

Since we are inside a LiveView template, we just need to use @socket instead of socket.

We see how in this way the code becomes simpler and using the browser inspector, we also see that for each click we just have two messages.


[...,"link",{"url":"http://localhost:4000/pictures/XesILKdmkwM"}]



[...,"phx_reply",{"response":{"diff": {...}}}]

With live_link, the browser changes immediately the URL without sending any click event (like we did before). Once changed, the new URL is sent to the server which re-renders the page and sends back a message with the differences to apply.

When using `live_link` and when `live_redirect`?

So, when to use live_link and when live_redirect?

In the Pictures example, the user interacts with the frontend clicking the links. It's clear that live_link is the right choice - it makes the code simpler and we get less exchanged messages.

In the animated URL example though, there is no user interaction (apart from starting/stopping the animation). Every 100 milliseconds the server tells the browser to change the URL, so in this live_redirect is the way to go.

Subscribe to get FREE Tutorials by email. Start learning Elixir and Phoenix to build features and apps!

Phoenix LiveView pushState support – Change the URL without refreshing the page

Alvise Susmel — Wed, 19 Jun 2019 16:11:17 +0000

Subscribe to get FREE Tutorials by email. Start learning Elixir and Phoenix to build features and apps!

It's now possible to try the pushState support available in Phoenix LiveView (which, remember, is still in beta)

What is pushState and why can be useful?

With Phoenix LiveView we can easily update parts of the page without having to change the location, refreshing the page or directly use JavaScript.

But when the page's content changes, the URL remains the same, making it difficult for a user to bookmark or share the current page state.

The LiveView pushstate support solves exactly this problem and, while changing the page content, we can dynamically update the URL without a full page refresh.

The reason why it's called pushState is because it refers to the history.pushState function in the HTML5 History API, which gives us the ability to change the URL and push a new page in the browser's history.

In this article we are going to test this functionality in Phoenix LiveView, with two different examples:

The first is a LiveView which shows picture thumbnails taken from unsplash.com. When we click on a thumbnail, the full picture is shown in the same page. We'll see how to use the LiveView pushState support to update the URL, making easy to share one specific picture.

In the second example we'll see something different, an animated URL with emojis. Maybe something we're not going to use in a real app, but something fun to build.

You can find these examples in the poeticoding/phoenix_live_view_example GitHub repo, which is a fork of the original chrismccord/phoenix_live_view_example.

If you haven't tried LiveView yet, subscribe to the newsletter to be updated with new Elixir and Phoenix content. New LiveView introductory content is coming soon!

LiveView Pictures page

In this example we build a simple LiveView page where we show a list of pictures thumbnails taken from Unsplash. When we click on the thumbnail, the full picture is shown in the page and the URL is updated to something that uniquely refers to that specific picture.

First, we add the live route in lib/demo_web/router.ex

defmodule DemoWeb.Router do

  scope "/", DemoWeb do

    live "/pictures", PicturesLive

  end
end

and then we create the file lib/demo_web/live/pictures_live.ex file where we define the new DemoWeb.PicturesLive LiveView module.

defmodule DemoWeb.PicturesLive do
  use Phoenix.LiveView
  alias DemoWeb.Router.Helpers, as: Routes

  @pictures %{
    "ySMOWp3oBZk" => %{
        author: "Ludomił", 
        img: "https://images.unsplash.com/photo-..."
      },
    ...
  }

  def render(assigns) do
    pictures = @pictures
    ~L"""
    <div class="row">
    <%= for {id, pic} <- pictures do %>
      <div class="column" 
       phx-click="show" phx-value="<%= id %>">
        <%= pic.author %>
        <img src="<%= picture_url(pic.img, :thumb) %>">
      </div>
    <% end %>
    </div>

    <%= if @selected_picture do %>
      <hr>
      <center>
      <label><%= @selected_picture.author %></label>
      <img src="<%= picture_url(@selected_picture.img, :big) %>">
      </center>
    <% end %>
    """
  end

  def mount(_session, socket) do
    socket = assign(socket, :selected_picture, nil)
    {:ok, socket}
  end

  def handle_event("show", id, socket) do
    picture = @pictures[id]
    {:noreply, assign(socket, :selected_picture, picture)}
  end

  defp picture_url(img, :thumb),
    do: "#{img}?w=250fit=crop"
  defp picture_url(img, :big),
    do: "#{img}?w=800&h=500fit=crop"

end

For simplicity we use a Map @pictures, where the keys are the picture IDs and the values are a Maps with :img URL and :author name.

At the bottom we find a multi-clause function picture_url/2, which we use to get the thumbnail and large image URL by appending w, h and fit parameters to the img URL string.

In the render/1 function we loop through the pictures Map, showing the thumbnails and making them clickable.

def render(assigns) do
  pictures = @pictures
  ~L"""
  <div class="row">
  <%= for {id, pic} <- pictures do %>
    <div class="column" 
    phx-click="show" phx-value="<%= id %>">
     <%= pic.author %>
     <img src="<%= picture_url(pic.img, :thumb) %>">
    </div>
  <% end %>
  ...

Since pictures is a map, in the generator we pattern match both the key and value {key, value} <- map.

For each picture we show its thumbnail using the picture_url(pic.img, :thumb) function, which returns the thumbnail URL.

With phx-click="show" we make the div with author name and image clickable. It means that when the user clicks the element, a "show" event is sent to the LiveView process, along with the phx-value value.

This event is handled by the handle_event function.

def handle_event("show", id, socket) do
  picture = @pictures[id]
  {:noreply, assign(socket, :selected_picture, picture)}
end

The id argument is the value passes with phx-value HTML attribute. In this function we use the id to get the picture map and assign it to :selected_picture.

After clicking the picture thumbnail, LiveView re-renders the view. This time the @selected_picture is bound to a picture map (which initially is set to nil in the mount function), so LiveView renders the HTML with the :big image.

It works, but as we saw before, it doesn't change the URL, making it difficult to share the page state.

live_redirect to change the URL using pushState

Let's see now how to change the URL without refreshing the page.
We start by adding a new route in lib/demo_web/router.ex

defmodule DemoWeb.Router do

  scope "/", DemoWeb do

    live "/pictures", PicturesLive
        live "/pictures/:id", PicturesLive

  end
end

The new route with :id is triggered by passing the picture id in the URL - the case where a picture is selected.

Back to our PictureLive module, we now have to add the handle_params/3 function.

def handle_params(%{"id" => id}=_params, _uri, socket) do
  picture = @pictures[id]
  {:noreply, assign(socket, :selected_picture, picture)}
end

In this function we do exactly what we were doing when handling the show event. The handle_params is invoked just after mount/2, when the user loads the page, and when changing the URL with live_link/2 and live_redirect/2.

To handle the /pictures route, we add below another handle_params clause, where we set the :selected_picture to nil.

def handle_params(%{"id" => _uri, socket) do
 ...
end

# catchall
def handle_params(_, _uri, socket) do
  {:noreply, assign(socket, :selected_picture, nil)}
end

We now update the handle_event/3 function

def handle_event("show", id, socket) do
{:noreply, 
  live_redirect(
    socket, 
      to: Routes.live_path(socket, DemoWeb.PicturesLive, id)
  )
}
end

Instead of assigning the picture (like we did before), we use live_redirect/2 which changes the URL to /pictures/:id (using pushState).
handle_params(%{"id" => id}, _uri, _socket) is then called, using the id to assign the selected_picture.

Animated URL 🌝

Let's see now how to create an animation in the address bar, with LiveView and emoji.

As we did for the previous example, we add the two routes lib/demo_web/router.ex

defmodule DemoWeb.Router do

  scope "/", DemoWeb do

    live "/moon", MoonLive
    live "/moon/:moon", MoonLive

  end
end

And we define the MoonLive module in lib/demo_web/live/moon_live.ex

defmodule DemoWeb.MoonLive do
  use Phoenix.LiveView
  alias DemoWeb.Router.Helpers, as: Routes

  @moons ["🌑", "🌒", "🌓", "🌔", "🌝", "🌖", "🌗", "🌘"]
  @moons_count Enum.count(@moons)

  def render(assigns) do
    ~L"""
    <button phx-click="start">start</button>
    <button phx-click="stop">stop</button>
    """
  end

  def mount(_session, socket) do
    {:ok, socket}
  end

  def handle_params(_, _uri, socket) do
    {:noreply, socket}
  end

  def handle_event("start", _, socket) do
    socket =
      socket
      |> assign(:moon_idx, 0)
      |> assign(:running, true)
    Process.send_after(self(), "next_moon", 100)
    {:noreply, socket}
  end

  def handle_event("stop", _, socket) do
    {:noreply, assign(socket, :running, false)}
  end

  def handle_info("next_moon", socket) do
    idx = rem(socket.assigns.moon_idx, @moons_count)
    moon = Enum.at(@moons, idx)

    socket = assign(socket, :moon_idx, idx + 1)

    if socket.assigns.running, 
    do: Process.send_after(self(), "next_moon", 100)

    {:noreply, 
    live_redirect(socket, 
    to: Routes.live_path(socket, DemoWeb.MoonLive, moon),
    replace: true)}
  end

end

@moons is a list with 8 frames ["🌑", "🌒", "🌓", "🌔", "🌝", "🌖", "🌗", "🌘"] we are going to use in the animation, each one is an emoji.

render/1
The render is pretty simple: we just have two buttons, start and stop. Each one sends and event to the LiveView process.

handle_event("start", _, socket)
Clicking the start button, we send a start event to the LiveView process. This event is handled by handle_event("start", _, socket) which sets :running to true and initialize the :moon_idx to 0. We will use this :moon_idx to know at which frame of the @moons list we are.

With Process.send_after(self(), "next_moon", 100) we send a delayed message (100ms) to the current LiveView process (self()), starting the animation.

handle_info("next_moon", socket)
The next_moon message is processed by handle_info("next_moon", socket).

In this function we get the right frame

idx = rem(socket.assigns.moon_idx, @moons_count)
moon = Enum.at(@moons, idx)

We increase the :moon_idx

socket = assign(socket, :moon_idx, idx + 1)

We check if :running is still true and we continue the animation sending another delayed "next_moon" message, which will be handled by the same function

if socket.assigns.running, 
  do: Process.send_after(self(), "next_moon", 100)

And like we did in the previous example, we use live_redirect/2 to update the URL. We pass moon which is the string with the emoji we want to show on the address bar

{:noreply, 
 live_redirect(socket, 
   to: Routes.live_path(socket, DemoWeb.MoonLive, moon), 
   replace: true)
}

With the replace: true option we change the current url without polluting the browser's history.

handle_event("stop", _, socket)
Clicking the Stop button, we send a stop event and the handle_event("stop", _, socket) function sets :running to false, stopping the animation.

def handle_event("stop", _, socket) do
  {:noreply, assign(socket, :running, false)}
end

Add a Progress Bar in the Phoenix File Upload app

Alvise Susmel — Wed, 29 May 2019 12:22:38 +0000

Part 1: Step-by-Step Tutorial to Build a Phoenix App that Supports User Upload
Part 2: Creating Thumbnails of uploaded Images and PDF in Phoenix
Part 3: this article

In the previous two parts we've seen how to create a Phoenix application with a multipart upload form, that creates thumbnails of images and PDF.

We select a file to upload and once uploaded we see it in the upload list with a thumbnail (if image or PDF). Since we are using it in localhost the upload is fast, even with large files.

But in a real world scenario the upload could take minutes (or even hours). We have to show a proper upload interface with a nice process bar.

To better follow this article, you can download the part-2 code at poeticoding/phoenix_uploads_articles:part-2.
Once we have started a Postgres server and installed Imagemagick (for thumbnails), we have an upload form that looks like this

Simulate slow connection with Chrome

With Chrome is possible to simulate a real case internet connection using throttling. You just need to open the Inspector, open the Network conditions panel

and choose the throttling option you want, like Slow 3G. You can even add your custom throttling profile.

If you try to upload a file bigger than one megabyte, you'll see that the interface is stuck, you wait ... without knowing for how long.

At least the browser shows, on the bottom left, a small bar displaying the progress, but obviously it's not enough for us and our app.

To make a nice upload page we need to start playing a little bit with JavaScript and CSS.

jQuery in Phoenix

There are great libraries out there that could do everything for us, like for example DropzoneJS. But in this article I want to show you how, with JavaScript and the jQuery, we can take control of upload form events and build our progress bar.

So let's start by adding jQuery in our Phoenix app. The easiest way is to add jquery under dependencies in assets/package.json

"dependencies": {
    ...
    "jquery": "^3.4.1"
}

I've started recently to use Visual Studio Code (before I was using Sublime Text), which supports Elixir really well thanks to ElixirLS.

It has also a nice feature I've noticed while editing the package.json configuration file: when adding jquery VS Code lists the packages available on npmjs and once selected jquery it suggests the latest stable version. it's a simple feature but a really helpful one.

After adding this new dependency we run npm install in the assets/ directory, installing the missing packages which will be saved in the assets/node_modules folder.

Now, we create a new javascript file, assets/js/upload.js and we import it into assets/js/app.js. In this new upload file we will write our JavaScript code that refers to the upload form.

// assets/js/app.js

import "phoenix_html"

import "./upload"

In this way webpack, which starts automatically in development with mix phx.server, includes the upload.js script in the app.js file served by Phoenix.

<body>
  ...
   <script type="text/javascript" src="/js/app.js"></script>
</body>

Upload a file using jQuery and Ajax

Let's now see how to submit the multipart form using jQuery, so we can monitor the upload's progress with JavaScript.

At first we give an id to the upload form to be able to easily refer to it with a jQuery selector. We find the form in lib/poetic_web/templates/upload/new.html.eex

<%= form_for @conn, Routes.upload_path(@conn, :create),
[multipart: true, id: "upload_form"], fn f-> %>
...
<% end %>

And then we focus on the upload.js file.

// assets/js/upload.js

import jQuery from "jquery"

jQuery(document).ready(function($){
    let $form = $("#upload_form");

    $form.submit(function(event){
        let formData = new FormData(this);
        startUpload(formData, $form);

        event.preventDefault();
    })
})

We start by importing jQuery and when the document is loaded, we catch the form submit event by passing a handler to the submit function.

Since we want to upload the file ourself using jQuery

we create a new FormData, which we'll use later
we pass formData to the startUpload function we are going to implement in a second
we also pass the $form jQuery object to startUpload - it will be useful later
and stop the form submission event with event.preventDefault(), so we can submit it ourself in the startUpload function.

Let's see now the startUpload function

// assets/js/upload.js

function startUpload(formData, $form) {
    jQuery.ajax({
        type: 'POST',
        url: '/uploads',

        data: formData,
        processData: false, //IMPORTANT!

        cache: false,
        contentType: false,

        xhr: function () {
            let xhr = jQuery.ajaxSettings.xhr();
            if (xhr.upload) {

                xhr.upload.addEventListener(
                    'progress', 
                    handleProgressEvent, 
                        false
                      );

            }
            return xhr;
        },

        success: function (data) {
            console.log("SUCCESS", data)
        },

        error: function (data) {
            console.error(data);
        }
    })
}

We use the the ajax function to submit the form, making a POST request to /uploads path, sending formData to the server. To avoid any data transformation from jQuery it's important to set processData to false.

The xhr parameter expects a callback function which creates and returns a XMLHttpRequest (XHR) object, used to make an HTTP request to the server. We pass a callback function which creates xhr, a XMLHttpRequest object, and with xhr.upload.addEventListener(...) we start listening to progress events, which are handled by handleProgressEvent function.

Almost there... last part before we are able to test this out. Let's write a handleProgressEvent function that just prints the event.

// assets/js/upload.js
function handleProgressEvent(progressEvent) {
    console.log(progressEvent);
}

Let's see what it's printed when we upload a file. Remember to enable the throttling - with a slow connection you can see many more events printed on the console.

Calculate and show the progress

The ProgressEvent object passed to handleProgressEvent() has everything we need to calculate the upload progress percentage.

ProgressEvent {
    total: 3698228,
    loaded: 49152
    ...
}

total is the total file size (in byte) and loaded is the current uploaded size.

Let's start with something simple showing in the upload page an HTML label with the progress percentage.

First, we need to add a label in the upload form in new.html.eex file

new.html.eex *

<%= form_for @conn, Routes.upload_path(@conn, :create),
 [multipart: true, id: "upload_form"], fn f-> %>

    <%= file_input f, :upload, class: "form-control" %>
    <%= submit "Upload", class: "btn btn-primary" %>

  <div class="upload-progress">
    <p>Upload progress: 
        <label class="progress-percentage">0%</label>
    </p>
  </div>

<% end %>

Instead of using the selector "#upload_form label.progress-percentage" directly inside the handleProgressEvent(e) function would create a tight coupling between the label and the handler.

We define a new function called createProgressHandler($form) which accepts the form jQuery object and returns a handler function

// assets/js/upload.js

function createProgressHandler($form) {
  let $label = $form.find("label.progress-percentage");

  return function handleProgressEvent(progressEvent) {
    let progress = progressEvent.loaded / progressEvent.total,
        percentage = progress * 100,
        percentageStr = `${percentage.toFixed(2)}%`;

        $label.text(percentageStr);

  }
}

In this way the handler function has access to $label and it's able to update its text. The handler calculates the progress and updates the label text with the percentageStr string.

To make it work we need to also change a line in the startUpload function.

// assets/js/upload.js

function startUpload(formData, $form) {
    jQuery.ajax({
                ...
        xhr: function () {
            ...
                xhr.upload.addEventListener(
                        'progress', 
                createProgressHandler($form), 
                false);
        }
    ...

Instead of passing the handler function directly to addEventListener, we call createProgressHandler($form) which returns the handler function that will be called for each progress event.

Let's try again to upload a file.

🎉👩‍💻👨‍💻🎉

Great, it works. It's not aesthetically pleasant, but at least it shows dynamically the upload's progress.

You've maybe noticed that once reached 100%, the success callback is called printing the server response to the JavaScript console. This response is the upload list page HTML (GET /uploads). In our case, since we've receive the response via jQuery success callback, the browser isn't redirected to /uploads and we just see a page with the progress stuck at 100%.

In general, it would be better to have an API that sends us a JSON response with the details of the created file - we could then show this data to confirm that the upload succeeded. For simplicity, in the case of success, we just ignore the data and redirect the browser to the uploads page.

// assets/js/upload.js

function startUpload(formData, $form) {
  ...
  jQuery.ajax({
    ...
    success: function (data) {
      window.location = "/uploads"
    },
    ...
  })
}

HTML5 progress bar

It's now time to try to nicely show the progress with a progress bar. We can start using the progress HTML5 tag, without having to import any library.

<%= form_for @conn, Routes.upload_path(@conn, :create),
 [multipart: true, id: "upload_form"], fn f-> %>

    <%= file_input f, :upload, class: "form-control" %>
    <%= submit "Upload", class: "btn btn-primary" %>

    <div class="upload-progress">
        <progress max="100" value="0"></progress>
        <label class="progress-percentage"></label>
    </div>

<% end %>

By default, the result is a thin blue bar.

I'm neither a front-end developer nor a CSS expert, but we can get a nicer progress bar just playing around with the bar's CSS. We create a new /assets/css/upload.css file adding the CSS below

/* assets/css/upload.css */

progress {
    position:relative;
    width: 100%;
    height: 25px;
    appearance: none;
    -webkit-appearance: none;
}

progress::-webkit-progress-bar {
    background-color: #eee;
    border-radius: 8px;
}

progress::-webkit-progress-value {
    background-color: #276bd1;
    border-radius: 8px;
}

We are able to customize size and colors using the progress element itself and some CSS progress bar pseudo-elements (this code works on Chrome and Safari, to support Firefox and other browsers we'd need to add some extra CSS).

This progress bar doesn't have an attribute to easily show a label at the center. But we can move our current label label.progress-percentage at the center of the bar.

/* assets/css/upload.css */
.upload-progress {
    position: relative;
}

label.progress-percentage {
    position: absolute;
    top: 0;
    text-align: center;
    width: 100%;
    color: white;
    font-weight: bold;
    text-shadow: 1px 1px 1px #444;
}

Similarly to what we did for upload.js, we import it on assets/css/app.css

/* assets/css/app.css */

@import "./phoenix.css";

@import "./upload.css";

Adding the CSS

progress {display: none; }

hides the bar by default. We want to show the bar only when the upload starts. When the startUpload(...) JavaScript function is called, we show the progress bar.

// assets/js/upload.js

function startUpload(formData, $form) {
    let $progress = $form.find("progress");
    $progress.show()
        ...
}

We now need to amend the createProgressHandler and handleProgressEvent functions so they can update both the progress bar and the label.

// assets/js/upload.js

function createProgressHandler($form) {
    let $progress = $form.find("progress"),
        $label = $form.find("label.progress-percentage");

    return function handleProgressEvent(progressEvent) {
        let progress = progressEvent.loaded/progressEvent.total,
            percentage = progress * 100,
            percentageStr = `${percentage.toFixed(2)}%`; //xx.xx%

        $label.text(percentageStr)

        //PROGRESS BAR
        $progress
        .attr("max", progressEvent.total)
        .attr("value", progressEvent.loaded);
    }
}

As you can see, to update the progress bar we just need to set the max and value attributes, which are respectively total size of the file and current uploaded bytes.

Move the upload form

At the moment the upload form is in its own page /uploads/new. To have everything in the same place, we can move the form into the upload list page.

Instead of copying & pasting the code inside the upload list page, we remove the action :new from the routes and rename the template file new.html.eex to upload_form.html.eex.

We can now render the form into templates/upload/index.html.eex, using the PoeticWeb.UploadView.render function and passing the connection

<%= PoeticWeb.UploadView.render("upload_form.html", conn: @conn) %>

<table class="table">
    <thead>
        <th>Thumbnail</th>
        <th>ID</th>
        <th>Filename</th>
        <th>Size</th>
        <th>Type</th>
        <th>Time</th>
    </thead>
    <tbody>
  ...

Choose file and Upload buttons

An extra small change: it would be nice to have just one Upload button to choose the file and, once the file is selected, to automatically start the upload.

We start by changing the upload_form.html.eex file, wrapping the file_input and a button inside a div with class upload-btn-wrapper. We also remove the submit button.

<%= form_for ... %>

    <div class="upload-btn-wrapper">
        <button class="btn btn-primary">Upload a file</button>
        <%= file_input f, :upload, class: "form-control" %>
    </div>

    ...
<% end %>

We add to upload.css some CSS specific to the wrapper, overlaying the file input with the button.

.upload-btn-wrapper {
  position: relative;
  overflow: hidden;
  display: inline-block;
}
.upload-btn-wrapper input[type=file] {
  width: 100%;
  height: 100%;
  position: absolute;
  left: 0;
  top: 0;
  opacity: 0;
}

And we add in upload.js the JavaScript code to automatically start the upload once the file is selected.

// assets/js/upload.js

jQuery(document).ready(function ($) {
    let $form = $("#upload_form"),
        $fileInput = $form.find("input[type='file']");

    $form.submit(function (event) { ... }

    $fileInput.on("change", function (e) {
        $form.trigger("submit");
    });
});

Wrap up

If you want to try the code of this part, you find it on the GitHub repo poeticoding/phoenix_uploads_articles:part-3_progress-bar.

In this article I preferred to use only jQuery so we could interact ourself with the upload JavaScript events and understand the dynamics. But when building an application we can't reinvent the wheel every time. There are a lot of great JavaScript libraries out there that can make our life easier.

If you want to bring the progress bar a step further, give a try to progressbar.js which is a JavaScript library that displays beautiful progress bars with different shapes, colors and animations. To use it, you just need to include it in the dependencies in package.json and import it in upload.js, like we did with jQuery.

jQuery File Upload is a pretty famous JavaScript library (it has more than 30,000 stars on GitHub) which handles the upload and progress bar. It's really well documented and it's still maintained.

A library I mentioned at the beginning is DropzoneJS. I haven't played a lot with it yet, but it seems a great full-optional library. It creates a drop-zone box in the page where we can drag & drop our files - DropzoneJS will take care of the rest sending the file to the server and showing a nice UI with progress bar and thumbnails. With this library we can easily set the maximum file size, choose the supported file types, generate thumbnails on the client side and many other things.

Creating Thumbnails of uploaded Images and PDF in Phoenix

Alvise Susmel — Wed, 15 May 2019 18:28:04 +0000

In Step-by-Step Tutorial to Build a Phoenix App that Supports User Upload we've built from scratch a Phoenix application where users can upload and download their files. We saw how to deal with uploads, saving them locally, using Ecto and Postgres database to save file's details like size, filename, hash etc.

We are now going to add a new layer, where we create thumbnails for image and PDF files. Let's see quickly how to get the previous part's code up and running, so we can start building from it.

$ git clone git@github.com:poeticoding/phoenix_uploads_articles.git
$ cd phoenix_uploads_articles 
# in phoenix_uploads_articles directory
$ git checkout tags/part-1

To be on the same page, once cloned the repository, let's checkout to the commit with tag part-1, which is the situation at the end of previous article.

We then need to get the app dependencies and build the javascript and css assets.

$ mix deps.get && mix deps.compile
$ cd assets && npm install && \
  node node_modules/webpack/bin/webpack.js \
  --mode development

We need a Postgres server up and running, where we save the uploaded files details. In Postgres using Docker you can see in detail how to run a Postgres server with Docker on your local machine.

Once the the Postgres server is ready, we can create the database and uploads table using the Ecto mix tasks.

$ mix ecto.create
Generated poetic app
The database for Poetic.Repo has been created

$ mix ecto.migrate
[info] == Running 20190412141226 Poetic.Repo.Migrations.CreateUploads.change/0 forward
[info] create table uploads
[info] create index uploads_hash_index
[info] == Migrated 20190412141226 in 0.0s

One last thing - at the bottom of config/dev.exs you find the uploads_directory option set to my local directory /Users/alvise/uploads_dev. It's important you create a folder where to store the uploads (better outside the project's directory) and set the absolute path in the configuration file. You could also set it using an environment variable (which is the best way in production).

We are now ready to run the Phoenix server and upload a file.

$ mix phx.server
[info] Running PoeticWeb.Endpoint with cowboy 2.6.3 at 0.0.0.0:4000 (http)
[info] Access PoeticWeb.Endpoint at http://localhost:4000
...

ImageMagick

To create images' and PDF's thumbnails, we need to install ImageMagick. If you never used ImageMagick before, it's one of the most famous and used tools to process images programmatically. We will use it to read any image format and convert it to a resized jpeg that we can display in the uploads page.

On my mac I've installed it using Homebrew, this is definitely the easiest way to install it on macOS.

$ brew install imagemagick
$ magick --version
Version: ImageMagick 7.0.8-44 Q16 x86_64 ...
...

If you are on linux, it depends which distribution you are using. You can install it using the binaries available on the ImageMagick website, Unix Binary Release. If you are using Debian or Ubuntu you can use apt-get

# Ubuntu 18.04
$ apt-get install imagemagick
$ mogrify -version
Version: ImageMagick 6.9.7-4 Q16 x86_64 20170114
...

Ubuntu 18.04, for example, has a slightly older ImageMagick version, which is not an issue for what we need to do.

ImageMagick 6.9 doesn't have the magick command like in the Mac installation. To check the version you can run the mogrify command, which is one of the ImageMagick tools and the one we'll use to create thumbnails.

Create thumbnails with Mogrify in Elixir

Mogrify is a wrapper around the ImageMagick mogrify command, which brings a great Elixir-like experience on processing images. We add it in our mix.exs under deps

$ mix deps.get
Resolving Hex dependencies...
Dependency resolution completed:
...
  mogrify 0.7.2
...

The only Mogrify's dependency is ImageMagick, which we've already installed.

You can download the phoenix_uploads.png image and save it into the assets/static/images/ folder, so you can use it do some resizing experiments in the Elixir's interactive shell iex.

$ iex -S mix
iex> Mogrify.open("assets/static/images/phoenix_uploads.png") \
...> |> Mogrify.resize_to_limit("300x300") \
...> |> Mogrify.save(path: "phoenix_uploads_thumb.jpg")
%Mogrify.Image{
  animated: false,
  buffer: nil,
  dirty: %{},
  ext: ".jpg",
  format: nil,
  frame_count: 1,
  height: nil,
  operations: [],
  path: "phoenix_uploads_thumb.jpg",
  width: nil
}

with resize_to_limit("300x300") the image is resized maintaining proportions, limiting the width and hight to a maximum of 300 pixels.
with Mogrify.save(path: "phoenix_uploads_thumb.jpg") we start the processing. The library at this point runs the ImageMagick mogrify command. With the :path option we specify the path where we want to store the file, and using a different extension (like .jpg in this case) the image is converted to the new image format.

We find our resized image in the root directory of the project. We can use Mogrify to check width and hight of this new image.

iex> ls
...
phoenix_uploads_thumb.jpg

iex> Mogrify.open("phoenix_uploads_thumb.jpg") \
...> Mogrify.verbose()
%Mogrify.Image{
  ...
  ext: ".jpg",
  format: "jpeg",
  height: 199,
  width: 300
  path: ...
}

`thumbnail?` field in Upload Ecto Schema

The user should be able to upload any file format, but we can create thumbnails just for images and PDF. We need then to have a new field in Upload schema reflecting a boolean column in our database, where we store if the upload has a thumbnail or not.

We are going to alter the current uploads database table, adding a new has_thumb boolean column. To do so, we generate a new Ecto migration

$ mix ecto.gen.migration add_uploads_has_thumb
* creating priv/repo/migrations/20190514143331_add_uploads_has_thumb.exs

The task creates a migration template file with an empty change function we need to fill.

# priv/repo/migrations/20190514143331_add_uploads_has_thumb.exs
defmodule Poetic.Repo.Migrations.AddUploadsHasThumb do
  use Ecto.Migration

  def change do
        alter table(:uploads) do
          add :has_thumb, :boolean, default: false
    end
  end
end

In this way we alter the uploads table adding the has_thumb column. All the existing records will have this value set to false.

To apply this migration we run the migration task

$  mix ecto.migrate
[info] == Running 20190514143331 Poetic.Repo.Migrations.AddUploadsHasThumb.change/0 forward
[info] alter table uploads
[info] == Migrated 20190514143331 in 0.0s

Let's add the new thumbnail? field in the uploads schema defined in the Poetic.Documents.Upload module, mapping this field to the has_thumb column, using the :source option

# lib/poetic/documents/upload.ex
defmodule Poetic.Documents.Upload do
  schema "uploads" do
    ...
    field :thumbnail?, :boolean, source: :has_thumb
    ...
  end
end

Upload module and changeset

After adding the thumbnail? field to the schema, it's now time to write the functions where we create the thumbnail and update the upload database record.

Let's start from the Upload.changeset/2 function first. As it is, if we try to change the thumbnail? value, nothing happens

iex> alias Poetic.Documents.Upload
iex> Upload.changeset(%Upload{},%{thumbnail?: true})
#Ecto.Changeset<
  action: nil,
  changes: %{},
  ...
>

We need to add the :thumbnail? atom to the valid fields passed to cast

# lib/poetic/documents/upload.ex
def changeset(upload, attrs) do
  upload
  |> cast(attrs, [:filename, :size, :content_type, :hash, :thumbnail?])
  ...
end

# on iex
iex> Upload.changeset(%Upload{}, %{thumbnail?: true})
#Ecto.Changeset<
  action: nil,
  changes: %{thumbnail?: true},
  ...
>

Thumbnails path

Now, where do we store our thumbnails?

We could store them in priv/static/images directory of the project, in this way we would use the Plug.Static plug that serves static assets. Just copying the the image into the priv/static/images directory, the thumbnail could be retrieved at the URL http://localhost:4000/images/thumbnail_filename.jpg. In this way is really easy to serve new images. Plug.Static also uses etags for HTTP caching.

But we are going with a different approach though, saving the thumbnail into the same directory where we store the uploaded files. In this way we put everything together and we can have a better control on serving these thumbnails based on user's permission.

Let's now define a thumbnail_path/1 function that returns a valid and unique absolute path for our thumbnail

# lib/poetic/documents/upload.ex
def thumbnail_path(id) do
  [@upload_directory, "thumb-#{id}.jpg"]
  |> Path.join()
end

As we did for the local_path/2, we use the id to uniquely identify the thumbnail. In general it's a good idea to have a dedicated directory for each single upload - in this way we can have the freedom to group multiple files that refer to the same upload, without overcrowding the uploads directory. Since this adds a bit of complexity (for example creating a directory for each upload), we save the thumbnails directly in the @upload_directory.

Image's thumbnail

To generate thumbnails we use a Mogrify code similar to what we've tried before on iex

# lib/poetic/documents/upload.ex
 def mogrify_thumbnail(src_path, dst_path) do
   try do
     Mogrify.open(src_path)
     |> Mogrify.resize_to_limit("300x300")
     |> Mogrify.save(path: dst_path)
   rescue
     File.Error -> {:error, :invalid_src_path}
     error -> {:error, error}
   else
     _image -> {:ok, dst_path}
   end
 end

In general I prefer to avoid the try/rescue construct to handle errors, it's usually better to pattern match {:error, _} {:ok, _} tuples.

The Mogrify.open/1 function raises a File.Error when the source image file doesn't exist, so we need to use try/rescue to properly handle the error and return a tuple.

At the moment, with the current Mogrify (version 0.7.2) , Mogrify.save/2 doesn't return or raise an error if it fails, which is not something that can be left to fate. We will see later, when creating thumbnails for PDF files, how to run ImageMagick commands in Elixir handling open and save errors.

Using pattern matching we can define a create_thumbnail/1 function for each content_type we want. We start with image content type to support different image formats (image/png, image/jpg etc.)

# lib/poetic/documents/upload.ex

def create_thumbnail(%__MODULE__{
  content_type: "image/" <> _img_type
}=upload) do

  original_path = local_path(upload.id, upload.filename)
  thumb_path = thumbnail_path(upload.id)
  {:ok, _} = mogrify_thumbnail(original_path, thumb_path)

  changeset(upload, %{thumbnail?: true})
end

def create_thumbnail(%__MODULE__{}=upload), 
    do: changeset(upload, %{})

The function arguments is an %Upload{} struct, with content_type starting with image/. We use the id and filename fields to get the original file path and thumbnail_path/1 function to get the destination path.

For simplicity, we expect that mogrify_thumbnail/2 returns a {:ok, _} tuple - when it returns {:error, _} a MatchError is raised. If unhandled, it can lead to a HTTP 500 Internal Server Error error message that is shown to the user. There are many ways of handling these type of errors: we could silently fail and set a generic thumbnail, or return an Ecto Changeset with an error on the thumbnail? field. At the end it depends on how we want our app to behave in these situations.

In the last line of the function we convert the upload struct to an Ecto changeset, setting the thumbnail? value to true. This function doesn't apply any change to the upload's record - to persist this change we will pass the changeset to Repo.update/1 function.

The second create_thumbnail/1 it's a catchall clause. It handles the cases where the file is not an image. It just returns a changeset with no changes.

Documents context

Now that we have everything we need to create our thumbnail, we just need to decide in which part of the code we want to generate it.

Let's start with the easiest way, adding a new step inside the Documents.create_upload_from_plug_upload/1 function. We can append a new step to the with in the transaction.

# lib/poetic/documents.ex
defmodule Poetic.Documents do

  def create_upload_from_plug_upload(%Plug.Upload{...}) do
    ...

    with ...,
     {:ok, upload} <- 
         ... |> Repo.insert(),
     ...
     {:ok, upload} <- 
         Upload.create_thumbnail(upload) |> Repo.update() 
    do
      ...
    else
      ...
    end

    ...
  end
end

If we now run our app and upload an image, we still don't see any thumbnail shown in the uploads page - but looking inside the uploads folder, we see that the thumbnail has been successfully created during the upload process.

Serve and show the thumbnails

We've just seen that the Documents context successfully creates thumbnails for image files, but these aren't shown in the uploads page. Since thumbnail images are in the uploads directory, to serve them we need first to add a route in the router and an action in the PoeticWeb.UploadController.

# lib/poetic_web/router.ex
...
scope "/", PoeticWeb do
  resources "/uploads", UploadController, only: [:index, :new, :create, :show] do
    get "/thumbnail", UploadController, :thumbnail, as: "thumbnail"
  end
end

$ mix phx.routes
...
upload_thumbnail_path  GET  /uploads/:upload_id/thumbnail  PoeticWeb.UploadController :thumbnail
...

Inside the uploads resources, we define a get route which points to the UploadController.thumbnail action.

With mix phx.routes we see that the new route is listed and the upload_thumbnail_path helper is created. Notice that the key used to the pass the id in the parameters is now upload_id.

In the UploadController we write the thumbnail/2 function

# lib/poetic_web/controllers/upload_controller.ex
def thumbnail(conn, %{"upload_id" => id}) do
  thumb_path = Upload.thumbnail_path(id)

  conn
  |> put_resp_content_type("image/jpeg")
  |> send_file(200, thumb_path)
end

We pattern match the upload_id and get the thumbnail path. We then set the content type to image/jpeg and send the file to the client. We can see immediately if it works, uploading a file and making a GET request with the browser

Let's now change the lib/poetic_web/templates/upload/index.html.eex file to show the thumbnail in the list

<%= for upload <- @uploads do %>

<tr>
    <td>
        <%= if upload.thumbnail? do 
            img_tag Routes.upload_thumbnail_path(@conn, :thumbnail, upload.id)
          else
            img_tag "/images/generic_thumbnail.jpg"
          end 
        %>
    </td>

    <td>...</td>

</tr>
<% end %>

If the upload has a thumbnail, we show the thumbnail using img_tag and the upload_thumbnail_path helper (which returns the path to get the thumbnail for the given upload id).
in the case it hasn't a thumbnail, we show a generic thumbnail placed in assets/static/images.

PDF's thumbnail

Until now, we've just focused on thumbnail of images. In the case of PDF files, we can create a thumbnail of the first page.

The best tool to create an image from a page of a PDF is the ImageMagick's convert command, which isn't covered by the Mogrify wrapper.

So we need to build our Elixir function that runs the convert command, based on System.cmd/3.

First, let's see how we create a PDF's thumbnail

# on linux/mac/unix
$ convert -density 300 -resize 300x300 'programming-phoenix.pdf[0]' 'pdf_thumb.jpg'

The [0] after the pdf path is used to choose which page we want to convert, starting from 0.

If convert returns an error like convert: no images defined it could be related to ghostscript 9.26, which is used to process PDFs. To temporarily fix this issue, waiting for the next version, I've downgraded ghostscript to version 9.25.

$ brew uninstall ghostscript
$ brew install https://raw.githubusercontent.com/Homebrew/homebrew-core/b0b6b9495113b0ac5325e07d5427dc89170f690f/Formula/ghostscript.rb

On Windows, from ImageMagick 7, the command is a little different. We need to prepend the magick

> magick.exe convert ...

Fantastic, so we add now a new function to the Upload module

# lib/poetic/documents/upload.ex
def pdf_thumbnail(pdf_path, thumb_path) do
  args = ["-density", "300", "-resize", 
          "300x300","#{pdf_path}[0]", 
          thumb_path]

  case System.cmd("convert", args, stderr_to_stdout: true) do
    {_, 0} -> {:ok, thumb_path}  
    {reason, _} -> {:error, reason}
  end
end

The arguments are passed as a list and we use the stderr_to_stdout option to catch anything printed by the command.

in the case convert exits with 0, it means the thumbnail was created correctly.
for any other exit code, we return {:error, reason} where reason is the command's output string

If we try to make it fail, providing an invalid PDF path, we get an {:error, reason} tuple as expected.

iex> Poetic.Documents.Upload.pdf_thumbnail("invalid.pdf", "thumb.jpg")
{:error,
 "convert: unable to open image 'invalid.pdf': No such file or directory @ error/blob.c/OpenBlob/3497.\nconvert: no images defined `thumb.jpg' @ error/convert.c/ConvertImageCommand/3300.\n"}

Since we are catching all the output on stdout and stderr, the error string could be a bit too verbose. We can process the string, for example getting just the first line until the @ symbol.

Same for an invalid output path

iex> Poetic.Documents.Upload.pdf_thumbnail("programming-phoenix.pdf", "/tmp/invalid/thumb.jpg")
{:error,
 "convert: unable to open image '/tmp/invalid/thumb.jpg': No such file or directory @ error/blob.c/OpenBlob/3497.\n"}

We can now use this function to extend the support to the application/pdf content type.

# lib/poetic/documents/upload.ex
def create_thumbnail(%__MODULE__{
  content_type: "application/pdf"
}=upload) do
    original_path = local_path(upload.id, upload.filename)
    thumb_path = thumbnail_path(upload.id)
    {:ok, _} = pdf_thumbnail(original_path, thumb_path)
    changeset(upload, %{thumbnail?: true})
end

We see that once added this new create_thumbnail clause into Upload module, the PDF's thumbnails are created and shown successfully

Dealing with long-running HTTP Requests and Timeouts in Phoenix

Alvise Susmel — Wed, 01 May 2019 18:36:58 +0000

Interested in learning about Elixir, Phoenix, and software architectures? Subscribe to my newsletter for general musings and weekly in-depth how to's.

Phoenix is fast and highly concurrent and it can process HTTP requests in less than a millisecond. It is our priority to serve the requests as fast as possible, but the reality is that sometimes processing a request can take too long. This forces Phoenix to trigger a timeout and close the connection.

Generally our Phoenix action doesn't just render a page; it deals with databases, remote resources, external services, API etc. while processing the request, and these steps can take time.

We've seen in the Step-by-Step Tutorial to Build a Phoenix App that Supports User Upload how to deal with uploaded files, saving them locally.

The UploadController.create action is simple, but it does a series of things: it calculates the file's hash, copies the file into a local directory and saves some info into the database. Each of these steps can take time, especially if the file is large or the database is under heavy load.

Usually, the files are saved in cloud storage (like AWS S3). This means we also need to consider the time it takes to upload the file (sometimes large) to the cloud storage service.

Simulate a Long-Running HTTP Request

To easily simulate a long-running request, we can use Process.sleep/1 in the action that is created by default with a new Phoenix project, PageController.index/2.

# lib/poetic_web/controllers/page_controller.ex

defmodule PoeticWeb.PageController do
  use PoeticWeb, :controller

  def index(conn, _params) do
        Process.sleep 65_000

      render(conn, "index.html")
  end

end

We force the action's process to sleep for 65 seconds before rendering the index page. We then make an HTTP request with curl, triggering the PageController.index/2 action.

$ curl -v localhost:4000

* Connected to localhost (127.0.0.1) port 4000 (#0)
> GET / HTTP/1.1
> Host: localhost:4000
> User-Agent: curl/7.54.0
> Accept: */*
>
* Empty reply from server
* Connection #0 to host localhost left intact
curl: (52) Empty reply from server

After 60 seconds, curl raises and error: Empty reply from server.

Phoenix doesn't log any timeout error, which makes timeouts harder to spot, the only thing we notice is that the usual [info] Sent 2xx in .. is missing.

[info] GET /
[debug] Processing with PoeticWeb.PageController.index/2
  Parameters: %{}
  Pipelines: [:browser]

The reason Phoenix doesn’t log a timeout error is that it does not handle this type of timeout. Cowboy - the HTTP server shipped with Phoenix - takes care of timeouts.

How can we solve this?

Phoenix, by default, gives us 60 seconds to process an HTTP request and respond back. And let's be honest... that's PLENTY of time in most cases.

But what can we do if we need more than one minute to process the request? (if we need, for example, to upload a really large file to AWS S3)

We can use a job queue like rihanna, which is a great choice, especially if we are sure that our task will take a number of minutes. We immediately return a status response to the client, deferring the actual processing to some point in future. However, this entails further work: we need a Postgres database (which we will have to monitor and maintain) and we need to implement some sort of polling mechanism on the client side to check that the job has finished successfully.
We can take advantage of awesome Phoenix channels, which make easy to have a persistent connection with the client, so we can receive the request and push the result back upon completion.
In some cases, it's fine (and easier) to just slightly increase the timeout. Thanks to the BEAM (Erlang virtual machine), Phoenix is highly concurrent and can easily handle thousands (or even millions) of active connections, so a long-running request shouldn't have a significant effect on other requests.

Cowboy timeout

We focus on the latter and simpler solution. Phoenix is based on Plug, which uses Cowboy as its default web server. It's Cowboy that actually handles timeouts, closing the connections.

Looking at the dependencies, in the mix.exs file, we can see plug_cowboy, the Plug adapter for Cowboy web server.

# mix.exs
defp deps do
  [
    ...
    {:plug_cowboy, "~> 2.0"}
  ]
end

From the version 1.4, Phoenix uses Cowboy 2 which has a new set of timeout options:

request_timeout (default 5_000 milliseconds), Time in ms with no requests before Cowboy closes the connection.
This is the maximum time, in which the client has to send the HTTP request. We do not touch this.
inactivity_timeout (default 300_000 milliseconds), Time in ms with nothing received at all before Cowboy closes the connection.
idle_timeout (default 60_000 milliseconds), Time in ms with no data received before Cowboy closes the connection.

The connection in the previous example was closed by idle_timeout. Once the client sends the HTTP request, Cowboy waits for 60 seconds to receive data from the client. Since the client is waiting for our response, after 60 seconds the web server closes the connection.

To slightly increase the timeout, we need to change the Endpoint configuration in the phoenix config file of the environment config/{dev,prod,test}.exs.

#config/dev.exs
config :poetic, PoeticWeb.Endpoint,
  http: [
    port: 4000,

    protocol_options: [
        idle_timeout: 70_000
    ]
  ],
  ...

Adding protocol_options: [idle_timeout: 70_000] we increase the timeout from 60 to 70 seconds, enough to be able to render the index page in our example.

After restarting Phoenix, we see that this time the server, after idling for 60 seconds, answers correctly and logs the processing time.

[debug] Processing with PoeticWeb.PageController.index/2
  Parameters: %{}
  Pipelines: [:browser]
[info] Sent 200 in 65009ms

Linked Process and Task

Having a timeout helps to clean stale connections and release retained resources.

Sometimes, to serve an HTTP request, we need to take advantage of concurrency, spawning processes to get remote resources, connect to the database or call an external API.

It's important to spawn processes linked to the connection process. This ensures that when a connection is closed due to a timeout, all the spawn processes are terminated.

If we just need to process some data concurrently, as we've seen in The Primitives of Elixir Concurrency, it's usually better to use Elixir Task module instead of using directly primitives like spawn_link and receive.

Task.async spawns a process linked to the caller's process, so if the request times out, the task is terminated automatically.

# lib/poetic_web/controllers/page_controller.ex

defmodule PoeticWeb.PageController do
  use PoeticWeb, :controller

  def index(conn, _params) do
        task1 = Task.async(fn ->
      # long running task 1
      IO.inspect(self(), label: "PID task1")
      Process.sleep 80_000
    end)

    task2 = Task.async(fn ->
      # long running task 2
      IO.inspect(self(), label: "PID task2")
      Process.sleep 75_000
    end)

    Task.await(task1, :infinity)
    Task.await(task2, :infinity)

      render(conn, "index.html")
  end

end

In this example we start two tasks and then await for the result. Using the Erlang observer we can see the two tasks sleeping.

Since both tasks take longer than the default idle_timeout of 60 seconds, Cowboy closes the connection. We see that once the connection's process is terminated, all the linked processes are terminated including the two tasks.

Interested in learning about Elixir, Phoenix, and software architectures? Subscribe to my newsletter for general musings and weekly in-depth how to's.

Step-by-Step Tutorial to Build a Phoenix App that Supports User Uploads

Alvise Susmel — Wed, 17 Apr 2019 15:00:23 +0000

Interested in learning about Elixir, Phoenix, and software architectures? Subscribe to my newsletter for general musings and weekly in-depth how to's.

There are many ways to let users upload their files to our Phoenix app, including reliable libraries that manage the uploads for us, saving the files securely on cloud storage.

However, in this tutorial we will focus on the foundation - we will only use the tools shipped with Phoenix, rather than third-party libraries.

We will create a fully-functional Phoenix app from scratch, running and configuring our Postgres database and Ecto schemas, handling uploads with Plug and storing the files locally.

Once we've done this, we'll able to download the files.

Now, let’s get to coding! 👩‍💻👨‍💻

New Phoenix app

Let's start by creating a new Phoenix 1.4 application called poetic

$ mix phx.new poetic && cd poetic

Once created, make sure the dependencies are correctly downloaded

$ mix deps.get
Resolving Hex dependencies...

We need Node.js in our system so we can build the assets inside the assets directory. I find nvm is pretty useful and easy to use if you need to handle different node versions on the same machine.

If you just want to compile these assets, you can download the installer of the latest version from the Node.js website. Once you've installed the latest version of Node.js, you can build the assets using the npm install and webpack.js commands

$ cd assets && npm install && node node_modules/webpack/bin/webpack.js --mode development

Postgres 🐘 using Docker 🐳

The Phoenix app we've just created comes with Ecto, which helps us to deal with relational databases like PostgreSQL. A database is necessary to store details about the uploaded files.

I think that, in this case, using Docker is the easiest and fastest way to get a PostgreSQL database up and running on our local system.

If you're not sure how to use Docker, you can take a look at How to run a Docker Container resource, to gain an initial understanding of how to run docker containers.

First, we create a docker volume, to keep the database data persistent

$ docker volume create poetic-postgres`

With a volume we can safely destroy and recreate a container without incurring any data loss.

We then launch a container running the official Postgres docker image, version 11-alpine.

$ docker container run --name postgres -p 5432:5432 -e POSTGRES_PASSWORD=postgres -v poetic-postgres:/var/lib/postgresql/data --rm postgres:11-alpine
LOG:  listening on IPv4 address "0.0.0.0", port 5432
LOG:  database system is ready to accept connections

We've started a Postgres server with these options:

-p 5432:5432 It means --publish port_local_machine:port_container. The port in the local machine is opened and the connection is forwarded to the container port.
-e POSTGRES_PASSWORD=postgres We pass the environment variable POSTGRES_PASSWORD to set the password postgres to the default postgres user.
-v poetic-postgres:/var/lib/postgresql/data We mount the docker volume we've created before to the /var/lib/postgresql/data directory, which is the directory Postgres uses to store its data.
--name postgres The name we give to the container which we can use later to refer to it.
--rm Once stopped, the container is removed automatically while the volume is preserved.

To remove the container, given it was launched with the --rm option, we simply need to stop it and it will be removed automatically. We can press CTRL+C in the terminal where we started postgres. Alternatively, using another terminal we can stop the container with the command:

$ docker container stop postgres

When the container is removed the volume is not deleted.

We can use the same docker container run ... command we used before to start a new postgres container, using the same volume and this will restore the old data.

Configuration

Ecto configuration

Now that our Postgres server is up and running, we need to configure the Phoenix app to connect to the database.

Let's take a look at config/dev.exs, where we find our Ecto Repo configuration

# config/dev.exs
...
config :poetic, Poetic.Repo,
    username: "postgres",
    password: "postgres",
    database: "poetic_dev",
    hostname: "localhost",
    pool_size: 10

By default, this configuration matches the username, password and hostname of our server, so we don't need to touch it.

However, at the moment there is no poetic_dev database in our Postgres. To create it, let's run this ecto task on the terminal

$ mix ecto.create
The database for Poetic.Repo has been created

This creates the poetic_dev database.

If you haven't seen the Poetic.Repo module yet, it's the module we use to run queries to the database and which inherits the Ecto.Repo functions (insert, get, ...)

# lib/poetic/repo.ex
defmodule Poetic.Repo do
  use Ecto.Repo,
    otp_app: :poetic,
    adapter: Ecto.Adapters.Postgres
end

Uploads directory

Let's now add a configuration to set the absolute path of the uploads directory, outlining where we're going to locally store the uploads, in this case for the dev environment.

# config/dev.exs
config :poetic,
   uploads_directory: "/Users/alvise/uploads_dev"

I prefer to keep the uploads directory out of the Phoenix app folder, so in my case, I've created an uploads_dev folder in my home directory.

If we then want to use this app in production - and we want to take advantage of cloud storage instead of refactoring it to use S3 (which is obviously a great option!) - we could use a service like AWS EFS where we mount a network filesystem in a directory of our cloud server.

We then just need to set the production configuration to use the mounted network file-system

# config/prod.exs
config :poetic,
  uploads_directory: System.get_env("POETIC_UPLOADS_DIRECTORY") || "/uploads"

In production, like in this case, it's usually better to use environment variables to pass our settings to a configuration file.

Upload module

We could use the phx.gen.html generator which creates schema, migration, context, controller, view, test and html files for us.

However, I find the generated scaffold a bit too much for what we need. Instead, I like to use this generator to learn more about the phoenix patterns, observing how the components are supposed to be used together.

The only generator we are going to use here is to create the Ecto schema and migration. We'll manually create the remaining necessary files.

Ecto schema and migration

Once the user has uploaded a file, we want to save different information about it into our database. Let's generate schema and migration files for the upload using the phx.gen.schema task

$ mix phx.gen.schema Documents.Upload uploads filename:string size:integer content_type:string hash:string 
* creating lib/poetic/documents/upload.ex
* creating priv/repo/migrations/20190412141226_create_uploads.exs

The first parameter Documents.Upload is the schema name, which creates a Poetic.Documents.Upload module, where Documents is our context (we will see more about this later).

uploads is the table that will be created in the database.

The remaining parameters define the fields in the schema and columns in the migration file.

Let's take a look first at the migration file

# priv/repo/migrations/20190412141226_create_uploads.exs

defmodule Poetic.Repo.Migrations.CreateUploads do
  use Ecto.Migration

  def change do
    create table(:uploads) do
      add :filename, :string
      add :size, :integer
      add :content_type, :string
      add :hash, :string

      timestamps()
    end
  end
end

This file reflects the changes we want to do in our database, creating the uploads table with filename, size, content_type and hash columns. By default Ecto also creates the id column, which is the primary key.

With timestamps() Ecto adds two timestamp columns, inserted_at and updated_at, which are managed automatically for us each time we insert or update a record in the table.

Let's make a few small changes to our migration


create table(:uploads) do
    ...
    add :size, :bigint
    add :hash, :string, size: 64
    ...
end

create index(:uploads, [:hash])

The first thing we've changed is the size column's type. The integer Postgres type is an integer between -2147483648 and +2147483647. To save a value bigger than 2GB, we need a bigint - which is plenty of space to store the kind of file we need to handle in our app (up to 8192 Petabytes!).

To hash the file, the algorithm we are going to use is SHA-256, where the hexadecimal digest is a 64-character string. We've set the size of the string using the size: 64 option and created an index on this column with create index(:uploads, [:hash]), so if we need to search for files using the hash, the search will be much faster.

Now let's take a look at the Upload Ecto schema file, which is used to map the database record into an Elixir struct.

# lib/poetic/documents/upload.ex
defmodule Poetic.Documents.Upload do
  use Ecto.Schema
  import Ecto.Changeset

  schema "uploads" do
    field :content_type, :string
    field :filename, :string
    field :hash, :string
    field :size, :integer

    timestamps()
  end

end

The fields in the schema are fields of the %Poetic.Documents.Upload{} struct. They map the database type (like bigint, varchar, etc.) to an Elixir type (integer, string, etc.).

iex> alias Poetic.Documents.Upload
iex> %Upload{}
%Poetic.Documents.Upload{
  __meta__: #Ecto.Schema.Metadata<:built, "uploads">,
  content_type: nil,
  filename: nil,
  hash: nil,
  id: nil,
  inserted_at: nil,
  size: nil,
  updated_at: nil
}

In this module we also find the changeset/2 function, where we set our validation rules and the required fields.

# lib/poetic/documents/upload.ex
defmodule Poetic.Documents.Upload do
  ...
  def changeset(upload, attrs) do
    upload
    |> cast(attrs, [:filename, :size, :content_type, :hash])
    |> validate_required([:filename, :size, :content_type, :hash])

    # added validations
    |> validate_number(:size, greater_than: 0) #doesn't allow empty files
    |> validate_length(:hash, is: 64)
  end
end

We've added two other validations:

validate_number(:size, greater_than: 0) Checks that the given size is greater than 0 (no empty files).
validate_length(:hash, is: 64) Checks that the hash is a 64 characters string.

It's now time to create the uploads table by running the migration task

$ mix ecto.migrate
[info] == Running 20190412141226 Poetic.Repo.Migrations.CreateUploads.change/0 forward
[info] create table uploads
[info] == Migrated 20190412141226 in 0.0s

To see if everything was created correctly, we can use the psql client in the running postgres Docker container

$ docker container exec -it postgres psql -U postgres
psql (10.7)
Type "help" for help.

postgres=# \l
                                  List of databases
    Name     |  Owner   | Encoding |  Collate   |   Ctype    |   Access privileges
-------------+----------+----------+------------+------------+-----------------------
 poetic_dev  | postgres | UTF8     | en_US.utf8 | en_US.utf8 |
...
postgres=# \c poetic_dev
You are now connected to database "poetic_dev" as user "postgres".

postgres=# \d
 Schema |       Name        |   Type   |  Owner
--------+-------------------+----------+----------
 public | schema_migrations | table    | postgres
 public | uploads           | table    | postgres
 public | uploads_id_seq    | sequence | postgres

postgres=# \d+ uploads
    Column    |              Type              
--------------+--------------------------------
 id           | bigint                         
 filename     | character varying(255) 
 size         | integer                        
 content_type | character varying(255)         
 hash         | character varying(64)         
 inserted_at  | timestamp(0) without time zone 
 updated_at   | timestamp(0) without time zone 
 Indexes:
    "uploads_pkey" PRIMARY KEY, btree (id)
    "uploads_hash_index" btree (hash)

Then, we run iex and use Ecto and Upload to create a record

poetic$ iex -S mix

iex> alias Poetic.Repo
iex> alias Poetic.Documents.Upload
iex> %Upload{} \
...> |> Upload.changeset(%{
    filename: "image.jpg", 
    content_type: "image/jpeg",
    hash: String.duplicate("a",64), #fake hash
    size: 1_000
}) |> Repo.insert()

[debug] QUERY OK db=2.4ms decode=0.8ms queue=0.8ms
INSERT INTO "uploads" ...

{:ok, %Poetic.Documents.Upload{id: 1, ...}

Upload.changeset/2 returns an Ecto.Changeset which is passed to the Repo.insert/2 function to create the upload record in the database.

SHA-256 function

Using what we've seen in Hashing a File in Elixir, we add the sha256(chunks_enum) function in the Poetic.Documents.Upload module. This calculates and returns the SHA-256 hexadecimal digest of the given stream.

# lib/poetic/documents/upload.ex
defmodule Poetic.Documents.Upload do
    ...
    def sha256(chunks_enum) do
        chunks_enum
        |> Enum.reduce(:crypto.hash_init(:sha256),&(:crypto.hash_update(&2, &1)))
        |> :crypto.hash_final()
        |> Base.encode16()
        |> String.downcase()
    end
end

To calculate the hash of a file we can use the following method

iex> File.stream!("assets/static/images/phoenix.png", [], 2048) \
...> |> Poetic.Documents.Upload.sha256()
"07aa9b01595fe10..."

local_path

Now, we need a simple function that tells us where to store the file, once received.

# lib/poetic/documents/upload.ex
defmodule Poetic.Documents.Upload do
    @upload_directory Application.get_env(:poetic, :uploads_directory)

    def local_path(id, filename) do
        [@upload_directory, "#{id}-#{filename}"]
        |> Path.join()
    end
end

This function returns an absolute path, placing the file in the uploads directory we defined previously in config/dev.exs.

There are several ways to make the path unique; in this case we just add the id (primary key of the upload's database record) to the beginning of the filename.

In general, if we have many uploads, it's not a good idea to save them all in one directory. A better practice would be to use an upload feature, such as the creation date, to produce a directory hierarchy, for example "2019/04/15/#{id}-#{filename}".

UploadController

In the PoeticWeb.UploadController controller we are going to implement the functions to answer the following HTTP requests:

GET /uploads/new Render a page with a form to upload our file
POST /uploads Receive the upload and stores it locally
GET /uploads List the uploaded files
GET /uploads/:id Download a file

We start by changing our PoeticWeb.Router router, using resources to match the HTTP requests to the controller actions

# lib/poetic_web/router.ex
defmodule PoeticWeb.Router do
    ...

  scope "/", PoeticWeb do
    pipe_through :browser

    resources "/uploads", UploadController, only: [:index, :new, :create, :show]
  end
end

Using the phx.routes mix task we can list all the routes

$ mix phx.routes
...
upload_path  GET   /uploads         PoeticWeb.UploadController :index
upload_path  GET   /uploads/new     PoeticWeb.UploadController :new
upload_path  GET   /uploads/:id     PoeticWeb.UploadController :show
upload_path  POST  /uploads         PoeticWeb.UploadController :create

And now the uploads routes should be listed.

:new action

To start writing the UploadController we must create a new file lib/poetic_web/controllers/upload_controller.ex in which we define the PoeticWeb.UploadController module

# lib/poetic_web/controllers/upload_controller.ex
defmodule PoeticWeb.UploadController do
  use PoeticWeb, :controller

  def new(conn, _params) do
    render(conn, "new.html")
  end
end

The new/2 function renders the new.html template, where we are going to write our upload form.

This template file has to be called new.html.eex and placed in lib/poetic_web/templates/upload (where upload is the lowercase name of the controller).

<%= form_for @conn, Routes.upload_path(@conn, :create), [multipart: true], fn f-> %>

    <%= file_input f, :upload, class: "form-control" %>
    <%= submit "Upload", class: "btn btn-primary" %>

<% end %>

This EEx template creates a multipart form where the user can choose the file to send. When the user submits the form, it will make a POST /uploads request (action returned by Routes.upload_path(@conn, :create)).

To render the template, we need a view for this controller. Phoenix assumes a strong naming convention from controllers, to views, to the templates they render. The UploadController requires a UploadView to render templates in the lib/poetic_web/templates/upload directory.

defmodule PoeticWeb.UploadView do
  use PoeticWeb, :view
end

Before testing our code, let's first add the create/2 function to the controller and inspect what the form sends to the action

defmodule PoeticWeb.UploadController do
    ...

  def create(conn, %{"upload" => %Plug.Upload{}=upload}) do
        IO.inspect(upload, label: "UPLOAD")
        text conn, "ok"
  end

end

We expect an upload key in the params, with a %Plug.Upload{} struct which holds our uploaded file.

Plug manages the upload for us, saving the data in a temporary file that we find in the path field of the struct; when the request terminates, the temporary file is automatically deleted.

It's time to run the Phoenix server to test the upload.

To see the form, we need to go to http://localhost:4000/uploads/new

$ mix phx.server
[info] Running PoeticWeb.Endpoint with cowboy 2.6.3 at 0.0.0.0:4000 (http)
[info] Access PoeticWeb.Endpoint at http://localhost:4000

I'm going to upload the phoenix.png image found in the project's assets. Once submitted, we should see this inspection log on the terminal

%Plug.Upload{
  content_type: "image/png",
  filename: "phoenix.png",
  path: "/var/folders/lt/t88p62t91mg1mxdlmv_rlkl80000gn/T//plug-1555/multipart-1555258629-39672758309128-2"
}

There are three important fields that Plug gives us: filename, content_type and path - which is the path of the temporary file.

Documents context

Once the UploadController.create/2 function is invoked, it has to create an upload record in the database and store the file in the uploads directory.

Instead of putting all the logic into the create function, it's much better to write this logic into a context. A context is a great way to decouple the controllers from the logic of the application (like using Repo to create, or load, records).

Create upload from %Plug.Upload{}

Let's create the file lib/poetic/documents.ex with our context Poetic.Documents, which we use as the public interface to manage the uploads.

# lib/poetic/documents.ex
defmodule Poetic.Documents do
    import Ecto.Query, warn: false

    alias Poetic.Repo
    alias Poetic.Documents.Upload

    def create_upload_from_plug_upload(%Plug.Upload{
        filename: filename,
        path: tmp_path,
        content_type: content_type
    }) do

        # upload creation logic
    end

end

The actual hash calculation, database record creation and file's copy take place as part of the create_upload_from_plug_upload/1 function.

#create_upload_from_plug_upload function

hash = 
    File.stream!(tmp_path, [], 2048) 
    |> Upload.sha256()

with {:ok, %File.Stat{size: size}} <- File.stat(tmp_path),

  {:ok, upload} <- 
        %Upload{}
        |> Upload.changeset(%{
                filename: filename, content_type: content_type,
                hash: hash, size: size }) |> Repo.insert(),

    :ok <- File.cp(
                tmp_path, 
                Upload.local_path(upload.id, filename))

do 
    {:ok, upload}

else
    {:error, reason}=error -> error
end

Using pattern matching, we can easily extract the filename, content_type and path from the Plug.Upload struct passed to the function.

We first calculate the SHA-256 hash of the file, and then we use the with construct to check that all of the three passed clauses match.

In the first clause we use File.stat and pattern matching to bind the size variable to the size of the file.
If the previous clause matches, we now have everything we need to insert our upload into the database.
Once the upload is successfully created in the database, we copy the file into our uploads directory using the Upload.local_path(upload.id, filename) function.

If everything goes well, the function returns {:ok, upload}, where upload is a Poetic.Documents.Upload struct.

If one of the clauses doesn't match, there is an error and the function returns the {:error, reason} tuple.

To test this function on iex, we pass a %Plug.Upload{} struct that points to a valid file (remember to set an absolute path).

iex> alias Poetic.Documents
iex> upload = %Plug.Upload{
    filename: "phoenix.png", 
    content_type: "image/png",
    path: "/absolute_path_to/phoenix.png", 
}
iex> Documents.create_upload_from_plug_upload(upload)
{:error, :enoent}

Uh oh - something's gone wrong 🤔

enoent means No such files or directory, and in this case we've received this error because we didn't create the uploads directory (which in my case would be /Users/alvise/uploads_dev).

We can also see that it's difficult to understand to which clause the error refers. There is an effective way of managing this problem using tuple-wrapping.

But there's another problem, so let's see what we have inside the uploads table

poetic_dev=# select * from uploads;
id |  filename   | size  | ...
 5 | phoenix.png | 13900 | ...

(1 rows)

Although we had an error, the uploads record is still in the database.

This happens because the function that returns an error File.cp/2 fails after we've inserted the upload into the database.

Repo.transaction

Hopefully, we can use transactions to pack everything into an atomic operation. Doing it this way means that, if anything fails, the changes are rolled-back (i.e., if we match an error, we ask to rollback).

Repo.transaction fn ->
    with ...
    do
        upload
    else
        {:error, reason} -> 
            Repo.rollback(reason)
    end
end

If everything goes well, the Repo.transaction returns {:ok, upload} while, in the case of an error, it returns {:error, reason}.

Let's fix the error by creating the uploads directory, then try again.

iex> %Plug.Upload{
    filename: "phoenix.png", 
    content_type: "image/png",
    path: "/absolute_path_to/phoenix.png", 
} |> Document.create_upload_from_plug_upload()

{:ok,
  %Poetic.Documents.Upload{
        ...
        content_type: "image/png",
    filename: "phoenix.png",
    hash: "07aa9b01...",
    id: 7,
    size: 13900,
    inserted_at: ~N[2019-04-15 12:25:24],
    updated_at: ~N[2019-04-15 12:25:24]
  }
}

Fantastic - it worked 🎉! The upload is created in the database and the file is copied into the uploads_dev directory.

$ ls /Users/alvise/uploads_dev/
7-phoenix.png

UploadController `:create` action

We can use the Documents context we just created to implement the create/2 function in the Upload controller

# PoeticWeb.UploadController
alias Poetic.Documents

def create(conn, %{"upload" => %Plug.Upload{}=upload}) do
    case Documents.create_upload_from_plug_upload(upload) do
        {:ok, upload}->
            put_flash(conn, :info, "file uploaded correctly")
            redirect(conn, to: Routes.upload_path(conn, :index))
        {:error, reason}->
            put_flash(conn, :error, "error upload file: #{inspect(reason)}")
            render(conn, "new.html")
    end
end

We pass the upload to Documents.create_upload_from_plug_upload/1 which handles everything for us.

when it returns successfully, we set a flash info message and redirect to the index page where we list all the uploads.
when it returns an error, we set a flash error message and redirect to the new page.

UploadController - `:index` action

To list the uploads, firstly we add a simple function in the Documents context, called list_uploads. This returns all the uploads in the database.

defmodule Poetic.Documents do
    ...
    def list_uploads do
        Repo.all(Upload)
    end
end

In the UploadController, we then create the index/2 function where we get the uploads list and render the index.html template.

defmodule PoeticWeb.UploadController do
    ...
  def index(conn, _params) do
        uploads = Documents.list_uploads()
        render(conn, "index.html", uploads: uploads)
  end
end

We pass the uploads list to the rendering function and - in the template lib/poetic_web/templates/upload/index.html.eex - we loop over the list to render a table of uploads.

<table class="table">
    <thead>
        <th>ID</th>
        <th>Filename</th>
        <th>Type</th>
        <th>Time</th>
    </thead>
    <tbody>

        <%= for upload <- @uploads do %>
            <tr>
                <td><%= upload.id %></td>
                <td><%= upload.filename %></td>
                <td><%= upload.content_type %></td>
                <td><%= upload.inserted_at %></td>
            </tr>
        <% end %>

    </tbody>
</table>

Let's try it

$ mix phx.server
Running PoeticWeb.Endpoint with cowboy
Access PoeticWeb.Endpoint at http://localhost:4000
...

Download

To request a download we use the show/2 function, which we invoked using the GET /uploads/:id HTTP request.

We start amending the index.html.eex template, by adding a download link to the row of each upload.

<%= for upload <- @uploads do %>
    ...
    <td>
        <%= link "download", to: Routes.upload_path(@conn, :show, upload.id) %>
    </td>
<% end %>

Then, in the Documents context, we add the get_upload!(id) function which, when given an id returns an Upload struct

# lib/poetic/documents.ex
defmodule Poetic.Documents do
  ...
    def get_upload!(id) do
        Upload
        |> Repo.get!(id)
    end
end

Finally, let's implement the show/2 function.

Here, we use the given id to obtain the upload from the database, along with the local_path where the file is saved, and send the file using send_download.

# PoeticWeb.UploadController

def show(conn, %{"id" => id}) do
    upload = Documents.get_upload!(id)
    local_path = Upload.local_path(upload.id, upload.filename)
    send_download conn, {:file, local_path}, filename: upload.filename
end

We pass then connection to send_download and a tuple with the file's path we want to send and the filename of the file.

Going with our browser to http://localhost:4000/uploads we now see the download link we can use to download a file

Upload size limit

If we try to upload a large file such as a 100MB tiff image, we receive the following error

By default the Plug multipart parser reads a maximum of 8_000_000 bytes. To increase this limit we need to amend the lib/poetic_web/endpoint.ex file

# lib/poetic_web/endpoint.ex
plug Plug.Parsers,
    parsers: [:urlencoded, :multipart, :json],

changing :multipart with {:multipart, length: num_of_bytes}

plug Plug.Parsers,
    parsers: [:urlencoded, {:multipart, length: 500_000_000}, :json],

In this case, we've set a limit of 500MB. After restarting the server, we see that following this change, we can now upload a huge 200MB tiff image.

Wrap up

In this tutorial we saw in depth how to build a Phoenix app from scratch, letting users upload their files using a multipart form. We learnt how to easily run PostgreSQL with Docker on our machine, to receive uploads using Plug.Upload, and how to use Ecto to store uploads' details into the database.

There are many libraries out there that help you to deal with uploads and cloud storage. A great one is Arc, which integrates well with Ecto and AWS S3.

You can find all the code related to this article in the GitHub repo: poeticoding/phoenix_uploads_articles.

Interested in learning about Elixir, Phoenix, and software architectures? Subscribe to my newsletter for general musings and weekly in-depth how to's.

Hashing Files in Elixir

Alvise Susmel — Wed, 10 Apr 2019 15:47:11 +0000

Are you passionate about Elixir and Phoenix? Subscribe to the Poeticoding newsletter and join happy regular readers and receive new posts by email.

A hash function is a function that converts a variable size sequence of bytes (a string, a file content etc.) to a fixed size sequence of bytes, called digest. This means that hashing a file of any length, the hash function will always return the same unique sequence of bytes for that file. It's a sort of digital fingerprint, usually represented by an hexadecimal string of length between 32 and 128 characters.

The hash of a file is useful, for example, to understand if the content of two files is identical, or if the content was corrupted during a download.

There are different hash functions, MD5, SHA-1, SHA-2, SHA-3 etc. , many of them available in Elixir.

Update 👨‍💻: I had initially written the examples below using MD5 algorithm (which is the weakest in the list), just because I thought to be the fastest one. @Hauleth pointed out that SHA-1 and SHA-256 should be faster on new CPUs due to Intel SHA Extensions, so I rewrote the examples using SHA-256

Hashing a string

Let's start hashing a string using the SHA-256 algorithm.

iex> :crypto.hash(:sha256,"I love Elixir")
<<164, 35, 167, 235, 69, 224, 253, 77, 180, 92, 77, 172, 37,...>>
iex> :crypto.hash(:sha256,"I love Elixir!")
<<209, 119, 188, 230, 168, 124, 98, 212, 119, ...>>

We use the hash/2 function in the :crypto Erlang module.

The first argument is the name of the hash algorithm we want to use, in this case :sha256, the second argument is the sequence of bytes we want to hash, in this case a string. It returns a sequence of bytes.

We see how the output changes just by appending a "!" character.

We can use Base.encode16/1 to get the hexadecimal string representation

iex> :crypto.hash(:sha256,"I love Elixir!") \
...> |> Base.encode16() \
...> |> String.downcase()
"d177bce6a87c62d4772f404fcad2f8c2d9606c04f99942b71d7c521eb79c4c3b"

If you are on a Linux or Mac machine, you can use a command line tool like sha256sum to see that the digest corresponds

$ echo -n 'I love Elixir!' |  sha256sum
d177bce6a87c62d4772f404fcad2f8c2d9606c04f99942b71d7c521eb79c4c3b -

Hashing a file

Calculating the hash of a file is conceptually the same as calculating the hash of a string. A file is a sequence of bytes and we could use the same :crypto.hash(:sha256, file_content_binary) function. But we saw that most of the time is not a good idea to load the whole file into memory!

We can use File.stream! and a different set of functions available in :crypto to read and process a file in chunks.
Let's see first a simple example using the same string we've used before, divided into chunks

iex> [chunk_1, chunk_2] = ["I love ", "Elixir!"]
iex> hash_ref = :crypto.hash_init(:sha256)
#Reference<...36636>
iex> hash_ref = :crypto.hash_update(hash_ref, chunk_1)
#Reference<...36647>
iex> hash_ref = :crypto.hash_update(hash_ref, chunk_2)
#Reference<...36655>
iex> digest = :crypto.hash_final(hash_ref)
<<209, 119, 188, 230, 168, 124, 98, 212, 119, ...>>
iex> digest |> Base.encode16() |> String.downcase()
"d177bce6a87c62d4772f404fcad2f8c2d9606c04f99942b71d7c521eb79c4c3b"

We process the sequence in chunks getting the same result we've gotten previously.

hash_ref = :crypto.hash_init(:sha256)

File.stream!(file_path)
|> Enum.reduce(hash_ref, fn chunk, prev_ref-> 
    new_ref = :crypto.hash_update(prev_ref, chunk)
    new_ref
end)
|> :crypto.hash_final()
|> Base.encode16()
|> String.downcase()

We get a hash reference from :crypto.hash_init(:sha256), which is passed to Enum.reduce as the first accumulator.
We use Enum.reduce to read each chunk from the file and add it to the calculation. The :crypto.hash_update/2 returns a new reference which is then set as the new accumulator.
Once processed all the chunks the final reference is then piped into the :crypto.hash_final/1 function which returns the SHA-256 digest of the file.

We can write the reduce function in a nicer and more compact way

File.stream!(file_path)
|> Enum.reduce(:crypto.hash_init(:sha256),&(:crypto.hash_update(&2, &1)))
|> :crypto.hash_final()
|> Base.encode16()
|> String.downcase()

`File.stream!` chunks vs lines

By default File.stream! emits lines instead of just chunks. Emitting lines is slower than emitting chunks, I think because the stream needs to look for newlines while splitting the chunk in strings.

To force the stream to emit chunks we use File.stream!/3

iex> File.stream!(file_path, [], 2_048)
%File.Stream{
    line_or_bytes: 2048,
    modes: [:raw, :read_ahead, :binary],
    path: file_path,
    raw: true
}

setting a chunk size of 2048 bytes.

I made a quick benchmark (you can find on this gist) where we see that streaming chunks is faster and also better memory wise.

Name             ips        average  deviation         median         99th %
chunks       23.29 K       42.93 μs    ±63.44%       41.98 μs       83.98 μs
lines         9.21 K      108.54 μs    ±42.52%       93.98 μs      275.98 μs

Comparison:
chunks       23.29 K
lines         9.21 K - 2.53x slower +65.61 μs

Memory usage statistics:

Name      Memory usage
chunks         2.11 KB
lines         20.84 KB - 9.88x memory usage +18.73 KB

Wrap up

We've seen what a hash function is and how to easily calculate the hash of a file using Elixir.

In the past (unfortunately I think still in the present 😅), hash functions were used to store passwords in the database. If you need to securely handle and store passwords, please use the bcrypt_elixir library!

Are you passionate about Elixir and Phoenix? Subscribe to the Poeticoding newsletter and join happy regular readers and receive new posts by email.

Elixir Stream and large HTTP responses: processing text

Alvise Susmel — Wed, 03 Apr 2019 16:02:19 +0000

Are you passionate about Elixir and Phoenix? Subscribe to the Poeticoding newsletter and join happy regular readers and receive new posts by email.

Part 1 – Elixir Stream to process large HTTP responses on the fly
Part 2 – this article

You find the code, of this and the previous article, on poeticoding/httpstream_articles GitHub repo. The code you find at this repo is not meant for production use, is just part of the experiments of these articles.

In the last article we've put together the concepts of HTTP async response and processing large files using Elixir Streams, to be able to easily download and save a large file using our HTTPStream module.

The stream we've built around HTTPoison is able to emit the chunks of an HTTP chunked response. In this way we avoid the huge memory impact we could have saving the whole response into memory.

In the last example of the previous article, each chunk is streamed down to the pipeline's functions, saved into our local file and then garbage collected. Then, it was easy to add compression just including the StreamGzip.gzip function in the pipeline.

Text response

Previously we where treating the response just as a binary, delegating the compression to the StreamGzip library and savings the result into a file.

In this part we want to process the lines of a large text file. To make it fun and a bit more realistic, we are going to get some inspiration from the first day of the Advent of code challenge. There are also some cool videos made by José Valim, where he uses Elixir to solve the challenges of the Advent of Code.

I've generated and uploaded a 125Mb text file at this link

https://poeticoding-data.sfo2.digitaloceanspaces.com/httpstream/numbers.txt

with 30 million lines. In each line there is a random integer from -1000 to 1000

767
138
-701
98
...

We want to:

1. process this text file on the fly, while downloading it, line by line.
2. like we did in the first part, we want to avoid big memory spikes (we cannot load the full response into memory)
3. last but not least, if we want to process just the first 30 lines of text, we just want to download the first few chunks needed.

To do just some tests, you can also find a smaller version of 4.5Mb with 1 million lines

https://poeticoding-data.sfo2.digitaloceanspaces.com/httpstream/numbers_small.txt

which is the same you find in the poeticoding/httpstream_articles repo.

What we are going to build

To process an HTTP response line by line, we need something that works like

File.stream! "numbers.txt" #, [], :line

which creates a stream that opens a file and, by default, emits each line. We want to do something similar with our HTTP async response.

Instead of changing the current HTTPStream module implementation, we want to take full advantage of the Elixir Stream's composability, writing a function we just add to the pipeline, that converts the chunks to lines.

Chunks to Lines

Ok, but why can't we just use the implementation we've done already for the binary file?

The problem with the previous implementation is that the stream we've built emits chunks without distinguishing the lines. We need an Elixir Stream that emits lines instead of chunks.

Unfortunately this is not as easy at it seems: splitting chunks into lines is not enough. This could be true only in a easy and specific case:

["767\n138\n","-701\n98\n","-504\n22"]

where each element of the list is a chunk and each chunk ends with a new line. In this easy case we just need to split each chunk, filter out the empty strings, emitting the result

iex> String.split("767\n138\n")
["767","138",""]

The problem though, is that we do not have any guarantee that all the chunks will be in this form. It's actually more than likely that multiple parts of the same line will be over different chunks

["767\n13","8\n-701\n98\n-","504\n22"]

You see how in this case the first two chunks don't end with a newline character \n and the second chunk starts with the final part of the line started in the previous chunk.

Stream.transform

We are now going to implement a function HTTPStream.lines(chunks_enum) which takes a stream of chunks as an input and returns a stream of lines.

We'll then use this function as part of our pipeline in this way:

HTTPStream.get("https://.../numbers.txt")
|> HTTPStream.lines()
|> Stream.map(fn line -> ... end)
...

We don't want to use Enum.reduce since this function is greedy and it would hold all the lines in memory. Instead, we can use Stream.transform which is lazy

Stream.transform(enum, initial_accumulator, fn element, acc ->
...
{list_of_elements_to_emit, new_accumulator}
end)

The first element of the returned tuple is a list of elements we are going to emit, in our case lines.

We see from the image that the accumulator acc is the last part of the first chunk and is prepended to the second chunk.

We need a function, we've called next_lines, that splits a chunk into separate lines and returns a tuple with two elements: the lines we want to emit and the last part of the chunk.

Stream.transform(chunks_enum, "", fn chunk, prev ->
  {lines, last_part} = next_lines(chunk, prev)
  {lines, last_part}
end)

Our initial accumulator is an empty string. This empty string will be passed as prev while processing the first chunk.

Recursive implementation

We now need to write the next_lines(chunk,prev) function. We can implement it using recursion, going through each single UTF-8 character looking for newlines. Remember that to behave like File.stream! we need to preserve the newlines \n.

# https://github.com/poeticoding/httpstream_articles/blob/36bc2167b7024a990b04b28f9447fb9bc0e0310e/lib/http_stream.ex#L78

def next_lines(chunk, prev), do: next_lines(chunk,prev,[])

def next_lines(<<"\n"::utf8, rest::binary>>, prev, lines) do
next_lines(rest,"",[prev <> "\n" | lines])
end

def next_lines(<<c::utf8, rest::binary>>, prev, lines) do
  next_lines(rest,<<prev::binary, c::utf8>>,lines)
end

def next_lines(<<>>, prev, lines), do: {Enum.reverse(lines), prev}

Ok, there is a lot happening here. Let's start from the beginning

next_lines(chunk,prev\\"")
The first clause is just a helper. We pass a chunk, and the accumulator prev. The function calls next_lines/3 , passing an empty list of lines as third argument.
next_lines(<<"\n"::utf8, rest::binary>>, prev, lines)
We are pattern matching a sequence of UTF-8 characters. This function is called only when we reach a newline character. We then call recursively next_lines passing the rest of the chunk we need to process, setting the accumulator to an empty string "", passing list of lines where we've prepended the accumulated line, prev.
next_lines(<<c::utf8, rest::binary>>,prev,lines)
Since every time c is a newline the clause above is matched, in this clause c != "\n" so we just need to append it to prev and recursively call next_lines going through the rest of the chunk.
next_lines(<<>>,prev,lines)
<<>> is an empty binary and means we've reached the end of the chunk. For performance reason we've pepended the lines.
[prev <> "\n" | lines] is faster than lines ++ [prev], especially when the lines list is big. When we reach the end of our recursion, we need to reverse the lines' list.

Let's try this function on iex

# ["767\n13","8\n-701\n98\n-","504\n22"]
iex> {lines, prev} = HTTPStream.next_lines "767\n13"
{["767\n"], "13"}
iex> {lines, prev} = HTTPStream.next_lines "8\n-701\n98\n-", prev
{["138\n", "-701\n", "98\n"], "-"}
iex> {lines, prev} = HTTPStream.next_lines "504\n22", prev
{["-504\n"], "22"}
iex> prev
"22"

Perfect, exactly what we need 👍. We go through the chunks' list passing the obtained prev to the next call.

`HTTPStream.lines`

next_lines returns the same tuple we need to return in the reducer function passed to Stream.transform/3. We can then write HTTPStream.lines/1 in a nice and compact way

def lines(chunks_enum) do
chunks_enum
|> Stream.transform("",&next_lines/2)
end

Let's try it on iex

iex> ["767\n13","8\n-701\n98\n-","504\n22"] \
...> |> HTTPStream.lines() \
...> |> Enum.each(&IO.inspect/1)
"767\n"
"138\n"
"-701\n"
"98\n"
"-504\n"
:ok

Mmm 🤔 ... there is something wrong here. The last line "22" is missing.

Emitting last line

The reason why it's not emitted is because it doesn't end with a newline and it remains stuck as an accumulator (prev). We have to emit it when the stream is ended, but using Stream.transform/3 the reducer function doesn't know when the stream is going to end! (Please let me know in the comments if you know there is a way to catch the end of a stream)

A workaround we can use, to let next_lines/2 know when the stream reached the end, is to add an :end atom at the end of our chunks' stream. next_lines/2 than has to handle the case with a specific clause

def next_lines(:end,prev), do: { [prev], ""}

which emits the final line. The accumulator is set to an empty string but it could be anything at this point.

Let's try it again on iex

iex> ["767\n13","8\n-701\n98\n-","504\n22", :end] \
...> |> HTTPStream.lines() \
...> |> Enum.each(&IO.inspect/1)
"767\n"
"138\n"
"-701\n"
"98\n"
"-504\n"
"22"
:ok

Great, it works! 🎉

But now how can we easily add a :end atom at the end of our HTTP chunked response stream?

Emitting :end at the end of the streamed HTTP response

If you have an alternative way of doing this, please share it in the comments section below! 👩‍💻👨‍💻

We need to make a small but significant change to our HTTPStream.get(url) function.

# HTTPStream.get/1
def get(url) do
  Stream.resource(
    start_fun,

    # next_fun
    fn
      #first clause
%HTTPoison.AsyncResponse{id: id}=resp ->
    receive do
...
%HTTPoison.AsyncEnd{id: ^id}->
  # emitting :end
{[:end], {:end, resp}}
end

#second clause
    {:end, resp} -> 
    {:halt, resp}
end,

after_fun
  )
end

1. When we receive the %HTTPoison.AsyncEnd{} message we know that we've reached the end of the HTTP response. Instead of just halting the stream, we emit the :end and set a new accumulator {:end, resp}, where resp is the %HTTPoison.AsyncResponse{} struct.
2. After emitting :end, next_fun is called again. This time the accumulator is the one we've just set, {:end, resp}, which pattern matches the second clause of our next_fun.

Something I don't like about this change, is that now we always have to handle the final :end, especially when saving the stream into a file.

HTTPStream.get("https://.../large_image.tiff")
|> Stream.reject(&match?(:end,&1))
|> Stream.into(File.stream!("image.tiff.gz"))
|> Stream.run()

The function in second line pattern matches each chunk and filters out the :end atom.

It's maybe better to enable and disable the final :end via an option passed to HTTPStream.get(url, emit_end), like the version you see on GitHub.

Sum numbers in 30M lines remote file

Let's use what we've implemented to process a 125MB remote text file with 30M numbers, each one separated by a newline \n. While processing the lines on the fly, we sum the numbers to get the total.

"https://poeticoding-data.sfo2.digitaloceanspaces.com/httpstream/numbers.txt"
|> HTTPStream.get()
|> HTTPStream.lines()
|> Stream.map(fn line-> 
case Integer.parse(line) do
{num, _} -> num
:error -> 0
end
end)
|> Enum.sum()
|> IO.puts()

## OUTPUT
STATUS: : 200
HEADERS: : [
  {"Content-Length", "131732144"},
  {"Accept-Ranges", "bytes"},
  {"Content-Type", "text/plain"},
  ...
]

12468816

Fantastic, we got the result: 12468816! 🎉

With the Erlang Observer sometime I've seen some memory spike (still below 100Mb) and sometime the line is almost flat. I think this could be related to how big the chunks are.

In the GitHub repo you find a memory_test.exs script you can play with, to see the HTTPStream.line memory allocation with different chunks sizes. Even with a 4.5Mb file, if we exaggerate with the chunk size (like 2_000_000) we have a huge memory spike. With 2_000 the line is almost flat.

It would be great to be able to set a maximum chunks' size in the HTTPoison options, unfortunately I didn't find any option to do that.

String.split

Let's see another way of writing the HTTPStream.lines(chunks) function. In the previous implementation we've used recursion to go through each single character of the chunk and to find newlines.

If we don't need to preserve newlines, we can use String.split/2 along with Stream.transform/3.

def lines(enum) do
  enum
  |> Stream.transform("",fn 
    :end, prev -> 
      {[prev],""}
    chunk, prev ->
      [last_line | lines] = 
        String.split(prev <> chunk,"\n")
        |> Enum.reverse()
      {Enum.reverse(lines),last_line}
  end)
end

The idea is similar to what we did before. We split the chunk into lines and the last element of the list becomes the accumulator, which is concatenated to the next chunk.

See how we extract the last item of the list of lines.

lines = String.split(prev <> chunk, "\n")
[last_line | rev_lines] = Enum.reverse(lines)
{ Enum.reverse(rev_lines), last_line }

We split the concatenated string prev <> chunk obtaining a list of lines. We now need to get the last element of the list.
We reverse the list, creating a list of new elements. Now, the head of Enum.reverse(lines) is the last element of lines.
rev_lines is the list of lines we want to emit, but in the wrong order, so we emit Enum.reverse(rev_lines) and set last_line as the next accumulator.

Let's see an example

iex> chunks = ["767\n138\n-701\n98\n-5", "04\n22\n375"]
iex> [chunk | remaining_chunks] = chunks
iex> chunk
"767\n138\n-701\n98\n-5"

iex> lines = String.split(chunk,"\n")
["767", "138", "-701", "98", "-5"]
iex> [last_line | rev_lines] = Enum.reverse(lines)
["-5", "98", "-701", "138", "767"]
iex> last_line
"-5"
iex> lines_to_emit = Enum.reverse(rev_lines)
["767", "138", "-701", "98"]

#let's process another chunk
iex> [next_chunk | remaining_chunks] = remaining_chunks
iex> next_chunk
"04\n22\n375"
# we need to prepend last_line
iex> chunk = last_line <> next_chunk
"-504\n22\n375"
iex> lines = String.split(chunk,"\n")
["-504", "22", "375"]

It turns out that this implementation is much faster than the previous one. Let's see some benchmarking

Benchmark `HTTPStream.line`

If you want to run this benchmark on your computer, you find everything on poeticoding/httpstream_articles.

Let's consider the stream created by

File.stream!("numbers_small.txt")

which by default emits the lines of a file. We want to compare the speed of this function with HTTPStream.line.

Instead of using a remote file, we are going to use a smaller ~4Mb version, numbers_small.txt you can find on the GitHub repo.

We need to now find a way to simulate the stream of chunks made by HTTPStream.get.

chunk_stream = File.stream!("numbers_small.txt",[],16_000)

Passing chunks size as third argument of File.stream!/3, the stream, instead of lines, will emit chunks (in this case of 16kb).

In the script bench_lines.exs we use Benchee

# bench_lines.exs
chunk_stream = File.stream!("numbers_small.txt",[],16_000)
lines_stream = File.stream!("numbers_small.txt", [], :line)

stream_sum = fn enum ->
  enum
  |> Stream.map(fn line-> 
case Integer.parse(line) do
{num, _} -> num
:error -> 0
end
end)
|> Enum.sum()
end


Benchee.run(%{
"File.stream! :line" => fn ->
lines_stream 
|> stream_sum.()
end,
"with next_lines" => fn ->
chunk_stream
|> HTTPStream.lines(:next_lines)
|> stream_sum.()
end,
"with String.split" => fn ->
chunk_stream
|> HTTPStream.lines(:string_split)
|> stream_sum.()
end
},
  time: 10
)

$ mix run bench_lines.exs

Name                        ips        average
with String.split          3.35      298.30 ms
File.stream! :line          2.08      481.22 ms
with next_lines            1.14      875.01 ms

Comparison:
with String.split          3.35
File.stream! :line          2.08 - 1.61x slower +182.93 ms
with next_lines            1.14 - 2.93x slower +576.71 ms

The interesting thing is that the version "with String.split" is even faster than "File.stream! :line", while the first implementation we did is the slowest.

Honestly, I don't know why the version "with String.split" is the fastest one. Maybe some optimisation in the String.split/2 function? If you are interested about these details, I've opened a topic about this on the elixir forum: Streaming lines from an enum of chunks.

Reducing the chunk size from 16_000 to 2_000 we see how both "with String.split" and "with next_lines" are a bit faster

chunk_stream = File.stream!("numbers_small.txt",[],2000)

Name                        ips        average  
with String.split          3.79      263.67 ms
File.stream! :line          2.06      484.98 ms
with next_lines            1.42      706.48 ms

Comparison:
with String.split          3.79
File.stream! :line          2.06 - 1.84x slower +221.31 ms
with next_lines            1.42 - 2.68x slower +442.81 ms

With a smaller chunk all the split, reverse and concatenation operations are faster.

Sum the first 30 numbers

The stream of lines we've built is lazy, this means we can take just the first 30 lines and halt the stream, without downloading the whole HTTP response.

To take the first 30 lines we use Enum.take/2.

"https://poeticoding-data.sfo2.digitaloceanspaces.com/httpstream/numbers.txt"
|> HTTPStream.get()
|> HTTPStream.lines()
|> Stream.map(fn line-> 
case Integer.parse(line) do
{num, _} -> num
:error -> 0
end
end)

|> Enum.take(30)

|> Enum.sum()
|> IO.puts()

You find this code in sum_30_lines.exs

$ mix run sum_30_lines.exs
STATUS: : 200
HEADERS: : [
  {"Content-Length", "131732144"},
  {"Accept-Ranges", "bytes"},
  {"Content-Type", "text/plain"},
  ...
]
END_FUN

1393

It should be really quick. Once took 30 lines, the stream is halted and the HTTP connection is closed.

Are you passionate about Elixir and Phoenix? Subscribe to the Poeticoding newsletter and join happy regular readers and receive new posts by email.

DEV Community: Alvise Susmel

AWS S3 in Elixir with ExAws

Create an IAM user, configure permissions and credentials

Configure ex_aws and environment variables

Listing the buckets

put, list, get and delete

Multipart upload and large files

Multipart upload timeout

Download a large file

presigned urls and download streams - process a file on the fly

Understanding LiveView: build a Gallery app

From a counter to a gallery

A bit of refactoring: Gallery module

image_url/1

next_image_id/2 and prev_image_id/2

Refactoring GalleryLive

Thumbnails

Slideshow

What's next?

The Primitives of Phoenix LiveView

New CounterLive module and static HTML

Page request and WebSocket connection

Make it dynamic with assign/3

Phoenix bindings, handle_event/3 and update/3

How LiveView updates the counter on the browser

What's next

How to Setup Phoenix LiveView

Create a new Phoenix project

:phoenix_live_view in mix.exs

Signing salt in config/config.exs

LiveView Flash plug

Imports in lib/gallery_web.ex

Socket in lib/gallery_web/endpoint.ex

JavaScript and CSS

Done! Let's see if everything works

What's next

Phoenix LiveView live_link

Pictures example with live_redirect/2

Inspect LiveView messages

live_link

When using live_link and when live_redirect?

Phoenix LiveView pushState support – Change the URL without refreshing the page

LiveView Pictures page

live_redirect to change the URL using pushState

Animated URL 🌝

Add a Progress Bar in the Phoenix File Upload app

Simulate slow connection with Chrome

jQuery in Phoenix

Upload a file using jQuery and Ajax

Calculate and show the progress

HTML5 progress bar

Move the upload form

Choose file and Upload buttons

Wrap up

Creating Thumbnails of uploaded Images and PDF in Phoenix

ImageMagick

Create thumbnails with Mogrify in Elixir

thumbnail? field in Upload Ecto Schema

Upload module and changeset

Thumbnails path

Image's thumbnail

Documents context

Serve and show the thumbnails

PDF's thumbnail

Dealing with long-running HTTP Requests and Timeouts in Phoenix

Simulate a Long-Running HTTP Request

How can we solve this?

Cowboy timeout

Linked Process and Task

Step-by-Step Tutorial to Build a Phoenix App that Supports User Uploads

New Phoenix app

Postgres 🐘 using Docker 🐳

Configuration

Ecto configuration

Uploads directory

Upload module

Ecto schema and migration

SHA-256 function

local_path

UploadController

Configure `ex_aws` and environment variables

A bit of refactoring: `Gallery` module

`image_url/1`

`next_image_id/2` and `prev_image_id/2`

Refactoring `GalleryLive`

New `CounterLive` module and static HTML

Phoenix bindings, `handle_event/3` and `update/3`

`:phoenix_live_view` in `mix.exs`

Signing salt in `config/config.exs`

Imports in `lib/gallery_web.ex`

Socket in `lib/gallery_web/endpoint.ex`

Pictures example with `live_redirect/2`

When using `live_link` and when `live_redirect`?

`thumbnail?` field in Upload Ecto Schema

UploadController `:create` action

UploadController - `:index` action

`File.stream!` chunks vs lines

`HTTPStream.lines`

Benchmark `HTTPStream.line`