DEV Community: Michał Szajbe

Base64-encoded file uploads with Phoenix and Plug

Michał Szajbe — Sat, 02 May 2020 11:05:45 +0000

Handling external files is one of the basic tasks for many web applications. Profile pictures, attachments, resources fetched from remote services, all are examples of the above.

The most common way such files find their way into a web app is by means of an HTTP multipart request, usually performed through a web page with an HTML form and a file input tag or a client using app's HTTP API directly. Phoenix and, more specifically, Plug handles those very well, by wrapping the uploaded file information in a convenient struct:

%Plug.Upload{
  content_type: "image/jpg",
  filename: "rainbow.jpg",
  path: "/var/folders/nk/_hgjvryd2ml40g0jpyllt2080000gn/T//plug-1548/multipart-440190-911831-1"
}

Plug writes uploaded data on disk as a temporary file and provides its path along with original filename and type encapsulated in a struct. As such, it is passed to Phoenix controller with the rest of request params.

The convenience of Plug.Upload goes beyond the definition of the above struct. It also implements a GenServer that tracks and manages all file uploads during the life cycle of the process ¹. This includes generation of unique file names, creation of intermediate directories in tmp folder, creating and writing the file itself and, probably most importantly, cleanup of all the temporary files when the process terminates.

However, not all the clients will find the HTTP multipart interface suitable. Instead of reimplementing mentioned Plug.Upload functionality, it would be better to simply re-use it when handling files obtained by other means.

Let's explore some options.

Base64-encoded data

XML and JSON are just two examples of text-based formats widely used in service-to-service communication. They are easy to implement and very flexible, but are limited to text only. Neither of those formats, nor the protocols that make use of them, directly support mixing with non-text data.

The solution is to turn non-text (binary) data into plain text, that can be easily embedded in XML or JSON for transport and turned back into binary format at the destination. Base64 is probably the most common encoding scheme that helps to achieve this.

Consider the following JSON document, describing some person and embedding her picture as Base64-encoded string (truncated for readability):

{
  "name": "John Doe",
  "email": "john@example.org",
  "picture": "SGFuZGxpbmcgZXzEtMSIKfQpgYGA..."
}

In contrast to HTTP multipart requests, this time Phoenix controller will receive picture as string instead of %Plug.Upload{} struct. To plug into the above-mentioned conveniences of Plug.Upload we need to decode it and build the struct ourselves.

First, let's define a helper function that writes arbitrary binary data into a temporary file and wraps it with a %Plug.Upload{} struct ².

def binary_to_upload(binary) do
  with {:ok, path} <- Plug.Upload.random_file("upload"),
       {:ok, file} <- File.open(path, [:write, :binary]),
       :ok <- IO.binwrite(file, binary),
       :ok <- File.close(file) do
    %Plug.Upload{path: path}
  end
end

The important part is the call to Plug.Upload.random_file/1, which creates a file in tmp directory and sets up its tracking by the GenServer so that it can be later cleaned up.

Now the handling of Base64-encoded files is straightforward.

def base64_to_upload(str) do
  with {:ok, data} <- Base.decode64(str) do
    binary_to_upload(data)
  end
end

Base64 encoding is quite common, but it's not the only alternative. Let's see another one.

Data URLs

Mobile and JavaScript single page apps, even when using regular HTTP APIs, may prefer to upload files as Data URLs that include raw or Base64-encoded file contents.

The format is very simple, it consists of data: prefix, optional MIME type indicator, optional base64 token (for non-textual data) and the data itself. For example, data:text/plain;base64,SGVsbG8gd29ybGQh is a "Hello World!" string encoded as Data URL. You can try pasting it in your web browser's address bar and it should happily decode it.

The approach here is analogous to the previous example - our goal is to decode the original file and plug it into Plug.Upload.

The following code snippet uses ex_url package, which provides parser and decoder for Data URLs (note URL.Data.parse/1 function).

def data_url_to_upload(data_url) do
  with %{scheme: "data"} = uri <- URI.parse(data_url),
       %URL.Data{data: data} <- URL.Data.parse(uri) do
    binary_to_upload(data)
  end
end

Let's see it in action:

iex> upload = data_url_to_upload("data:text/plain;base64,SGVsbG8gd29ybGQh")
%Plug.Upload{
  content_type: nil,
  filename: nil,
  path: "/var/folders/nk/_hgjvryd2ml40g0jpyllt2080000gn/T//plug-1548/base64-data-1548887956-746460942629208-8"
}

iex> File.read(upload.path)
{:ok, "Hello World!"}

Looks good!

Summary

The essence of abstractions is preserving information that is relevant in a given context, and forgetting information that is irrelevant in that context.

– John V. Guttag ³

Base64-encoded files may be a special case in your app, but with little work they can be handled as easily as regular files uploaded via HTML forms.

Plug.Upload nicely abstracts away the process of handling temporary files. At the same time, it provides a convenient struct around them to work with.

With it at your disposal, irrespective of where the external files come from, what format they're in and what your application does with them, you can rely on a universal, stable interface to ease both feature development and testing.

Wherever in this article a word process is used, it refers to Erlang VM process. ↩
To keep the examples simple, I am not even trying to set content_type and filename fields in the structs. Depending on the situation they may or may not be available in the request data. ↩
Guttag, John V. (2013-01-18). Introduction to Computation and Programming Using Python (Spring 2013 ed.). Cambridge, Massachusetts: The MIT Press. ISBN 9780262519632. ↩

Understanding and fixing recompilation in Elixir projects

Michał Szajbe — Sat, 02 May 2020 10:59:59 +0000

Our goal as programmers is to deliver value by writing code. It should be efficient, maintainable, but most importantly correct.

Correctness can be checked at multiple stages of the development process, but the most immediate feedback we can get is from the compiler, test suite or the REPL.

We rarely write large chunks of code in one go, then test it as a whole, because that means unnecessary risk of wasted effort if it turns out not to work as expected. We’d rather make small, deliberate changes, then quickly test and adjust if needed before going forward.

For that however, our feedback loops must be as short as possible. We strive for fast test suites and invest time in optimizing the compilation process, because they directly affect our workflow. Slow feedback contributes to lost focus at the very least.

Consider the rate of feedback as your speed limit. ¹

When working with compiled languages, and Elixir is no exception here, we are destined to spend some time waiting for the compiler. And there may be times when even the smallest of changes result in recompilation of significant parts of the codebase. These situations quickly get annoying.

Ever experienced this?

$ touch some/project/module.ex
$ mix test

Running tests...
Compiling 791 files (.ex)

That's a long wait before tests even run! Let’s find why and how to fix this.

How compiler decides what to recompile

Elixir compiler uses lexical tracker to track references to modules, function dispatches, usage of aliases, imports and requires in the code, etc. It uses this information to build project modules' dependency graph and ultimately optimize its own work.

When module changes, the compiler finds its dependants by analyzing the dependency graph and marks them as stale. Next, dependants of these are marked as stale too. The process is repeated until the whole dependency graph is traversed and all the stale modules are identified.

Whether a stale module will be recompiled depends on the type of dependency it has to a module that "made" it stale.

Understanding module dependencies

Basically, when module A uses module B in any way, we say it depends on B.
Dependencies themselves are transitive. If A depends on B and B depends on C, then by implication A depends on C too.

There are three types of dependencies between Elixir modules.

Compile-time dependencies

Such dependecies are created when module A uses module B at compile time, for example by:

requiring module B
importing functions from B
using macros from B
delegating calls to B via defdelegate
implementing behaviour B
implementing protocol B

When B changes, A must be recompiled too.

Runtime dependencies

Runtime dependencies happen when module A interacts with B only at runtime, e.g. by calling its functions (either by fully qualified name or via an alias).

When B changes, A does not need to be recompiled.

Struct dependencies

This type of dependency is created when A uses %B{} struct.

A needs to be recompiled only when the definition of %B{} struct changes, because struct keys are checked at compile time. ²

Strategy for faster recompilation

Actually it’s not about the speed of the compiler, but the amount of work it has to do. The less cross-dependencies in the codebase, the less modules will need to be recompiled after something changes.

The obvious strategy would be to try to reduce compile-time dependencies, but reducing runtime and struct deps is equally important.

Consider following example:

defmodule A do
  @answer B.search()
  def get_answer, do: @answer
end

defmodule B do
  def search, do: C.search()
end

defmodule C do
  def search do
    # ...
  end
end

A has a compile-time dependency on B because it calls a function from that module at compile-time (when module attributes are defined).

B has only a runtime dependency on C because it calls a function from that module at runtime.

A doesn’t have a direct compile-time dependency on C, but if C changes, then A must be recompiled, even though B doesn’t have to!

$ touch lib/c.ex
$ mix compile --verbose
Compiling 2 files (.ex)
Compiled lib/c.ex
Compiled lib/a.ex

The process is as follows:

Compiler detects that C changed (on disk), marks it as stale and for recompilation.
Compiler sees that B depends on something stale (C), so marks it as stale as well, but doesn’t mark it for recompilation because it’s only a runtime dependency.
Compiler sees that A depends on something stale (B), marks it as stale, but this time it’s a compile-time dependency, so the file is marked for recompilation too.

Clearly A has an implicit compile dependency on C.

It was counter-intuitive to me at first ³. It happens because the compiler assumes that A may be using C’s code at compile time indirectly (through B).

This fact is going to shape our strategy when trying to avoid recompilation hell in our projects.

Fixing your project

In large projects, it’s not uncommon to see cycles in the dependency graph. If there happen to be a compile-time dependency between member modules of such cycle, any change will trigger a cascade-style recompilation of other modules in that cycle as well as ones depending on them and so on.

That’s why even changes that appear simple on the surface, sometimes get you few hundred files to recompile... and ruined workflow.

As removing cycles from the dependency graph is rarely a trivial task, it's better to try to prevent them happening in the first place. ⁴

Nonetheless, there are some dependency-breaking techniques listed below that may apply to any project.

As a prerequisite, get familiar with xref tool.

It’s built into mix and can help you indentify super-connected modules in your project and modules that have deep subtrees of dependencies ⁵. Simply use mix help xref to start.

There's also a short and practical overview of the tool written by Wojtek Mach on Dashbit's blog.

Keep you library code clean

There will probably be some well-connected modules in your business layer, like those in User or Account contexts. In general, business layer code is very likely to contain a lot of cross-module dependencies, even cycles.

Library code on the other hand is supposed to be generic, it should not depend on any module from your business layer. Otherwise it would transfer all such dependencies to any place it’s referenced from.

Here’s an example from actual project:

# web/i18n.ex
defmodule MyApp.I18n do
  @moduledoc """
  Internationalization with a gettext-based API.
  """

  use Gettext, otp_app: :my_app

  def set_locale(%User{locale: locale}),
    do: Gettext.put_locale(__MODULE__, locale)

  # …
end

Pattern-matching on %User{} struct creates a compile-time dependency on User module. It didn't seem to be a big deal until we realized that it creates a lot of indirect dependencies throughout the whole project because I18n module is very widely-used.

A simple change yielded a huge positive change in recompilation.

- def set_locale(%User{locale: locale}),
+ def set_locale(%{locale: locale}),
    do: Gettext.put_locale(__MODULE__, locale)

Don’t import everything

In Phoenix apps, web/router.ex builds a super-connected MyApp.Router.Helpers module. When you import it to use route helpers, you indirectly import a lot of its dependencies. To avoid that, alias it instead:

- import MyApp.Router.Helpers
+ alias MyApp.Router.Helpers, as: Routes

The same applies to any other well-connected module.

Change defdelegate to proxy functions

defdelegate defines functions via metaprogramming at compile time. Simple "proxy" functions would be runtime dependencies instead.

- defdelegate authorize(conn), to: Auth
+ def authorize(conn), do: Auth.authorize(conn)

Don’t define module attributes with remote functions

Module attributes are defined at compile-time, if they are set by using remote functions, compile-time dependencies are created. If you don’t need to use module attributes in guards, consider functions instead.

- @extension_whitelist FileExt.images()
+ defp extension_whitelist, do: FileExt.images()

Use remote types in typespec

Consider the example:

defmodule Hello do
  def say(%User{username: username}), do: "Hello, #{username}"
  def say(%Admin{name: name}), do: "Hello, #{name}"
end

Hello uses %User{} and %Admin{} structs, so we have just struct dependencies, as shown by xref.

$ mix xref graph
lib/hello.ex
├── lib/admin.ex (struct)
└── lib/user.ex (struct)

Now let’s add a pretty standard function @spec that list these structs as accepted argument types:

@spec say(%User{} | %Admin{}) :: binary()

Suddenly, we get more strict compile-time deps ⁶.

$ mix xref graph
lib/hello.ex
├── lib/admin.ex (compile)
└── lib/user.ex (compile)

In order to fix this, we should rather define remote types and use them instead of structs.

defmodule User do
  defstruct [:username]
  @type t() :: %__MODULE__{}
end

defmodule Admin do
  defstruct [:name]
  @type t() :: %__MODULE__{}
end

defmodule Hello do
  @spec say(User.t() | Admin.t()) :: any()
  def say(%User{username: username}), do: "Hello, #{username}"
  def say(%Admin{name: name}), do: "Hello, #{name}"
end

Summary

Sometimes a quick, small change may result in removal of a crucial dependency and break a cycle in your dependency graph, yielding tangible improvements in recompilation speed. Other times, some improvements may come at the expense of code readability and understandability, and simply will not be worth it.

Pay attention. The compiler, through slow recompilation, may be signalling problems in your code. It may prompt you to rethink your recent architectural decisions.

Issues are generally easier and cheaper to fix when detected early and recompilation that's slowing down is a plainly visible warning sign you probably should take seriously.

D. Thomas, A. Hunt. (2020). The Pragmatic Programmer, 20th Anniversary Edition. Pearson Education, Inc. ↩
As of Elixir 1.6. See Separate tracking structs from compile-time dependencies #6575 ↩
Until I received explanation from Jason Axelson. See Implicit compile-time dependencies in Elixir Forum. ↩
I personally look forward to using such tools as boundary which makes cross-module dependencies explicit. Umbrella projects can also be helpful in that aspect as they don’t allow cyclic dependencies between individual apps. ↩
Technically it’s a graph, not a tree, but xref displays it as such. ↩
I actually suspect this is a bug and plan to investigate it. ↩