DEV Community: Murphy Randle

Quick Tip: Formatting Elixir templates

Murphy Randle — Wed, 26 May 2021 17:51:29 +0000

Finding a VS Code plugin to automatically format my Elixir templates (.eex, .leex) has been some trouble. The Elixir Language Server extension doesn't do this by default. I've been using "Beautify" up until now, because at least it properly formats HTML. However, it'd leave Elixir's special forms (if, for, etc...) completely un-indented:

Reading through this issue, it's apparant that I'm not the only one who wants some indenting from a formatter: https://github.com/timmhirsens/vscode-elixir/issues/129

This comment solved it for me: https://github.com/timmhirsens/vscode-elixir/issues/129#issuecomment-653891686

After installing https://marketplace.visualstudio.com/items?itemName=ouven.vscode-yab-for-eex-leex and the htmlbeautifier gem, I configured the default formatting for my .leex files by bringing up the command palette, typing "format document with", then choosing "Configure Default Formatter...", and then selecting "YAB for eex/leex".

Then, after a file save, my indentation comes back! Well, mostly. Sadly there's no indentation support for form_for/3, since it's just a variable assignment folled at the end by a </form> tag, but having indentation for the rest is a huge help.

Deploying Elixir to Render.com

Murphy Randle — Tue, 04 May 2021 23:26:17 +0000

I've deployed a few different Elixir services to the hosting platform https://render.com and found their provided documentation to be lacking for real-world Phoenix apps. Especially if you're using Live View. Here's an augmentation to their guide with the extra steps you might be missing.

Initial Setup

Follow the official guide: https://render.com/docs/deploy-phoenix but don't deploy your service yet. You'll need to make some changes:

Add Evars

Add some new evars:

ELIXIR_VERSION = 1.10.3 or whatever version you like. But the default version is older and breaks the build for new Elixir apps
DATABASE_URL = The connection string for the database you'll be using

Migrations

After, do these things. MAKE SURE YOU REPLACE WITH YOUR APP NAME :

Add a new script for running migrations (./build_and_migrate.sh):

#!/usr/bin/env bash
# exit on error
set -o errexit

./build.sh

# Run migrations
_build/prod/rel/<app_name>/bin/<app_name> eval "Render.Release.migrate"

Make that new script executable.
Make a module in your application for running migrations (lib/<app_name>_web/render_release.ex):

defmodule Render.Release do
  @moduledoc """
    Responsible for custom release commands
  """
  @app :<app_name>

  def migrate do
    # Allows the migration script to connect to a database using SSL.
    Application.ensure_all_started(:ssl)

    for repo <- repos() do
      {:ok, _, _} = Ecto.Migrator.with_repo(repo, &Ecto.Migrator.run(&1, :up, all: true))
    end
  end

  def rollback(repo, version) do
    # Allows the migration script to connect to a database using SSL.
    Application.ensure_all_started(:ssl)

    {:ok, _, _} = Ecto.Migrator.with_repo(repo, &Ecto.Migrator.run(&1, :down, to: version))
  end

  defp repos do
    Application.load(@app)
    Application.fetch_env!(@app, :ecto_repos)
  end
end

Change the deploy command for your render app to be ./build_and_migrate.sh instead of ./build.sh

Using a Database that Requires SSL

Unless you're using a private database on Render, you'll want to connect to your PG instance using SSL. To do that, you've got to

Uncomment the line in releases.exs that says ssl: true for your app's repo:

...
config :<app_name>, <AppName>.Repo,
  ssl: true, <-- this line, it's commented by default.
  url: database_url,

And then

Add the :ssl application to extra_applications in mix.exs:

def application do
    [
      mod: {<AppName>.Application, []},
      extra_applications: [:logger, :runtime_tools, :ssl] <-- added :ssl to the end of this list.
    ]
  end

Troubleshooting

If you run into an issue with Phoenix Live View breaking and reloading the page every few seconds, you might have enabled force_ssl in your endpoint, and it could be trying to redirect your web socket traffic to an https scheme. Which is strange. I ran into this once, and haven't found a great workaround for it.

PgBouncer

If your database does pooling using pg_bouncer then open up releases.exs and add prepare: :unnamed to your repo config.

config :<app_name>, <AppName>.Repo,
  ssl: true,
  url: database_url,
  pool_size: String.to_integer(System.get_env("POOL_SIZE") || "10"),
  prepare: :unnamed <-- this line!

Authenticated Live View Tests

Murphy Randle — Mon, 03 May 2021 21:15:24 +0000

I've been running through Pragmatic Studio's Live View course, which I highly recommend, and wanted to make a specific note of how to authenticate users when writing Live View tests after adding phx_gen_auth to a Phoenix project.

Happily, this is a very short post, because phx_gen_auth does all the work for you. When you've run that generator, you get a function added to your conn_case.ex called register_and_log_in_user. This function takes a map with a connection in it, creates a new user, logs the user in, and returns it with a modified connection.

So now my test that used to fail because the page now requires authentication:

    test "saves new message", %{conn: conn} do
      {:ok, index_live, _html} = live(conn, Routes.message_index_path(conn, :index))

      assert index_live |> element("a", "New Message") |> render_click() =~
               "New Message

--------

     ** (MatchError) no match of right hand side value: {:error, {:redirect, %{flash: "SFMyNTY.g2gDdAAAAAFtAAAABWVycm9ybQAAACRZb3UgbXVzdCBsb2cgaW4gdG8gYWNjZXNzIHRoaXMgcGFnZS5uBgCMD68zeQFiAAFRgA.zriUcbZsaCi0a4Bqq-hpJK0WP8IH-MHusGt3DPw028g", to: "/users/log_in"}}}

Will pass if I just call that function at the front:

    test "saves new message", %{conn: conn} do
      # 🚨 Add this line to authenticate the test request.
      %{conn: conn} = register_and_log_in_user(%{conn: conn})
      {:ok, index_live, _html} = live(conn, Routes.message_index_path(conn, :index))

      assert index_live |> element("a", "New Message") |> render_click() =~
               "New Message

Murphy’s setup guide for a new Phoenix (1.5) project

Murphy Randle — Sat, 01 May 2021 22:16:54 +0000

What follows is an extremely concise list of steps I'm keeping around for reference when setting up a new Elixir / Phoenix project with Tailwind, Alpine, and user auth.

NOTE: Pretending the name of the app is "love_notes"

Generate it

mix phx.new love\_notes --live

Use UUIDs

in config/config.exs

config :love_notes, :generators,
  migration: true,
  binary_id: true,
  sample_binary_id: "11111111-1111-1111-1111-111111111111"

Setup Tailwind

from https://pragmaticstudio.com/tutorials/adding-tailwind-css-to-phoenix

Install

cd assets
npm install tailwindcss @tailwindcss/forms postcss autoprefixer postcss-loader@4.2 --save-dev

Postcss

In assets/postcss.config.js

module.exports = {
  plugins: {
    tailwindcss: {},
    autoprefixer: {},
  }
}

Webpack

In your config:

use: [
  MiniCssExtractPlugin.loader,
  'css-loader',
  'postcss-loader',// <-- add this
  'sass-loader'
]

Tailwind Config

cd assets
npx tailwindcss init

Modify the config to add purge directories, set up the jit, enable dark mode, and add the tailwind UI plugin:

module.exports = {
  purge: [
    '../lib/**/*.ex',
    '../lib/**/*.leex',
    '../lib/**/*.eex',
    './js/**/*.js'
  ],
  mode: 'jit',
  darkMode: 'media',
  theme: {
    extend: {},
  },
  variants: {
    extend: {},
  },
  plugins: [
    require('@tailwindcss/forms')
  ],
}

Modify script in package.json

"deploy": "NODE_ENV=production webpack --mode production"

Include the CSS in the main file

in assets/css/app.scss

@tailwind base;
@tailwind components;
@tailwind utilities;

If you want to use component classes

You can do it like this.

@layer components {
  .btn-indigo {
    @apply bg-indigo-700 text-white font-bold py-2 px-4 rounded;
  }
}

Adding Alpine JS

npm i alpinejs

In app.js :

import 'alpinejs'

// other stuff

let liveSocket = new LiveSocket("/live", Socket, {
  dom: { // <- Add this 'dom' section
    onBeforeElUpdated(from, to){
      if(from.__x){ window.Alpine.clone(from.__x, to) }
    }
  },
  params: {_csrf_token: csrfToken}
})

Adding Auth

Inside of mix.exs

      {:jason, "~> 1.0"},
      {:plug_cowboy, "~> 2.0"},
      # Add the following line 👇🏻
      {:phx_gen_auth, "~> 0.6", only: [:dev], runtime: false}
    ]

Get deps and run the generator:

mix deps.get
mix phx.gen.auth Accounts User users
mix deps.get # Do this again so that the deps added by the generator get fetched.

Elixir and the Wall of Tests

Murphy Randle — Thu, 04 Feb 2021 06:47:32 +0000

Here's an experience I tend to have over and over:

Generate a new Elixir / Phoenix project
Use the super cool Phoenix generators to make a new model
Change the model substantially
Realize that the tests generated with the model are now super broken

When I get into this situation running mix text produces a huge number of errors. For some reason, I always want to scroll to the top of the error list and work on the first one produced, which makes fixing all of the errors a painful process of save, clear terminal, run command, scroll up, repeat.

Run just one test 🧘

The first thing I do to get some presence of mind and move forward in this situation is stop running the tests. No, not all the tests, though it certainly can be tempting to delete the file and forget about testing. No, just tell the test runner to one a single test at a time.

In ex-unit, Elixir's testing framework which comes built-in to Phoenix, this can be done with what they call "tags".

What are tags?

Test tags are a very flexible feature that allows the programmer to categorize tests throughout the project, and include or exclude whole selections of test based on their tags.

For example?

One example of that comes to mind would be tagging "slow" tests and configuring them only to run before a merge into a repository's main branch.

Back to the topic

Okay, back on topic, here's how we use tags to run just one test at a time. First, find the first test in a broken suite and add this line above its declaration:

@tag :focus
test "does a thing when I do a thing", %{
...

Here I've declared that this test is tagged with the atom "focus". Now, I can run just this test by adding the --only tag to the test command:

mix test --only focus

Now, all of the other tests will be skipped, and only the tagged test will be run. When I've fixed that test, I'll move the tag to the next test in-line. If I've fixed all the tests in a describe block, and I want to verify that they're all working together, I can move that @tag :focus line right above the describe block, and all of the test inside of it will be run. Hurrah!

Keyboard fatigue 🥱

Another thing that gets tiresome when working through the wall of tests is command-tabbing to my terminal, up-arrowing, and enter-ing after every file change. mix_test_watch to the rescue! This package will re-run your tests for you after every file change, and it accepts all the same arguments as the normal test command!

Just add it to your mix.exs file:

      ...
      {:phoenix_ecto, "~> 4.1"},      
      # 👇 Add this line
      {:mix_test_watch, "~> 1.0", only: :dev, runtime: false},
      ...

And and change test to test.watch in your CLI command:

mix test.watch --only focus

That's a wrap 🌯

If you're like me, these two steps will help you actually see what's going wrong with the broken tests, instead of just closing your terminal in fear of that massive wall of red text and browsing Twitter until you forget why you were procrastinating in the first place.

UUIDs for User IDs

Murphy Randle — Sat, 30 Jan 2021 20:59:25 +0000

Integers By Default 🔢

Phoenix's generators save a ton of time writing boilerplate code. Pow is an Elixir package and Phoenix extension that offers a great way to get user authentication up and running in very little time. But by default Phoenix's generators use auto-incrementing integers for user IDs.

What's Wrong with Integer IDs 🤔?

I've been bitten nastily by integer IDs multiple times in my career.

One time a backup failure caused the counter on user IDs to be reset, and the account records were erased while the data still existed, un-attached. When new accounts got created, those IDs got re-used, and existing data got attached to the new accounts, super bad news. (This is also an argument for using a SQL store with strong relational guarantees, rather than the NoSQL store the company had adopted at the time).

On another occasion, user IDs were being randomly generated, but they were still rather small integers, and the likelihood of a collision was high due to the rate at which new accounts were being created. A simple oversight in the application code was performing an upsert on the newly generated user accounts instead of an insert. The Database would have detected a conflict and thrown an error in the case of an insert on a re-used user ID, but since the app was upserting, re-used user IDs just clobbered old accounts with new credentials. Yikes!

Neither of these problems were caused by integer IDs, but in both cases, using a longer, richer user ID would have massively reduced the likelihood that the bugs would have had negative outcomes.

Switching to UUIDs 👷🏿‍♀️

Let's say you've already gone through the process of generating a new Phoenix app, and you've already followed all the guides from Pow on getting set up, and then you realize that your user IDs are integers. Stop! Don't throw away all of your code and start over. I found myself in the same situation this morning, but thanks to Cam Stuart on GitHub, I got my IDs switched over in no time. Here are the steps.

Change the migration.

Open up priv/repo/migrations/<timestamp>_create_users.exs
Change

  def change do
    create table(:users) do
...

  def change do
    create table(:users, primary_key: false) do
...

Which tells the migration not to create the default auto-incrementing ID column. Then add a line to create your own ID column:

  def change do
    create table(:users, primary_key: false) do
    add :id, :uuid, primary_key: true
...

Change the Model

Next open up lib/boots/users/user.ex and add two module attributes above the "schema" declaration:

  @primary_key {:id, :binary_id, autogenerate: true} # Add this
  @foreign_key_type :binary_id # And add this
  schema "users" do
    ...

Those module attributes tell the schema that it should use UUIDs (here represented as :binary_id) for the ID column, and that it should auto-generate them. It also tells any other schemas that when they're making a foreign key that references this table, they should also use UUIDs.

Update the Test

If you've followed the Pow guide on adding API tokens, you'll have a failing test now. Open up test/your_app_web/api_auth_plug_test.exs and change the lines that create a test user with an integer ID:

    user =
      Repo.insert!(%User{id: 1, email: "test@example.com"})

To use a UUID instead:

    user =
      Repo.insert!(%User{id: "e2c54c31-e574-4c9f-8672-dad23449d4cf", email: "test@example.com"})

Change your Generators

Now that you've fixed your users, let's make it so that any new models you generate will use UUIDs by default. Open up config/config.exs and add the following lines to the end of the file:

config :your_app, :generators,
  migration: true,
  binary_id: true,
  sample_binary_id: "11111111-1111-1111-1111-111111111111"

🚨 Don't forget to change :your_app to the actual name of your app!

This should tell phoenix that whenever it's generating a new model, it should use binary IDs for them.

All Done!

Congratulations! Your app is now a little more robust 👏. It's worth noting that there has been some discussion about whether using random UUIDs significantly hurts Postgres's performance by causing it to do more random seeks. After my research, the evidence against UUIDs has appeared weak enough to me that I'd rather have the safety than be concerned about possible performance loss. I'd love to hear from you if you have more information on this, though!

On Pagination and Sort Order

Murphy Randle — Thu, 28 Jan 2021 18:37:44 +0000

The Bug 🐞

We've been testing the next generation of our sync engine at Day One prior to public release, and we found a funny bug. We have an endpoint for fetching a big list of entries that have changed since the last fetch. This endpoint is also how a new phone with no content would pull down all entries from a journal. But when we did try to sync all of our journal contents down to a new device, we were missing entries!

Enter Sherlock 🔎

We investigated the internal workings of the endpoint to see what was up. The missing entry was in the database, so that wasn't the issue. We tested the query that fetched the rows and the entry showed up, so that wasn't the issue. Internally we fetch groups of entries from the database in pages and write them out to the endpoint so we figured it must have been some flawed logic with the pagination. We logged the pagination pattern, but everything looked perfect.

Then we looked closer at the response returned by the server.

We were getting the right number of entries pulled down from that endpoint, but it turned out some of the entries were missing from the feed, and others were repeated!

This was a big indicator that we had a problem with sorting. We compared the entry that showed up twice with the entry that didn't show up at all. Sure enough, we found that they had the exact same "sync date", which is the column that our query sorted by.

    builder->orderBy([|"sync_date asc"|]);

The Fix

All we had to do was add a unique column into the sort clause:

    builder->orderBy([|"sync_date asc, id asc"|]);

And the problem was fixed.

But Why? 🤔

When paginating, the same query is executed over again each time a new page is fetched. Each time, a small window of the results are fetched by skipping over the number of items fetched by previous pages. In order for this to work, however, the overall result set has to be the same for every page **. If the result set changes as new pages are fetched, we run the risk of dropping or repeating items.

In our case, the sync date on an entry was pretty close to unique. It's got millisecond-accuracy. And most of the time, two entries aren't synced in the exact same millisecond. But, as we now know from testing on real journal data, there are entries with identical sync dates. When this happens, The order in which those two entries are returned in a result state is considered unstable. Our query had an unstable sort.

So something about skipping through the items in our result set caused Postgres to reverse the order of those two rows every time. One of them was missed, the other repeated.

Once we added a unique column to the sort, the sort became a stable sort. Which means that Postgres always knows how to order the results in the query, no matter how much skipping or limiting we do. With that change in place, skipping old rows no longer changed the order and all the entries were properly included in the result set.

Don’t Make Me Tap Twice

Murphy Randle — Mon, 25 Jan 2021 20:24:46 +0000

I've started rough work on a new app for digital "morning baskets". While I just used Expo when starting Storytime (another app I have in-progress), I decided to give the famous Ignite boilerplate from Infinite Red a chance. In short, it's fantastic. In just a few days of side-work time I've almost completed a fully-functioning minimum viable product (MVP) of this new app.

But I quickly ran into that annoying situation where you're editing a text field, and you want to press the "submit" button, but you've gotta tap it twice so that the keyboard gets dismissed before you can progress.

Thanks to a detailed blog post I found a quick solution. In a project generated by Ignite 6.x, we can open up app/components/screen/screen.tsx and find the place where ScrollView is used:

...
<ScrollView
    style={[preset.outer, backgroundStyle]}
    contentContainerStyle={[preset.inner, style]}
>
    {props.children}
</ScrollView>
...

All we have to do is add keyboardShouldPersistTaps="handled" to the props of ScrollView.

...
<ScrollView
    style={[preset.outer, backgroundStyle]}
    contentContainerStyle={[preset.inner, style]}
    keyboardShouldPersistTaps="handled" // <- Here!
>
    {props.children}
</ScrollView>
...

This instructs the scrollview to dismiss the keyboard if it receives a tap gesture which none of its children have handled, but to leave the keyboard alone if one of the children in the view handles the event. In my case, I navigate right after the button click, and this action dismisses the keyboard automatically anyway. So it resolves the problem for me!

Debugging tricky parameterized types

Murphy Randle — Fri, 11 Dec 2020 20:51:32 +0000

Debugging tricky parameterized types in Rescript

Parameterized functions are fantastic. I'm talking about the functions that operate around some abstract piece of data, without doing anything that would require specific knowledge about that data. Take, for example, the following fictional function:

Example

let maybeSaveThing = (maybeThing: option<debug>): option<debug> => {
  switch maybeThing {
  | Some(thing) => saveThing(thing)
  | None => None
  }
}

The code above is designed to take an option of something, save it somewhere if the option contains data, and return the result. We assume, for this example, that the saveThing function knows now to save anything. We can tell that the function shouldn't care what's contained in the option type by looking at the function signature. It takes an option<'a>, and returns an option<'a>.

A Sneaky Bug

But this code has a sneaky bug. We can see it if we look at the function's inferred type:

option<int> => option<int>

The compiler is telling us that the function takes an option of an integer, even though we explicitly stated in the type signature that the function should be generic. Why is that?

Why Don't we Get a Type Error?

Generics encapsulate all other types, that's why. For example, an option<int> is an option<'a> as far as the type system is concerned. So is an option<string> or a option<blah>. If a part of the code inside of a function returns a specific value where a generic value was expected, the type inference algorithm quietly turns the function's parameter into a specific type. The compiler won't tell you that your type annotation is wrong, because it isn't. An option<int> => option<int> is an option<'a> => option<'a>.

Why Are We Getting an `int`?

Spoiler alert, there's a call inside of our function that returns an int where the generic type was expected. Stick around and we'll see how to find it in a moment.

Discovering the Mismatch

We intended the type to be generic. It's troublesome that we don't discover this breakage until we want to use this function to save some type other than an int. The compiler will tell us that maybeSaveThing takes an option<int> and we're trying to pass in something like option<string>.

How can we discover these unintentional losses of genericism closer to the site of definition?

Enforcing Genericism

Adding an interface (.resi) file with the generic signature for the function will help the compiler out. Instead of inferring the type of the function to be specific, the compiler will know that you intend to keep that function generic, and it'll tell you that your implementation doesn't match the interface.

Finding the Mistake

Unfortunately, an error that the function definitions don't match doesn't go far to help us find the location of the mistake. When we're dealing with more than mere example code, and we have a complex function with lots of operations, it's tempting to abort the whole effort and scroll through Twitter instead.

But here's a trick that can speed the process up:

type debug

We've added a new type to our code that is the opposite of generic. It's so specific that no value fits it. And it's explicitly for debugging, so we know that no other area of the code will rely on this type for business logic.

If we swap out the generic argument in our function definition with debug temporarily:

type debug

let maybeSaveThing = (maybeThing: option<debug>): option<debug> => {
  switch maybeThing {
  | Some(thing) => saveThing(thing)
  | None => None
  }
}

It'll show us the line of code that is returning a specific, non-generic value:

  8 │ let maybeSaveThing = (maybeThing: option<debug>): option<debug> => {
   9 │ switch maybeThing {
  10 │ | Some(thing) => saveThing(thing)
  11 │ | None => None
  12 │ }

  This has type: option<int>
  Somewhere wanted: option<debug>

  The incompatible parts:
    int vs debug

Aha! The call to saveThing is returning a option<int>. If that had been a generic function, its return type would have been option<debug>. Let's look at its definition:

let saveThing = _thing => {
  Some(21)
}

There's the source of the problem. If we modify it to do something generic instead:

let saveThing = thing => {
  let _ = Js.Json.stringifyAny //-> blah blah do some saving in the background
  Some(thing)
}

Then the compiler is happy, and we know our code is generic again. We can remove debug and put the type parameters back in place.

On File Switching in Rescript

Murphy Randle — Wed, 09 Dec 2020 17:36:28 +0000

I use Rescript at work, and find that I often want to switch between interface and implementation files. Less often, I also want to inspect the compiled source of a file. This can be a small pain to do by hand, and small pains repeated often become big pains. So here are some VS Code extensions to help out.

For Opening Interface & Compiled Files

Extension: Open related files

This extension will automatically open files with the same name, but different extensions, so you'll get the .resi and the .bs.js files popped up in new tabs. Better yet, if you keep running the command, it toggles between .res, and .resi.

This one doesn't come with a default keymap, so you'll need to add one in your settings.

Another cool thing about this extension is that it can create interface files for you as well, just throw this into your settings.json:

  "openRelatedFiles.createFileMap" : {
    ".res": [
        ".resi"
    ],
    ".re": [
      ".rei"
    ]
  }

And then you can use the "Create related files" command to make an ".resi" file for you.

For Opening Child Files

The OCaml naming convention gives files hierarchy by their names. For example Foo.res would break out some complex logic "baz" into a subfile: Foo_Baz.res and import it. We can use an extension to easily browse these child files.

Quick Open Related Files

This pops open a picker of files that have a name with a prefix similar to our own:

Animating Text Content with Redash and Reanimated 2

Murphy Randle — Fri, 04 Dec 2020 03:18:53 +0000

On Cyber Monday I sprang for William Candillon's course on animation with React Native. I love his style. And I like how beautiful his examples are. However, it took me a few minutes to understand that he's basing all his work on two libraries, Redash, and Reanimated. These tools are super powerful, and I'm off to a strong start building lovely app animations, but I ran into a couple of problems while getting started.

TypeError: global.__reanimatedWorkletInit is not a function

I got this error the first time I tried to use Redash. I had installed both it and reanimated, imported a hook, and tried to use it. But the app immediately crashed. I resolved it by importing reanimated at the top of the file:

import "react-native-reanimated";
import ...

Animated string values would cause a rerender in the UI.

With useDerivedValue I transitioned a string into existence by animating its length. However, when I passed the Animation.SharedValue<string> to a Retext component and tested it out in my app, there was no animation.

Thanks to an answer this github issue I fixed that problem by whitelisting text as an animatable prop:

Animated.addWhitelistedNativeProps({ text: true });

After adding that line to the top of the file, my views updated without a problem as the text animated.

On media uploads, and annoying S3 APIs.

Murphy Randle — Tue, 01 Dec 2020 07:16:22 +0000

While working on Storytime Studio a couple of weeks ago I decided to upgrade my media uploading approach. I had started with the naive "I'll just upload everything to the server and put it in a folder on the hard drive" approach, because that was simple. And it worked really well until I tried to upload 15 minutes of audio (a few hundred megabytes) at once. The server crashed, every time. Even after upgrading the box from 1GB of memory to 2GB.

The server is written in Adonis JS right now (an experiment, I've never used that framework at all before, let alone in production) and it's taking care of parsing the multipart form body I was using. I assume that the body parser is streaming the content to a temporary file on disk. It shouldn't be a big memory hit, even with a file that's hundreds of megabytes. But instead of taking time to figure out why the server wasn't surviving even a moderately heavy upload, I decided it was time to switch to the direct-to-s3 upload model that I would eventually be moving to anyway.

Before I move on, let me make sure you know what S3 is. Amazon's S3 is a low-cost, durable option for storing even very large files indefinitely. It's a fantastic place to put media files and raw data. I'm using S3 as a loose term here, because I'm actually hosting my server on Linode, and I'll be using Linode's Object Storage, which implements the same API as S3. Most cloud platforms have some service like this.

What is the direct-to-S3 approach?

Short and sweet: The server generates a special URL for putting data directly in S3 and hands that back to the client. The client can use that URL to upload or download media for a limited time. After that time, the URL expires. The media itself doesn't have to go through the server, and since the URLs are time-limited, it's secure enough for most purposes.

The requests look like this when uploading audio:

sequenceDiagram participant Phone participant Server participant S3 Phone->>Server: Upload story meta Server->>Phone: Signed URL Phone->>S3: Upload story audio file

The audio content never actually touches my server, only the metadata. The audio goes straight to S3. This pattern drastically increases the amount of traffic my little application server will be able to handle, because it doesn't have to worry about buffering, writing or reading any of the media data coming from the clients. And even a small server can handle a considerable number of requests for little chunks of metadata. Thus, this is a "scalable" way to handle media upload.

This is what the requests look like for downloads:

sequenceDiagram participant Phone participant Server participant S3 Phone->>Server: Get story content Server->>Phone: Redirect to signed URL Phone->>S3: Follow redirect and download content

When the phone asks for story content from the server, the server just sends back a 302 Found status code, and sets the Location header to the newly generated S3 URL. The client then automatically visits that URL and downloads the content. Again, nothing has streamed through my little inexpensive server, and my scalability goes up.

API Pain

The current version of Amazon's S3 API was designed in 2006, and it shows its age. Though doing simple things isn't too uncomfortable, and there's not much drama if you don't make any mistakes, I've found debugging mistakes when they do happen to be a real pain. Switching to this direct-to-s3 approach took me multiple days, when it should have taken me an hour or two, because I was busy debugging APIs that didn't feel like they should have been breaking.

Initially I used the "aws-sdk" node module to generate signed URLs for Linode's object storage. I do this all the time at Day One (where we are using the real S3), and I've never had a problem with it. But no matter what I tried, I kept getting a response from Linode that said SignatureDoesNotMatch, and that's pretty much all it said 😡. Yeah, it seems they use that error to cover a number of possible mess-ups. So it took me a lot of experimentation to finally get anything working.

I ditched the aws-sdk module and followed their docs for their HTTP API instead. These docs taught me two important things:

When using Linode's object storage, the content-type of the media being PUT to the URL must be specified when creating the URL, and the upload must match that type.
When doing a GET or a DELETE, the content-type must not be specified when creating the URL.

Guess what happens if you mess one of those things up? SignatureDoesNotMatch. Ultimately that error message does make sense, once you understand that the content-type is part of the signature, but the information returned in the response is bad at actually helping the user to fix the problem.

Even though I ditched the S3 API library, I think now that I probably could have stuck with it if I had included and excluded the content-type attribute in the right places.

Peace

It's done now. I probably won't have to touch this part of the code again for a very long time, if ever. So even though the work was frustrating, I can close my eyes, take a deep breath in, let it out, and move on to other frustrating problems 🧘‍♂️.