DEV Community: Fly.io

Docker without Docker

Fly.io — Wed, 02 Jun 2021 15:42:48 +0000

We’re Fly.io. We take container images and run them on our hardware around the world. It’s pretty neat, and you should check it out; with an already-working Docker container, you can be up and running on Fly in well under 10 minutes.

Even though most of our users deliver software to us as Docker containers, we don’t use Docker to run them. Docker is great, but we’re high-density multitenant, and despite strides, Docker’s isolation isn’t strong enough for that. So, instead, we transmogrify container images into Firecracker micro-VMs.

Let's demystify that.

What’s An OCI Image?

They do their best to make it look a lot more complicated, but OCI images — OCI is the standardized container format used by Docker — are pretty simple. An OCI image is just a stack of tarballs.

Backing up: most people build images from Dockerfiles. A useful way to look at a Dockerfile is as a series of shell commands, each generating a tarball; we call these “layers”. To rehydrate a container from its image, we just start the the first layer and unpack one on top of the next.

You can write a shell script to pull a Docker container from its registry, and that might clarify. Start with some configuration; by default, we’ll grab the base image for golang:

image="${1:-golang}"
registry_url='https://registry-1.docker.io'
auth_url='https://auth.docker.io'
svc_url='registry.docker.io'

We need to authenticate to pull public images from a Docker registry – this is boring but relevant to the next section – and that’s easy:

function auth_token { 
  curl -fsSL "${auth_url}/token?service=${svc_url}&scope=repository:library/${image}:pull" | jq --raw-output .token
}

That token will allow us to grab the “manifest” for the container, which is a JSON index of the parts of a container.

function manifest { 
  token="$1"
  image="$2"
  digest="${3:-latest}"

  curl -fsSL \
    -H "Authorization: Bearer $token" \
    -H 'Accept: application/vnd.docker.distribution.manifest.list.v2+json' \
    -H 'Accept: application/vnd.docker.distribution.manifest.v1+json' \
    -H 'Accept: application/vnd.docker.distribution.manifest.v2+json' \
      "${registry_url}/v2/library/${image}/manifests/${digest}"
}

The first query we make gives us the “manifest list”, which gives us pointers to images for each supported architecture:

  "manifests": [
    {
      "digest": "sha256:3fc96f3fc8a5566a07ac45759bad6381397f2f629bd9260ab0994ef0dc3b68ca",
      "platform": {
        "architecture": "amd64",
        "os": "linux"
      },
    },

Pull the digest out of the matching architecture entry and perform the same fetch again with it as an argument, and we get the manifest: JSON pointers to each of the layer tarballs:

   "config": {
      "digest": "sha256:0debfc3e0c9eb23d3fc83219afc614d85f0bc67cf21f2b3c0f21b24641e2bb06"
   },
   "layers": [
      {
         "digest": "sha256:004f1eed87df3f75f5e2a1a649fa7edd7f713d1300532fd0909bb39cd48437d7"
      },

It’s as easy to grab the actual data associated with these entries as you’d hope:

function blob {
  token="$1"
  image="$2"
  digest="$3"
  file="$4"

  curl -fsSL -o "$file" \
      -H "Authorization: Bearer $token" \
        "${registry_url}/v2/library/${image}/blobs/${digest}"
}

And with those pieces in place, pulling an image is simply:

function layers { 
  echo "$1" | jq --raw-output '.layers[].digest'
}

token=$(auth_token "$image")
amd64=$(linux_version $(manifest "$token" "$image"))
mf=$(manifest "$token" "$image" "$amd64")

i=0
for L in $(layers "$mf"); do
  blob "$token" "$image" "$L" "layer_${i}.tgz"
  i=$((i + 1 ))
done

Unpack the tarballs in order and you’ve got the filesystem layout the container expects to run in. Pull the “config” JSON and you’ve got the entrypoint to run for the container; you could, I guess, pull and run a Docker container with nothing but a shell script, which I’m probably the 1,000th person to point out. At any rate here’s the whole thing.

You’re likely of one of two mindsets about this: (1) that it’s extremely Unixy and thus excellent, or (2) that it’s extremely Unixy and thus horrifying.

Unix tar is problematic. Summing up Aleksa Sarai: tar isn’t well standardized, can be unpredictable, and is bad at random access and incremental updates. Tiny changes to large files between layers pointlessly duplicate those files; the poor job tar does managing container storage is part of why people burn so much time optimizing container image sizes.

Another fun detail is that OCI containers share a security footgun with git repositories: it’s easy to accidentally build a secret into a public container, and then inadvertently hide it with an update in a later image.

We’re of a third mindset regarding OCI images, which is that they are horrifying, and that’s liberating. They work pretty well in practice! Look how far they’ve taken us! Relax and make crappier designs; they’re all you probably need.

Speaking of which:

Multi-Tenant Repositories

Back to Fly.io. Our users need to give us OCI containers, so that we can unpack and run them. There’s standard Docker tooling to do that, and we use it: we host a Docker registry our users push to.

Running an instance of the Docker registry is very easy. You can do it right now; docker pull registry && docker run registry. But our needs are a little more complicated than the standard Docker registry: we need multi-tenancy, and authorization that wraps around our API. This turns out not to be hard, and we can walk you through it.

A thing to know off the bat: our users drive Fly.io with a command line utility called flyctl. flyctl is a Go program (with public source) that runs on Linux, macOS, and Windows. A nice thing about working in Go in a container environment is that the whole ecosystem is built in the same language, and you can get a lot of stuff working quickly just by importing it. So, for instance, we can drive our Docker repository clientside from flyctl just by calling into Docker’s clientside library.

If you're building your own platform and you have the means, I highly recommend the CLI-first tack we took. It is so choice. flyctl made it very easy to add new features, like databases,
private networks, volumes, and our bonkers SSH access system.

On the serverside, we started out simple: we ran an instance of the standard Docker registry with an authorizing proxy in front of it. flyctl manages a bearer token and uses the Docker APIs to initiate Docker pushes that pass that token; the token authorizes repositories serverside using calls into our API.

What we do now isn’t much more complicated than that. Instead of running a vanilla Docker registry, we built a custom repository server. As with the client, we get a Docker registry implementation just by importing Docker’s registry code as a Go dependency.

We’ve extracted and simplified some of the Go code we used to build this here, just in case anyone wants to play around with the same idea. This isn’t our production code (in particular, all the actual authentication is ripped out), but it’s not far from it, and as you can see, there’s not much to it.

Our custom server isn’t architecturally that different from the vanilla registry/proxy system we had before. We wrap the Docker registry API handlers with authorizer middleware that checks tokens, references, and rewrites repository names. There are some very minor gotchas:

Docker is content-addressed, with blobs “named” for their SHA256 hashes, and attempts to reuse blobs shared between different repositories. You need to catch those cross-repository mounts and rewrite them.
Docker’s registry code generates URLs with _state parameters that embed references to repositories; those need to get rewritten too. _state is HMAC-tagged; our code just shares the HMAC key between the registry and the authorizer.

In both cases, the source of truth for who has which repositories and where is the database that backs our API server. Your push carries a bearer token that we resolve to an organization ID, and the name of the repository you’re pushing to, and, well, our design is what you’d probably come up with to make that work. I suppose my point here is that it’s pretty easy to slide into the Docker ecosystem.

Building And Running VMs

The pieces are on the board:

We can accept containers from users
We can store and manage containers for different organizations.
We've got a VMM engine, Firecracker, that we've written about already.

What we need to do now is arrange those pieces so that we can run containers as Firecracker VMs.

As far as we're concerned, a container image is just a stack of tarballs and a blob of configuration (we layer additional configuration in as well). The tarballs expand to a directory tree for the VM to run in, and the configuration tells us what binary in that filesystem to run when the VM starts.

Meanwhile, what Firecracker wants is a set of block devices that Linux will mount as it boots up.

There's an easy way on Linux to take a directory tree and turn it into a block device: create a file-backed loop device, and copy the directory tree into it. And that's how we used to do things. When our orchestrator asked to boot up a VM on one of our servers, we would:

Pull the matching container from the registry.
Create a loop device to store the container's filesystem on.
Unpack the container (in this case, using Docker's Go libraries) into the mounted loop device.
Create a second block device and inject our init, kernel, configuration, and other goop into.
Track down any persistent volumes attached to the application, unlock them with LUKS, and collect their unlocked block devices.
Create a TAP device, configure it for our network, and attach BPF code to it.
Hand all this stuff off to Firecracker and tell it to boot.

This is all a few thousand lines of Go.

This system worked, but wasn't especially fast. Part of the point of Firecracker is to boot so quickly that you (or AWS) can host Lambda functions in it and not just long-running programs. A big problem for us was caching; a server in, say, Dallas that's asked to run a VM for a customer is very likely to be asked to run more instances of that server (Fly.io apps scale trivially; if you've got 1 of something running and would be happier with 10 of them, you just run flyctl scale count 10). We did some caching to try to make this faster, but it was of dubious effectiveness.

The system we'd been running was, as far as container filesystems are concerned, not a whole lot more sophisticated than the shell script at the top of this post. So Jerome replaced it.

What we do now is run, on each of our servers, an instance of containerd. containerd does a whole bunch of stuff, but we use it as as a cache.

If you're a Unix person from the 1990s like I am, and you just recently started paying attention to how Linux storage works again, you've probably noticed that a lot has changed. Sometime over the last 20 years, the block device layer in Linux got interesting. LVM2 can pool raw block devices and create synthetic block devices on top of them. It can treat block device sizes as an abstraction, chopping a 1TB block device into 1,000 5GB synthetic devices (so long as you don't actually use 5GB on all those devices!). And it can create snapshots, preserving the blocks on a device in another synthetic device, and sharing those blocks among related devices with copy-on-write semantics.

containerd knows how to drive all this LVM2 stuff, and while I guess it's out of fashion to use the devmapper backend these days, it works beautifully for our purposes. So now, to get an image, we pull it from the registry into our server-local containerd, configured to run on an LVM2 thin pool. containerd manages snapshots for every instance of a VM/container that we run. Its API provides a simple "lease"-based garbage collection scheme; when we boot a VM, we take out a lease on a container snapshot (which synthesizes a new block device based on the image, which containerd unpacks for us); LVM2 COW means multiple containers don't step on each other. When a VM terminates, we surrender the lease, and containerd eventually GCs.

The first deployment of a VM/container on one of our servers does some lifting, but subsequent deployments are lightning fast (the VM build-and-boot process on a second deployment is faster than the logging that we do).

Some Words About Init

Jerome wrote our init in Rust, and, after being cajoled by Josh Triplett, [we released the code (https://github.com/superfly/init-snapshot), which you can go read.

The filesystem that Firecracker is mounting on the snapshot checkout we create is pretty raw. The first job our init has is to fill in the blanks to fully populate the root filesystem with the mounts that Linux needs to run normal programs.

We inject a configuration file into each VM that carries the user, network, and entrypoint information needed to run the image. init reads that and configures the system. We use our own DNS server for private networking, so init overrides resolv.conf. We run a tiny SSH server for user logins over WireGuard; init spawns and monitors that process. We spawn and monitor the entry point program. That’s it; that’s an init.

Putting It All Together

So, that's about half the idea behind Fly.io. We run server hardware in racks around the world; those servers are tied together with an orchestration system that plugs into our API. Our CLI, flyctl, uses Docker's tooling to push OCI images to us. Our orchestration system sends messages to servers to convert those OCI images to VMs. It's all pretty neato, but I hope also kind of easy to get your head wrapped around.

The other "half" of Fly is our Anycast network, which is a CDN built in Rust that uses BGP4 Anycast routing to direct traffic to the nearest instance of your application. About which: more later.

Building a Distributed Turn-Based Game System in Elixir

Fly.io — Wed, 26 May 2021 17:38:25 +0000

One of the best things about building web applications in Elixir is LiveView, the Phoenix Framework feature that makes it easy to create live and responsive web pages without all the layers people normally build.

Many great Phoenix LiveView examples exist. They often show the ease and power of LiveView but stop at multiple browsers talking to a single web server. I wanted to go further and create a fully clustered, globally distributed, privately networked, secure application. What's more, I wanted to have fun doing it.

So I set out to see if I could create a fully distributed, clustered, privately networked, global game server system. Spoiler Alert: I did.

What I didn't have to build

What I find remarkable is what I didn't need to build.

I didn't build a Javascript front end using something like React.js or Vue.js. That is the typical approach. Building a JS front-end means I need JS components, a front-end router, a way to model the state in the browser, a way to transfer player actions to the server and a way to receive state updates from the server.

On the server, I didn't build an API. Typically that would be REST or GraphQL with a JSON structure for transferring data to and from the front-end.

I didn't need other external systems like Amazon SQS, Kafka, or even just Redis to pass state between servers. This means the entire system requires less cross-technology knowledge or specialized skills to build and maintain it. I used Phoenix.PubSub which is built on technology already in Elixir's VM, called the BEAM. I used the Horde library to provide a distributed process registry for finding and interacting with GameServers.

As for Fly.io's WireGuard connected private network between geographically distant regions and data centers? I don't even know how I would have done that in AWS, which is why I've always given up on the idea.

What I did build

What I built was just a proof of concept, but I'm surprised at how it came together. I ended up with a platform that can host many different types of games, all of which:

Can be multi-player
Offer a Jackbox-style 4-letter game code system
Have on-demand game and match creation
With a fast, response UI

And, just one little extra detail: the platform supports multiple connected servers operating together in clusters. Elixir for the win!

I created this as an open source project on Github, so you can check it out yourself.

https://github.com/fly-apps/tictac

Technology

I've worked with enough companies and teams to imagine several different approaches to build a system like this. Those approaches would all require large multi-disciplinary teams like a front-end JS team, a backend team, a DevOps team, and more. In contrast, I set out to do this by myself, in my spare time, and with a whole lot of "life" happening too.

Here's what I chose to use:

Elixir programming language – A dynamic, functional language for building scalable and maintainable applications.
Phoenix Framework – Elixir's primary web framework
Phoenix LiveView – Rich, real-time user experiences with server-rendered HTML delivered by websockets
libcluster – Automatic cluster formation/healing for Elixir applications.
Horde – Elixir library that provides a distributed and supervised process registry.
Fly.io – Hosting platform that enables private networked connections and multi-region support.

Application Architecture

There are many guides to getting started with LiveView, I'm not focusing on that here. However, for context, this demonstrates the application architecture when running on a local machine.

The "ABCD" in the graphic is a running game identified by the 4-letter code "ABCD".

Let's walk it through.

A player uses a web browser to view the game board. The player clicks to make a move.
The browser click triggers an event in the player's LiveView. There is a bi-directional websocket connection from the browser to LiveView.
The LiveView process sends a message to the game server for the player's move.
The GameServer uses Phoenix.PubSub to publish the updated state of game ABCD.
The player's LiveView is subscribed to notifications for any updates to game ABCD. The LiveView receives the new game state. This automatically triggers LiveView to re-render the game immediately pushing the UI changes out to the player's browser.
All connected players see the new state of the board and game.

We need a game

I needed a simple game to play and model for this game system. I chose Tic-Tac-Toe. Why?

It's simple to understand and play.
Easy to model.
Doesn't bog down the project with designing a game.
Quick to play through and test it being "over".

I want to emphasize that this system can be used to build many turn-based, multi-user games! This simple Tic-Tac-Toe game covers all of the basics we will need. Besides, Tic-Tac-Toe was even made into a TV Show!

This is what the game looks like with 2 players.

The game system works great locally. Let's get it deployed!

Hosting on Fly.io

Following the Fly.io Getting Started Guide for Elixir, I created a Dockerfile to generate a release for my application. Check out the repo here:

https://github.com/fly-apps/tictac

The README file outlines both how to run it locally and deploy it globally on Fly.io.

What is special about hosting it on Fly.io? Fly makes it easy to deploy a server geographically closer to the users I want to reach. When a user goes to my website, they are directed to my nearest server. This means any responsive LiveView updates and interactions will be even faster and smoother because the regular TCP and websocket connections are just that much physically closer.

But for the game, I wanted there to be a single source of truth. That GameServer can only exist in one place. Supporting a private, networked, and fully clustered environment means my server in the EU can communicate with the GameServer that might be running in the US. But my EU players have a fast and responsive UI connection close to them. This provides a better user experience!

Here is what I find compelling about Fly.io for hosting Elixir applications:

Secure HTTPS automatically using Let's Encrypt. I didn't do anything to set that up!
Distributed nodes use private network connections through WireGuard.
Nodes auto-clustered using libcluster and the DNSPoll strategy. (See here for details)
Geographically distributed servers near my users are clustered together.
This was the easiest multi-region yet still privately networked solution I've ever seen! (I have experience with AWS, DigitalOcean, and Heroku)

Conclusion

For a proof-of-concept, I couldn't be happier! In a short time, by myself, I created a working, clustered, distributed, multi-player, globe-spanning gaming system!

The pairing of Elixir + LiveView + Fly.io is excellent. Using Elixir and LiveView, I built a powerful, resilient, and distributed system in orders of magnitude shorter time and effort. Deploying it on Fly.io let let me easily do something I would never have even tried before, namely, deploying servers in regions around the globe while keeping the application privately networked and clustered together.

Whenever I've thought of creating a service with a global audience, I'd usually scapegoat the idea saying, "Well I don't know how I'd get the translations, so I'll just stick with the US. It's a huge market anyway." In short, I've never even considered a globally connected application because it would be "way too hard".

But here, with Elixir + LiveView + Fly.io, I did something by myself in my spare time that larger teams using more technologies struggle to deliver. I'm still mind blown by it!

What will you build?

Tic-Tac-Toe is a simple game and doesn't provide "hours of fun". I know you can think of a much cooler and more interesting multi-player, turn-based game that you could build on a system like this. What do you have in mind?

You should know about Server-Side Request Forgery

Fly.io — Mon, 25 Jan 2021 18:09:17 +0000

This is a post about the most dangerous vulnerability most web applications face, one step that we took at Fly to mitigate it, and how you can do the same.

Server-side request forgery (SSRF) is application security jargon for “attackers can get your app server to make HTTP requests on their behalf”. Compared to other high severity vulnerabilities like SQL injection, which allows attackers to take over your database, or filesystem access or remote code injection, SSRF doesn’t sound that scary. But it is, and you should be nervous about it.

The deceptive severity of SSRF is one of two factors that makes SSRF so insidious. The reason is simple: your app server is behind a security perimeter, and can usually reach things an ordinary Internet user can’t. Because HTTP is a relatively flexible protocol, and URLs are so expressive, attackers can often use SSRF to reach surprising places; in fact, leveraging HTTP SSRF to reach non-SSRF protocols has become a sport among security researchers. A meaty, complicated example of this is Joshua Maddux’s TLS SSRF trick from last August. Long story short: in serious applications, SSRF is usually a game-over vulnerability, meaning attackers can use it to gain full control over an application’s hosting environment.

The other factor that makes SSRF nerve-wracking is its prevalence. As an industry, we’ve managed to drastically reduce instances of vulnerabilities like SQL injection by updating our libraries and changing best practices; for instance, it would be weird to see a mainstream SQL library that didn’t use parameterized queries to keep attacker meta-characters out of query parsing. But applications of all shapes and sizes make server-side HTTP queries; in fact, if anything, that’s becoming more common as we adopt more and more web APIs.

There are two common patterns of SSRF vulnerabilities. The first, simplest, and most dangerous comprises features that allow users to provide URLs for the web server to call directly; for instance, your app might offer “web hooks” to call back to customer web servers. The second pattern involves features that incorporate user data into URLs. In both cases, an attacker will try to leverage whatever control you offer over URLs to trick your server into hitting unexpected URLs.

Fortunately, there’s a mitigation that frustrates attackers trying to exploit either pattern of SSRF vulnerabilities: SSRF proxies.

You should know about Smokescreen

Imagine if your application code didn’t have to be relentlessly vigilant about every URL it reached out to, and could instead assume that a basic security control existed to make sure that no server-side HTTP query would be able to touch internal resources? It’s easy if you try! What you want is to run your server-side HTTP through a proxy.

We've been putting Smokescreen to work at Fly, and it's so useful, we thought we should share. Smokescreen is an egress proxy that was built at Stripe to help manage outgoing connections in a sensible and safe way.

Smokescreen’s job is to make sure your outgoing requests are sane, sensible and safe. SmokeScreen was created by Stripe to ensure that they knew where all their outgoing requests were going. Specifically, it makes sure that the IP address requested is a publicly routed IP and that means checking any request isn't destined for 10.0.0.0/8, 172.16.0.0/12, 192.168.0.0/16, or fc00::/7 (the IPv6 "ULA" space, which includes Fly's 6PN private addresses.

Out of the box

There's more SmokeScreen can do, but before we get to that, let's talk about how Smokescreen determines who you are. By default, Smokescreen uses the client certificate from a TLS connection, extracts the common name of the certificate and uses that as the role. There is another mechanism documented for non-TLS connections using a header but doesn't seem to be actually wired up Smokescreen (probably because it's way too simple to present to be another system). So you'll have to use TLS CA certs for all the systems connecting through Smokescreen and that is an administrative pain.

Getting Basic

We wanted Smokescreen to be simpler to enable, and with Fly we have the advantage of supporting Secrets for all applications. Rather than repurposing TLS CAs to provide a name, we can store a secret with the Smokescreen proxy and with the app that sends requests to the outside world. That secret? For the example, we've gone with a PROXY_PASSWORD that we can distribute to all inside the Fly network.

Here's the Fly Github repository for the Fly Smokescreen.

I'm on the list...

In all cases, what Smokescreen does is turn the identity of an incoming request into a role. That role is then looked up in the acl.yaml file. Here's the Fly example ACL:

---
version: v1
services:
  - name: authed
    project: users
    action: report


default:
    project: other
    action: enforce

We've gone super simple on the roles here. There's one and that's authed. You're either authed or you fall through to default. The project field is there to make logging more meaningful by associating roles with projects.

The control of what happens with requests comes from the action field; this has three settings: open lets all traffic through, report lets all traffic through but logs the request if it's not on the list, and enforce only lets through traffic on the list. The list in this example isn't there, so report logs all requests and enforce blocks all requests.

Adding allowed-domains and a list of domains lets you fine tune these options. For a general purpose block-or-log egress proxy, this example is enough. Smokescreen has more ACL control options, including global allow and deny lists if you want to maintain simple but specfic rules but want to block a long list of sites.

Smokescreen inside

If you are interested in how this modified Smokescreen works, look in the main.go file. This is where the smokescreen code is loaded as a Go package. The program creates a new configuration for Smokescreen with a alternative RoleFromRequest function. It's this function that extracts the Proxy-Authorization password and checks it against the PROXY_PASSWORD environment variable. If it passes that test, it returns authed as a role. Otherwise, it returns an empty string, denoting no role. It's this function that you may want to customize to create your own mappings from username and password combinations to Smokescreen roles.

Deploy now

Fly

This is where we show how to deploy on Fly first:

fly init mysmokescreen --import source.fly.toml --org personal
fly set secret PROXY_PASSWORD="somesecret"
fly deploy

And that's it for Fly; there'll be a mysmokescreen app set up with Fly's internal private networking DNS (and external DNS if we needed that, which we don't here), and it'll be up and running. Turn on your Fly 6PN (Private Networking) VPN and test it with:

curl -U anyname:somesecret -x mysmokescreen.internal:4750 https://fly.io

And that will return the Fly homepage to you. Run fly logs and you'll see entries for the opening and closing of the proxy's connection to fly.io. What's neat with the Fly deployment is that with just two commands you can deploy the same application globally.

Docker - locally

If you're on another platform, you should be able to reuse the Dockerfile. Running locally, you just need to do:

docker build -t smokescreen .
docker run -p 4750:4750 --env PROXY_PASSWORD=somesecret smokescreen

And to test, in another session, do:

curl -U anyname:somesecret -x localhost:4750 https://fly.io

You'll see the log output appearing in the session where you did the docker run. We leave it as an exercise to readers to deploy the application to their own Cloud.

Using a Proxy from an app

To wrap up this article, we present two code examples, one in Go and one in Node, that take from the environment a PROXY_URL pointed at our smokescreen and a PROXY_PASSWORD for that smokescreen and issue a simple GET for an https: URL.

On Fly, the PROXY_URL can be as simple as "http://mysmokescreen.internal:4750/". Fly's 6PN network automatically maps deployed applications' names and instances into the the .internal TLD for DNS. On other platforms, you'll have to configure a hostname for your smokescreen and make sure you change it everywhere if you move your proxy.

Calling through an authenticated Proxy from Go

This example uses only the system libraries. There are no extra modules needed.

package main

import (
    "encoding/base64"
    "fmt"
    "io/ioutil"
    "log"
    "net/http"
    "net/url"
    "os"
)

func main() {
    // Set the proxy's URL
    proxy, ok := os.LookupEnv("PROXY_URL")
    if !ok {
        log.Fatal("Set PROXY_URL environment variable")
    }

    // And parse it.
    proxyURL, err := url.Parse(proxy)
    if err != nil {
        log.Fatal("The proxyURL is unparsable: " + proxy)
    }

    proxyPASS, ok := os.LookupEnv("PROXY_PASSWORD")

    if !ok {
        log.Fatal("Set PROXY_PASSWORD environment variable")
    }

    // Get you a transport that understands Proxies and Proxy authentication
    transport := &http.Transport{Proxy: http.ProxyURL(proxyURL)}

    // Create a usename:password string
    auth := "anyname:" + proxyPASS

    // Base64 that string with "Basic " prepended to it
    basicAuth := "Basic " + base64.StdEncoding.EncodeToString([]byte(auth))

    // Put a header into the proxy connect header
    transport.ProxyConnectHeader = http.Header{}

    // And then add in the Proxy-Authorization header with our auth string
    transport.ProxyConnectHeader.Add("Proxy-Authorization", basicAuth)

    // Now we are ready to get pages, just create HTTP clients which use the
    // Proxy transport.

    client := &http.Client{Transport: transport}

    rawURL := "https://fly.io"

    request, err := http.NewRequest("GET", rawURL, nil)
    if err != nil {
        fmt.Println(err)
        return
    }

    response, err := client.Do(request)
    if err != nil {
        fmt.Println(err)
        return
    }
    fmt.Println("Read ok")

    if response.StatusCode != 200 {
        fmt.Println(response.Status)
    }

    bs, err := ioutil.ReadAll(response.Body)

    fmt.Println(string(bs))

}

Calling through an authenticated Proxy from Node.js

This example uses the https-proxy-agent package.

var url = require('url');
var https = require('https');
var HttpsProxyAgent = require('https-proxy-agent');

// Create a URL for our proxy from the env var
var proxyOpts = new URL(process.env.PROXY_URL);

// Get a password from the environment too
var proxyPass= process.env.PROXY_PASSWORD;

// Inject a Proxy-Authentication header into the proxy using the password
proxyOpts.headers = {
  'Proxy-Authentication': 'Basic ' + (`anyname:${proxyPass}`).toString('base64')
};

// Create an HTTPS Proxy Agent with our accumulated options
var agent = new HttpsProxyAgent(proxyOpts);

var options = new URL("https://fly.io");

options.agent = agent;

https.get(options, function (res) {
  console.log('"response" event!', res.headers);
  res.pipe(process.stdout);
});

Smokescreen summarized

We've shown you examples of setting up a custom Smokescreen with password authentication. You'll find all the code for setting that up at the Fly Github repository for this Smokescreen. Have fun sanitizing your outgoing web requests.

Certificates for your Cloud backend

Fly.io — Tue, 24 Nov 2020 17:59:02 +0000

When you're building a cloud application, the last thing you want is to have to stop while you get yourself some certificates to secure your servers. It costs time and money and wouldn't you rather just print your own certificates for your own use? I mean, you trust you don't you? Assuming you do, then you can likely make use of mkcert when building out that cloud app.

If you know mkcert, you may not think of it as something for a cloud application's backend. Mkcert was originally developed with another problem in mind, local development. When putting together a web application to run in development on a local machine, it is literally impossible to get a certificate for it from a certificate authority - everyone has a localhost.

You can try and make your own self-signed certificates to get around that problem. That turns into a bit of a red flag carnival when you use them, with applications complaining about not being able to trust the certificates you've signed as they can't be checked with anyone. As the developers of mkcert say, "Managing your own CA is the best solution, but usually involves arcane commands, specialized knowledge, and manual steps."

Why mkcert

That's why they came up with mkcert which does all that arcane stuff for you, from making a root CA certificate to generating locally trusted certificates. And more importantly, it does it without the strange world of OpenSSL commands and error messages which can quite literally reduce people to tears.

Mkcert can get it all up and running and it'll talk to your local trust store and make your generated certificates trustable. And there are no weird commands or errors. It's great, but what's this got to do with certificates in the cloud.

Well, what mkcert does is also enough to enable TLS for many applications and servers, so you can turn to mkcert for some fast certificate provisioning for your cloud apps.

Applying mkcert

Let's show you how it works. Recently at Fly, we put together an example of a Redis installation with TLS enabled. To make it easy to reproduce and secure, we used mkcert.

The first step is to install mkcert. You'll find the instructions for that in it's README.

Once it's installed you run mkcert -install and that's where the magic happens. It creates a functioning local certificate authority (CA) with no intervention and installs that CA locally and, if Firefox is around, into Firefox. The only part of that we're interested in is the fact that it has created a CA because it's from that CA that we can extract the CA root file. This is the certificate that other applications can use to verify any certificates that this mkcert installation will generate.

Run mkcert -CAROOT to get the file name for the CA root file. Copy it somewhere - or combine it all into one command:

mkdir -p certs
cp "$(mkcert -CAROOT)/rootCA.pem" certs/rootCA.pem

You now have what is essentially a master key tester for your certificates. We'll be copying this file around our application's infrastructure and... will you stop panicking at the back there. First of all, this is a certificate that's been signed using another key so you can't make new certificates with it, only verify them.

Secondly, your application's infrastructure should be working like a sealed box, with only specific and auditable entry paths. Beyond common base images, it should be composed of assets and container images that you control access to.

Copying the CA Root file around inside a closed environment should not be an issue, and if it is, well, you have bigger problems to be concerned with. If you have external access to the system it should be secured by real, verifiable certificates anyway.

Cutting Server Keys

Anyway, now we have a CA Root file we can use to verify keys, we can get around to cutting some keys.

The first key we need in our Redis TLS example is one for the server, which has a hostname of appkata-redistls.fly.dev. Mkcert will generate a key and certificate for the server

mkcert -key-file certs/redis-server.key -cert-file certs/redis-server.crt appkata-redistls.fly.dev

That's the certificates done, all we need to do now is configure Redis with those certificates. The certs directory we created should be copied into /etc/ before this is done (as part of the image building) and with that in place we add:

tls-port 7379
tls-cert-file /etc/certs/redis-server.crt
tls-key-file /etc/certs/redis-server.key
tls-ca-cert-file /etc/certs/rootCA.pem
tls-auth-clients no

To the redis.conf file. That's very Redis specific, but in more generic terms, you pass the cert, key, and ca-cert file to the application's TLS configuration; either through a config file or through command-line flags.

When the server is brought online, it'll use those certificates and that CA root as a basis for validating the certificate. Of course, as no connecting client will know how to verify the locally generated certificates, you'll have to give the client a copy of the rootCA.pem file so it can do that for itself. Something like this:

redis-cli -h appkata-redistls.fly.dev \
           --tls \
          --cacert certs/rootCA.pem

An incoming client, equipped with that file, will be able to connect over TLS and authenticate with a password if you've set one. But all this does is encrypt the connection.

Keys for Clients

We can tighten up security further if we ask the server to demand that anyone connecting has a valid client certificate. The first step in that is turning that requirement on. Over to the redis.conf file and let's flip the tls-auth-clients line to read:

tls-auth-clients yes

Now you can't connect to the server - with Redis, you can connect but you'll be disconnected when you try any command. What we need now are some client certificates, ones that say "This client is allowed to access this named server". Again mkcert makes it simple. Just add --client to the mkcert and it will make client certificates.

mkcert --client -key-file redis-client.key -cert-file redis-client.crt appkata-redistls.fly.dev

Now, you can go to a remote system, with these files, redis-client.key, redis-client.crt and the rootCA.crt file and use them to connect as application such as redis-cli:

redis-cli -h redis-example.fly.dev \
          --cert redis-client.crt \
          --key redis-client.key \
          --tls \
          --cacert certs/rootCA.pem \

Summary

We've gone over the processes involved in creating keys and certificates for servers and client and setting up a Redis server, without and with client authentication. It's important to know that these mkcert certificates we've generated have a limited life span, at which point you'll have to manually generate and install them again. So while they are great for your development processes, you'll want something more automated when it comes to production. Which we'll come to in a future article.

Deno on Fly using Buildpacks

Fly.io — Tue, 05 May 2020 15:09:41 +0000

We really like Deno at Fly.io. It really is a better Node and we’d love people to build more with it. Our plan to aid in the adoption of Deno? “Let's make building and deploying Deno apps on Fly as simple as other languages”. Now, you can configure, build and deploy Deno code in just two commands, and we’d like to show you how we did it.

What’s Deno?

The deno.land website bills Deno as a secure runtime for JavaScript and TypeScript. In practice, it’s like working with a streamlined Node where TypeScript is the norm, with a solid Rust foundation. It also incorporates many of the lessons learned over Node’s development and growth.

As we write this, Deno is heading rapidly towards its first 1.0 release candidate and we are really looking forward to a post-1.0 Deno. We believe as more people discover Deno’s elegance as a platform it’ll become much more popular. So how do you build a Deno app for Fly now?

Straight to the chase

Name your app’s entry point server.ts
Create your Fly app with flyctl apps create --builder flyio/builder
flyctl deploy and watch it all build and deploy.

That’s it.

Behind the builder

The “chase” above was notably short. How? Let's look at what’s powering this build process. Fly’s builder support landed in early April and is based on the Cloud Native Buildpacks.io (CNB) specifications and implementations for building cloud native container images.

A Cloud Native build is made up of stacks, builders and buildpacks which come together to make a repeatable build process, for different languages and frameworks. Used together, they deliver container images that are ready to run. For the Deno builder, we created our own stack, builder and buildpack to build Deno.

Stacks

Everything in CNB is built on a “stack”, the base operating system in which the builders operate. For the Fly stack, we selected Ubuntu bionic, added in curl and unzip utilities (to support the Deno installer) and built the three images - base, build and run - which will be used in the subsequent steps.

Builders

A builder in CNB is a container loaded with all the OS level tooling needed to construct an image. It’s fundamentally an OS image of some form. It’s not tied to any specific container platform such as Docker, it’s designed to run wherever you can run a container.

That said, the Buildpacks.io developers do have the pack commmand that uses Docker to run these containers - especially useful on Windows and macOS where there is no native Linux container support.

The builder brings up a container running the build stack and then steps through the buildpacks associated with the builder to work out which build script to run in that container.

Buildpacks

The buildpacks are smaller bundles of scripts (or Go programs) which have two jobs, detect and build.

detect

The detect script tries to work out if the directory contents it has been given are appropriate to build with its build script. For example, a detect script for Ruby would look for a Gemfile and go “aha! this is the song of my people! run my build script”.

Now, Deno doesn't have an obvious packaging file like Ruby or well, anything else, due to it being so self-contained. So, we made a rule. If there’s a server.ts file in the directory, we’ll assume it's a Deno application and signal to run our build script.

build

The build script runs in the Builder’s container and does the work of assembling tools and building the layers to create our image. In the case of our Deno buildpack, that includes downloading Deno into the image and running the Deno deno cache command so all the dependencies are ready to run. Finally, it writes out a set of launch commands to start up the application.

permissions

Now, one thing Deno does is restrict access to resources by default. You have to specify which resources are available to any program with --allow-* command-line flags. Of course this is hard to pass through the command chain, so we don’t. What we do is add a .permissions file, containing the command-line arguments you’d expect to be added to the command-line, to the directory with the code we’re building. In the example app, this file contains:

--allow-env --allow-net

Allowing environment variable and network access. If there’s no .permissions file, currently we default to no permission flags, in sync with the default Deno experience. You can incrementally add appropriate permissions to your app as you iterate your code.

If you want to allow all access, you can put -A into .permissions, allow all access and sort out permissions later (well, that’s what you’ll say but you know it’ll slip and you’ll do a production deployment with it still in place.

The arguments are added to the deno command when the container and its launch file are built.

Step by Step

Creating an app

So, let’s build a Deno app. We’re going to use dinatra, a Deno module which gives Sinatra-like capabilities to Deno apps. You’ll of course want to install Deno and create a directory for your app. Then make a server.ts file and put this in it:

import {
    app,
    get,
    post,
    redirect,
    contentType,
  } from "https://denopkg.com/syumai/dinatra@0.11.0/mod.ts";

  app(
    get("/hello", () => `Hello`),
    get("/hello/:id", ({ params }) => `Hello to ${params.id}`)
  );

And that’s an entire simple web server application. Don’t forget permissions though: create a .permissions file with --allow-net in it; all this application wants is network access.

Extra steps

You may want to test your app before deploying it. You have two options.

The first is to just run it before packaging it into a container image.

Running the deno command with the permissions:

deno --allow-net server.ts

will be all you need in that case

The second is to package up the container image and run that image. You’ll need Docker and Buildpacks.io's pack installed locally to do this. Use pack to create the image:

pack build test-server-app --builder flyio/builder

Then use Docker to run the test-server-app image:

docker run -p 8080:8080 test-server-app

Deploying to Fly

Now we can create a Fly app for this code by running

flyctl apps create --builder flyio/builder

And we can deploy it with flyctl deploy

What Next

For Fly Buildpacks

We’ve made all the Buildpack code available in a GitHub repository for anyone who wants to improve the process. We’ll be looking to add new buildpacks to it too, and enhance existing buildpacks.

It’s worth noting that you can build your own local buildpack and use it with the fly-builder stack for local/test builds.

For Deno on Fly

We’re ready for your next Deno app on Fly, even if it’s your first. With a simple build and deploy process, it's easier than ever.

Making Datasets Fly with Datasette and Fly

Fly.io — Mon, 30 Mar 2020 14:20:21 +0000

The creator of Datasette, the tool for Dataset sharing and exploration, has added Fly to the platforms you can use to publish and share data. We take a look at Datasette and show how well it works with Fly.

I've always liked finding a good dataset. With a background in databases and writing, I know a good dataset can bring a demo to life, be it a census of Squirrels in Central Park or a survey of grocery purchases in London. Datasets can also provide valuable foundations for citizen journalism and, under analysis, provide insights.

The key to making these datasets work for people is making them accessible and available, which is where Datasette comes in. It's a project by Simon Willison designed to make sharing datasets easy. It all hinges on SQLite - essentially, you load up an SQLite database with data and then hand it to Datasette which presents it through a web site, to the world.

First, the data!

I'm going to use the New York Central Park Squirrel Census Data for my data source because squirrels rock. Go there and click the View Data button to see the data presented in the very fine NYC OpenData viewer.

I want the raw data though so I'll click on Export and select CSV which will kick off an immediate download. We now have a 2018_Central_Park_Squirrel_Census_-_Squirrel_Data.csv file to work with.

Making an Sqlite database

Ensure you have SQLite3 installed on your system; it'll already be there on a macOS system as it is heavily used throughout the OS. If you run sqlite3 filename.db the data will be persisted to that file rather than just held in memory, so let's begin.

❯ sqlite3 squirrels.db
SQLite version 3.28.0 2019-04-15 14:49:49
Enter ".help" for usage hints.
sqlite>

And the first thing we need to do is set the mode to CSV. If you look at the documentation, you'll see the .mode command listed as setting the output mode. That's not quite completely true, it's also used as a hint to the import command.

sqlite> .mode csv

Now we are ready to import with .import. This command takes the CSV filename and a table name to import into. If the table isn't there, it'll use the first row in the CSV file to create the table columns and then import. (ProTip: If the table already exists it just imports everything, including the header row so always drop the table first).

sqlite> .import 2018_Central_Park_Squirrel_Census_-_Squirrel_Data.csv squirrels

And we can do a quick check on what actually got imported with the .schema command.

sqlite> .schema squirrels
CREATE TABLE squirrels(
  "X" TEXT,
  "Y" TEXT,
  "Unique Squirrel ID" TEXT,
  "Hectare" TEXT,
  "Shift" TEXT,
  "Date" TEXT,
  "Hectare Squirrel Number" TEXT,
 ...
  "Lat/Long" TEXT,
  "Zip Codes" TEXT,
  "Community Districts" TEXT,
  "Borough Boundaries" TEXT,
  "City Council Districts" TEXT,
  "Police Precincts" TEXT
);

There are a lot of columns about squirrels. Now I can exit sqlite3 and get ready to apply Datasette to the database.

Installing Datasette

There are a couple of ways to install and run Datasette. I'm going to go with the one that is simplest for most developers:

make sure Python3 is installed (brew install python3 on macOS using HomeBrew)
pip3 install datasette - You may get an error on macOS as it stops the file being copied to a system directory. Don't worry, just repeat the command with --user on the end and add the directory it suggests into your path.

And we're ready to test running datasette squirrels.db:

❯ datasette squirrels.db
Serve! files=('squirrels.db',) (immutables=()) on port 8001
INFO:     Started server process [2812]
INFO:     Waiting for application startup.
INFO:     Application startup complete.
INFO:     Uvicorn running on http://127.0.0.1:8001 (Press CTRL+C to quit)

Navigating to http://127.0.0.1:8001 and clicking on the Squirrel table, we should see the squirrels data available:

And clicking through you can start using Datasette's UI to compose views of particular facets of the data, write your own SQL queries, export data as JSON or CSV or access the data through a REST/JSON API.

The next stop is making it available to the world.

Publishing to Fly

While other platforms are already built into Datasette, the Fly publishing element of Datasette is a new plugin, created by Datasette's developer. That means it has to be installed separately with pip3 install datasette-publish-fly

With that installed, you can run

datasette publish fly squirrels.db --app squirrels

The --app flag lets you set the app name and it will be rejected if it clashes with an existing app. You may, if you are in multiple organizations, be asked to pick one of those too. Once you've done that, the publish command takes over, builds an image and deploys it onto the Fly platform. If you want to know what IP address and hostname the app is on, run flyctl info -a <appname> like so:

❯ flyctl info -a squirrels

App
  Name     = squirrels
  Owner    = dj
  Version  = 0
  Status   = running
  Hostname = squirrels.fly.dev

Services
  PROTOCOL   PORTS
  TCP        80 => 8080 [HTTP]
             443 => 8080 [TLS, HTTP]

IP Addresses
  TYPE   ADDRESS                                CREATED AT
  v4     77.83.142.59                           23m21s ago
  v6     2a09:8280:1:7bc8:bf19:7779:aef7:8f18   23m21s ago

And that also tells us where we need to browse: squirrels.fly.dev. We're online and we can dig down into a table view where we can compose queries.

Deploying with Plugins

Datasette Plugins aren't just for publishing; there are a whole range of additional capabilities waiting to be slotted in. Take datasette-cluster-map, for example. It looks for latitude and longitude columns in the data and turns the data into an interactive map using them. Let's see how we use this with Fly.

Tuning the tables

The Squirrel data has X and Y coordinates which match Longitude and Latitude; we'll need to rename those columns first. Now, for a long time, sqlite lacked the ability to rename columns, so you'll find many workarounds on the web if you search. The good news is, though, that since 2018 you have been able to rename columns with the alter table rename column command. So I'll just load up the sqlite3 database and alter those columns:

❯ sqlite3 squirrels.db
SQLite version 3.28.0 2019-04-15 14:49:49
Enter ".help" for usage hints.
sqlite> alter table squirrels rename column X to Longitude;
sqlite> alter table squirrels rename column Y to Latitude;
sqlite> .schema squirrels
CREATE TABLE squirrels(
  "Longitude" TEXT,
  "Latitude" TEXT,
...
sqlite> .exit
❯

Run Locally Again

I now need to install that plugin with pip3:

❯ pip3 install datasette-cluster-map

And run datasette locally:

❯ datasette squirrels.db
Serve! files=('squirrels.db',) (immutables=()) on port 8001
INFO:     Started server process [39385]
INFO:     Waiting for application startup.
INFO:     Application startup complete.
INFO:     Uvicorn running on http://127.0.0.1:8001 (Press CTRL+C to quit)

And if we browse to that http://localhost:8001/ we'll see the index page as we did before. Now to deploy to Fly.

Deploying and Plugins

The difference here from when we previously published to Fly is that we have to list the plugins we need installed with our Datasette. The --install flag takes care of that, so now I can publish to Fly with:

datasette publish fly squirrels.db --app squirrels-mapped --install datasette-cluster-map

And that will include the cluster-map plugin. If I now browse to "https://squirrels-mapped.fly.dev", and click in on the squirrels table:

Our view of the Squirrels data now includes a cluster map over Central Park that we can click in on and get a closer view. When sightings resolve to a single squirrel, you can hover over it to get all the details.

Datasette and Fly

So what does Fly add to Datasette? Well, as well as being super simple to deploy, you may have noticed that all the connections we're making are HTTPS secured, with Let's Encrypt certificates being generated automatically. If you want more, it's simple to use your own custom domain or hostname with your Fly/Datasette deployment. You can also deploy all around the world so your dataset is available where people need it to be. And there's also the edge networking/SSL termination which makes interaction that bit snappier. There are a whole lot more to explore in Datasette - check out the documentation - and it's a great way to discover how you can make your applications Fly.

Thanks to Simon Willison, not only for Datasette and the Fly plugin, but also for his feedback on this article. And to the Squirrels of New York's Central Park for taking part in the census. Want to learn more about Fly? Head over to our Fly Docs for lots more, including a Hands On where you can get a free account and deploy your first app today.