DEV Community: StackPath

Pwnd Password Checking on the Edge

Karl L. Hughes — Mon, 03 Aug 2020 18:06:33 +0000

One of the most effective ways to secure your users’ data is to ensure that they select a strong password. Unfortunately, putting the burden on your users rarely works:

“The bad news is that most users don't pick strong passwords. This has been proven time and time again...Even worse, most users re-use these bad passwords across multiple websites.” - Jeff Atwood, Founder of StackOverflow

Many application developers implement password rules that require a particular password length, a combination of characters, or various forms of capitalization to address this issue. Some of these commonly-used password rules are not that helpful, but long, hard to guess passwords are generally good for security. Checking password length is easy enough, but how can you ensure that the user’s password is hard to guess?

One method is to check if they’re using a password that’s been compromised before.

If you’ve worked in web development for long, you may have heard of the NIST security guidelines. In their Digital Identity Guidelines, they specify:

“When processing requests to establish and change memorized secrets, verifiers SHALL compare the prospective secrets against a list that contains values known to be commonly-used, expected, or compromised. For example, the list MAY include, but is not limited to: Passwords obtained from previous breach corpuses.” - NIST Digital Identity Guidelines

When users set their passwords, NIST recommends that you ensure they don’t use a password previously exposed in a data breach. While you might hear about some large data breaches on the news, there’s no way you can keep up with all the data breaches happening around the world. If you tried, you’d never have time to get any coding done!

Have I Been Pwned?
A few years ago, Troy Hunt - a well-known security researcher - set up a website to address this very issue. Users can enter their password on HaveIBeenPwned.com to see if it’s been part of a data breach and, therefore, not safe to use.

Unfortunately, this isn’t something most people do, but Have I Been Pwned also has an API. Using the REST API, you can check that a user signing up for your application does not use a compromised password before you let them register. When used in addition to requiring a long, sufficiently random password, this can significantly increase your platform’s data security and minimize brute force password attacks.

Checking for Pwned Passwords on the Edge
If you’re using an authentication service, you will need to figure out the best way to implement password strength checking into your signup flow. While you could modify your authentication flow, you can also host your pwned password checking service on the edge with StackPath.

Enforcing strict password rules will annoy some users, but you can minimize this annoyance by making responses as fast as possible. In this tutorial, you’ll see how to use StackPath’s edge hosting and the Pwned Password API to ensure that users are signing up with uncompromised passwords, and you'll see how edge hosting can offer a 14x performance improvement over traditional hosting when caching responses.

Creating the Node Application

To demonstrate the use of the Pwned Password API on StackPath, I’ve created an Express application in a Docker image. This project's code is available on Github if you’d like to follow along. It uses the same Dockerfile and Express server outlined here, so you may want to start by reading that tutorial if you’re not familiar with Node or Docker.

Signup requests from a web browser will be sent to the Node application. When the client posts a password, the Node app will call the Pwned Password API. The whole flow can be visualized here:

The Pwned Password API takes the first five characters of a SHA1 hash of the password and returns a list of hashed password suffixes to the Node application. This use of a partial hash minimizes any risk in posting secure data to a third-party service.

If you want to see the raw results that the Pwned Password API returns, visit https://api.pwnedpasswords.com/range/21BD1 in your browser. You will see a list of hashes followed by the number of times they’ve been seen in plain text on the internet:

0018A45C4D1DEF81644B54AB7F969B88D65:1
00D4F6E8FA6EECAD2A3AA415EEC418D38EC:2
011053FD0102E94D6AE2F8B83D76FAF94F6:1
...

In order to create the partial hash, prefixes, and suffixes in Node, you can use the sha1 library and built-in substring function:

// Shah-1 hash the password and break it into a prefix and suffix
const hashedPassword = sha1(password).toUpperCase();
const prefix = hashedPassword.substring(0, 5);
const suffix = hashedPassword.substring(5);

Next, call the Pwned Password API using the prefix and check for the presence of suffix in the response. In this case, I’m using the node-fetch library, but you can use whichever HTTP request package you prefer.

Because performance is one of the primary reasons to use edge hosting for password validation, you can cache the response from the Pwned Password API. While caching hashed passwords would normally have some security implications, these are pubicly available compromised password hashes, so it's less of a concern.

Here's the code that handles caching and checking against the API's results:

// Check against cache
const cachedResults = hashedPasswordCache.get(prefix);
if (cachedResults) {
  // Check if the cached result includes the suffix of the hash
  return generateResponse(res, cachedResults, suffix);
} else {
  // Send the first 5 chars to pwned password API
  fetch('https://api.pwnedpasswords.com/range/' + prefix)
    .then(tmp => tmp.text())
    .then(body => {
      // Cache the response for future use
      hashedPasswordCache.set(prefix, body);
      // Check if the response includes the suffix of the hash
      return generateResponse(res, body, suffix);
    }).catch(err => {
    // Handle any API errors
    console.error(err);
    res.status(400).json({message: 'Something went wrong.'});
  });
}

This demo lets the user know if their password has been pwned or not, but in production, you probably want to pass valid usernames and passwords along to your authentication service to create the user’s account and log them in.

Next, you need to build your application as a Docker image. Open your terminal and navigate to the repository. Build the Dockerfile using your Docker Hub username:

docker build -t <YOUR_USERNAME>/stackpath-pwned .

And push the image to Docker Hub:

docker push <YOUR_USERNAME>/stackpath-pwned:latest

Now your image is publicly available. In the next step, you'll deploy the application to the edge using StackPath.

Deploying the Application

Once you’ve created the Node application that calls the Pwned Password API and responds appropriately, you can deploy it to StackPath. StackPath’s edge hosting network runs your Docker containers on servers worldwide, so you get the fastest response times possible.

First, create a new Workload and enter the Docker image you just pushed to Docker Hub above:

You don’t need any environment variables for this project, but you should check the box to “Add Anycast IP Address” and make sure port 80 is open.

Next, choose the PoPs (points of presence) where you want your application to be hosted. I wanted to spread my nodes across the world for benchmarking, but you should select locations according to your use case. I used:

Chicago
San Jose
New York
London
Singapore

Within a couple of minutes, your StackPath application will be running, and you'll be able to access it from the Anycast IP address shown on your Workload's Overview.

Benchmarking Against Traditional Hosting

Generally, the closer your code is to your users, the faster it will run. To demonstrate the performance benefits that StackPath offers over traditional hosting in the case of pwned password checking, I set up a DigitalOcean droplet in the San Francisco region with similar specs to the StackPath Workload (2GB, 1vCPU).

I used a response timer Docker image to test the application's speed in four locations around the world. The initial uncached response times are pretty similar because in both hosting environments, a request must be made to the Pwned Password API. But, when the result is cached and an API call isn't necessary, StackPath provides significantly faster responses.

On average, responses from StackPath were almost 14x faster than those from a traditional web server.

Conclusions

Passwords are just one piece in the quest for application security, but they’re a cornerstone of implementing strong security practices. By enforcing strong passwords and requiring users to select passwords that haven’t been compromised, you can lower their accounts' risk. To combat any delay in checking passwords, you can utilize edge hosting with StackPath, as demonstrated above.

API Consolidation and Load Balancing at the Edge with Traefik and StackPath

Tomas Fernandez — Mon, 03 Aug 2020 15:58:30 +0000

Traefik is an open source edge router for the cloud. Like a hostess at a restaurant that meets incoming guests and shows them to their tables, traefik intercepts requests at the edge of the network and relays them to their services.

In this tutorial, we'll learn how to install and configure highly available, low-latency API gateways with traefik on StackPath's edge computing network.

API Gateway Setup

In the course of this tutorial, we'll deploy the following infrastructure:

Gateways: one or more traefik gateways distributed over different PoPs
Config DB: a single database container with the gateway's routes
Services: some API servers to test the routes

Step 1 - Prepare the API Servers

To learn how traefik works, we’ll deploy some test APIs in StackPath. You can replace these services with other containers or VMs. Or, if you already have APIs running elsewhere, you can route traffic to them from StackPath.

To get started, log in or create a StackPath account. The first time you log in to StackPath, you’ll be asked to create a Stack. Stacks are like projects, a group of deployed resources and services. In this tutorial, we’ll focus on deploying edge computing services.

After logging in, click Workloads > Create Workload in the left navigation menu.

Set the workload type to Container. Then, type the name of the Docker image that implements your API service. As an example, we can use containous/whoami, a service that always responds with its IP address and hostname.

Click Continue to Settings.

In the Settings section, you can define environment variables and startup commands for the container. You can also expose ports to the internet.

Unless you need direct access to your containers, avoid opening any ports in this section. We should route all public connections from the gateway.

Click Continue to Spec.

Select a machine in Spec and set the name of the Deployment Target. Deployment targets are used to configure auto-scaling.

Choose one PoP and set the number of instances to two.

Click Create Workload and wait a few seconds until your containers are running.

Step 2 - Install Etcd

Traefik has two types of settings: static and dynamic, with the former being activated when the gateway boots up and the latter taking effect without interruption. Dynamic settings are stored in distributed databases like Consul, ZooKeeper, or etcd.

Here we’ll use etcd, an easy-to-use key-value database for the cloud.

Create a second workload on StackPath. Like you did in Step 1, go to Workloads > Create Workload. Set the name of the workload to “my-api-config”. Then, select Container in Workload Type, and type quay.io/coreos/etcd:v3.4.0 in Image.

Click Continue to Settings.

You don’t have to open any ports here as all communication with etcd can take place over the private network.

We’ll configure etcd with Environment Variables. There are three settings we’re interested in.

ETCD_DATA_DIR - Tells etcd where the database is located. We’ll point it to a persistent mount to preserve the data across reboots and upgrades. Set it to /etcd-data.

ETCD_LISTEN_CLIENT_URLS - Defines the ports and interfaces that etcd binds to. Set it to http://0.0.0.0:2379.

ETCD_ADVERTISE_CLIENT_URLS - This is the DNS name or IP address the clients use to connect with etcd. StackPath provides a free internal Discovery DNS service that simplifies this step enormously.

Discovery DNS

StackPath's Discovery DNS service helps you connect your containers and VMs using predictable names instead of IP addresses. DNS entries are automatically created when you update workloads. You can use hostnames, workload names, ports, and SRV records to define your network.

For instance, we can find all healthy containers in a given workload following this formula:

WORKLOAD_NAME.STACK_NAME.edgeengine.internal

So, if the workload name for etcd is my-api-config and the stack is called my-default-stack, then its FQDN (Fully Qualified Domain Name) is my-api-config.my-default-stack.edgeengine.internal.

In this tutorial, we’ll only deploy a standalone etcd instance. Later, you might decide to upgrade it to a cluster. If you follow this pattern, the system naturally grows as you add new instances.

Set this value to http://my-api-config.YOUR_STACK_NAME.edgeengine.internal:2379.

Click Continue to Spec.

On Additional Volume set the mount path to /etcd-data and set the volume size.

Set the Deployment target and click Create Workload.

Click Overview in the left menu and scroll down to the Manage Your Instances section and enable Remote Management.

Click on your etcd instance and scroll down to the Instance Details section.

Click on the Run & Open Terminal button next to Remote Management. A terminal window opens.

Define the username and password environment variables. We’ll need to repeat the export commands each time we connect to the etcd instance.

export ETCDCTL_USER=root
export ETCDCTL_PASSWORD=TYPE_YOUR_ETCD_PASSWORD

Create a new username and enable authentication.

etcdctl user add "$ETCDCTL_USER":"$ETCDCTL_PASSWORD"
etcdctl user grant-role root $ETCDCTL_USER
etcdctl auth enable

Use the following command to get etcd ready for the traefik configuration:

etcdctl put traefik -- true

OK

Leave the terminal window open. We’ll use it to configure Traefik next.

Step 3 - Deploy Traefik

Traefik gateways can issue certificates, buffer and filter requests, and authenticate users right on the edge of your network where it makes the most sense. In this step, we’ll create a third workload with the traefik containers.

Create a new workload in StackPath. Select Container in Workload Type. On Image, type traefik:2.2.

Click Continue to Settings and enable the Add Anycast IP option so users are always routed to the closest gateway. Then, on Public Ports, add ports 80 (HTTP) and 443 (HTTPS). Traefik also offers an optional dashboard on port 8080. The dashboard shows all sorts of interesting information so you may want to open that port too.

Next, we need to tell Traefik where to find the configuration database. Use the + buttons in Commands to add the following lines—one line per command. Fill in your etcd DNS and password.

traefik
--providers.etcd.endpoints=my-api-config.YOUR_STACK_NAME.edgeengine.internal:2379
--providers.etcd.username=root
--providers.etcd.password=YOUR_ETCD_PASSWORD

If you wish to enable the dashboard, also add these commands:

--api
--api.insecure=true

Click Continue to Spec

On Deployment Target, set the name of the deployment and choose as many PoPs and instances as you need. There is no limit to the number of instances and PoPs you can deploy. Prioritize locations that are close to you and your users to reduce latency.

Depending on the load you expect, two or three nodes may be enough. Or perhaps you find that you need 20 or 30 nodes spread all over the world. The good news is that you can scale it up and down at the click of a button—in seconds, without interruptions.

Click Create Workload

Click Overview, scroll down, and copy the gateway’s anycast IP.

Step 4 - Configure Routes and Services

In this section we’ll try out two common gateway scenarios: load balancer and API consolidation.

Traefik uses routers to match incoming requests with their destinations. Routers check for specific patterns in headers, hostnames, paths, or query strings, and forwards them to the corresponding service. Replies are sent back to the client as if they had originated from the gateway.

Load Balancer Scenario

A load balancer distributes incoming connections among two or more API servers.

Traefik uses a tree to represent the routers on the configuration database. To define the routes, we only have to create the relevant nodes.

traefik
└── http
    ├── middlewares
    ├── routers
    └── services

First, we need to find out the DNS names of the API services we plan to proxy. On StackPath, go to the test service workloads you created in Step 1 and take note of their hostnames.

Here we benefit from StackPath’s DNS discovery service. We can target any particular instance as HOSTNAME.STACK_NAME.edgeengine.internal.

So, in this case, the FQDN for the services are:

my-api-services-my-api-server-group-sea-0.my-default-stack.edgeengine.internal
my-api-services-my-api-server-group-sea-1.my-default-stack.edgeengine.internal

Note: If you are proxying API services deployed outside of StackPath, the DNS resolver tactic won’t work. You’ll need to provide for DNS resolution or IP addresses yourself.

Go back to the etcd terminal window and type the following commands to define a load balancer service called whoami:

etcdctl put traefik/http/services/whoami/loadbalancer/servers/0/url -- http://YOUR_API_SERVICE_1_HOSTNAME
etcdctl put traefik/http/services/whoami/loadbalancer/servers/1/url -- http://YOUR_API_SERVICE_2_HOSTNAME

Next, we have to set the rules that send incoming traffic into the load balancer. This router matches requests to the /whoami path (for example, example.com/whoami).

etcdctl put traefik/http/routers/whoami/rule -- 'Path(`/whoami`)'
etcdctl put traefik/http/routers/whoami/service -- whoami

Once configured, you can check the new route on the dashboard (if you have enabled it).

To try the load balancer, you can use curl or a browser.

curl -w "\n" http://GATEWAY_ANYCAST_IP/whoami

You can then test how the load balancer is working by making two consecutive requests to the gateway’s anycast IP.

Hostname: my-api-services-my-api-server-group-sea-0
IP: 10.128.144.3
[...]

On the second request, you should get a different API service IP.

Hostname: my-api-services-my-api-server-group-sea-1
IP: 10.128.144.4
[..]

API Consolidation Scenario

API consolidation is another everyday scenario for a gateway. Traefik allows developers to solidify all their API services under a single endpoint. In this example, we’ll learn how to merge two APIs services.

Imagine we have two different APIs, foo and bar, which we want to present as a single entity. The easiest way to achieve this is by using different paths:

example.com/foo is relayed to the foo service
example.com/bar is relayed to the bar service.

First, create both services in the database.

etcdctl put traefik/http/services/foo/loadbalancer/servers/0/url -- http://YOUR_API_SERVICE_1_HOSTNAME
etcdctl put traefik/http/services/bar/loadbalancer/servers/0/url -- http://YOUR_API_SERVICE_2_HOSTNAME

Since the foo and bar services do not know about the gateway, we have to remove /foo and /bar from the requests. While the requests are passing through the gateway, we can rewrite them using middlewares.

Create a StripPrefix middleware to remove the extra paths.

etcdctl put traefik/http/middlewares/strip-foobar/stripPrefix/prefixes/0 -- /foo
etcdctl put traefik/http/middlewares/strip-foobar/stripPrefix/prefixes/1 -- /bar

Finally, create two routes to match /foo and /bar.

etcdctl put traefik/http/routers/foo/service -- foo
etcdctl put traefik/http/routers/foo/middlewares/0 -- strip-foobar
etcdctl put traefik/http/routers/foo/rule -- 'Path(`/foo`)'

etcdctl put traefik/http/routers/bar/service -- bar
etcdctl put traefik/http/routers/bar/middlewares/0 -- strip-foobar
etcdctl put traefik/http/routers/bar/rule -- 'Path(`/bar`)'

The dashboard now shows:

Once again, you can try the gateway. The /foo path should respond with one IP.

curl GATEWAY_ANYCAST_IP/foo

Hostname: my-api-services-my-api-server-group-sea-0
IP: 10.128.144.3
[..]

While the /bar path should always return the other IP:

curl GATEWAY_ANYCAST_IP/bar

Hostname: my-api-services-my-api-server-group-sea-1
IP: 10.128.144.4
[..]

Step 5 - Add More Middlewares

In addition to StripPrefix, traefik ships with many other middlewares to filter, buffer, and authenticate requests. For instance, we can try RateLimit, which prevents abuse of the APIs and mitigates Denial of Service (DoS) attacks.

First, configure the middleware with the following commands.

etcdctl put traefik/http/middlewares/ratelimit-foobar/rateLimit/average -- 10
etcdctl put traefik/http/middlewares/ratelimit-foobar/rateLimit/burst -- 20
etcdctl put traefik/http/middlewares/ratelimit-foobar/rateLimit/period -- 1

Then, add it to the foo and bar routers.

etcdctl put traefik/http/routers/foo/middlewares/1 -- ratelimit-foobar
etcdctl put traefik/http/routers/bar/middlewares/1 -- ratelimit-foobar

And that's it. Clients exceeding 10 requests per second on average or a burst of 20 requests will get a 429 error code (too many requests).

Check the full list of middlewares here. And, as an exercise, try combining the CircuitBreaker and retry middlewares to keep track of service health and retry failed requests.

Step 6 - Securing the Gateway

Traefik has many options to secure the gateway. Before using it in production, check the following settings.

Dashboard

The dashboard shows potentially sensitive information about your architecture. If you don't need it, you can disable it by removing the --api and --api.insecure startup options in the container workload.

If you plan to keep using the dashboard, ensure that it’s adequately secured.

Use StackPath network policies to control what IPs can access the dashboard.
Add a password by setting up one of the authentication methods described here.

Authentication

Traefik ships with several authentication middlewares. If your API service requires users to authenticate, you can set up one of the following middlewares.

BasicAuth uses basic HTTP authentication to control access of known users. Usernames and passwords are stored encrypted on the dynamic configuration.
DigestAuth authenticates using HTTP digests. The user credentials are stored on the dynamic configuration.
ForwardAuth controls access permissions using a custom-defined external API service.

SSL/TLS

Traefik can act as a TLS terminator. If your API servers are in a private network you may not need HTTPS transport between them and the gateway. In such scenarios, traefik can offload the encryption workload.

Before using the setup in production, you should activate TLS on your gateway. There are two methods for setting up certificates.

Standard certificates: you can add a disk volume on your traefik container to store the certificates.
Let's Encrypt: you can use Let's Encrypt to get free and automated certificates. This setting is static and must be enabled when traefik starts up.

Once configured, you can add the RedirectScheme middleware to redirect all incoming HTTP to HTTPS.

Next Steps

In this tutorial, we have learned how to get started with traefik on StackPath by deploying some conventional gateway scenarios.

While we focused only on HTTP, traefik can also work with TCP and UDP traffic, which adds the possibility of routing content related to streaming, conferencing, and gaming.

Besides routing, traefik adds observability to your APIs.

As a next step, learn how to use the metrics and tracing integrations to monitor the network

Cover image source: https://joshuaavalon.io/setup-traefik-step-by-step

How to Deploy a FoundationDB Cluster at the Edge with StackPath

Tomas Fernandez — Wed, 29 Apr 2020 19:10:19 +0000

This tutorial was originally published in StackPath Blog.

FoundationDB is an open-source distributed NoSQL database released by Apple under the Apache 2.0 license. It is non-relational, so instead of using tables it organizes data as a set of ordered key-values. It can also handle massive volumes of data and is exceptionally well suited for write-intensive workloads.

FoundationDB has been designed from the ground up for distributed computing and, to get the most out of FoundationDB's performance features, you need to put the database cluster as close as possible to where it is needed. To do this, you can leverage StackPath's low-latency edge compute VMs to run FoundationDB clusters.

We'll show you how in the tutorial below. But first, let's review what makes FoundationDB great and determine if it's a good fit for your project.

FoundationDB: A Distributed, Multi-Model Database

As the name implies, FoundationDB focuses on working with simple data types such as strings, integers, floating-point numbers, and arrays. Higher-order types are available via installable Layers.

Currently, there are two official Layers:

Record Layer: a Java library that offers structured records and additional data types
Document Layer: a layer that offers a document-oriented MongoDB compatible API

Core features

Horizontal scaling: as we add more nodes to the cluster, the performance of the system increases proportionally
Fault tolerance: the database remains available even if some nodes in the cluster go down
Coordinated transactions: FoundationDB automatically routes transactions inside the cluster
Strong consistency: unlike other NoSQL databases that compromise on consistency (e.g. eventual consistency), FoundationDB uses strict serialization, the most robust consistency model available
Non-blocking: FoundationDB uses Multi-Version Concurrency Control (MVCC) for consistency; performance is not affected by slow clients and deadlocks are impossible
Self-tuning: FoundationDB automatically optimizes its settings for best performance
Network aware topology: FoundationDB is data center- and region-aware, using network information for tuning and fault tolerance.

Is FoundationDB right for your project?

The scalable nature of FoundationDB makes it a great fit for:

Read-intensive workloads
Write-intensive workloads
Workloads with strong transactional requirements
Databases distributed over several data centers and regions
Multi-petabyte databases

FoundationDB design deliberately limits the size of the transactions. Because of this, the following workloads are NOT compatible with FoundationDB:

Large transactions: there is a 10 MB limit on transaction size.
Long transactions: transactions longer than 5 seconds are canceled.
Large payloads: FoundationDB doesn’t support keys larger than 10 kB and values larger than 100 kB

Deploying FoundationDB on StackPath

Sizing virtual machines

Each FoundationDB process uses up to one CPU and is assigned its own data directory. We can run as many processes as CPU cores we have in the machine.

FoundationDB minimum specs for production are:

RHEL or CentOS 6 or 7
Ubuntu 12.04 or later
4 GB of RAM per CPU process
Storage as required (more on that below)

FoundationDB has two storage modes:

memory: Optimized for small databases. The whole data must fit in memory. This mode is ideal for fast-cache applications. FoundationDB still requires some disk space to ensure consistency.
ssd: This mode stores the data on disk and is intended for big databases. The amount of usable space depends on the chosen redundancy level.

FoundationDB has three levels of redundancy. Each redundancy level represents a different tradeoff between usable space and fault tolerance.

single: Best for 1 or 2 nodes. Doesn’t provide any fault tolerance. Maximizes space available for data. Good for development and cache storage.
double: Best for 3 or 4 nodes. Data is duplicated. The cluster can survive the failure of 1 or 2 nodes. The amount of usable free space is halved.
triple: Best for more than 5 nodes. Data is triplicated across the cluster nodes. Provides the highest safety level for a single-data-center cluster.

As a general rule, keep in mind the following when deciding what type of machine to choose:

In order to ease management, try to keep all machines in the cluster the same type and size.
Leave enough free space to accommodate sudden bursts of incoming data.
Memory is king for databases; dedicate all the memory you can afford to databases.
Increasing the redundancy level prevents data loss, but reduces the usable space for data.

Network and security

FoundationDB doesn’t have any built-in security mechanisms; there are no database users or passwords, nor read and write permissions. As a result, most deployments will likely only run inside a private network.

Access control must be controlled by other means:

Network permissions: each FoundationDB process listens on a separate port, starting from port TCP 4500 for the first process, 4501 for the second process, and so on
Connection file: clients must have a copy of the cluster file (fdb.cluster) to connect
Certificates: access can be secured by configuring mutual TLS certificates

Prerequisites

To get started you will need:

A StackPath account
Beginner-level knowledge of Linux servers: editors, executing commands, and connecting via SSH

Step 1 - Create an SSH key pair

You will need an SSH key to access your StackPath servers. If you already have one, you can skip to Step 2.

To create an SSH key pair in Linux or Mac:

ssh-keygen -C "your_email@example.com"

On Windows, install PuTTY and use puttygen.exe to generate the key pair.

Step 2 - Create the first VM

In this section, we’ll create our first Virtual Machine (VM) on StackPath.

To start, go to StackPath and log in or sign up for an account.

In the left menu, click Workloads and then Create Workload. Name your workload and, on Workload Type, select VM.

On Image select the most recent Ubuntu LTS version available (Ubuntu 18.04 LTS).

Click Continue to Settings.

On Public Ports, type 22 and select TCP.

On First Boot Key, copy the contents of your SSH public key (the file with the .pub extension, for instance: id_rsa.pub).

Click Continue to Spec.

Select the machine Spec that suits your needs. The SP3 machine has enough power to run two FoundationDB processes.

If you’re planning to use the ssd mode, add a volume for the data. On Additional Volume, type /var/lib/foundationdb and set the amount of storage you want to dedicate to the databases on this node.

On Deployment Target, type the name of the deployment group. Deployment targets are groups of instances with the same scaling needs.

Select one of the PoPs that are near you or your users.

For Instances per PoP, select 1.

Create the machine using the Create Workload button.

Step 3 - Install FoundationDB

Wait a few seconds for the machine. You’ll know it’s ready when it reaches the Running status.

Copy the Public IP, as shown in StackPath.

Open a terminal in your machine or start PuTTY.

Connect to the Public IP with SSH or PuTTY using the key generated in Step 1.

ssh ubuntu@YOUR_NODE1_IP

Go to the FoundationDB download page and pick the newest version of FoundationDB server and client packages for Ubuntu.

Copy the addresses of the packages and download both files to the server.

wget https://www.foundationdb.org/downloads/6.2.11/ubuntu/installers/foundationdb-clients_6.2.11-1_amd64.deb
wget https://www.foundationdb.org/downloads/6.2.11/ubuntu/installers/foundationdb-server_6.2.11-1_amd64.deb

Install them with dpkg.

dpkg -i foundationdb-clients_6.2.11-1_amd64.deb foundationdb-server_6.2.11-1_amd64.deb

After a few seconds, your first FoundationDB instance should be up and running.

$ ps -e | egrep "(fdb|backup_agent)"

14108 ?        00:00:02 fdbserver
14106 ?        00:00:00 fdbmonitor
14107 ?        00:00:00 backup_agent

FoundationDB has three types of processes:

fdbserver: is the core server process, each of these uses a single CPU. For dedicated machines, we want to run as many fdbserver processes as CPUs the system has. Each fdbserver has a dedicated port, starting from TCP 4500.
fdbmonitor: monitors the configuration files and the server processes. Whenever we make a change in the configuration, fdbmonitor automatically reloads the servers. fdbmonitor is also in charge of managing the backup_agent.
backup_agent: is responsible for taking backups and importing data into the database.

To get the server status information use:

fdbcli --exec status

At any point, you can start, stop, or restart the database with:

sudo systemctl stop foundationdb
sudo systemctl start foundationdb
sudo systemctl restart foundationdb

Step 4 - Configure the first node

In this section, we’ll prepare FoundationDB for production.

Copy the configuration to a temporary directory.

cp /etc/foundationdb/foundationdb.conf /tmp

Open the file with your preferred editor.

nano /tmp/foundationdb.conf

And find the [fdbserver] section and uncomment the datacenter_id line.

Set a data center name; you should use an identical string for all nodes in the same PoP. For example, for a node in New York, a good name would be:

datacenter_id = nyc

If your server has two CPUs: add the following line right before the [backup_agent] section; this tells FoundationDB to start a second process on port 4501.

[fdbserver.4501]

If your server has more than two CPUs: repeat the previous step until the number of processes matches the number of CPUs.

When you complete the modifications, the file should look like this:

[fdbmonitor]
user = foundationdb
group = foundationdb

[general]
restart_delay = 60
cluster_file = /etc/foundationdb/fdb.cluster

[fdbserver]
command = /usr/sbin/fdbserver
public_address = auto:$ID
listen_address = public
datadir = /var/lib/foundationdb/data/$ID
logdir = /var/log/foundationdb
datacenter_id = nyc

[fdbserver.4500]
[fdbserver.4501]

[backup_agent]
command = /usr/lib/foundationdb/backup_agent/backup_agent
logdir = /var/log/foundationdb

[backup_agent.1]

Save the file and copy it back to /etc/foundationdb.

cp /tmp/foundationdb.conf /etc/foundationdb/foundationdb.conf

You should now find a second fdbserver process (or as many processes as you configured).

ps -e|egrep "(fdb|backup_agent)"
14106 ?        00:00:00 fdbmonitor
14107 ?        00:00:09 backup_agent
14412 ?        00:00:00 fdbserver
14413 ?        00:00:00 fdbserver

You can also check how many processes are running with fdbcli.

fdbcli --exec status

[...]
Cluster:
  FoundationDB processes - 2
  Zones                  - 1
  Machines               - 1
  Memory availability    - 4.1 GB per process on machine with least available

[...]

Step 5 - Make server listen for external connections

In this step, we’ll open the ports for external connections.

Keep in mind that if you have opened ports 4500 or 4501 in StackPath, your server may be publicly accessible. In that case, ensure that you've read all the security considerations carefully before continuing.

To allow external connections, install Python.

apt-get update && sudo apt-get install python -y

Then run the make_public.py script. FoundationDB binds itself to the server’s private IP.

/usr/lib/foundationdb/make_public.py
/etc/foundationdb/fdb.cluster is now using address 10.128.96.3

Restart the service.

sudo systemctl restart foundationdb

The first server is ready! Now we'll copy the configuration files to your machine.

End your SSH session with logout or CTRL+D and copy the config and cluster files to your computer.

scp ubuntu@YOUR_NODE1_IP:"/etc/foundationdb/*" .

Step 6 - Create more machines

In this step, we’ll create additional nodes to form a cluster. You can stop here if you plan on deploying only one machine.

To provision the rest of the machines, go back to your StackPath dashboard. Locate your FoundationDB workload and click the Gear icon to modify its settings.

Click Edit next to Spec, Storage & Deployment.

In the Deployment Target section, increase the number of instances to the number of nodes you want in the cluster.

Click Save Changes and wait for the new servers to reach the Running status.

FoundationDB clients and servers connect to the cluster using the information contained in the fdb.cluster file.

To add a node to a cluster, we have to copy this file from any server already in the cluster and put it on /etc/foundationdb on the new machine. Once restarted, the new node will automatically join the cluster and start synchronizing.

For each of the new servers, repeat these steps:

Copy configuration files you saved in your machine at the end of Step 5.

scp foundationdb.conf fdb.cluster ubuntu@YOUR_NEW_SERVER_IP

Repeat Step 3 to install FoundationDB with its default configuration.
Overwrite the default config and cluster files.

cp foundationdb.conf fdb.cluster /etc/foundationdb

Restart FoundationDB.

sudo systemctl restart foundationdb

Step 7 - Configure the cluster

In this section, we’ll finalize the configuration of the cluster. Since the servers are linked up, we only need to run the following commands once.

First, ensure you are connected via SSH to any of the servers in the cluster. Then, start an interactive fdbcli session.

fdbcli

Using cluster file `/etc/foundationdb/fdb.cluster'.

The database is unavailable; type `status' for more information.

Welcome to the fdbcli. For help, type `help'.

fdb>

Type status and check the number of machines in the Cluster.

> status

[...]
Cluster:
  FoundationDB processes - 6
  Machines               - 3
[...]

Set the coordinator mode to auto so the cluster can automatically nominate coordinators.

> coordinators auto

You can also set a name for the cluster.

> coordinators description=mycluster

After a few seconds, you should see that the coordinators are activated.

> coordinators

Cluster description: mycluster
Cluster coordinators (3): 10.128.96.3:4500,10.128.96.4:4500,10.128.96.5:4500
Type `help coordinators' to learn how to change this information.

If you wish to activate ssd storage mode, set the redundancy level now (single, double or triple).

> configure double ssd

Check the coordinator settings and the redundancy level with status.

> status

Configuration:
  Redundancy mode        - double
  Storage engine         - ssd-2
  Coordinators           - 3

[..]

Next Steps

Congratulations! You’ve learned about FoundationDB and how to deploy it using StackPath’s edge compute VMs. You’re on your way to using high performance, low-latency databases.

As a next step, learn how to use datacenter mode to extend your database over more data centers. Then, learn how regions can help you build a truly global distributed database so that data is always close to your users.

Finally, check out these links to continue learning about FoundationDB:

How to Run Clickhouse DBMS at the Edge with StackPath

Tomas Fernandez — Wed, 15 Apr 2020 16:12:21 +0000

This tutorial originally appeared on StackPath Blog.

Businesses use data analytics to extract meaningful, decision-making information for their day-to-day operations. Analytics can discover hidden relations, find unexpected patterns, learn about customer behavior, understand the past and predict the future.

In this tutorial, we’ll learn about Clickhouse, a column-oriented database for real-time reports. We will explore how it works and take it out for a test drive using StackPath Containers.

Before diving into the tutorial, first we’ll provide an overview of column-based databases, ClickHouse, and the benefits of running ClickHouse on StackPath.

Column-Based Databases

Compared with traditional row-based databases, column databases have a subtle yet profound difference in how data is stored on disk.

A traditional database stores data as rows:

ID	COMPANY	CEO_NAME	CEO_NAME	YEAR_TO
1	Microsoft	Bill Gates	Bill Gates	2000
2	Microsoft	Satya Nadella	Satya Nadella	--
3	Apple	Steve Cook	Steve Cook	--
4	Facebook	M. Zuckerberg	M. Zuckerberg	--
5	Apple	Steve Jobs	Steve Jobs	2001

A column-based database DBs serialize them by columns:

COLUMN	VALUE	VALUE	VALUE	VALUE	VALUE
COMPANY	Microsoft (1,2)	Microsoft (1,2)	Facebook (4)
CEO_NAME	Bill Gates (1)	Bill Gates (1)	Steve Cook (3)	M. Zuckerberg (4)	Steve Jobs (5)
YEAR_FROM	1975 (1)	1975 (1)	2011 (3)	2014 (4)	1997 (5)
YEAR_TO	2000 (1)	2000 (1)

Our expected workload determines what type of database we should choose. Row I/O is designed to get all the information for a small set of records while column I/O is designed to get a small subset of columns for a huge set of records (or even a whole table).

Now, imagine we ask the server to calculate how many years each of the persons on the tables above were CEOs of their respective companies. A row database has to retrieve each row, then get to the columns with the relevant years and discard the rest. A column database only needs two I/O operations (regardless of the size of the table): one for each date column. No data is discarded. A column system will perform much better in this case and scales much better with large tables.

As a result of their design, column databases have unique properties:

Efficient column aggregation and materialized views.
Wide tables with optional columns pose no problems. On the contrary, they are encouraged.
Because similar data compress best, column orientation uses less space on disk than rows.
Due to less reliance on locking, column architectures can run with a much higher degree of parallelization.
Batch inserts are more efficient on columns than on rows.
Column databases have less reliance on indexes, which reduces overhead in the system.

ClickHouse: a Distributed Column-Based DBMS

Clickhouse is a database management system (DBMS) created by Yandex, the second-largest web analytics platform in the world, and released as open-source software under the Apache 2.0 License.

Beyond Yandex, many other big companies have adopted Clickhouse for their analytics, including Cisco, Cloudflare, CERN, Bloomberg, and Spotify.

Clickhouse main features are:

True column-based DBMS.
A subset of the SQL Language feature-rich with analytics and statistics functions.
Fault tolerance and no-downtime upgrades thanks to replication.
Multi-petabyte scale support.
Several engine modes for databases and tables.
Parallel processing on a single node.
Distributed queries using multiple nodes.
Support approximated calculations through sampling.

ClickHouse supports two mechanisms to scale out databases. Sharding distributes big tables among multiple machines and requests get automatically parallelized. Replication creates redundant copies of the data and can be used to distribute and filter data to different data centers. Also, ClickHouse supports multi-master replication.

Is Clickhouse right for your project?

Clickhouse is good for:

OLAP and OLEP workloads as long as they don’t need full-text search.
Data Warehouse: Clickhouse can store and process petabytes of data and build customized reports on the fly.
Big tables: tables with hundreds or thousands of columns.
Applications that need a high read throughput.
Applications that make heavy use of aggregated columns or materialized views.

Clickhouse is not good for:

OLTP (Online Transactional Processing) workloads. Clickhouse doesn’t support full-fledged transactions.
Applications with heavy write requirements.
Applications with complex or random write or update activity.
Applications that need full-text search.

Running Clickhouse in StackPath

Databases have some unique requirements that can clash with public cloud
common offerings:

Cost: high per-GB costs can make downloads and uploads expensive.
Bandwidth: big databases can take a long time to load or export.
Latency: round trip times can add unacceptable latencies to the queries.

Edge Computing solves these issues by moving servers as close as possible to customers and sources of data. As for StackPath, it offers the best of both worlds: the convenience and flexibility of the cloud with on premise-like performance.

Single Instance Configuration

In this section, we’ll deploy a single ClickHouse edge computing node in order to get familiar with the system. Then, we’ll learn how to implement a cluster.

Prerequisites

First, we need to create a custom Docker image with our private config. Before continuing, ensure you have the following things:

Docker installed on your workstation. On Mac and Windows, we recommend installing Docker Desktop. On Linux machines, install Docker from repositories.
A Docker Hub account to store the images.

Preparing the Docker Image

Yandex supplies two ready-to-use Docker images:

clickhouse-server: is the server component; it stores the data, accepts client connections, and processes requests.
clickhouse-client: is the official command-line client, much like psql or mysql.

To download the images to your machine, use the following commands:

docker pull yandex/clickhouse-server
docker pull yandex/clickhouse-client

By default, the Clickhouse server includes a single user called default with a blank password. For testing purposes, having a blank password is not a problem, but since we’re going to deploy the node in production, we need to secure it.

To customize the user settings, we need to change a file called users.xml, which is inside the clickhouse-server image.

To do this, create a directory to store our files.

mkdir my-clickhouse-server
cd my-clickhouse-server

Start an instance of the container so we can copy out the file.

docker create -it --name clickhouse-server clickhouse server

Copy the file from the container to your machine.

docker cp clickhouse-server:/etc/clickhouse-server/users.xml .

Open users.xml with a text editor and locate the following lines:

<users>
    <default>
        <!-- This is the default user with a blank password -->
        <password></password>

        . . .

    </default>
</users>

Clickhouse supports several password settings:

Plaintext: insecure and not recommended.
Double SHA1: uses a cryptographic hash function to make the password harder to guess to anyone looking at our image. Double SHA1 strings go inside <password_double_sha1_hex></password_double_sha1_hex>.
SHA256: arguably the most secure alternative and the one we will use. This option uses <password_double_sha256_hex></password_sha256_hex>.

To generate a SHA256 password in Linux or Mac, execute the following command in a new terminal:

PASSWORD=$(base64 < /dev/urandom | head -c8); echo "$PASSWORD"; echo -n "$PASSWORD" | sha256sum | tr -d '-'

The first line printed is the password, the second is the encoded SHA256 string. Alternatively, you can use an online SHA256 generator to obtain the string. In either case, keep both the password and the SHA string at hand as we’ll need them both soon.

Going back to users.xml.

Remove the <password></password> line to get rid of the blank password. Then add the following line with your SHA256 string.

<password_sha256_hex>78bb49329e64a3df1ea5a48fb09ce22ed5223171cf140e3b08683ab3926a0b9b</password_sha256_hex>

For example:


<users>
    <default>
    <password_sha256_hex>78bb49329e64a3df1ea5a48fb09ce22ed5223171cf140e3b08683ab3926a0b9b</password_sha256_hex>

        . . .

    </default>
</users>

Build the Image

Create a file called Dockerfile with the following content:

FROM yandex/clickhouse-server
COPY users.xml /etc/clickhouse-server

Then, build the image.

docker build -t my-clickhouse-server .

Test if it starts correctly with this command:

docker run -it -d --name my-server my-clickhouse-server

Finally, from a different terminal, try connecting using your password:

docker run -it --rm --link clickhouse:clickhouse-server yandex/clickhouse-client --host clickhouse-server --user default --password YOUR_PASSWORD
ClickHouse client version 20.3.2.1 (official build).
Connecting to 151.139.47.86:9000 as user default.
Connected to ClickHouse server version 20.3.2 revision 54433.

clickhouse-server :)

Push the Image to the Registry

Before we can use the image in StackPath, we have to put it in a Docker registry. To do this, log in to Docker Hub with your account.

docker login -u YOUR_DOCKERHUB_USERNAME

Build the image using your Docker Hub username as a prefix.

docker build -t YOUR_DOCKERHUB_USERNAME/clickhouse-server:latest

Push the image to Docker Hub.

docker push YOUR_DOCKERHUB_USERNAME/clickhouse-server:latest

Deploy to StackPath

Once we have pushed the image to the Docker Registry, we can deploy it in StackPath.

Head over to StackPath. Login with your account or sign up for a new one. Then, click on the Create button and select Workload.

We’ll name the workload “clickhouse-servers” and set the type to Container. For Image, type the address to your Docker image: YOUR_DOCKERHUB_USERNAME/clickhouse-server:latest

If you are using a private repository or a different registry, check the Add Image Pull Credentials box and type in the pull details.

Click on Continue to Settings.

If you want the servers to be accessible to everyone, add the following ports under Public ports. This will assign Public IPs and open the ports to the Internet.

All containers get a Private IP assigned automatically:

Port 8123 TCP is the default HTTP API Endpoint.
Port 9000 TCP is the default port for the native clients.

Click on Continue to Spec.

In Spec, choose the machine size. Clickhouse developers recommend a minimum of 4GB of RAM to allow for complex queries. If in doubt, choose the smallest machine, you can always scale the machine up with a few clicks later on.

Next, we’ll need a volume to persist our data, under Additional volume type /var/lib/clickhouse and set its size on GBs.

Under Deployment Target, we will group servers with similar scaling needs. Type a descriptive name like “clickhouse-servers”. Then select the PoP, which is nearest to you or to your customers. Set the number of instances to 1.

Click on Create Workload.

Wait a few seconds until the instance is on the Running state.

Test the Connection

The following test will only work if you have opened ports 8123 and 9000:

Click on the running instance to get its details.
Then, scroll down to the Instance details section and copy the Public IP address.

Next, open a terminal and try the /ping endpoint with Curl or a browser. If the server is running it should respond with "Ok".

curl -w "\n" YOUR_PUBLIC_IP:8123/ping

Ok

We can also connect with Clickhouse Native Client.

docker run -it yandex/clickhouse-client --host YOUR_PUBLIC_IP --user default --password YOUR_PASSWORD

ClickHouse client version 20.3.2.1 (official build).
Connecting to 151.139.31.12:9000 as user default.
Connected to ClickHouse server version 20.3.2 revision 54433.

clickhouse-servers-clickhouse-servers-jfk-0 :)

Now you can execute Clickhouse SQL commands on your new server:

clickhouse-servers-clickhouse-servers-jfk-0 :) create database my_db;

CREATE DATABASE my_db

Ok.

0 rows in set. Elapsed: 0.393 sec.

clickhouse-servers-clickhouse-servers-jfk-0 :) show databases;

SHOW DATABASES

┌─name────┐
│ default │
│ my_db   │
│ system  │
└─────────┘

3 rows in set. Elapsed: 0.326 sec.

Sharded Cluster Configuration

In this section, we’ll deploy two more database instances to create a
3-node Edge Computing cluster.

Create More Instances

First, we’ll create the remaining instances so that we can get their
Private IPs.

On the left navigation menu, click on Workloads and select your container workload. Then, click on the Gear icon to edit the workload configuration.

Click on the Edit icon next to Spec, Storage & Deployment and Increase the number of Instances per PoP to 3. Click on Save Changes.

StackPath will create the remaining instances in a few seconds.

When the instances are ready, copy the Private IP address of each one:

Configure Clickhouse Shards

We have to tell the Clickhouse where all the nodes are. For this, we
need to first get the config.xml file from the Clickhouse image.

Open a terminal and type the following command to copy the file out of the container:

docker cp clickhouse-server:/etc/clickhouse-server/config.xml .

Then, open the file and locate the remote_servers line:

<remote_servers incl="clickhouse_remote_servers" >
    <!-- Test only shard config for testing distributed storage -->
    <test_shard_localhost>
        <shard>
            <replica>
                <host>localhost</host>
                <port>9000</port>
            </replica>
        </shard>
    </test_shard_localhost>

        . . . 
</remote_servers>

Remove all the lines inside the <remote_servers> </remote_servers> elements and type the following config. Replace the IP Addresses with your servers Private IPs:

<remote_servers incl="clickhouse_remote_servers" >
    <my_sharded_config>
        <shard>
            <replica>
                <host>SERVER_1_PRIVATE_IP</host>
                <port>9000</port>
            </replica>
        </shard>
        <shard>
            <replica>
                <host>SERVER_2_PRIVATE_IP</host>
                <port>9000</port>
            </replica>
        </shard>
        <shard>
            <replica>
                <host>SERVER_3_PRIVATE_IP</host>
                <port>9000</port>
            </replica>
        </shard>
    </my_sharded_config>
</remote_servers>

Edit your Dockerfile to add the modified config:

FROM yandex/clickhouse-server
COPY users.xml /etc/clickhouse-server
COPY config.xml /etc/clickhouse-server

Build a new version of the image (notice we’re using a different tag
here).

docker build -t YOUR_DOCKERHUB_USERNAME/clickhouse-server:sharded

Push the new image to the Docker Registry, same as before.

docker push YOUR_DOCKERHUB_USERNAME/clickhouse-server:sharded

Update StackPath Workload

The final step is to tell StackPath to use our new configuration. To do this, go to your StackPath Workload and click the Gear button again to change its configuration. Edit the Image name with the new tag: YOUR_DOCKERHUB_USERNAME/clickhouse-server:sharded.

Click on Save Changes.

StackPath will start recycling the containers. When this is done, you should find all the containers running the newest version:

Testing the Cluster

Again, we can use Curl or any browser to check each of the ClickHouse nodes. Just replace the IP addresses with your Private IPs.

curl -w "\n" YOUR_PUBLIC_IP_1:8123/ping
Ok

curl -w "\n" YOUR_PUBLIC_IP_2:8123/ping
Ok

curl -w "\n" YOUR_PUBLIC_IP_2:8123/ping
Ok

We also can connect to any of the instances to check the cluster health.
Run the following query to see the cluster status.

select * from system.clusters;

For example:

docker run -it yandex/clickhouse-client --host YOUR_PUBLIC_IP --user default --password YOUR_PASSWORD

ClickHouse client version 20.3.2.1 (official build).
Connecting to 151.139.31.12:9000 as user default.
Connected to ClickHouse server version 20.3.2 revision 54433.

clickhouse-servers-clickhouse-servers-jfk-0 :) select * from system.clusters;

SELECT *
FROM system.clusters

┌─cluster────────────┬─shard_num─┬─shard_weight─┬─replica_num─┬─host_name───┬─host_address─┬─port─┬─is_local─┬─user────┬─default_database─┬─errors_count─┬─estimated_recovery_time─┐
│ my_sharded_config  │         1 │            1 │           1 │ 10.128.96.2 │ 10.128.96.2  │ 9000 │        0 │ default │                  │            0 │                       0 │
│ my_sharded_config  │         2 │            1 │           1 │ 10.128.96.4 │ 10.128.96.4  │ 9000 │        1 │ default │                  │            0 │                       0 │
│ my_sharded_config  │         3 │            1 │           1 │ 10.128.96.3 │ 10.128.96.3  │ 9000 │        0 │ default │                  │            0 │                       0 │
└────────────────────┴───────────┴──────────────┴─────────────┴─────────────┴──────────────┴──────┴──────────┴─────────┴──────────────────┴──────────────┴─────────────────────────┘

3 rows in set. Elapsed: 2.140 sec.

And that's it! If you made it this far, you have successfully deployed a 3-node cluster on StackPath.

Next Steps

For the next step in your road to mastering ClickHouse, try setting up a second cluster on a different PoP and establishing table replication between the two data centers.

If you want to learn more, check out these related docs and projects:

What are Containers?
Clickhouse SQL Reference
Clickhouse Operations
Clickhouse official JDBC and ODBC drivers.
Clickhouse third party drivers

Optimizing First-Frame Bitrate for HLS with Serverless Edge Compute

Robert Gibb — Wed, 18 Dec 2019 20:25:01 +0000

In this article we’ll show you how to use StackPath’s serverless edge product with HLS to deliver the right bitrate to the right device with the lowest possible delay. Whether you’re using cloud serverless for HLS or a different streaming solution entirely, this article will introduce you to the possibilities of optimizing streams with serverless scripting and low-latency edge compute.

Note: This tutorial is detailed and requires considerable effort to follow, but completing it can significantly reduce content delivery costs for you and buffer time for your users. It could be the catalyst for inspiring an evolution of how your video content is consumed.

Introduction

When responding to an HLS request, the streaming server determines which video quality (i.e. ts file) the client will attempt to play before switching to a lower or higher quality video. This switch depends on available bandwidth and device type.

Switching to the wrong quality first degrades the user experience considerably. For instance, sending a high quality file to a low-end device may cause the video/device to freeze, even on a good connection. And sending a low quality file to a high-end device with a good connection may cause the viewer to experience a low quality viewing experience for a prolonged period.

It may seem that sending a medium quality file first is a good solution, but it’s actually quite lazy. Instead, you can solve for the best solution in every case by using serverless scripting.

Serverless scripting, also known as function-as-a-service (FaaS), allows you to optimize responses on a per-device, per-request basis without touching your origin server’s configuration or code. There’s also no loss of cacheability and you can decrease latency further by making the decision at the edge instead of in a far-off data center.

Understanding how HLS works

For developers who are not already aware, HLS works by sending the client an index, also known as a manifest. This index contains a list of ts files called chunks that make up the video. The client requests the appropriate chunk based on play time.

In an adaptive bitrate implementation, the manifest provides links to several alternative playlists called variants instead of listing chunks directly. All variants have an identical number of chunks and video content, but they differ in bitrate and resolution (i.e. quality).

According to the HLS spec, clients should request the top listed quality in the manifest, play it, and calculate the available bandwidth (i.e. chunk size and download time). Based on the calculation, the client switches to a higher or lower quality until the bitrate of the video is lower than the maximum available bandwidth. This ensures seamless play.

(The image above is a sample HLS index listing three variants, each with a different bandwidth, resolution, and index file.)

To summarize, choosing the video quality based on the available bandwidth is the responsibility of the client, but choosing the default quality to start with is the responsibility of the server. This default quality is usually either the highest one, or simply the first one uploaded to the server. In all cases, it is the same regardless of which device requests the video and where the video is requested from.

Understanding the problems with HLS

We’ve established that the default video quality isn’t often the best to include in the initial response with HLS, but let’s look at some different scenarios to further understand why.

Scenario #1: High-End Device | High Quality Video | Low Bandwidth | Any Screen Size

In this scenario, the device downloads the master playlist (playlist.m3u8) and starts with best quality as dictated by the server. The device downloads the first chunk and it takes a whole minute (62,352ms). During this time, the viewer is waiting, not watching anything. This is a terrible video streaming experience.

Scenario #2: High-End Device | Medium Quality Video | Medium Bandwidth | Small Screen

In this scenario, the device spends seven seconds downloading a 10-second second chunk. After this, it starts to play.

So we’ve fixed the problem in Scenario #1 but we still waited seven seconds for the video to start. But we’re playing in a 800x450 pixel player and medium quality has a resolution of 1280x720 with a bitrate to match. Therefore, low quality is beyond enough here and waiting seven seconds for a higher quality file is unnecessary. If we started with a screen-appropriate quality we’d risk less delay before optimization catches up.

Scenario #3: High-End Device | Medium Quality Video | High Bandwidth | Big Screen

In this scenario, the device downloads the initial ts file quickly and, upon downloading, plays 10 seconds of medium quality video before switching to high quality.

So we’ve eliminated the problem in Scenario #2 but we still had to tolerate 10 seconds of lower quality video before optimizing up. And the next time we load another video (even on the same site), we’re going to go through all of this again. It would be better if we could remember the current quality and start with it.

Note: There’s an edge case scenario containing a low-end device and high-quality start but there’s not much to expand on. As you might expect, the video starts to stutter and lag.

Solutions for HLS problems

Client-side solutions

Naturally, all these scenarios are much better solved for on the client side. For example, VideoJS, one of the most popular libraries for HLS playback on the web, will immediately start with the highest quality whose resolution fits the current player size, eliminating one of the issues above.

But not all players do this. For instance, ExoPlayer, Google's HLS player on Android devices, performs no such capping. However, it does let developers customize its selection logic to fit their use case.

Server-side solutions with serverless

The vast majority of media servers make use of a static master playlist file and custom development is often required to create an adaptive system. But even if you have the logic for changing the order of file delivery at your origin server based on device type, for instance, what do you do with your CDN? Cache the generated master playlist for each one of hundreds of different user agents? With the request sent all the way to origin every time you see a new one? Or perhaps you don’t cache the master playlist and delegate processing to your origin completely? Either way, you’re still slowing everything down and adding a excessive load to your origin.

Edge serverless solves this by allowing you to implement custom logic right at the edge. Moreover, this approach is backend agnostic. A serverless script can manipulate the fetched resources as a man-in-the-middle with zero reliance on any feature of the backend system.

Objective

This tutorial will show you how to use serverless scripting to optimize HLS quality by device at request time, at the edge server, without any contribution from your origin server or any effect on cacheability.

Serverless scripting allows you to write custom logic (implemented by js code) to process requests at the edge server. It can modify a client's request before fetching it and also modify the response. The possibilities with this serverless use case are endless and, therefore, the techniques in this tutorial serve as more of a demonstration than a definitive solution.

Prerequisites

1) An origin serving a multi-quality HLS stream. This may be powered by nginx-rtmp, Mist, Wowza, Nimble, or any form of media server. As an example we'll use the url http://awesomeMedia/, serving an HLS stream at http://awesomeMedia/hls.m3u8. Here is the example playlist served at that URL:

#EXTM3U
#EXT-X-VERSION:3
#EXT-X-STREAM-INF:BANDWIDTH=5000000,RESOLUTION=1920x1080
1080p.m3u8
#EXT-X-STREAM-INF:BANDWIDTH=2800000,RESOLUTION=1280x720
720p.m3u8
#EXT-X-STREAM-INF:BANDWIDTH=1400000,RESOLUTION=842x480
480p.m3u8
#EXT-X-STREAM-INF:BANDWIDTH=800000,RESOLUTION=640x360
360p.m3u8

Normally an encoder in your media pipeline will take care of auto generating this. In the file above, we are serving four different qualities at 1080p, 720p, 480p, and 360p—each with a defined bitrate appropriate to the resolution, as defined in the #EXT-X-STREAM-INF lines above.

2) A StackPath account with serverless scripting enabled. Refer to this tutorial for instructions on enabling serverless for your site for $10/month. Alternatively, you can test the concept for free using the Sandbox.

Step 1: Set up your site with serverless

The first step is to sign up for a StackPath account, add your site, and enable Serverless Scripting.

1) Sign up for a StackPath account here. Then select Website & Application Services and choose the services you require.

The Edge Delivery Bundle is recommended but the bare minimum is the CDN service.

2) Log in to the StackPath Control Panel and create/select an active Stack.

3) Select Sites in the left-hand navigation bar and click Create Site.

4) Check Serverless Scripting, enter your domain (or IP address), and select Continue.

5) Finally, take note of your site’s edge address, which you can now use for delivery of your content all over the globe. Alternatively, you can use the StackPath DNS service to have your domain name resolve to the ideal edge and always serve content from StackPath’s CDN.

In this example, our HLS video is now accessible at http://j3z000b6.stackpathcdn.com/HLS.m3u8 with the .ts chunks available on the same path.

Step 2: Create your script

Now that we have a site with Serverless Scripting enabled we can deploy our first script. We can do this manually through the web control panel or through the Serverless Scripting CLI. The web option is suitable for testing or manual deployment. It’s simple, intuitive, and comes with a nice web editor. But if you’re developing with your own tools or integrating with a CI/CD pipeline you’ll appreciate the ease and automatability of a CLI single-line deployment.

In this step our objective is to deploy a script to process requests for the path http://awesomeMedia/HLS.m3u8. When a user visits the site, rather than fetching the file HLS.m3u8 straight from cache (or origin), the serverless engine will execute the script to decide what happens. The script could choose to block the request, deliver the file normally, modify the file, or write a new response altogether.

Option 1: Using the control panel

Navigate to the control panel and click Scripts. Then click the Add Script button. You’ll be taken straight to the code editor. Enter a name for your script and the route it will run for. You can also enter multiple routes, or use wildcards.

Option 2: Using the serverless CLI

To use the Serverless CLI we’ll have to install the CLI package, create a configuration file named sp-serverless.json in the project directory, and publish. But first we’ll need an API key and ID information.

1) Using the dashboard, take note of your Stack ID and Site ID.

2) To generate API keys for use by the CLI, go to API Management in the dashboard and click Generate Credentials. Then record the Client ID and API Client Secret.

3) Install the CLI by running the following command in a terminal.

npm install -g @stackpath/serverless-scripting-cli

4) To configure the CLI, run the following command.

sp-serverless auth

This will set up authentication for the CLI to use your account. It will ask you for the details above interactively.

5) Clone the skeleton repository by running:

git clone https://github.com/stackpath/serverless-scripting-examples/tree/master/hls-initialization.git

Or create your files manually. If you do this, create a directory for the project. Then create the following two files.

A file named sp-serverless.json with the following content. Substitute your Stack ID and Site ID where indicated.

{
  "stack_id": "",
  "site_id": "",
  "scripts": [
    {
      "name": "HLS Optimizer",
      "paths": [
        "HLS.m3u8"
      ],
      "file": "default.js”
     }
  ]
}

This file indicates the deployment details to the Serverless CLI. The paths array indicates the routes your script will be active for and multiple scripts/paths can be configured using the same file. You can add extra scripts for extra routes by copying the first object in the scripts array and modifying as needed.

A file named default.js with the following content. This will be the script you’ll deploy. For now it’s the default skeleton script and has no effect on response.

addEventListener("fetch", event => {
  event.respondWith(handleRequest(event.request));
});
/**
 * Fetch and return the request body
 * @param {Request} request
 */
async function handleRequest(request) {
  try {
    /* Modify request here before sending it with fetch */
    let response = await fetch(request);

    /* Modify response here before returning it */
    return response;
  } catch (e) {
    return new Response(e.stack || e, { status: 500 });
  }
}

6) Finally, deploy by issuing the following command.

sp-serverless deploy

This is the command to run every time you update your script or make changes to its configuration in the sp-serverless.json file.

From this point forward this is a development tutorial. You can skip this and use the finished product by going directly to Step 6 at the end of the tutorial, with one caveat: you’ll need to deploy a container as detailed in Step 3. Alternatively, if you’d like to follow along without creating a StackPath account you can use the Sandbox.

Step 2: Parse the HLS master playlist

If using the Sandbox, start by filling in the origin route you’d like to apply the script to in the URL textbox (e.g. http://awesomeMedia/HLS.m3u8) and select Raw as the display mode (optional). The Sandbox provides console access, so that you can issue console.log commands and view stack traces live. Other options for debugging serverless scripts are to return information in the body of the response, or in its headers.

The default (skeleton) script just returns the original manifest as is. The essential ingredient is simply a request handler bound to fetch that receives the original request from the client, fetches the response, and returns it untouched.

addEventListener("fetch", event => {
  event.respondWith(handleRequest(event.request));
});

/**
 * Fetch and return the request body
 * @param {Request} request
 */
async function handleRequest(request) {
  try {
    /* Modify request here before sending it with fetch */

    let response = await fetch(request);

    /* Modify response here before returning it */

    return response;
  } catch (e) {
    return new Response(e.stack || e, { status: 500 });
  }
}

The code is helpfully marked with suggested locations for modifying the request, the response, and returning them.

First, let's pass the request upstream and examine the response before returning it.

Modify the script as shown below.

/* Modify response here before returning it */
const upstreamContent = await response.text();
/* Log the response body to the console */
console.log(upstreamContent)

return new Response(upstreamContent, {
    status: response.status,
    statusText: response.statusText,
    headers: response.headers
    });

After clicking Run Script we are greeted with the result in the console, showing the contents of the master playlist:

[info] #EXTM3U
#EXT-X-VERSION:3
#EXT-X-STREAM-INF:BANDWIDTH=800000,RESOLUTION=640x360
360p.m3u8
#EXT-X-STREAM-INF:BANDWIDTH=1400000,RESOLUTION=842x480
480p.m3u8
#EXT-X-STREAM-INF:BANDWIDTH=2800000,RESOLUTION=1280x720
720p.m3u8
#EXT-X-STREAM-INF:BANDWIDTH=5000000,RESOLUTION=1920x1080
1080p.m3u8

Note: We’ve changed the original line returning the response in the code. Rather than simply returning the same response object, we’re generating a new one with the same header and content. This is because attempting to return the original object after having already resolved the promise of text() will result in an error.

We now have the HLS master playlist in a variable so it’s time to parse it for available variants (qualities).

For a master playlist with several variants, the playlist starts with several “header” tags, then defines every variant in a line beginning with “EXT-X-STREAM-INF”. More information can be found in the HLS spec. But for the purposes of this script a regex solution is ideal, as implemented in this new function:

function parseM3u8(body) {
  // Parse M3U8 manifest into an object array, containing bitrate, resolution, and codec
  var regex = /^#EXT-X-STREAM-INF:BANDWIDTH=(\d+)(?:,RESOLUTION=(\d+x\d+))?,?(.*)\r?\n(.*)$/gm;
  var qualities = [];
  while ((match = regex.exec(body)) != null) {
    qualities.push({bitrate: parseInt(match[1]), resolution: match[2], playlist: match[4], codec: match[3]});
  }
  return qualities;
}

This function continues to look for #EXT-X-STREAM-INF lines and extracts bandwidth, resolution, playlist link, and, finally, preserves any extra information found (usually codec information). To explore how it works, you can use a handy tool called RegExr.

To test this function, we can add a console log line anywhere in the handler function:

console.log(parseM3u8(upstreamContent))

[info] [ { bitrate: 800000,
    resolution: '640x360',
    playlist: '360p.m3u8',
    codec: '' },
  { bitrate: 1400000,
    resolution: '842x480',
    playlist: '480p.m3u8',
    codec: '' },
  { bitrate: 2800000,
    resolution: '1280x720',
    playlist: '720p.m3u8',
    codec: '' },
  { bitrate: 5000000,
    resolution: '1920x1080',
    playlist: '1080p.m3u8',
    codec: '' } ]

And…success! The test worked as expected.

Step 3: Parsing the User Agent

By examining the headers of the request before sending it upstream we can find the user agent of the client browser. In the marked location of the skeleton script for modifying requests, we log the user agent by adding two lines of code.

/* Modify request here before sending it with fetch */
const userAgent = request.headers.get('User-Agent');
console.log(userAgent)

[info] Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:69.0) Gecko/20100101 Firefox/69.0

Aside from the original headers sent by the browser, there is a set of additional StackPath headers that are added to the original request. These contain the IP address, location, server region, and other information about the request. You can find a list of them in the StackPath Developer Docs.

In order to make meaningful decisions about which quality to prioritize and which video resolutions to send, there are three main pieces of information we need to learn about from the client device:

Type (desktop/phone/tablet)
Screen resolution
Power measurements to be used for deciding whether to cap at a certain quality for devices with poor processing capabilities (optional)

While there are many JavaScript libraries for parsing the user agent, none of them can tell us the screen resolution of a mobile device. Doing so not only requires a complicated parser that understands all the different ways UA strings are formed, but also a database with information about the devices once they’re identified. This parser and database combination is formally called a Device Description Repository (DDR).

Today, there are plenty of commercial DDR services. But only one is both reliable and free: OpenDDR. This service can be deployed in the form of a docker container, providing a simple API that returns JSON-formatted data for a given UA that we will call from within our script.

Calling the API in Serverless

Before deploying OpenDDR, we’ll use the demo endpoint to develop the necessary function in our script and test the integration. The function fires a new http request to the API, parses the response, and passes the returned information to the main handler.

 /* Modify request here before sending it with fetch */
    const userAgent = request.headers.get('User-Agent');
    console.log(userAgent);
    /* Consult the DDR microservice in regards to the user agent */
    const deviceData = await getDeviceData(userAgent);
    console.log(deviceData);

async function getDeviceData(ua) {
  try {
    const res = await fetch('http://openddr.demo.jelastic.com/servlet/classify?ua='+ua);
    const data = await res.json();
    return data.results.attributes;
  } catch (e) {
    throw new Error('DDR communication failed')
  }
}

[info] Mozilla/5.0 (Linux; Android 8.0; Pixel 2 Build/OPD3.170816.012) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/67.0.3396.87 Mobile Safari/537.36
[info] { model: 'Pixel XL',
  ajax_support_getelementbyid: 'true',
  marketing_name: 'Pixel XL',
  is_bot: 'false',
  from: 'openddr',
  displayUnit: 'pixel',
  displayWidth: '1440',
  device_os: 'Android',
  id: 'PixelXL',
  xhtml_format_as_attribute: 'false',
  dual_orientation: 'true',
  nokia_series: '0',
  device_os_version: '8.0',
  nokia_edition: '0',
  vendor: 'Google',
  cpu: 'Qualcomm Snapdragon 821',
  mobile_browser_version: '67.0.3396.87',
  ajax_support_events: 'true',
  is_desktop: 'false',
  cpuRegister: '64-Bit',
  image_inlining: 'true',
  ajax_support_inner_html: 'true',
  ajax_support_event_listener: 'true',
  mobile_browser: 'Chrome',
  ajax_manipulate_css: 'true',
  displayHeight: '2560',
  cpuCores: 'Quad-Core',
  is_tablet: 'false',
  memoryInternal: '32/128GB, 4GB RAM',
  inputDevices: 'touchscreen',
  ajax_support_javascript: 'true',
  cpuFrequency: '2150 MHz',
  is_wireless_device: 'true',
  ajax_manipulate_dom: 'true',
  is_mobile: 'true',
  xhtml_format_as_css_property: 'false' }

The response from the DDR contains more info than the UA itself has. In particular, we are interested in displayWidth and displayHeight, but the information in there is pretty extensive.

Note: For a desktop (or laptop) device, the display resolution is still rarely identifiable using this method. For example, the UA for Google Chrome 77.0 on Windows 10 is the same regardless of your display. Also, viewers may rarely play the video in full screen depending on the type of content you’re serving.

Deploying a container

The function above is currently making use of the demo API endpoint offered on the OpenDDR website and is unsuitable for production use. Therefore, we’re going to be deploying our own instance using an Edge Container.

StackPath’s Edge Containers offer a way of hosting containers in diverse locations so that no request has to travel all the way to a centralized computing node. This is particularly useful for stateless applications such as the one we’re deploying. (A more detailed tutorial can be found here.)

To start, head back to the control panel and click Workloads. Then click Continue to add Edge Compute to your Stack.

Now, click Create Workload.

Now you can configure your container:

Name: (Any name works)
Image: Enter 0x41mmar/openddr-api:latest, which refers to this container
Anycast IP: Disabled for now (more on this below)
Public Ports: Enter 8080, the only port exposed by the container image above
Spec: The resources allocated to each instance. SP-1 (1 vCPU, 2GB Mem) should be enough.
Deployment Target: Locations where the container will be deployed. Select a name and one or more locations

Note: Unlike the ephemeral IPs assigned to containers when they’re started, the Anycast IP is a static IP assigned to the workload for its entire lifetime. The Anycast IP has significant performance benefits. Traffic towards an Anycast IP will enter the StackPath Network at the closest Edge Location and be routed to the instance using [StackPath’s private backbone(https://blog.stackpath.com/network-backbone-speed/).

Finally, click Create Workload. You’ll be redirected to the container’s overview page where you’ll see the container starting. When it’s fired up, you’ll see the IP addresses associated with it. Make note of the public IP.

Finally, let’s replace the demo URL in the classifier function with our own container’s, using the IP above.

const res = await fetch('http://101.139.43.73:8080/classify?ua='+ua);

Step 4: Decision Tree

The information required to make a decision about the optimization has been collected. Now the objective is to decide on the order of variants in the master playlist. We’ll create a simple algorithm to determine the best way to sort the variants in the file based on the following rules:

Never start with a variant whose resolution is higher than the display, even if the device will optimize up afterwards.
If the best quality variant satisfying condition (1) has a relatively high bitrate (>=4 Mbps), start one step lower.
If device is quite old/low-end, start at the lowest quality and cap to display resolution to avoid performance issues.
If nothing is known for sure (desktop devices), assume resolution is appropriate for 720p and apply the rules above.

Rule 4 makes sense because, according to StatCounter, at least 52.91% of desktop/laptop devices in use today have a display above 720p, but only 19% have 1080p displays.

The resulting algorithm is shown in the diagram below. Note that this is not a definitive solution and can be changed easily to fit your unique scenario.

This is straightforward to implement in JavaScript, as is the execution of the sorting and capping. This translates into the following function.

function decisionTree(deviceData, qualities) {
  /* Logic deciding on ordering and capping of available qualities */
  /* Returns config object to the spec of the output function above */

  // get higher dimension of resolution
  var res = Math.max.apply({},[deviceData.displayHeight,deviceData.displayWidth]);
  //is it a desktop device? assume 1280x720
  if (deviceData.is_desktop == "true")
    return {top: 2, cap: false, res: 1280};
  else {
    // mobile device. Is it an old device?
    if ((deviceData.device_os == "iOS" && parseInt(deviceData.device_os_version) < 7) || (deviceData.device_os == "Android" && parseInt(deviceData.device_os_version) < 6) || (deviceData['release-year'] < 2012))
return {top: -1, cap: true, res: res};
   else
     return {top: 2, cap: false, res: res};
  }

  //default
  return {top: 2, cap: false, res: res}
}

All we have to return is the assumed display resolution, our order decision, and capping decision. The code is more or less a direct implementation of the graph above. One thing to note is the nature of variables returned by OpenDDR, as most of them are strings and require some conversion.

More complex decision making is easy to implement in much the same way. For example, Apple provides guidelines on bitrates for particular resolutions.

Step 5: Output and Putting All Together

Now we need to rewrite the master playlist from scratch using the available information and decisions from the decision tree above. The function below is extensively commented and easily readable, but is basically sorting qualities, moving the required quality to the top of the list, and, if needed, deleting one that are too high.

First, we check if certain qualities need to be removed by checking the boolean config.cap. If some do, we filter the qualities array to remove offending variants unless there is only one of them.

Second, we sort the available variants in descending order by bitrate, unless it is required to place the lowest one on top (config.top=-1). If the top quality is to be the first, we’re done. If we’re to apply the 4Mbps rule, then we progressively test qualities for the required conditions (<=display, <4Mb).

function writeSortedManifest(qualities, config) {
  /* Sort qualities, optionally cap at a certain resolution, */
  /* then rewrite into correct HLS master playlist syntax */
  /* config = {cap: bool, top: int, res: int} */
  // top = 1: highest quality first, 2: highest quality within res or next one if >4Mbps, 0: middle quality first, -1: lowest quality first

  //cap
  //remove qualities with a resolution higher than a certain value (player resolution) 
  if (config.cap) {
    newQualities = qualities.filter((x)=> Math.max.apply({},x.resolution.split('x')) <= config.res );
  // anything left?
  if (newQualities.length > 0)
    qualities = newQualities;
  }

  // sort array so either best or worst quality is the first
  // dir = 1 for descending, -1 for ascending. use 1 for anything except top==-1
  // if top==-1, this is all we need to do
  var dir = config.top==-1 ? -1: 1;
  qualities.sort((a,b) => (a.bitrate>b.bitrate)? -1*dir: dir)

  //if applying resolution rule (top==2), process from top to bottom to find variant satisfying conditions
  if (config.top==2) {
    for (var i in qualities) {
      // assume it's this one for now
      var topChoice = qualities[i];

      // convert "1280x720" to 1280; accomodate top dimension and the rest will be fine if using a sane aspect ratio
      var topDim = qualities[i].resolution.split('x').sort()[1];

      // For this variant:
      // is res <= display?
      if (topDim <= config.res) {
        // yes! but is bitrate <4Mbps?
        if (qualities[i].bitrate < 4000000) {
          // great! done here.
          break;
        }
        else if (qualities[i+1]) {
          //it's not, so choose next option if it exists
          topChoice = qualities[i+1];
          break;
        }
        else {
          // next option doesn't exist? ok, fine. We'll take this one
          break;
        }
      }
    }
  // Now let's move the top choice top
  qualities.splice(0,0,topChoice);
  }

  //if middle quality required to be the first, move it there
  if (config.top==0) {
    var m = Math.floor(qualities.length/2);
    var middleItem = qualities[m];
    qualities.splice(m,1);
    qualities.splice(0,0,middleItem); 
  }

Going back to the main handler function, we just need to call the functions and pass the data as needed.

async function handleRequest(request) {
  try {
    const userAgent = request.headers.get('User-Agent');
    var deviceData, response;
    [deviceData, response] = await Promise.all([getDeviceData(userAgent), fetch(request)]);
    const upstreamContent = await response.text();
    var variants = parseM3u8(upstreamContent);
    var config = decisionTree(deviceData, variants);
    var output = writeSortedManifest(variants, config)
    /* Return modified response */
    response.headers.set("Content-Length", output.length)
    return new Response(output, {
    status: response.status,
    statusText: response.statusText,
    headers: response.headers
    });
  } catch (e) {
    return new Response(e.stack || e, { status: 500 });
  }
}

Notice how we’ve made sure to send the two external IO requests asynchronously. The DDR request has no dependency on the upstream fetch, and vice versa. This saves us time.

One little quirk in this code is the line right before the return statement that calculates the new content-length. This is necessary because the length (in bytes) of the body may have changed after being regenerated due to the ambiguity of new line characters. Our generator function only uses new line characters (\n), but the original file may or may not have used carriage returns (\r). A simple alternative would be to preserve the original line breaks with a slight modification of the regex and generator function, and have the final line be the exact same length as the original.

Step 6: Final Deployment and Testing

If you’re using the web control panel to upload your code you can find the complete code on GitHub here. Or, if you’re using the CLI and have followed the instructions in Step 1, you’ll have cloned a repository that already contains this code. Simply modify your sp-serverless.json file to use hls.js rather than default.js and replace the URL in line 53 (getDeviceData) with that of your own openddr container, per Step 3.

{
  "stack_id": "",
  "site_id": "",
  "scripts": [
    {
      "name": "HLS Optimizer",
      "paths": [
        "HLS.m3u8"
      ],
      "file": "hls.js"
     }
  ]
}

Note that the file will have had a script ID added automatically if you’ve deployed before. If you have, be sure to leave it in, then simply enter the following:

sp-serverless deploy

And that’s it!

With the script saved and deployed, we can now test the response with a variety of devices. The fastest way to do this is to use the Developer Tools available in your favorite browser. You can also use an extension/addon for the purpose of device switching.

Here are the results for a few different devices:

Device	UA	First Variant	Capping	Comment
Windows Laptop	Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/61.0.3163.100 Safari/537.36	720p	N/A	Guessing display is 720p capable, and bitrate of 720p variant is 2.8Mbps, acceptable
Google Pixel 2	Mozilla/5.0 (Linux; Android 8.0; Pixel 2 Build/OPD3.170816.012) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/67.0.3396.87 Mobile Safari/537.36	720p	N/A	Definitely 1080p capable, but we’re not risking that high a bitrate on first request, falling down to 720p
Galaxy Ace 3	Mozilla/5.0 (Linux; Android 4.2.2; GT-S7275R Build/JDQ39) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/48.0.2564.95 Mobile Safari/537.36	360p	360p is the only quality sent	Old low-end device, anything more is quite wasted on it
HTC One M8	Mozilla/5.0 (Linux; Android 4.4.2; HTC6525LVW Build/KOT49H) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/31.0.1650.59 Mobile Safari/537.36	360p	N/A	Old device now, but not low-end, and display is good. Starting low but sending all qualities.

Conclusion

This article demonstrates the potential for Serverless Scripting as a solution for HLS streaming optimization. The ideal solution, as always, remains proper optimization on the client side. But with so many different players implementing so many different behaviors a solution of this type can make a significant difference to the end viewer’s experience. Serverless Scripting makes this quite easy and accessible, particularly when compared to the alternative of dynamic processing at the origin.

The decision algorithm implemented may or may not be fitting for a particular audience or system, but we hope that it is explained and demonstrated satisfactorily for any developer to modify it to fit her own needs.

Taking this even further

A number of improvements can be made to the script to make it even more powerful.

Cookies: The entire problem of first-variant selection stems from the fact that the bandwidth is unknown on first play. But what if the viewer has just watched another stream from your site? Use cookies to remember what variant the client last settled on within a certain time window.
Client-side device detection: If the client is viewing the video inside a web or phone app you control, a bit of JS can provide much better information than we can find with the user agent alone. It’s possible to have some JS on your website that writes this information in a cookie for the Serverless Script to consume, and thus mitigate the lack of information about desktop devices (for instance). Naturally, this kills the quality of being server-agnostic and not requiring any changes to your origin configuration or deployment, but the trade off can be useful.
Error Checking: Upgrade the script to deal with various error scenarios. What if the DDR API is unresponsive? Information ambiguous? Decide on which assumptions to make and how to order variants.
Statistics: The numbers used in the assumptions above (e.g. a high bitrate is 4Mbps) are based on industry recommendations and general experience, but proper statistics are lacking. For example, we could enhance the script by making more statistically sound assumptions about available bandwidth for a particular region—or even for a particular class of device in a particular region, and so on.
Statistics II: Following from the point above, we can use a microservice with a db (similar to the DDR one above) to keep track of which devices, IPs, and regions settle on certain qualities, or which have problems starting the stream. This is useful for studying your audience and optimizing your configuration, but bonus points if you use the information automatically while sorting variants to adjust the rules as needed. For instance, if we have enough data to see that iPhones on Vodaphone 4G in Manhattan settle on 2Mbps most often, just start them there. This takes the previous point to its logical conclusion, but beware of the law of large numbers.

How We Perform Frontend Testing on Our Customer Portal

Thomas Ladd — Fri, 08 Nov 2019 17:56:35 +0000

An effective automated testing strategy is crucial for ensuring teams can deliver quality updates to web applications quickly. We are lucky to have a lot of great options in the space right now for testing. However, with a lot of options comes the difficulty of sorting through which one(s) to pick. Then, once the tools are chosen, you need to decide when to use each one.

At StackPath we're very happy with the level of confidence we've achieved in our customer portal. So, in this post, we will share the set of tools we use to test our customer portal and inspire confidence in its performance.

Testing Principles

Before diving into specific tools, it’s worth thinking about what good tests look like. Prior to starting work on the customer portal, we wrote down the principles we wanted to follow when writing tests. Going through that process first helped us decide which tools to select.

The four principles we wrote down (with a little bit of hindsight thrown in) are listed below.

1. Tests should be thought of as an optimization problem

An effective testing strategy is about maximizing value (confidence in the application working) and minimizing cost (time spent maintaining tests and running tests). Questions we often ask when writing tests related to this principle are:

What’s the possibility that this test actually catches a bug?
Is this test adding value and does that value justify its cost?
Could I derive the same level of confidence as I do from this test with another test that is easier to write/maintain/run?

2. Avoid excessive mocking

One of my favorite explanations of mocking is Justin Searls’s talk at Assert.js 2018. He goes into a lot more detail and subtlety than I will here, but in the talk, he refers to mocking as punching holes in reality, and I think that’s a very instructive way of looking at mocks. While mocking does have a place in our tests, we have to weigh the reduction of cost the mock provides by making the test easier to write and run against the reduction in value caused by punching that hole in reality.

Previously, engineers on our team relied heavily on unit tests where all child dependencies were mocked using enzyme’s shallow rendering API. The shallow rendered output would then be verified using Jest snapshots. All of these sorts of tests followed a similar template:

it('renders ', () => {
  const wrapper = shallow();
  // Optionally interact with wrapper to get the component in a certain state
  expect(wrapper).toMatchSnapshot();
});

These sorts of tests punch a ton of holes in reality. You can pretty easily get to 100% test coverage with this strategy. The tests take very little thought to write, but without something testing all of the numerous integration points they provide very little value. The tests may all pass, but I’m not too sure if my app works or not. Even worse, all of the mocking has a hidden cost that pops up later.

3. Tests should facilitate refactoring—not make it more painful

Tests like the one shown above make refactoring more difficult. If I notice I have the same repeated code over and over again and extract it to another component later, every test I had for components that use that new component will fail. The shallow rendered output is different; where before I had the repeated markup, now I have the new component.

A more complicated refactoring that involves adding some components and removing others results in even more churn as I have to add new test files and remove others. Regenerating the snapshots is easy, but what value are these tests really providing me? Even if they could catch a bug, I’m more likely to miss it amongst the number of snapshot changes and just accept the newly generated ones without thinking too hard about it.

So these sorts of tests don’t help much with refactoring. Ideally, no test should fail when I refactor and no user-facing behavior is changed. Conversely, if I do change user-facing behavior, at least one test should fail. If our tests follow these two rules, they are the perfect tool for ensuring I didn’t change any user-facing behavior while refactoring.

4. Tests should mimic how a user actually uses the application

If I want my tests to only fail when user-facing behavior changes, it follows that my tests ought to interact with my application in the same sort of way an actual user would. For example, my tests should actually interact with form elements and type in input fields the same way a user would. They should never reach into a component and manually call lifecycle methods, set state, or anything else that is implementation-specific. Since the user-facing behavior is what I’m ultimately wanting to assert, it’s logical that the tests should be operating in a way that closely matches a real user.

Testing Tools

Now that we’ve defined what our goals are for our tests, let’s look at what tools we ultimately chose.

Typescript

We use TypeScript throughout our codebase. Our backend services are written in Go and communicate using gRPC, which allows us to generate typed gRPC clients for use in our GraphQL server. The GraphQL server’s resolvers are typed using generated types from graphql-code-generator. Finally, our queries, mutations, and subscriptions components/hooks are also generated with full type coverage. End-to-end type coverage eliminates an entire class of bugs resulting from the shape of data not being what you expect. Generating types from schema and protobuf files ensures our entire system remains consistent across the stack.

Jest (Unit Tests)

We use Jest as our unit testing framework along with @testing-library/react. In these tests, we test functions or components in isolation from the rest of the larger system. We typically test functions/components that are used frequently throughout the app and/or have a lot of different code paths that are difficult to target all of in an integration or end-to-end (E2E) test.

For us, unit tests are about testing the fine-grained details. Integration and E2E tests do a good job of handling the broad strokes of the application generally working, but sometimes you need to make sure little details are correct and it would be too costly to write an integration test for each possible case.

For instance, we want to ensure that keyboard navigation works for our dropdown select component, but we don’t need to verify every instance of it in our app. We test the behavior in depth in isolation so that we can just focus on higher-level concerns when testing the pages that use that component.

Cypress (Integration Tests)

Cypress integration tests are at the core of our testing suite. When we started building out the StackPath portal they were the first tests we wrote because they deliver a lot of value for fairly small cost. Cypress renders our whole app in a browser and runs through test scenarios. Our entire frontend is running exactly as it would for a user. The network layer, however, is mocked. Every network request that would go to our GraphQL server is instead mocked with fixture data.

Mocking the network layer provides a number of benefits:

Tests are faster. Even if your backend is super fast, the number of calls made for an entire test suite run add up. With the responses being mocked, they can return instantly.
Tests are more reliable. One of the difficulties with full E2E tests is accounting for variability in the network and stateful backend data. When every request is mocked that variability is gone.
Hard-to-replicate scenarios can be simulated with ease. For instance, it would be difficult to reliably force calls to fail. If we want to test that our app responds correctly when a call fails, being able to force that failure is helpful.

While mocking our entire backend may seem like a problem, all of our fixture data is typed using the same generated TypeScript types our app uses, so it is guaranteed to be at least structurally equivalent to what an unmocked backend would return. For most of our tests, we are happy with the tradeoff that mocking the network provides.

The developer experience with Cypress is also really good. The tests run in the Cypress Test Runner which shows your tests on the left and your app running in a main iframe performing those tests. After a test run, you can highlight individual steps in your tests to see what your app was doing at that point. Since the test runner is itself running in a browser, you also have access to developer tools to help debug tests.

Oftentimes when writing frontend tests it can take a lot of time to assess what a test is actually doing and what state the DOM is in at a particular point in the test. Cypress makes this part really easy because you can just see it happening right in front of you.

These tests exemplify a lot of our state testing principles. Cost to value ratio is favorable, the tests very closely mimic how an actual user interacts with the app, and the only thing being mocked is the network layer.

Cypress (E2E Tests)

Our E2E tests are also written in Cypress, but for these we do not mock the network (or anything else). Our tests run against our actual GraphQL server which communicates with actual instances of our backend services.

E2E tests are immensely valuable because they can definitively tell you if something works or not. Nothing is being mocked, so it’s using the app exactly as a user would. E2E tests are higher cost as well though. They are slower, take more thought to prevent intermittent failures, and take more work to ensure your tests are always in a known state before running.

Tests typically need to start from a known state, do some operations, and then arrive at some other known expected state. With the integration tests, this is easy to accomplish because the API calls are mocked and thus are the same every test run. For E2E tests, it’s more complicated because the backend storage now holds state which could be mutated as the result of a test. Somehow, you have to ensure that when you start a test, you’re in a known state.

At the beginning of our E2E test run, we run a script that seeds a new account with new stacks, sites, workloads, monitors, etc by making API calls directly. Each test run operates on different instances of the data, but the test setup is identical. The seed script emits a file with the data our tests use when running (instance ids and domains mostly). This seed script is what allows us to get into a known state before running our tests.

Since these E2E tests are higher cost, we write less of them than integration tests. We cover the critical functionality of our app: user registration/login, creating and configuring a site/workload, etc. From our extensive integration tests, we know that our frontend generally works, so these just need to ensure that there isn’t something slipping through the cracks when hooked up to the rest of the system.

Downsides to this multipronged testing strategy

While we’ve been really happy with our tests and the general stability of our app, there are definitely downsides to going with this sort of multipronged testing strategy.

First, it means everyone on the team needs to be familiar with multiple testing tools instead of just one. Everyone needs to know Jest, @testing-library/react, and Cypress. Not only do we have to know how to write tests in these different tools; we also have to make decisions all the time about which tool to use. Should I write an E2E test covering this functionality or is just writing an integration test fine? Do I need unit tests covering some of these finer-grain details as well?

There is undoubtedly a mental load here that isn’t present if you only have one choice. In general, we start with integration tests as the default and then add on an E2E test if we feel the functionality is particularly critical and backend-dependent. Or we start with unit tests if we feel integration tests cannot reasonably cover the number of different details involved.

We definitely still have some gray areas, but patterns start to emerge after going through this thought process enough times. For instance, testing form validation tends to be done in unit tests due to the number of different scenarios and everyone on the team is aware of that at this point.

Another downside to this approach is that collecting test coverage, while not impossible, is more difficult. While chasing test coverage can result in bad tests just for the sake of making a number go up, it can still be a useful automated way of finding holes in your tests. The trouble with having multiple testing tools is that you have to combine test coverage to find out which parts of your app are truly not covered. It’s possible, but it’s definitely more complicated.

Conclusion

While some challenges exist when using many different testing tools, each tool serves its purpose and we think is worth including in our overall testing strategy. When starting a new application, or adding tests to an existing one, integration tests are a great place to start. Adding some base-level E2E tests around absolutely critical functionality early on is a good idea as well.

With those two pieces in place, you should be able to make changes to your application with pretty reasonable confidence. If you start to notice bugs creeping in, stop and assess what sort of tests could have caught those bugs and if it indicates a deficiency in the overall strategy.

We definitely did not arrive at our current test setup overnight and it is something we expect to keep evolving as we continue growing. For the time being though, we feel good about our current approach to testing.