DEV Community: Robert Gibb

Building Our E-Commerce Platform with Serverless FaaS

Robert Gibb — Sun, 26 Sep 2021 22:57:33 +0000

This post was published here on behalf of Devashish Pandey, a lead software development engineer at fabric who previously worked on platform and API development at SecureDB and StormDB.

With so many "something-as-a-service" options at our disposal for infrastructure, in 2017 we opted for function-as-a-service (FaaS) to build our platform of e-commerce services. Our mission was, and remains, to build for e-commerce services what AWS built for web services, or what we call the commerce fabric of the internet.

Building with FaaS and only paying for function execution time rather than constantly running servers (i.e. serverless) allowed us to serve customers early on without incurring high infrastructure costs. These customers were large, household names and referrals from our founding team who had transformed digital at Staples. Part of this transformation involved moving from the monolithic e-commerce platform IBM Websphere to a service-oriented architecture with open-source and custom-built software.

E-commerce veterans who knew our founding team wanted to transform digital for their companies, which led us to build our platform of modular commerce services in 2017. Using serverless FaaS with AWS Lambda supported these services reliably and efficiently, and it continued to support them through 2020 when we raised our $9.5M seed round and attracted customers outside of our circle of friends.

But as we raise more money (most recently our $100M Series B) and onboard global brands like GNC, life on the serverless backend is not as easy as it once was. As a result, some challenges that were inconsequential at the time are now more pronounced. In this post, I will explain how we direct these serverless challenges and show you how serverless FaaS powers our platform of commerce services.

Our Serverless Architecture

The current iteration of the fabric platform can be hosted in either AWS, GCP, Azure, and even Knative since all of these are supported by the serverless framework we use. However, in 2017 when we adopted serverless, AWS had the most mature offering so we made the simple choice in choosing Lambda for FaaS.

Flash forward to today and we still use Lambda and sparingly route requests through AWS Fargate, a container-as-a-service (CaaS) environment. Fargate is mainly used when we can't meet latency service level agreements (SLAs) with Lambda due to cold start problems. However, since Lambda powers 95% of all processes within our platform, we've found ways to direct these issues which I'll talk about below.

As for the way our products are structured, a data store layer is used, followed by a business layer in which most logic is programmed as you would with any other backend scenario. Lambda orchestrates event-driven workflows at the business and data storage layers.

The actual product offerings consist of commerce applications like a product information manager (PIM) and order management system (OMS), referred to as Co-Pilot Apps in the diagram below. (Co-Pilot is the name of the UI that merchandisers, marketers, and other business users interact with.) We also offer e-commerce storefronts that fast-track storefront development. Each type of offering has a separate yet connected business layer.

On the top level, each layer is supported by Lambda in a serverless manner. API gateway routes interface between them, and the logic layer and external services layer interface through integrations.

Every action, process module, and element of the platform is isolated into standalone functions in Lambda. The resulting layout achieves full separation of concern, with a single gateway route connecting with a single layer action/process at any given time. This cancels out the most common issues with resource sharing and allows each action/module to scale independently.

Managing Serverless

There are numerous tools engineers can use to manage serverless setups of the type I've discussed thus far. However, mainly because it was one of the earlier market representatives, we decided to use the Serverless Framework to manage serverless infrastructure through infrastructure as code (IaC).

Adopting the Serverless Framework as our infrastructure modeling and standardization tool enables quick spring-ups of environments. Further, the vendor-agnostic nature of the framework leaves room for any migration toward a cloud-agnostic, client-requested offering down the road. Using the Serverless Framework, we could support this offering while maintaining the same variables for mushroomed fabric platform instances.

In addition to these benefits, we found getting started with the Serverless Framework incredibly easy, even for a robust platform of applications and APIs like ours. If you want to try it yourself, just launch your terminal and install the node package manager. After that, a simple serverless command sets your environment up for configuration and management through the framework's CLI. A serverless.yml file is then created, which is where it all started for us in 2017!

Below is what a typical serverless.yml file configuration looks like using the Serverless Framework. The scripting language used is proprietary serverless.js, specific for a node.js app running on a MongoDB data store layer. This setup is similar to our own.

The provider section of the config file allows for the full exploration of cloud services providers. However, we continue to use AWS as our go-to provider as we've found that other providers cause capacity bottlenecks as we continue supporting more of the world's largest retailers and brands. After AWS is selected, each function across the implementation is associated with a handler and specific API gateways.

Using this template has dramatically improved the accuracy and speed of infrastructure configurations as we scale with serverless.

Serverless Challenges

As good as the story sounds so far, we have experienced our fair share of challenges with serverless and continue to experiment with ways to maintain business efficiencies while improving capacity and performance.

Capacity

While our current setup has enough capacity to support existing customers, there's always concern about having enough serverless infrastructure to support more, larger customers---especially during holiday shopping bursts. To get ahead of this, we are experimenting with augmenting Lambda with Amazon EKS using serverless containers with AWS Fargate.

Cloud-agnostic support is an option we are exploring as well. But before allowing customers to dictate their cloud provider, we need to better understand capacity with GCP and Azure. With customers calling the shots for infrastructure, capacity issues could also arise with AWS. For instance, if a customer selects an AWS region of deployment outside USA East and West, burst concurrency diminishes.

Performance

What makes serverless efficient for your business can cause inefficiencies with application performance, particularly when using Lambda functions. These inefficiencies occur during the cold start duration when Lambda is downloading your code and starting a new execution environment.

To reduce inefficiencies, we have tested and implemented several performance optimizations. One workaround we use for avoiding cold starts is periodic warmup events that counteract environment timeouts. We are also exploring Lambda@Edge (Lambda + Amazon Cloudfront) to offset any latency caused by Lambda. However, while doing this, we are trying not to defeat the cost-benefit variable for which Lambda was chosen in the first place.

Stick with Serverless?

Despite these challenges, we remain loyal to serverless infrastructure and putting product development ahead of server management. As one of the fastest-growing tech startups in e-commerce this year, serverless has served us well in supporting new customers while enabling fast iterations of new commerce services like Subscriptions and Loyalty Management.

While AWS Lambda is not yet perfect, we have noticed that, as fabric grows, so too does Lambda in terms of capacity. This gives us breathing room to test iterations of Lambda and other serverless offerings from AWS while continuing conversations and experimentation around matching the right architecture with the right infrastructure.

How Event Driven Systems Work in Commerce

Robert Gibb — Tue, 20 Oct 2020 13:46:50 +0000

Author's Note: This post was created in collaboration with Sajjad Heydari.

Assume someone is waiting for a package to be delivered to their house. The impatient customer keeps looking out their window, waiting for the delivery truck to pull up. The patient customer keeps doing other things until they hear the doorbell ring. This everyday example can easily be transformed to the computer science domain where two types of systems exist: a polling system and an event driven system.

The polling system acts like the impatient customer. In a commerce-related scenario, it keeps polling the system for new updates such as orders and payment authorizations while the event driven system relies on asynchronous event handlers to notify it of updates in the system.

The event driven system is easier to develop and more efficient, but it requires special infrastructure to work. The good news is that some software and platforms like Fabric support event driven systems. In commerce, these systems create more streamlined operations so customers get their orders faster.

In this post, we will define what events are and how retailers can leverage them to create a system that is more efficient. But, before doing that, let’s look at an example of how an event-driven system might support a retailer and its customers.

Example of an Event Driven System

After adding multiple items to his cart, Joe goes to the checkout page on a retailer’s website. He fills in his address and payment information, then places his order. But what happens next?

With a polling system, Joe's order is stored in a database. Some time later, a script or person queries the database for any new orders and processes the payment. Another script or person in the warehouse then queries the database and starts the delivery procedure.

This way of handling payment and fulfillment is slow and prone to error. Now more than ever, handling events in real time is important in commerce given the rise in online ordering.

Polling a system periodically can create a bottleneck in operations, especially if there is an unexpected spike in order activity. Furthermore, querying a database when no new orders have been placed results in unnecessary infrastructure costs.

In an event-driven system, here is how Joe's order is processed:

Joe's order is stored in a database like before, but this time an event is emitted to an event router.
The event router looks up what that can handle this type of event and discovers that the payment processing script is responsible for it.
The payment processing script calls the payment processor (e.g. Stripe) with the appropriate payment data.
The payment processing script emits a payment authorized event. This event initiates an order confirmation email to Joe and a notification to the warehouse for fulfillment.

Parts of an Event Driven System

The Event

Event based systems in computer science go back to the 1950s where they were first designed to handle asynchronous events such as Input/Output). Edsger Wybe Dijkstra designed the first interrupt handler on the Electrologica X-1 that is still used in modern systems to this day. The modern day's events are the same concept, only they are used in different environments.

When a CPU interrupt emerges, the system changes its behavior to address the incoming interrupt. This could be a new keystroke pressed or a notification that this process is about to be terminated. The interrupt doesn't tell the process what to do, only that some event has happened. It is the process's duty to handle the interrupt and decide what to do with it.

The same thing happens with events in event driven systems. When an event is created, the event itself shouldn't decide what happens; it should only describe what has happened, be it a change in state or an update. For instance, in a modern commerce tech stack, the event router may send an abandoned cart event to an email platform like Klaviyo that decides to enroll Joe in an abandoned cart email sequence.

The Event Router

How an event is structured and emitted relies heavily on the event router. There are many different types of event routers, including AWS EventBridge, Apache Kafka, and Microsoft Event Hub.

Events are JSON documents that follow a specific scheme. The top-level fields are the same across all event types, but the detail of each event is application dependent.

For example, if Joe successfully ordered an item from a retailer, his "order created" event would look something like this...

{
  "cartId": "314919ab8aed637123f804eb",
  "orderCurrency": "USD",
  "orderTotal": 27.64,
  "orderId": "314919ab8aed637123f804eb",
  "shipTo": [
    {
      "address": {
      "name": {
        "first": "John",
        "last": "Smith"
      },
      "phone": {
        "number": "5555555555",
        "kind": "mobile"
      },
      "email": "user1@gmail.com",
      "street1": "1250 main street",
      "street2": "",
      "city": "HAVRE DE GRACE",
      "state": "MD",
      "country": "US",
      "zipCode": "21078-3213",
      "kind": "shipping"
      },
      "shipmentCarrier": "FedEx"
    }
  ],
  "items": [
    {
      "itemId": 4912,
      "price": 122.43,
      "quantity": 1,
      "total": 122.43
    },
    {
      "itemId": 8123,
      "price": 21.99,
      "quantity": 3,
      "total": 65.97
    }
  ]
}

The most straightforward way to handle events in a custom way is by using a function-as-a-service (FaaS) solution like AWS Lambda that natively supports Java, Go, Powershell, Node.js, C#, Python, and Ruby, but provides a Runtime API to allow usage of any other programming language.

Developing Lambda functions is easier and cheaper to operate since AWS only charges for the time that they are running. This is known as serverless, another technology we use at Fabric to make commerce more extensible and scalable.

Benefits of Event Driven Systems

Event driven systems are ideal for commerce since many commerce setups contain different services and microservices that benefit from being decoupled. Payments, promotions, pricing, subscriptions, search, and reviews are just a few of these services.

Decoupling allows for a faster development cycle since each service can be developed independently, cheaper runtime since each service can be scaled independently, and faster page load times. Event driven systems facilitate the decoupling of services as different services can talk to each other through the event router without directly depending on each other.

Another benefit of event driven systems are their potential to handle faulty situations. If two services rely on each other and one of them fails for any reason, the other one wouldn't be able to function properly. But through the help of event routers, if a service becomes inaccessible for any reason (such as updates or errors), the event router stores the event until it can be delivered to its intended handlers. For instance, in a scenario where the payment processing script is unavailable, the shopping cart should still be functional.

Conclusion

In this post we provided an overview of how event driven systems streamline commerce. For a more detailed view of how events and event routers allow different services to communicate better in a commerce tech stack, check out this video from AWS.

Optimizing First-Frame Bitrate for HLS with Serverless Edge Compute

Robert Gibb — Wed, 18 Dec 2019 20:25:01 +0000

In this article we’ll show you how to use StackPath’s serverless edge product with HLS to deliver the right bitrate to the right device with the lowest possible delay. Whether you’re using cloud serverless for HLS or a different streaming solution entirely, this article will introduce you to the possibilities of optimizing streams with serverless scripting and low-latency edge compute.

Note: This tutorial is detailed and requires considerable effort to follow, but completing it can significantly reduce content delivery costs for you and buffer time for your users. It could be the catalyst for inspiring an evolution of how your video content is consumed.

Introduction

When responding to an HLS request, the streaming server determines which video quality (i.e. ts file) the client will attempt to play before switching to a lower or higher quality video. This switch depends on available bandwidth and device type.

Switching to the wrong quality first degrades the user experience considerably. For instance, sending a high quality file to a low-end device may cause the video/device to freeze, even on a good connection. And sending a low quality file to a high-end device with a good connection may cause the viewer to experience a low quality viewing experience for a prolonged period.

It may seem that sending a medium quality file first is a good solution, but it’s actually quite lazy. Instead, you can solve for the best solution in every case by using serverless scripting.

Serverless scripting, also known as function-as-a-service (FaaS), allows you to optimize responses on a per-device, per-request basis without touching your origin server’s configuration or code. There’s also no loss of cacheability and you can decrease latency further by making the decision at the edge instead of in a far-off data center.

Understanding how HLS works

For developers who are not already aware, HLS works by sending the client an index, also known as a manifest. This index contains a list of ts files called chunks that make up the video. The client requests the appropriate chunk based on play time.

In an adaptive bitrate implementation, the manifest provides links to several alternative playlists called variants instead of listing chunks directly. All variants have an identical number of chunks and video content, but they differ in bitrate and resolution (i.e. quality).

According to the HLS spec, clients should request the top listed quality in the manifest, play it, and calculate the available bandwidth (i.e. chunk size and download time). Based on the calculation, the client switches to a higher or lower quality until the bitrate of the video is lower than the maximum available bandwidth. This ensures seamless play.

(The image above is a sample HLS index listing three variants, each with a different bandwidth, resolution, and index file.)

To summarize, choosing the video quality based on the available bandwidth is the responsibility of the client, but choosing the default quality to start with is the responsibility of the server. This default quality is usually either the highest one, or simply the first one uploaded to the server. In all cases, it is the same regardless of which device requests the video and where the video is requested from.

Understanding the problems with HLS

We’ve established that the default video quality isn’t often the best to include in the initial response with HLS, but let’s look at some different scenarios to further understand why.

Scenario #1: High-End Device | High Quality Video | Low Bandwidth | Any Screen Size

In this scenario, the device downloads the master playlist (playlist.m3u8) and starts with best quality as dictated by the server. The device downloads the first chunk and it takes a whole minute (62,352ms). During this time, the viewer is waiting, not watching anything. This is a terrible video streaming experience.

Scenario #2: High-End Device | Medium Quality Video | Medium Bandwidth | Small Screen

In this scenario, the device spends seven seconds downloading a 10-second second chunk. After this, it starts to play.

So we’ve fixed the problem in Scenario #1 but we still waited seven seconds for the video to start. But we’re playing in a 800x450 pixel player and medium quality has a resolution of 1280x720 with a bitrate to match. Therefore, low quality is beyond enough here and waiting seven seconds for a higher quality file is unnecessary. If we started with a screen-appropriate quality we’d risk less delay before optimization catches up.

Scenario #3: High-End Device | Medium Quality Video | High Bandwidth | Big Screen

In this scenario, the device downloads the initial ts file quickly and, upon downloading, plays 10 seconds of medium quality video before switching to high quality.

So we’ve eliminated the problem in Scenario #2 but we still had to tolerate 10 seconds of lower quality video before optimizing up. And the next time we load another video (even on the same site), we’re going to go through all of this again. It would be better if we could remember the current quality and start with it.

Note: There’s an edge case scenario containing a low-end device and high-quality start but there’s not much to expand on. As you might expect, the video starts to stutter and lag.

Solutions for HLS problems

Client-side solutions

Naturally, all these scenarios are much better solved for on the client side. For example, VideoJS, one of the most popular libraries for HLS playback on the web, will immediately start with the highest quality whose resolution fits the current player size, eliminating one of the issues above.

But not all players do this. For instance, ExoPlayer, Google's HLS player on Android devices, performs no such capping. However, it does let developers customize its selection logic to fit their use case.

Server-side solutions with serverless

The vast majority of media servers make use of a static master playlist file and custom development is often required to create an adaptive system. But even if you have the logic for changing the order of file delivery at your origin server based on device type, for instance, what do you do with your CDN? Cache the generated master playlist for each one of hundreds of different user agents? With the request sent all the way to origin every time you see a new one? Or perhaps you don’t cache the master playlist and delegate processing to your origin completely? Either way, you’re still slowing everything down and adding a excessive load to your origin.

Edge serverless solves this by allowing you to implement custom logic right at the edge. Moreover, this approach is backend agnostic. A serverless script can manipulate the fetched resources as a man-in-the-middle with zero reliance on any feature of the backend system.

Objective

This tutorial will show you how to use serverless scripting to optimize HLS quality by device at request time, at the edge server, without any contribution from your origin server or any effect on cacheability.

Serverless scripting allows you to write custom logic (implemented by js code) to process requests at the edge server. It can modify a client's request before fetching it and also modify the response. The possibilities with this serverless use case are endless and, therefore, the techniques in this tutorial serve as more of a demonstration than a definitive solution.

Prerequisites

1) An origin serving a multi-quality HLS stream. This may be powered by nginx-rtmp, Mist, Wowza, Nimble, or any form of media server. As an example we'll use the url http://awesomeMedia/, serving an HLS stream at http://awesomeMedia/hls.m3u8. Here is the example playlist served at that URL:

#EXTM3U
#EXT-X-VERSION:3
#EXT-X-STREAM-INF:BANDWIDTH=5000000,RESOLUTION=1920x1080
1080p.m3u8
#EXT-X-STREAM-INF:BANDWIDTH=2800000,RESOLUTION=1280x720
720p.m3u8
#EXT-X-STREAM-INF:BANDWIDTH=1400000,RESOLUTION=842x480
480p.m3u8
#EXT-X-STREAM-INF:BANDWIDTH=800000,RESOLUTION=640x360
360p.m3u8

Normally an encoder in your media pipeline will take care of auto generating this. In the file above, we are serving four different qualities at 1080p, 720p, 480p, and 360p—each with a defined bitrate appropriate to the resolution, as defined in the #EXT-X-STREAM-INF lines above.

2) A StackPath account with serverless scripting enabled. Refer to this tutorial for instructions on enabling serverless for your site for $10/month. Alternatively, you can test the concept for free using the Sandbox.

Step 1: Set up your site with serverless

The first step is to sign up for a StackPath account, add your site, and enable Serverless Scripting.

1) Sign up for a StackPath account here. Then select Website & Application Services and choose the services you require.

The Edge Delivery Bundle is recommended but the bare minimum is the CDN service.

2) Log in to the StackPath Control Panel and create/select an active Stack.

3) Select Sites in the left-hand navigation bar and click Create Site.

4) Check Serverless Scripting, enter your domain (or IP address), and select Continue.

5) Finally, take note of your site’s edge address, which you can now use for delivery of your content all over the globe. Alternatively, you can use the StackPath DNS service to have your domain name resolve to the ideal edge and always serve content from StackPath’s CDN.

In this example, our HLS video is now accessible at http://j3z000b6.stackpathcdn.com/HLS.m3u8 with the .ts chunks available on the same path.

Step 2: Create your script

Now that we have a site with Serverless Scripting enabled we can deploy our first script. We can do this manually through the web control panel or through the Serverless Scripting CLI. The web option is suitable for testing or manual deployment. It’s simple, intuitive, and comes with a nice web editor. But if you’re developing with your own tools or integrating with a CI/CD pipeline you’ll appreciate the ease and automatability of a CLI single-line deployment.

In this step our objective is to deploy a script to process requests for the path http://awesomeMedia/HLS.m3u8. When a user visits the site, rather than fetching the file HLS.m3u8 straight from cache (or origin), the serverless engine will execute the script to decide what happens. The script could choose to block the request, deliver the file normally, modify the file, or write a new response altogether.

Option 1: Using the control panel

Navigate to the control panel and click Scripts. Then click the Add Script button. You’ll be taken straight to the code editor. Enter a name for your script and the route it will run for. You can also enter multiple routes, or use wildcards.

Option 2: Using the serverless CLI

To use the Serverless CLI we’ll have to install the CLI package, create a configuration file named sp-serverless.json in the project directory, and publish. But first we’ll need an API key and ID information.

1) Using the dashboard, take note of your Stack ID and Site ID.

2) To generate API keys for use by the CLI, go to API Management in the dashboard and click Generate Credentials. Then record the Client ID and API Client Secret.

3) Install the CLI by running the following command in a terminal.

npm install -g @stackpath/serverless-scripting-cli

4) To configure the CLI, run the following command.

sp-serverless auth

This will set up authentication for the CLI to use your account. It will ask you for the details above interactively.

5) Clone the skeleton repository by running:

git clone https://github.com/stackpath/serverless-scripting-examples/tree/master/hls-initialization.git

Or create your files manually. If you do this, create a directory for the project. Then create the following two files.

A file named sp-serverless.json with the following content. Substitute your Stack ID and Site ID where indicated.

{
  "stack_id": "",
  "site_id": "",
  "scripts": [
    {
      "name": "HLS Optimizer",
      "paths": [
        "HLS.m3u8"
      ],
      "file": "default.js”
     }
  ]
}

This file indicates the deployment details to the Serverless CLI. The paths array indicates the routes your script will be active for and multiple scripts/paths can be configured using the same file. You can add extra scripts for extra routes by copying the first object in the scripts array and modifying as needed.

A file named default.js with the following content. This will be the script you’ll deploy. For now it’s the default skeleton script and has no effect on response.

addEventListener("fetch", event => {
  event.respondWith(handleRequest(event.request));
});
/**
 * Fetch and return the request body
 * @param {Request} request
 */
async function handleRequest(request) {
  try {
    /* Modify request here before sending it with fetch */
    let response = await fetch(request);

    /* Modify response here before returning it */
    return response;
  } catch (e) {
    return new Response(e.stack || e, { status: 500 });
  }
}

6) Finally, deploy by issuing the following command.

sp-serverless deploy

This is the command to run every time you update your script or make changes to its configuration in the sp-serverless.json file.

From this point forward this is a development tutorial. You can skip this and use the finished product by going directly to Step 6 at the end of the tutorial, with one caveat: you’ll need to deploy a container as detailed in Step 3. Alternatively, if you’d like to follow along without creating a StackPath account you can use the Sandbox.

Step 2: Parse the HLS master playlist

If using the Sandbox, start by filling in the origin route you’d like to apply the script to in the URL textbox (e.g. http://awesomeMedia/HLS.m3u8) and select Raw as the display mode (optional). The Sandbox provides console access, so that you can issue console.log commands and view stack traces live. Other options for debugging serverless scripts are to return information in the body of the response, or in its headers.

The default (skeleton) script just returns the original manifest as is. The essential ingredient is simply a request handler bound to fetch that receives the original request from the client, fetches the response, and returns it untouched.

addEventListener("fetch", event => {
  event.respondWith(handleRequest(event.request));
});

/**
 * Fetch and return the request body
 * @param {Request} request
 */
async function handleRequest(request) {
  try {
    /* Modify request here before sending it with fetch */

    let response = await fetch(request);

    /* Modify response here before returning it */

    return response;
  } catch (e) {
    return new Response(e.stack || e, { status: 500 });
  }
}

The code is helpfully marked with suggested locations for modifying the request, the response, and returning them.

First, let's pass the request upstream and examine the response before returning it.

Modify the script as shown below.

/* Modify response here before returning it */
const upstreamContent = await response.text();
/* Log the response body to the console */
console.log(upstreamContent)

return new Response(upstreamContent, {
    status: response.status,
    statusText: response.statusText,
    headers: response.headers
    });

After clicking Run Script we are greeted with the result in the console, showing the contents of the master playlist:

[info] #EXTM3U
#EXT-X-VERSION:3
#EXT-X-STREAM-INF:BANDWIDTH=800000,RESOLUTION=640x360
360p.m3u8
#EXT-X-STREAM-INF:BANDWIDTH=1400000,RESOLUTION=842x480
480p.m3u8
#EXT-X-STREAM-INF:BANDWIDTH=2800000,RESOLUTION=1280x720
720p.m3u8
#EXT-X-STREAM-INF:BANDWIDTH=5000000,RESOLUTION=1920x1080
1080p.m3u8

Note: We’ve changed the original line returning the response in the code. Rather than simply returning the same response object, we’re generating a new one with the same header and content. This is because attempting to return the original object after having already resolved the promise of text() will result in an error.

We now have the HLS master playlist in a variable so it’s time to parse it for available variants (qualities).

For a master playlist with several variants, the playlist starts with several “header” tags, then defines every variant in a line beginning with “EXT-X-STREAM-INF”. More information can be found in the HLS spec. But for the purposes of this script a regex solution is ideal, as implemented in this new function:

function parseM3u8(body) {
  // Parse M3U8 manifest into an object array, containing bitrate, resolution, and codec
  var regex = /^#EXT-X-STREAM-INF:BANDWIDTH=(\d+)(?:,RESOLUTION=(\d+x\d+))?,?(.*)\r?\n(.*)$/gm;
  var qualities = [];
  while ((match = regex.exec(body)) != null) {
    qualities.push({bitrate: parseInt(match[1]), resolution: match[2], playlist: match[4], codec: match[3]});
  }
  return qualities;
}

This function continues to look for #EXT-X-STREAM-INF lines and extracts bandwidth, resolution, playlist link, and, finally, preserves any extra information found (usually codec information). To explore how it works, you can use a handy tool called RegExr.

To test this function, we can add a console log line anywhere in the handler function:

console.log(parseM3u8(upstreamContent))

[info] [ { bitrate: 800000,
    resolution: '640x360',
    playlist: '360p.m3u8',
    codec: '' },
  { bitrate: 1400000,
    resolution: '842x480',
    playlist: '480p.m3u8',
    codec: '' },
  { bitrate: 2800000,
    resolution: '1280x720',
    playlist: '720p.m3u8',
    codec: '' },
  { bitrate: 5000000,
    resolution: '1920x1080',
    playlist: '1080p.m3u8',
    codec: '' } ]

And…success! The test worked as expected.

Step 3: Parsing the User Agent

By examining the headers of the request before sending it upstream we can find the user agent of the client browser. In the marked location of the skeleton script for modifying requests, we log the user agent by adding two lines of code.

/* Modify request here before sending it with fetch */
const userAgent = request.headers.get('User-Agent');
console.log(userAgent)

[info] Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:69.0) Gecko/20100101 Firefox/69.0

Aside from the original headers sent by the browser, there is a set of additional StackPath headers that are added to the original request. These contain the IP address, location, server region, and other information about the request. You can find a list of them in the StackPath Developer Docs.

In order to make meaningful decisions about which quality to prioritize and which video resolutions to send, there are three main pieces of information we need to learn about from the client device:

Type (desktop/phone/tablet)
Screen resolution
Power measurements to be used for deciding whether to cap at a certain quality for devices with poor processing capabilities (optional)

While there are many JavaScript libraries for parsing the user agent, none of them can tell us the screen resolution of a mobile device. Doing so not only requires a complicated parser that understands all the different ways UA strings are formed, but also a database with information about the devices once they’re identified. This parser and database combination is formally called a Device Description Repository (DDR).

Today, there are plenty of commercial DDR services. But only one is both reliable and free: OpenDDR. This service can be deployed in the form of a docker container, providing a simple API that returns JSON-formatted data for a given UA that we will call from within our script.

Calling the API in Serverless

Before deploying OpenDDR, we’ll use the demo endpoint to develop the necessary function in our script and test the integration. The function fires a new http request to the API, parses the response, and passes the returned information to the main handler.

 /* Modify request here before sending it with fetch */
    const userAgent = request.headers.get('User-Agent');
    console.log(userAgent);
    /* Consult the DDR microservice in regards to the user agent */
    const deviceData = await getDeviceData(userAgent);
    console.log(deviceData);

async function getDeviceData(ua) {
  try {
    const res = await fetch('http://openddr.demo.jelastic.com/servlet/classify?ua='+ua);
    const data = await res.json();
    return data.results.attributes;
  } catch (e) {
    throw new Error('DDR communication failed')
  }
}

[info] Mozilla/5.0 (Linux; Android 8.0; Pixel 2 Build/OPD3.170816.012) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/67.0.3396.87 Mobile Safari/537.36
[info] { model: 'Pixel XL',
  ajax_support_getelementbyid: 'true',
  marketing_name: 'Pixel XL',
  is_bot: 'false',
  from: 'openddr',
  displayUnit: 'pixel',
  displayWidth: '1440',
  device_os: 'Android',
  id: 'PixelXL',
  xhtml_format_as_attribute: 'false',
  dual_orientation: 'true',
  nokia_series: '0',
  device_os_version: '8.0',
  nokia_edition: '0',
  vendor: 'Google',
  cpu: 'Qualcomm Snapdragon 821',
  mobile_browser_version: '67.0.3396.87',
  ajax_support_events: 'true',
  is_desktop: 'false',
  cpuRegister: '64-Bit',
  image_inlining: 'true',
  ajax_support_inner_html: 'true',
  ajax_support_event_listener: 'true',
  mobile_browser: 'Chrome',
  ajax_manipulate_css: 'true',
  displayHeight: '2560',
  cpuCores: 'Quad-Core',
  is_tablet: 'false',
  memoryInternal: '32/128GB, 4GB RAM',
  inputDevices: 'touchscreen',
  ajax_support_javascript: 'true',
  cpuFrequency: '2150 MHz',
  is_wireless_device: 'true',
  ajax_manipulate_dom: 'true',
  is_mobile: 'true',
  xhtml_format_as_css_property: 'false' }

The response from the DDR contains more info than the UA itself has. In particular, we are interested in displayWidth and displayHeight, but the information in there is pretty extensive.

Note: For a desktop (or laptop) device, the display resolution is still rarely identifiable using this method. For example, the UA for Google Chrome 77.0 on Windows 10 is the same regardless of your display. Also, viewers may rarely play the video in full screen depending on the type of content you’re serving.

Deploying a container

The function above is currently making use of the demo API endpoint offered on the OpenDDR website and is unsuitable for production use. Therefore, we’re going to be deploying our own instance using an Edge Container.

StackPath’s Edge Containers offer a way of hosting containers in diverse locations so that no request has to travel all the way to a centralized computing node. This is particularly useful for stateless applications such as the one we’re deploying. (A more detailed tutorial can be found here.)

To start, head back to the control panel and click Workloads. Then click Continue to add Edge Compute to your Stack.

Now, click Create Workload.

Now you can configure your container:

Name: (Any name works)
Image: Enter 0x41mmar/openddr-api:latest, which refers to this container
Anycast IP: Disabled for now (more on this below)
Public Ports: Enter 8080, the only port exposed by the container image above
Spec: The resources allocated to each instance. SP-1 (1 vCPU, 2GB Mem) should be enough.
Deployment Target: Locations where the container will be deployed. Select a name and one or more locations

Note: Unlike the ephemeral IPs assigned to containers when they’re started, the Anycast IP is a static IP assigned to the workload for its entire lifetime. The Anycast IP has significant performance benefits. Traffic towards an Anycast IP will enter the StackPath Network at the closest Edge Location and be routed to the instance using [StackPath’s private backbone(https://blog.stackpath.com/network-backbone-speed/).

Finally, click Create Workload. You’ll be redirected to the container’s overview page where you’ll see the container starting. When it’s fired up, you’ll see the IP addresses associated with it. Make note of the public IP.

Finally, let’s replace the demo URL in the classifier function with our own container’s, using the IP above.

const res = await fetch('http://101.139.43.73:8080/classify?ua='+ua);

Step 4: Decision Tree

The information required to make a decision about the optimization has been collected. Now the objective is to decide on the order of variants in the master playlist. We’ll create a simple algorithm to determine the best way to sort the variants in the file based on the following rules:

Never start with a variant whose resolution is higher than the display, even if the device will optimize up afterwards.
If the best quality variant satisfying condition (1) has a relatively high bitrate (>=4 Mbps), start one step lower.
If device is quite old/low-end, start at the lowest quality and cap to display resolution to avoid performance issues.
If nothing is known for sure (desktop devices), assume resolution is appropriate for 720p and apply the rules above.

Rule 4 makes sense because, according to StatCounter, at least 52.91% of desktop/laptop devices in use today have a display above 720p, but only 19% have 1080p displays.

The resulting algorithm is shown in the diagram below. Note that this is not a definitive solution and can be changed easily to fit your unique scenario.

This is straightforward to implement in JavaScript, as is the execution of the sorting and capping. This translates into the following function.

function decisionTree(deviceData, qualities) {
  /* Logic deciding on ordering and capping of available qualities */
  /* Returns config object to the spec of the output function above */

  // get higher dimension of resolution
  var res = Math.max.apply({},[deviceData.displayHeight,deviceData.displayWidth]);
  //is it a desktop device? assume 1280x720
  if (deviceData.is_desktop == "true")
    return {top: 2, cap: false, res: 1280};
  else {
    // mobile device. Is it an old device?
    if ((deviceData.device_os == "iOS" && parseInt(deviceData.device_os_version) < 7) || (deviceData.device_os == "Android" && parseInt(deviceData.device_os_version) < 6) || (deviceData['release-year'] < 2012))
return {top: -1, cap: true, res: res};
   else
     return {top: 2, cap: false, res: res};
  }

  //default
  return {top: 2, cap: false, res: res}
}

All we have to return is the assumed display resolution, our order decision, and capping decision. The code is more or less a direct implementation of the graph above. One thing to note is the nature of variables returned by OpenDDR, as most of them are strings and require some conversion.

More complex decision making is easy to implement in much the same way. For example, Apple provides guidelines on bitrates for particular resolutions.

Step 5: Output and Putting All Together

Now we need to rewrite the master playlist from scratch using the available information and decisions from the decision tree above. The function below is extensively commented and easily readable, but is basically sorting qualities, moving the required quality to the top of the list, and, if needed, deleting one that are too high.

First, we check if certain qualities need to be removed by checking the boolean config.cap. If some do, we filter the qualities array to remove offending variants unless there is only one of them.

Second, we sort the available variants in descending order by bitrate, unless it is required to place the lowest one on top (config.top=-1). If the top quality is to be the first, we’re done. If we’re to apply the 4Mbps rule, then we progressively test qualities for the required conditions (<=display, <4Mb).

function writeSortedManifest(qualities, config) {
  /* Sort qualities, optionally cap at a certain resolution, */
  /* then rewrite into correct HLS master playlist syntax */
  /* config = {cap: bool, top: int, res: int} */
  // top = 1: highest quality first, 2: highest quality within res or next one if >4Mbps, 0: middle quality first, -1: lowest quality first

  //cap
  //remove qualities with a resolution higher than a certain value (player resolution) 
  if (config.cap) {
    newQualities = qualities.filter((x)=> Math.max.apply({},x.resolution.split('x')) <= config.res );
  // anything left?
  if (newQualities.length > 0)
    qualities = newQualities;
  }

  // sort array so either best or worst quality is the first
  // dir = 1 for descending, -1 for ascending. use 1 for anything except top==-1
  // if top==-1, this is all we need to do
  var dir = config.top==-1 ? -1: 1;
  qualities.sort((a,b) => (a.bitrate>b.bitrate)? -1*dir: dir)

  //if applying resolution rule (top==2), process from top to bottom to find variant satisfying conditions
  if (config.top==2) {
    for (var i in qualities) {
      // assume it's this one for now
      var topChoice = qualities[i];

      // convert "1280x720" to 1280; accomodate top dimension and the rest will be fine if using a sane aspect ratio
      var topDim = qualities[i].resolution.split('x').sort()[1];

      // For this variant:
      // is res <= display?
      if (topDim <= config.res) {
        // yes! but is bitrate <4Mbps?
        if (qualities[i].bitrate < 4000000) {
          // great! done here.
          break;
        }
        else if (qualities[i+1]) {
          //it's not, so choose next option if it exists
          topChoice = qualities[i+1];
          break;
        }
        else {
          // next option doesn't exist? ok, fine. We'll take this one
          break;
        }
      }
    }
  // Now let's move the top choice top
  qualities.splice(0,0,topChoice);
  }

  //if middle quality required to be the first, move it there
  if (config.top==0) {
    var m = Math.floor(qualities.length/2);
    var middleItem = qualities[m];
    qualities.splice(m,1);
    qualities.splice(0,0,middleItem); 
  }

Going back to the main handler function, we just need to call the functions and pass the data as needed.

async function handleRequest(request) {
  try {
    const userAgent = request.headers.get('User-Agent');
    var deviceData, response;
    [deviceData, response] = await Promise.all([getDeviceData(userAgent), fetch(request)]);
    const upstreamContent = await response.text();
    var variants = parseM3u8(upstreamContent);
    var config = decisionTree(deviceData, variants);
    var output = writeSortedManifest(variants, config)
    /* Return modified response */
    response.headers.set("Content-Length", output.length)
    return new Response(output, {
    status: response.status,
    statusText: response.statusText,
    headers: response.headers
    });
  } catch (e) {
    return new Response(e.stack || e, { status: 500 });
  }
}

Notice how we’ve made sure to send the two external IO requests asynchronously. The DDR request has no dependency on the upstream fetch, and vice versa. This saves us time.

One little quirk in this code is the line right before the return statement that calculates the new content-length. This is necessary because the length (in bytes) of the body may have changed after being regenerated due to the ambiguity of new line characters. Our generator function only uses new line characters (\n), but the original file may or may not have used carriage returns (\r). A simple alternative would be to preserve the original line breaks with a slight modification of the regex and generator function, and have the final line be the exact same length as the original.

Step 6: Final Deployment and Testing

If you’re using the web control panel to upload your code you can find the complete code on GitHub here. Or, if you’re using the CLI and have followed the instructions in Step 1, you’ll have cloned a repository that already contains this code. Simply modify your sp-serverless.json file to use hls.js rather than default.js and replace the URL in line 53 (getDeviceData) with that of your own openddr container, per Step 3.

{
  "stack_id": "",
  "site_id": "",
  "scripts": [
    {
      "name": "HLS Optimizer",
      "paths": [
        "HLS.m3u8"
      ],
      "file": "hls.js"
     }
  ]
}

Note that the file will have had a script ID added automatically if you’ve deployed before. If you have, be sure to leave it in, then simply enter the following:

sp-serverless deploy

And that’s it!

With the script saved and deployed, we can now test the response with a variety of devices. The fastest way to do this is to use the Developer Tools available in your favorite browser. You can also use an extension/addon for the purpose of device switching.

Here are the results for a few different devices:

Device	UA	First Variant	Capping	Comment
Windows Laptop	Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/61.0.3163.100 Safari/537.36	720p	N/A	Guessing display is 720p capable, and bitrate of 720p variant is 2.8Mbps, acceptable
Google Pixel 2	Mozilla/5.0 (Linux; Android 8.0; Pixel 2 Build/OPD3.170816.012) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/67.0.3396.87 Mobile Safari/537.36	720p	N/A	Definitely 1080p capable, but we’re not risking that high a bitrate on first request, falling down to 720p
Galaxy Ace 3	Mozilla/5.0 (Linux; Android 4.2.2; GT-S7275R Build/JDQ39) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/48.0.2564.95 Mobile Safari/537.36	360p	360p is the only quality sent	Old low-end device, anything more is quite wasted on it
HTC One M8	Mozilla/5.0 (Linux; Android 4.4.2; HTC6525LVW Build/KOT49H) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/31.0.1650.59 Mobile Safari/537.36	360p	N/A	Old device now, but not low-end, and display is good. Starting low but sending all qualities.

Conclusion

This article demonstrates the potential for Serverless Scripting as a solution for HLS streaming optimization. The ideal solution, as always, remains proper optimization on the client side. But with so many different players implementing so many different behaviors a solution of this type can make a significant difference to the end viewer’s experience. Serverless Scripting makes this quite easy and accessible, particularly when compared to the alternative of dynamic processing at the origin.

The decision algorithm implemented may or may not be fitting for a particular audience or system, but we hope that it is explained and demonstrated satisfactorily for any developer to modify it to fit her own needs.

Taking this even further

A number of improvements can be made to the script to make it even more powerful.

Cookies: The entire problem of first-variant selection stems from the fact that the bandwidth is unknown on first play. But what if the viewer has just watched another stream from your site? Use cookies to remember what variant the client last settled on within a certain time window.
Client-side device detection: If the client is viewing the video inside a web or phone app you control, a bit of JS can provide much better information than we can find with the user agent alone. It’s possible to have some JS on your website that writes this information in a cookie for the Serverless Script to consume, and thus mitigate the lack of information about desktop devices (for instance). Naturally, this kills the quality of being server-agnostic and not requiring any changes to your origin configuration or deployment, but the trade off can be useful.
Error Checking: Upgrade the script to deal with various error scenarios. What if the DDR API is unresponsive? Information ambiguous? Decide on which assumptions to make and how to order variants.
Statistics: The numbers used in the assumptions above (e.g. a high bitrate is 4Mbps) are based on industry recommendations and general experience, but proper statistics are lacking. For example, we could enhance the script by making more statistically sound assumptions about available bandwidth for a particular region—or even for a particular class of device in a particular region, and so on.
Statistics II: Following from the point above, we can use a microservice with a db (similar to the DDR one above) to keep track of which devices, IPs, and regions settle on certain qualities, or which have problems starting the stream. This is useful for studying your audience and optimizing your configuration, but bonus points if you use the information automatically while sorting variants to adjust the rules as needed. For instance, if we have enough data to see that iPhones on Vodaphone 4G in Manhattan settle on 2Mbps most often, just start them there. This takes the previous point to its logical conclusion, but beware of the law of large numbers.

What is pub/sub messaging? A simple explainer.

Robert Gibb — Fri, 12 Jul 2019 17:26:46 +0000

Definition

Pub/sub is shorthand for publish/subscribe messaging, an asynchronous communication method in which messages are exchanged between applications without knowing the identity of the sender or recipient.

Overview

Four core concepts make up the pub/sub model:

Topic – An intermediary channel that maintains a list of subscribers to relay messages to that are received from publishers
Message – Serialized messages sent to a topic by a publisher which has no knowledge of the subscribers
Publisher – The application that publishes a message to a topic
Subscriber – An application that registers itself with the desired topic in order to receive the appropriate messages

Advantages and disadvantages of pub/sub

As with all technology, using pub/sub messaging comes with advantages and disadvantages. The two primary advantages are loose coupling and scalability.

Loose coupling

Publishers are never aware of the existence of subscribers so that both systems can operate independently of each other. This methodology removes service dependencies that are present in traditional coupling. For example, a client generally cannot send a message to a server if the server process is not running. With pub/sub, the client is no longer concerned whether or not processes are running on the server.

Scalability

Pub/sub messaging can scale to volumes beyond the capability of a single traditional data center. This level of scalability is primarily due to parallel operations, message caching, tree-based routing, and multiple other features built into the pub/sub model.

Scalability does have a limit though. Increasing the number of nodes and messages also increases the chances of experiencing a load surge or slowdown. On top of that, the advantages of the pub/sub model can sometimes be overshadowed by the message delivery issues it experiences, such as:

A publisher may only deliver messages for a certain period of time regardless of whether the message was received or not.
Since the publisher does not have a window into the subscriber it will always assume that the appropriate subscriber is listening. If the subscriber isn’t listening and misses an important message it can be disastrous for production systems.

How Pub/Sub Works

In the overview we covered how a publisher sends a message to a topic and how the topic forwards the message to the appropriate subscriber. From a topology point of view it is a simple process.

When it comes to coding the publish or the subscribe process the model can be a bit more confusing. Consider the following Java code which is used to create a topic.

Topic createTopic(String topicName) throws IOException {
  String topic = getTopic(topicName); // adds project name and resource type
  Pubsub.Projects.Topics topics = pubsub.projects().topics();
  ListTopicsResponse list = topics.list(project).execute();
  if (list.getTopics() == null || !list.getTopics().contains(new Topic().setName(topic))) {
      return topics.create(topic, new Topic()).execute();
  } else {
      return new Topic().setName(topic);
  }
}

Cloud or edge providers often simplify this code. Google Cloud, for example, has simplified topic creation into a single line of code.

gcloud beta pubsub topics create topicName

Examples of Pub/Sub

Publish/subscribe messaging has a multitude of use cases, some of which include:

Balancing workloads
Asynchronous workflows
Event notifications
Data streaming

Faye

Faye is an open source system based on pub/sub messaging. The code below shows you how to start a server, create a client, and send messages using Faye.

var http = require('http'),
    faye = require('faye');

var server = http.createServer(),
    bayeux = new faye.NodeAdapter({mount: '/'});

bayeux.attach(server);
server.listen(8000);

var client = new Faye.Client('http://localhost:8000/');

client.subscribe('/messages', function(message) {
  alert('Got a message: ' + message.text);
});

client.publish('/messages', {
  text: 'Hello world'
});

Faye is much more straightforward than standard JavaScript and it was created specifically for Node.js and Ruby servers. It’s often used for online instant messaging which is a use case for pub/sub that most people experience on a daily basis.

Pub/sub on the edge

As with a lot of edge computing benefits, pub/sub thrives with the speed that comes with being on the edge. A publisher must send a message to a topic, sometimes located far away in the physical world, that a subscriber is listening to. To travel across a room, a message may need to travel halfway around the world through Internet exchange points and physical distance always creates latency.

On the edge that message can travel two to four times faster across the room or even across the world.

Key Takeaways

Publish/subscribe messaging is when a publisher sends a message to a topic and the message is forwarded to a subscriber.
The concept of pub/sub is easy to understand but every coding and programming language handles it differently, making it a little more challenging to learn across all platforms.
On the edge, message delivery times can be two to four times faster by using a network backbone and multiple points of presence.