DEV Community: Andrew Betts

Build "For you" recommendations using AI on Fastly!

Andrew Betts — Wed, 07 Aug 2024 11:48:17 +0000

Forget the hype; where is AI delivering real value? Let's use edge computing to harness the power of AI and make smarter user experiences that are also fast, safe and reliable.

Recommendations are everywhere, and everyone knows that making web experiences more personalized makes them more engaging and successful. My Amazon homepage knows that I like home furnishings, kitchenware and right now, summer clothing:

Today, most platforms make you choose between either being fast or being personalized. At Fastly, we think you — and your users — deserve to have both. If every time your web server generates a page, it is only suitable for one end user, you can't benefit from caching it, which is what edge networks like Fastly do well.

So how can you benefit from edge caching, and yet make content personalized? We've written a lot before about how to break up complex client requests into multiple smaller, cacheable backend requests, and you'll find tutorials, code examples and demos in the personalization topic on our developer hub.

But what if you want to go further and generate the personalisation data at the edge? The "edge" - the Fastly servers handling your website's traffic, are the closest point to the end user that's still within your control. A great place to produce content that's specific to one user.

The "For you" use case

Product recommendations are inherently transient, specific to an individual user and likely to change frequently. But they also don't need to persist - we don't typically need to know what we've recommended to each person, only whether a particular algorithm achieves better conversion than another. Some recommendation algorithms need access to a large amount of state data, like what users are most similar to you and their purchase or rating history, but often that data is easy to pregenerate in bulk.

Basically, generating recommendations usually doesn't create a transaction, doesn't need any locks in your data store, and makes use of input data that's either immediately available from the current user's session, or created in an offline build process.

Sounds like we can generate recommendations at the edge!

A real world example

Let's take a look at the website of the New York Metropolitan Museum of Art:

Each of the 500,000 or so objects in the Met's collection has a page with a picture and information about it. It also has this list of related objects:

This seems to use a fairly straightforward system of faceting to generate these relationships, showing me other artworks by the same artist, or other objects in the same wing of the museum, or which are also made of paper or originate in the same time period.

The nice thing about this system (from a developer perspective!) is that since it's only based on the one input object, it can be pre-generated into the page.

What if we want to augment this with a selection of recommendations that are based on the end user's personal browsing history as they navigate around the Met's website, not just based on this one object?

Adding personalized recommendations

There's lots of ways we can do this, but I wanted to try using a language model, since AI is happening right now, and it's really different from the way the Met's existing related artworks mechanism seems to work. Here's the plan:

Download the Met's open access collection dataset.
Run it through a language model to create vector embeddings – lists of numbers suitable for machine learning tasks.
Build a performant similarity search engine for the resulting half a million vectors (representing the Met’s artworks) and load it into KV store so we can use it from Fastly Compute.

Once we've done all that, we should be able to, as you browse the Met's website:

Track the artworks you visit in a cookie.
Look up the vectors corresponding to those artworks.
Calculate an average vector representing your browsing interests.
Plug that into our similarity search engine to find the most similar artworks.
Load details about those artworks from the Met's Object API and augment the page with personalized recommendations.

Et voilà, personalized recommendations:

OK, so let's break that down.

Creating the dataset

The Met's raw dataset is a CSV with lots of columns and looks like this:

Object Number,Is Highlight,Is Timeline Work,Is Public Domain,Object ID,Gallery Number,Department,AccessionYear,Object Name,Title,Culture,Period,Dynasty,Reign,Portfolio,Constituent ID,Artist Role,Artist Prefix,Artist Display Name,Artist Display Bio,Artist Suffix,Artist Alpha Sort,Artist Nationality,Artist Begin Date,Artist End Date,Artist Gender,Artist ULAN URL,Artist Wikidata URL,Object Date,Object Begin Date,Object End Date,Medium,Dimensions,Credit Line,Geography Type,City,State,County,Country,Region,Subregion,Locale,Locus,Excavation,River,Classification,Rights and Reproduction,Link Resource,Object Wikidata URL,Metadata Date,Repository,Tags,Tags AAT URL,Tags Wikidata URL
1979.486.1,False,False,False,1,,The American Wing,1979,Coin,One-dollar Liberty Head Coin,,,,,,16429,Maker," ",James Barton Longacre,"American, Delaware County, Pennsylvania 1794–1869 Philadelphia, Pennsylvania"," ","Longacre, James Barton",American,1794      ,1869      ,,http://vocab.getty.edu/page/ulan/500011409,https://www.wikidata.org/wiki/Q3806459,1853,1853,1853,Gold,Dimensions unavailable,"Gift of Heinz L. Stoppelmann, 1979",,,,,,,,,,,,,,http://www.metmuseum.org/art/collection/search/1,,,"Metropolitan Museum of Art, New York, NY",,,
1980.264.5,False,False,False,2,,The American Wing,1980,Coin,Ten-dollar Liberty Head Coin,,,,,,107,Maker," ",Christian Gobrecht,1785–1844," ","Gobrecht, Christian",American,1785      ,1844      ,,http://vocab.getty.edu/page/ulan/500077295,https://www.wikidata.org/wiki/Q5109648,1901,1901,1901,Gold,Dimensions unavailable,"Gift of Heinz L. Stoppelmann, 1980",,,,,,,,,,,,,,http://www.metmuseum.org/art/collection/search/2,,,"Metropolitan Museum of Art, New York, NY",,,

Simple enough to transform that into two columns, an ID and a string:

id,description
1,"One-dollar Liberty Head Coin; Type: Coin; Artist: James Barton Longacre; Medium: Gold; Date: 1853; Credit: Gift of Heinz L. Stoppelmann, 1979"
2,"Ten-dollar Liberty Head Coin; Type: Coin; Artist: Christian Gobrecht; Medium: Gold; Date: 1901; Credit: Gift of Heinz L. Stoppelmann, 1980"
3,"Two-and-a-Half Dollar Coin; Type: Coin; Medium: Gold; Date: 1927; Credit: Gift of C. Ruxton Love Jr., 1967"

Now we can use the transformers package from Hugging Face AI toolset, and generate embeddings of each of these descriptions. We used the sentence-transformers/all-MiniLM-L12-v2 model, and used principal component analysis (PCA) to reduce the resulting vectors to 5 dimensions. That gives you something like:

[
  {
    "id": 1,
    "vector": [ -0.005544120445847511, -0.030924081802368164, 0.008597176522016525, 0.20186401903629303, 0.0578165128827095 ]
  },
  {
    "id": 2,
    "vector": [ -0.005544120445847511, -0.030924081802368164, 0.008597176522016525, 0.20186401903629303, 0.0578165128827095 ]
  },
  …
]

We have half a million of these, so it's not possible to store this entire dataset within the edge app's memory. And we want to do a custom type of similarity search over this data, which is something a traditional key-value store doesn't offer. Since we’re building a real-time experience, we also really want to avoid having to search half a million vectors at a time.

So, let's partition the data. We can use KMeans clustering to group vectors that are similar to each other. We sliced the data into 500 clusters of varying sizes, and calculated a center point called a “centroid vector” for each of those clusters. If you plotted this vector space in two dimensions and zoomed in, it might look a bit like this:

The red crosses are the mathematical center points of each cluster of vectors, called centroids. They can work like wayfinders for our half-million-vector space. For instance, if we want to find the 10 most similar vectors to a given vector A, we can first look for the nearest centroid (out of 500), then conduct our search only within its corresponding cluster–a much more manageable area!

Now we have 500 small datasets and an index that maps centroid points to the relevant dataset. Next, to enable real-time performance, we want to precompile search graphs so that we don't need to initialize and construct them at runtime, and can use as little CPU time as possible. A really fast nearest-neighbor algorithm is Hierarchical Navigable Small Worlds (HNSW), and it has a pure Rust implementation, which we're using to write our edge app. So we wrote a small standalone Rust app to construct the HNSW graph structs for each dataset, and then used bincode to export the memory of the instantiated struct into a binary blob.

Now, those binary blobs can be loaded into KV store, keyed against the cluster index, and the cluster index can be included in our edge app.

This architecture lets us load parts of the search index into memory on demand. And since we’ll never have to search more than a few thousand vectors at a time, our searches will always be cheap and fast.

Building the edge app

The application that we run at the edge needs to handle several types of requests:

HTML pages: We fetch these from metmuseum.org and transform the response to add extra front-end <script> and <style> tags, so we can inject a bit of our own front end processing and content
The Fastly script and style resources referenced by those extra tags, which we can serve directly out of the edge app's binary.
The recommender endpoint, which generates and returns the recommendations.
All other (non-HTML) requests: Images, and the Met's own scripts and stylesheets, which we proxy directly from their domain without alteration.

We initially built this app in JavaScript, but ended up porting the recommender part to Rust because we liked the HNSW implementation in instant-distance.

The client side JavaScript does a few interesting things:

Using IntersectionObserver, we trigger an event when the user scrolls down the page to the related objects section. This is a super efficient API that's much better than using older methods like onscroll.
Make a fetch to our special recommendations API endpoint (which we can then handle at the edge and return object information)
Compose some HTML using a template built into a client-side function
Append that HTML to the page and move the intersection observer to the new element so as you scroll through the recommendations, we keep loading more.

This way, we can deliver the main HTML payload without invoking our recommendation algorithm, but the recommendations are delivered fast enough that we can load them as you scroll and they'll almost certainly be there by the time you get to them.

I like doing things this way because getting that first above-the-fold view to the user as fast as possible is absolutely paramount. Anything that you can't see unless you scroll can be loaded later, and especially if it is a complex piece of personalized content - there's no point generating it if the user isn't planning to scroll.

Closing thoughts

So now you have the best of both worlds: the ability to serve highly personalized content, almost never requiring any blocking fetches to origin, and an optimized HTML payload that renders incredibly fast, allowing your application to enjoy effectively limitless scalability and near perfect resilience.

It's not a perfect solution. It'd be great if Fastly offered more higher level features to expose edge data via query mechanisms other than a simple key lookup (let us know if that would help you!) and this specific mechanism has obvious flaws - if I have separate interests in two or more very different things (say 19th century oil paintings and ancient Roman amphora) I would get recommendations which would be the theoretical semantic "middle point" between those, not a very useful result.

Still, hopefully this demonstrates the principle that figuring out how to do work at the edge often results in outsized benefits in terms of scalability, performance and resilience.

Let us know what you build on our community thread!

The elusive perfect language switcher

Andrew Betts — Tue, 23 Jul 2024 15:29:53 +0000

Lots of websites get tripped up trying to offer users a choice of language. But a few simple steps can ensure your users get exactly the language they want.

I have the dubious honor in my social circle of being one of the few who speaks only one language fluently. Fortunately for me the language I speak is a pretty popular one, but when I visit websites when I'm overseas, even ones I use a lot, it's surprising how often they suddenly think I want the site in Spanish, Japanese or Polish. This is one of a bunch of things websites often get wrong when designing a language-localized experience.

We can group these language-switching fails into roughly three categories: bad assumptions, poor labeling, and unintuitive persistence. Let's address these and then consider a checklist of good practices for language switchers (and also some ideas for how to use edge computing to make this easier!).

Bad assumptions: the most common mistakes
Poor labeling: avoid red flags!
Broken persistence: what not to forget
Edge computing to the rescue! (so predictable but I like my job etc.)

Bad assumptions

A user's physical location is not the best indicator of the language they understand. It's correlated: it's certainly true that a user based in Mexico is much more likely to speak Spanish than a user in Finland. But this logic results in a really poor experience when people are traveling away from their home country. Using the Accept-Language header (which communicates the user's device system locale) is a much better way to figure out what language to select.

For retailers, it's also risky to assume that delivery location, language and currency should all change together. If I want to buy a gift for a friend overseas, I might want a foreign delivery location but still shop in my native language and currency.

What if someone migrates to a new country but still prefers their native language? Now you might have a currency and location that seem to match but with a language that doesn't fit. The more flexibility you can build into your system design, the less convoluted the solutions you'll have to come up with to deal with these edge cases.

Google Flights offers a masterclass in doing this correctly - language preference is detected from browser locale, location from your IP address, and currency synced to location - but all are independently adjustable:

In summary:

Don't assume that the user wants the predominant language of the country they are physically located in. Use browser locale in preference to the user's physical location when choosing a language automatically.
Don't assume that currency, language and delivery location are all "in sync"

Poor labeling

Flags are a somewhat controversial way of representing language choice. Neilsen, a UX consultancy, found that users do associate their country's flag with their language, but where a language is spoken in multiple countries, this can be quite a political minefield.

My favorite example of a bizarrely labeled language switcher from a German sofa company.

It's not completely wrong to associate languages with places. The English spoken in the UK is clearly not (entirely) the same as that spoken in the US, and the commonly used IETF language tags standard uses this as the basis for classifying language variants: en-GB, and en-US, for example.

As with so many standards that attempt to classify and group human behavior, some compromise is needed. Most correctly you might say that en-IN is "a dialect of the English language which has its cultural center of gravity in India" but how exactly you define that dialect and who gets to do so is highly debatable, and obviously any variant of any language could be spoken in any physical location in the world.

So, using flags can be a nice experience for users but is hard to make work across a large user population unless you have invested in multiple variants.

Businesses clearly need to be pragmatic, and will invest time and money only where there is a market for their products or services. It's actually quite uncommon for websites (except the very largest) to bother providing services in multiple variants of the same language, and where there is a link between locations and languages, it's often more likely to be because the business has a local version of the entire site, which is then offered in whatever languages are deemed to be relevant to that location.

Booking.com doesn't have regional sites, and uses flags for language choice, but is able to offer the correct regional variants of the languages that match the flags (check out the multiple English, Spanish and Portuguese variants here):

Great job Booking.com team!

Now consider IKEA, which unlike Booking.com is selling physical products that vary between countries, and therefore they do have regional sites each offered in a different selection of languages.

IKEA Portugal, for example, offers Portuguese and English, but doesn't specify which variant of either of those they are using ('pt/en' doesn't mean "the variant of English spoken in Portugal", but rather "the Portugal website, rendered in English"):

That's a pretty common pattern for sites whose product offering varies between countries. Here's Storytel's Brazil site, which is only available in Portuguese (presumably pt-BR?):

In summary, disassociating language and location is pretty hard and ultimately the most practical solution depends on to what extent your product or service is different in different markets.

The other labeling issue I find interesting is the icon or symbol used to indicate that language selection is available. The most common choices are a globe, some kind of specific "languages" icon, the name of the currently selected language, or a list of available languages with the current one highlighted.

The Language Icon project has been trying to standardize this for 15 years, and today doesn't enjoy very much traction, while most major global sites seem to have adopted a globe. The advantage of the big ol' list of languages approach (normally in the footer) is that if I see a wall of text I don't understand, I can press CMD+F and search the page for the word "English" and find the link to the English version.

The worst offense here is to have a menu option somewhere called "Change language". Any English speaker who has ever been pranked by friends who changed their phone's UI to Korean will be familiar with the challenges this presents.

Labeling is really important. Some things to bear in mind:

Write the names of languages in the language itself. Offer "Deutsch", not "German"
Don't expose users to labels designed for computers. No-one should have to know what "DE" means.
Make clear the distinction between "service location" and "language region"
Use a globe symbol for a language selector
Consider listing available languages somewhere on the page

Unintuitive persistence

The final horseman of the language apocalypse is the challenge of persisting language choices, and when to do so. Users have an annoying tendency to log out, use more than one device, and even send links to friends and family who might prefer different languages!

A great example of this problem happens if you are visiting a friend overseas and they send you a link to a Google Maps location where they want to meet you. Odds are, you'll click that link and it will display Google Maps in your friend's language, not yours.

On the face of it that seems odd because in principle the way the web works is that the same URL should be available in multiple languages and the server gives you the one that matches your Accept-Language preference. However, this idealized idea of 'content negotiation' is rarely used in its purest form on the web, not just because of technical headaches but because it sometimes isn't actually the right experience.

On a news / long-form content site, if content is available in multiple languages, it's rarely exactly equivalent. I might send the French version of an article to a friend who normally prefers English, because it reads better in French, or includes content not available in the English version.

Sharing is a minefield. But problems with persistence can arise even when just dealing with one user. What if a user registers for an account on your site, and has never made an explicit language selection, but then changes their device language. Since they have an account, you might feel that in this situation the account settings have taken over from automatic selection, but since they never made an explicit choice, shouldn't we still be led by their browser locale?

This is where we need to make smart decisions about how to persist language choice, and the hierarchy of precedence for the different language signals. I prefer the following principles:

The "user preference" is an explicit choice saved on your account, or the Accept-Language header if there's no saved preference.
A language specific URL overrides the user preference, but an inconsistency between the two should be signalled to the user.

For example, if a friend sends me a link to www.example.com and I've never been there before, my friend might have seen it in French but I'd get it in English, because of Accept-Language.

However, if they sent me a link to www.example.com/fr/products/123, then I'd see the page in French, with an alert asking me (in English) if I want to switch to English.

Edge computing to the rescue

I'm obviously now going to tell you how Fastly is the answer to all your language challenges, right? Well, it does actually make a lot of sense to manage aspects of language selection at the edge. One of the most important things to do is to normalize the Accept-Language value. In Fastly VCL you can do that like this:

set req.http.Accept-Language = accept.language_lookup("en:de:fr:nl", "de", req.http.Accept-Language);

This will turn a value such as en-GB,en-US;q=0.9,en;q=0.8 into en, and means that you can store far fewer versions of your pages.

Also consider redirecting users to a language-specific URL when they arrive at your homepage. A custom VCL subroutine will do the trick here:

sub locale_redirect {
  declare local var.urlLocale STRING;
  declare local var.prefLocale STRING;
  set var.urlLocale = if (req.url.path ~ "^/(en|de|fr|nl)/", re.group.1, "");
  set var.prefLocale = if (
    req.http.cookie:lang, 
    req.http.cookie:lang, 
    req.http.Accept-Language
  );
  if (var.urlLocale == "" && var.prefLocale != "") {
    error 600 var.prefLocale;
  }
}

Whatever you do at the edge, I hope some of this helps you build more insightful, more intuitive language-sensitive experiences for your users!

Also tell us what you build, at community.fastly.com.

Is purging still the hardest problem in computer science?

Andrew Betts — Wed, 01 May 2024 08:37:44 +0000

One of the most common reasons for customers sending support tickets to Fastly is for help with purging content from cache - either it stays too long, or doesn't stay long enough. We have the best purging mechanism of any edge network, so what's going on?

It's said there are only two hard problems in computer science: cache invalidation, naming things and off-by-one errors. Caching stuff close to where your users are is just one of many things you can do with Fastly these days, but it's still one of the easiest and most effective ways to make your website faster, more reliable and more engaging. Traditionally, you just set a TTL (time to live) and let the cache just hold onto it for that amount of time.

However, while serving outdated content might have been acceptable in the past it just doesn't cut the mustard today. News editors need stories to be up to date. Prices in stores must be correct and consistent. Application versions must be compatible. Purging the right things quickly is essential.

The three types of purge

How do you purge the right set of things? We offer three flavors of purging:

Single item / URL purge
Purge all
Surrogate key

On the off chance that you just want to purge one single item by its URL, that's a URL purge and on Fastly you do it by sending an HTTP PURGE request to the very URL you want to purge:

curl -X PURGE "https://www.example.com/hot-deals"

URL purges are simple and easy, but affecting just a single URL is not really very scalable, and if that URL is ever requested with a query string like ?page=2, that's a different URL! Sorry, you will have to purge that and every other variant as well.

The natural complement to the single purge, then, is the purge all. Very much for the "nuke it from orbit" crowd, this blunt instrument has no equal. A quick call to a fixed API endpoint and your entire service's cache is wiped:

curl -X POST "https://api.fastly.com/service/{SERVICE_ID}/purge_all" -H "Fastly-Key: {YOUR_FASTLY_API_TOKEN}"

Most cases are more nuanced than either of these scenarios. Purge anything that mentions product 36253; purge all images; purge all URLs under the /products/shirts prefix; purge all the 720p variants from all season 2 episodes of show 414562; purge all the resources associated with the "about us" page… the list of possible purge criteria is endless. For all your precision purging needs, look no further than surrogate keys, which let you apply tags to content and then purge everything that has a particular tag.

For example, you could serve a response from your origin server with the following header listing out all the tags you want to put on that response:

Surrogate-Key: html product-123 product-456 region-eu /products /products/shirts /products/shirts/123

Then, later, send a purge for one of the tags you put on that response:

curl -X POST "https://api.fastly.com/service/{SERVICE_ID}/purge/product-123" -H "Fastly-Key: {YOUR_FASTLY_API_TOKEN}"

This will surgically remove all the cached content that has that tag, without affecting the rest of your cache.

The fastest purge... in the world.

Fastly purges your stuff really fast. When you send us a URL or surrogate key purge, it will be received by the Fastly POP (point-of-presence) closest to you, and then from there it is replicated rapidly and efficiently to every other POP in the world. Copies of your objects will start to purge within 5ms and every Fastly POP in the world should be done in around 150ms. It's almost always complete within 250ms.

That's the fastest purge of any edge network or CDN (by quite a margin). The laws of physics give us a theoretical fastest purge time of about 65ms, the time it takes light to travel (one way) from one side of the planet to the other, but we also need to account for some processing time, buffering and retransmit delays, and the fact that cables do not go in completely straight lines, so 150ms is about the fastest you can ever expect a global purge to be.

(Note that "purge all" invokes a completely different mechanism and takes up to 1 minute to complete)

Cache more stuff!

When you can purge things in 150ms, you can cache more, and not worry about it. Historically it was risky to cache content on a CDN if it might change, or if it contained anything that was specific to the user browsing the site.

Some things never change

Content that never changes is great for caching, obviously. And you can help create more of that by using immutable URLs for all your assets. These contain a hash of the content of the file, like /scripts/main.142c04bb.min.js, and should be served with a cache policy such as:

Cache-Control: public, max-age=31536000, immutable

Web frameworks like Next.js will usually include this feature, but do check that they set the caching headers correctly!

For content that changes constantly or is otherwise genuinely uncacheable, use:

Cache-Control: private, no-store

But is it uncacheable, really?

Event driven content

What about content that doesn't change often but when it does, it needs to update immediately? Things like news articles - which typically remain static after publishing but may need an urgent update to correct an error?

We call this event-driven content. For this use case, you'll want to configure caching like this:

Cache-Control: public, max-age=3600
Surrogate-Control: public, max-age=31536000
Surrogate-Key: content article-12345

This tells Fastly to cache the content "forever" (for a year) but the browser should only cache for a short period (1 hour in this example). We also tag the content with any relevant surrogate key tags that will help you target it when you want to purge. When the article is updated, send a URL purge or a surrogate key purge. If you are using a Fastly plugin in your CMS, it might do this for you automatically!

Private or authenticated content

What if the content changes when the user logs in, or is subtly different depending on the user's privileges, memberships or subscriptions? If there are a lot of possible variations, consider personalizing the content at the edge, but if there's only a few, you can use the Vary header! Start by normalizing the authentication state on the request before it is forwarded to the origin (e.g. by decoding a JSON web token session cookie). You can then add headers to the request such as:

User-role: admin
User-region: europe
User-is-subscriber: true

And then your response content can be marked up with headers such as:

Cache-Control: private, no-store
Surrogate-Control: public, max-age=31536000
Surrogate-Key: content article-12345
Vary: User-is-subscriber, User-region

This works well if the number of permutations of the request headers you are varying on is small. We allow up to 50 variations per object (in each POP) in our VCL services and have slightly different rules for Compute services.

Purge power moves

So you can often cache more than you think you can, and using cache at the edge is one of the most effective ways to improve end-user experiences and reduce costs. Customers often come up with really interesting patterns for using purging - here are some to inspire you!

Soft purge and serving stale

Generally when you purge things you expect them to die immediately, the clue being very much in the word "purge". But there are a couple of scenarios where maybe we can do something better. What about things that are super popular, being requested thousands of times a minute, or even per second? When you purge things like that, it can create a poor experience for not just one user but everyone who wants that resource before we've managed to re-cache it.

Or what about when your origin servers go down? Some system could unhelpfully purge content at a bad moment when the origin is not available to serve a new version.

Both of these scenarios can be improved by supporting stale serving. Add the stale-while-revalidate and stale-if-error directives to your Cache-Control header to activate this behavior:

Cache-Control: public, max-age=31536000, stale-while-revalidate=60, stale-if-error=31536000

Now, when issuing your purge, set the soft purge flag, and instead of purging the object, we'll instead mark it as stale. This is a real superpower and frankly unless you have a good reason not to, you should consider doing this in pretty much every scenario where you purge.

Path prefix purging

Want to purge everything under /products/? Use surrogate keys to tag your responses with every path segment prefix:

Surrogate-Key: /products/winter-season/shirts/378245 /products/winter-season/shirts /products/winter-season /products

Now issue a surrogate key purge for "/products" or "/products/winter-season" and we'll delete everything with those prefixes (in 150ms!)

Slow purges

Sometimes you want to purge a lot of content, but you don't want it to all disappear at once, because that will cause unreasonably high load on your origin servers. Yann Harmon from Contentful has a great solution to this, adding a random-number surrogate key tag (from a set of 100) to every response. For example:

Surrogate-Key: purge-group-23

You can now issue purges for all 100 purge group tags (purge-group-1, purge-group-2, purge-group-3, etc), with whatever pause you want in between each one, to draw out the purge over a longer period of time, and give content time to re-cache. It's unusual to have customers want to purge more slowly but sometimes it's useful!

Calculated TTLs at the edge

It's generally best to determine cache policy at the point that content is generated, typically on your origin servers, and write the instructions for Fastly into headers like Cache-Control and Surrogate-Control. But you can also set the TTL of content within the logic of your Fastly service if you like.

This can allow you to do almost anything, for example one customer decided to have their cache reset once a day at a predetermined time, but didn't want to rely on issuing a purge on a schedule, so instead wrote logic to adjust the TTL of each response so that they always expire at the next reset point.

Combating race conditions

Sometimes it seems like a purge didn't work, and that's typically because it happened too early. If you purge Fastly before the upstream server has updated, we might just pull the old version of the content and re-cache it. Take care to ensure that your origin is serving the new version before you purge Fastly.

This also often happens when you're using our shielding feature, which puts two layers of Fastly between the end user and your origin. If your purge is processed by other Fastly POPs before it reaches the shield POP, the purged POPs might re-cache from the shield cache. This is typically only a problem with purge all, because it takes significantly longer than a URL purge or a surrogate key purge.

The way a "purge all" works actually provides a solution to this problem because this kind of purge is done by increasing the cache version number, and you can read the current version number from the configuration of your Fastly service. By comparing the version number of the upstream cache with the version of the local cache, it's possible to identify when a purge is in progress and responses should not yet be cached.

Or you could, you know, just purge twice.

Conclusion

Edge networks can now do incredibly smart stuff, and at Fastly we spent a lot of time building more and more features you can make use of to improve the security, performance and scalability of your apps and websites. But caching is still one of the most powerful things you can do, and something you should spend time getting right. It can have dramatic effects on your costs and environmental impact too.

Taking advantage of the speed and precision of Fastly's purging machinery allows you to get the maximum value out of edge caching. Tell us how you're doing it at community.fastly.com.

11 things I do in every Fastly service I create

Andrew Betts — Tue, 26 Mar 2024 11:46:30 +0000

The variety of things you can do in edge computing these days is huge, and Fastly customers are constantly impressing me with their creativity and the complexity of some of the problems that can be solved entirely at the edge. But there are some things that almost regardless of your use case, you probably should be doing.

I've set up a lot of Fastly services, and along the way I've realized that pretty much every time, I add the same patterns to do the same useful stuff. Whether it's an e-commerce site, something statically generated, a SaaS service, or anything else, there's a set of stuff that is almost universally applicable. Let's check them off here, with examples for Fastly's Delivery (VCL) services.

(If JavaScript, Rust or Go are more your thing, try Fastly Compute services. It's free to sign up and you can do a lot of the same things I've listed here)

Shielding

If your Fastly service uses backends, i.e. you have origin servers behind Fastly, it is almost always a good idea to enable shielding, which focuses "miss" traffic into one Fastly POP before sending it to your origin. You then have two chances to get a hit in the cache, and the cache in the shield POP will typically become larger and more comprehensive than other POPs, so this can reduce traffic to your origin servers by a huge amount: up to two orders of magnitude!

Shielding needs to be enabled as a property of the backend configuration, either in the control panel or when creating a backend via the Fastly API.

Want to know where to shield? Use the shield chooser tool!

HTTP/3

Although Fastly negotiates HTTP/2 automatically if your TLS configuration supports it, at time of writing we require you to explicitly enable H3 by adding an Alt-Svc header to your responses - but the good news is that you can enable this in the control panel:

Or drop a single line of VCL into vcl_recv:

h3.alt_svc();

Force TLS

Another quick win is forcing all clients to use TLS. We do accept connections on plain HTTP in VCL services, so it's a good idea to redirect them to HTTPS. This can be done using a toggle in the control panel, right next to the HTTP/33 option.

Or use custom VCL:

if (req.protocol != "https") {
  error 608;
}

We recommend throwing error codes in the 6xx range and then mapping them to conventional HTTP status codes in vcl_error, where you can also turn the parameter into a Location header:

if (obj.status == 608) {
  set obj.http.location = "https://" req.http.host req.url;
  set obj.status = 308;
  set obj.response = "Moved permanently";
  return (deliver);
}

(by the way, if you're using a Compute service, this redirect happens automatically, and no insecure traffic ever reaches your service)

HTTP Strict Transport Security

You should also return a strict-transport-security header to ensure that clients never make an insecure connection. If you use the control panel to configure Force TLS, HSTS will be set up as well, but if you're using custom VCL, adding response headers just before sending to the client can be done in vcl_deliver:

set resp.http.strict-transport-security = "max-age=31536000; includeSubDomains; preload"

Compression

The final option that is configurable in the control panel, enable compression under Content > Compression, and we will automatically compress any uncompressed responses from origin.

If you want to do this in VCL, take care to update the Vary header to include "Accept-Encoding":

if (beresp.status == 200 && beresp.http.content-type ~ "^(?:text/|application/json)") {
  if (req.http.Accept-Encoding == "br") {
    set beresp.brotli = true;
  } elsif (req.http.Accept-Encoding == "gzip") {
    set beresp.gzip = true;
  }
  set beresp.http.Vary:Accept-Encoding = "";
}

CORS

Trying to build handling for Cross-origin resource sharing into your backend app is often a pain. Instead, handle CORS preflights at the edge and ensure clients get the right CORS rules. In vcl_recv you can trap OPTIONS requests:

if (req.method == "OPTIONS" && req.http.Origin) {
  error 612;
}

Then, in vcl_error, construct the preflight response:

if (obj.status == 612) {
  set obj.status = 204;
  set obj.http.access-control-allow-origin = req.http.origin;
  set obj.http.access-control-max-age = "86400";
  return(deliver);
}

Finally, in vcl_deliver, you can add the headers specifying what is allowed, since you need to add these to all responses, not just the preflight ones:

set resp.http.access-control-allow-methods = "GET,HEAD,POST,OPTIONS";
set resp.http.access-control-allow-headers = "Content-Type, x-requested-with";

Redirects

Handling redirection of out of date URLs is a really big and easy win. Put your redirects in an edge dictionary, or if you have pattern-matching redirects, write simple VCL to map the patterns:

declare local var.dest STRING;

// Lookup in table / edge dictionary
if (table.contains(my_redirects, req.url.path)) {  
  set var.dest = table.lookup(my_redirects, req.url.path);

// Special case pattern
} else if (req.url.path ~ "^/products/(\w+)/([0-9]+)") {
  set var.dest = "/catalog/categories/" re.group.1 "/products/" re.group.2;
}
if (var.dest) {
 error 601 var.dest;
}

Then in vcl_error, construct the redirect:

if (obj.status == 601) {
  set obj.http.location = obj.response;
  set obj.status = 308;
  set obj.response = "Permanent redirect";
  return (deliver);
}

Normalizing requests for better cache performance

If you have a clear idea of what constitutes a valid URL for your site, it's helpful to use VCL to filter out anything else:

if (fastly.ff.visits_this_service == 0 && req.restarts == 0) {

  # Sort query params into alphabetical order
  set req.url = querystring.sort(req.url);

  # Media assets have a specific set of allowed query params
  if (req.url.path ~ "/media_[0-9a-f]{40,}[/a-zA-Z0-9_-]*\.[0-9a-z]+\z") {
    # width, height, format, optimize
    set req.url = querystring.filter_except(req.url,
      "width" querystring.filtersep()
      "height" querystring.filtersep()
      "format" querystring.filtersep()
      "optimize"
    );

  # Paths ending .json have a different, specific set of allowed params
  } else if (req.url.ext == "json") {
    # limit, offset, sheet
    set req.url = querystring.filter_except(req.url,
      "limit" querystring.filtersep()
      "offset" querystring.filtersep()
      "sheet"
    );

  # Otherwise, query params are not allowed at all
  } else {
    set req.url = req.url.path;
  }
}

Not all websites have the luxury of knowing what parameters might be on the query strings of requests - and sometimes even if you do, you don't want to turn today's reality into tomorrow's limitation. After all ossification of the web is a major problem.

Blocking unwanted traffic

If you want a comprehensive solution to block malicious traffic, check out our Next-Gen WAF at Edge, but you can also construct simple rules in VCL using Access Control Lists or even just basic if statements:

if (fastly.ff.visits_this_service == 0 && req.restarts == 0) {
  if (
    req.http.user-agent ~ "/bot/" || // Header pattern
    !req.http.accept ||              // Required header
    req.http.via ||                  // Banned header
    client.geo.country_code ~ "^(fr|uk|ru)$" || // Country of origin
    client.ip ~ my_acl ||            // IP is in an ACL
    client.as.number == 12345 ||     // Client's ISP
  ) {
    error 625;
  }
}

Then in vcl_error:

if (obj.status == 625) {
  set obj.status = 403;
  set obj.response = "Forbidden";
  synthetic "Not allowed";
  return(deliver);
}

Strip upstream headers

If you're using AWS S3, Google Cloud Storage or similar as a backend, they will likely return a whole bunch of response headers that you don't want. Remove them with some VCL in vcl_fetch:

unset beresp.http.server;
unset beresp.http.x-generator;
unset beresp.http.via;
unset beresp.http.x-github-request-id;
unset beresp.http.x-amz-delete-marker;
unset beresp.http.x-amz-id-2;
unset beresp.http.x-amz-request-id;
unset beresp.http.x-amz-version-id;
unset beresp.http.x-goog-component-count;
unset beresp.http.x-goog-expiration;
unset beresp.http.x-goog-generation;
unset beresp.http.x-goog-metageneration;
unset beresp.http.x-goog-stored-content-encoding;
unset beresp.http.x-goog-stored-content-length;

Doing this in vcl_fetch makes sense because it happens before the object is written to cache.

Debugging

By default, Fastly includes certain response headers when a Fastly-Debug header is present in the request. It's handy to use this mechanism to add any other debugging information you want to emit. For example, you could include the final request path that was used to look up the object in cache, and the version of your Fastly service that handled the request:

if (req.http.fastly-debug) {
  set resp.http.fastly-service-version = req.vcl.version;
  set resp.http.fastly-req-path = req.url.path;
}

You're off to a great start!

These patterns are a great way to learn about the power of the edge to simplify and decouple layers of your service architecture, and apply to almost any use case. Implementing these also helps you understand some common best practices of VCL and set you up to implement solutions more specific to your use cases.

Whatever you create, share it with us below or at community.fastly.com.

Open source at Fastly is getting opener

Andrew Betts — Fri, 15 Mar 2024 17:48:15 +0000

Like most tech companies these days, Fastly is heavily reliant on open source software and the ecosystem of services and frameworks that modern software development weaves together. We need to talk about what it means to 'be open,' how we can recognize the broader ecosystem we're part of, and how we're making big changes to improve.

We’re working to make our little corner of the open source world the friendliest, most welcoming, easiest to understand, and most inclusive place it can be. You’ll see the results of that work in three key new parts of our open platform we’re sharing today: an updated code of conduct, a new project directory, and clearly-defined support levels for our open work.

Making open understandable

Recently, we identified a number of problems that made things unnecessarily complicated for users of our open source work. We had a lot of public repositories on our GitHub but few people could easily understand which were active, whether support could be expected and whether contributions were invited. Many of our customers also use Fastly in conjunction with other tools, frameworks and services (like Gatsby, Heroku, NextJS, Vercel, Splunk, New Relic, Wordpress to name a few random ones), and adapters exist to support those interops, but it was often hard to figure out to what extent we support those adapters, whether they work for particular use cases, and who maintains them. This needed to be cleared up.

Making sure you have a good experience

So, it’s not enough to provide a lot of great open source tools — you need to have a reliable, trusted experience when you use them. We started with the basics. GitHub has a very useful tool under Insights > Community standards on each repo, which provides a checklist for community health. This felt like a good starting point to help people understand the intent and support behind our public code:

Then, we took it a step further. We also created a clearly-documented support model to bundle up a bunch of expectations and behaviors into simple, recognisable buckets. Four levels can cover all projects quite well:

Product: An official part of Fastly's services. Expect complete documentation and support engineers trained to help you out.
Tier 1 OSS: "Active" projects that we are working on frequently and where you can expect enthusiastic engagement.
Tier 2 OSS: "Maintenance mode" projects where we keep on top of dependencies and security issues but may not always be able to engage with community interest.
Archived: Projects that we don't intent to revisit, which may no longer work, and should be considered out of date and likely insecure.

This can be nicely represented in a comparison matrix. You can see this on our new open source directory:

To tie all this together, we made an org profile for our GitHub organization page. There are some great examples of these, like Microsoft and PayPal. For Fastly's, we wanted something friendly and human so we used Imagemagick to create a montage of all the avatars of the Fastly team members who have contributed to the open source projects in our org:

This is also a good spot to link to policies that apply across our open source work, and promote the great work being done by the organizations we support as part of our Fast Forward program. More about that in a minute.

Wow I had no idea we had that!

There's a ton of gold in our public repos, and scandalously, few people know it's there. Sorting through what we've open sourced over the years, we developed a category system, and most things fall into one of these buckets:

Tools: Stuff you run in your own environment, usually to help you interact with Fastly, like our compute static publisher utility or the Fastly CLI.
Plugins: Things you use inside third party products to help them integrate with Fastly, like our Wordpress plugin or Terraform provider.
Demos: Applications that you can run on or with Fastly, demonstrating creative uses of the Fastly platform, like this JavaScript implementation of OAuth 2.0 you can run at the edge.
API clients: Dedicated adapters to access api.fastly.com in a variety of languages.
Compute Starter kits: The Fastly Compute platform allows you to run your code on our giant edge network in a variety of languages. Starter kits offer complete example Compute applications that are great for scaffolding new projects.
Compute SDKs: These run within Fastly Compute apps and allow you to access platform features from your preferred language.
Compute libraries: Code you can import into Compute apps to make common things easy or integrate with third party services or frameworks, like our adapters for OpenTelemetry or Next.JS.

Not all our open source repos are Fastly related either, Fastly engineering teams have opened up a bunch of things that we've built just to solve problems within low level parts of our stack. For example utilities for dnstap in Rust, or Prometheus instrumentation for Sidekiq.

Do we really need a 9-year old fork of the Linux kernel?

One of the things that stifles openness is noise. Sifting through our public repos, some are very obviously redundant and just distracting. So we started by archiving a whole bunch of them - over half, in fact. We wrote some Google Apps Script in a spreadsheet to import and analyze the state of all our public repos:

This allows for some basic filters to be applied to find the most obvious candidates for archiving:

No updates in over a year
No open issues
Does not have forks
Primary committer is no longer at Fastly
Minimal watchers / stargazers

Repos where all these things are true could be archived in bulk, and we reached out to the team closest to the code to give them the opportunity to delete the repo entirely if it was of no use at all. Archiving works well for everything else - archived repos remain accessible and cloneable, so any unknown project that depends on them will continue to work, and it's reversible, so we can change our minds later if we need to. But archiving also sends a clear message that we are not paying attention to this code anymore, so use it at your own risk and don't expect any help!

This captured a lot of fun examples, like a fork of the linux kernel that we last committed to 9 years ago and last merged from upstream 11 years ago 😅. But once these really obvious ones are out of the way, the real work begins.

Code of conduct and responsible disclosure

Once we finished cleaning up our repos, we wanted to add one more essential piece: A code of conduct. As a values-driven organization, we hold ourselves and those we interact with to a high standard of behavior. (That even extends to the kinds of customers we allow on our network because we choose to do business with those who reflect our values.)

But… which one to adopt, or should we write a new one? With so many excellent options, we quickly decided adopting an existing code of conduct would be the best path forward.

The best way to build for our community is to build in collaboration with the community, so we simply asked for input on our community forum. We received so many excellent responses and suggestions from community members and even other Fastly employees. People suggested the Contributor Covenant, the Django Code of Conduct, and Elastic’s CoC.

After reviewing the suggestions against our needs, we chose to fork Python’s Code of Conduct, which is a fork of the example policy from Geek Feminism wiki, created by the Ada Initiative and other volunteers, and is under a Creative Commons Zero license. (This felt particularly fitting, considering Python is among our highest-traffic and longest-standing Fast Forward program members.) From there, we edited the Code to suit our circumstances and community needs. Our version is published on our GitHub profile.

Finally, we needed a working group to help enforce the code of conduct, review possible violations, and determine a plan of action when infractions happen. So we formed Fastly’s OSS governance group, pulling from different backgrounds and disciplines across Fastly, including our inclusion and diversity, legal, engineering, and marketing teams. You can see the members and enforcement policies on our GitHub.

Fast Forward

Fastly is deeply committed to the open internet: we use open source software throughout our stack, contribute to open standards, and encourage employees to participate in and use open-source software.

Given the benefits we derive from open source and standards, we must give back to open source builders and communities. That’s why we created Fast Forward — Fastly’s mission to build the open internet together. This broad-reaching initiative comprises four key programs and policies: Our OSS contribution guidelines, our Customer Community Policy, our contributions to open standards, and the Fast Forward program.

Through the Fast Forward program, we give free services and support to open source projects and the nonprofits that support them. We support many of the world’s top programming languages (like Python, Rust, Ruby, and the wonderful Scratch), foundational technologies (cURL, the Linux kernel, Kubernetes, OpenStreetMap), and projects that make the internet better and more fun for everyone (Inkscape, Mastodon, Electronic Frontier Foundation, Terms of Service; Didn’t Read).

We are always looking for new projects and ways to support the open internet. If you’re an open-source project that needs help scaling and protecting your website or applications, apply to the program.

Looking forward

At Fastly, open source is a part of our heritage, essential to our operations, and central to our future. What’s more, we’re on a mission to build the good internet — whether you're building or browsing, an internet that is accessible and safe for everyone.

Take a look at our open source homepage, Fast forward program, or join us on our community forum!

The crucial difference between robots.txt and x-robots-tag

Andrew Betts — Mon, 27 Nov 2023 16:14:28 +0000

“Robots” directives control how search engine crawlers are allowed to index your website, and are one of the oldest web standards. But does it matter if you use the meta tag, HTTP header, or the robots.txt file? Yes, very much so.

I hadn’t given this a huge amount of thought until I recently had a conversation with Harry Roberts, at the perf.now() conference. Harry made me realise that there is a significant difference between robots.txt and the other approaches to restricting indexing, which if not correctly understood might mean your site goes unindexed by Google and friends completely.

Here’s an example robots.txt file telling a search crawler that it is not allowed to crawl anything under the /api path on that domain:

User-Agent: *
Disallow: /api

This means the crawler will not even request anything (except the robots.txt file). But, where the robots.txt allows it, and the crawler does make a request for something on my site, I can still stop that response from being indexed by adding a meta tag to the markup, if the response is an HTML page:

<meta name="robots" content="noindex">

Of course not everything is an HTML page, and Google and other search engines will often show documents like PDFs in search results, so a generally better approach is to use an HTTP header, which has the same effect and can be used on any type of response:

x-robots-tag: noindex

This brings us to the trade offs between using robots.txt vs these per-response directives.

Sub-resources must be fetchable

Search engines crawl using a real browser, and will therefore download all the images, CSS and scripts needed to render the page. If a script on the page makes API calls, the crawler will make those requests too.

Imagine you have a site that requires an API response to fully render a page (eg as part of a React app that doesn’t do server side rendering). The content inserted into the page from that API response will be indexable if the page itself is indexable, regardless of whether the API response had a noindex directive on it. However, if the API request path is disallowed by robots.txt, the crawler won’t even make the request at all, and the page will be incomplete.

Pages need to be discoverable

By using robots.txt to prevent resources from being fetched, you also prevent the crawler from discovering links to other resources. Conversely, a per-resource noindex directive says “don’t index me”, but doesn’t stop the crawler from scanning the page to find things that are indexable (you can separately tell crawlers not to follow links in a page with the nofollow directive).

In my experience crawlers have a creepy ability to find things even if you think they’re not linked to anywhere, but it’s certainly possible to make a piece of content accidentally undiscoverable.

Don’t invite unnecessary crawler traffic

You might be getting the sense at this point that you should lean into using resource-level noindex directives instead of robots.txt. But that comes with a different risk: of simply creating unnecessary requests to your servers. Crawlers tend to create traffic that’s more expensive to service, because they hit every page of your site. This forces your application stack to generate each page separately, even including obscure long-tail content for which crawlers might be the only regular clients.

Certainly for situations where, for example, you want an entire domain to be off limits to a crawler, having it download every page only to find it has a noindex directive is pretty wasteful.

It would also be easy to forget to include the noindex on some resources and end up having a site indexed by accident. Robots.txt is certainly easier to apply comprehensively.

Getting the balance right

So, taking the above scenarios into account, there are a few rules of thumb that make for a generally pretty good robots strategy:

where a whole domain should be off limits to crawlers, for example for staging/preview URLs, use robots.txt.
Where resources should not be indexed but might be necessary to render indexable pages, they should be marked as noindex using an x-robots-tag header.

Looking back at the earlier example of using robots.txt to disallow /api: this seems likely to be a risky decision. Maybe responses from /api would be better labelled with a noindex directive instead.

Use an edge network

If you use an edge network like Fastly, it’s often possible to adjust the headers of responses before they are served to the end user. This can be handy for quickly adjusting the indexability of resources. Check out this example to learn how to add and remove headers. You could even consider generating and serving the robots.txt file from the edge too.

Visualize your Fastly traffic on a real time globe using Glitch

Andrew Betts — Wed, 09 Aug 2023 13:37:34 +0000

On the Fastly developer hub, we recently added a visualization of the traffic flowing across the Fastly edge network. It was a big hit and customers ask me how they can visualize their own traffic like this.

Fear not. I've repackaged the React component we made to do this as a Glitch app, so you can create your very own globe. Not only that, but a lot of the feedback I was getting was stuff like "This would look great on the giant screen in our lobby" so we made a non-interactive version designed to fill a 16:9 screen. Let's dive in.

Make a log processor app

First, you need somewhere for Fastly to send your log data. A Glitch app is ideal for this (Glitch is a collaborative and free code playground which is now part of Fastly).

Remix the fastly-globeviz-data Glitch app
Note the URL that Glitch assigns to your new app, eg monthly-ferret-taxi.glitch.me or glitch.com/edit/#!/monthly-ferret-taxi.

What does this app do? It's a one-file ExpressJS app that has a few route handlers:

POST requests to / are unpacked and each line is treated as an event to emit on a Server sent events stream.
GET requests to / receive a streaming response that includes any events as they are created.
A GET request to /.well-known/fastly/logging/challenge is answered as required by Fastly's log challenge rules for HTTP endpoints.

Now you have your own log collector, it's time to send some data to it!

Instrument your Fastly service

Configure your Fastly service to emit traffic data to the Glitch app you just created.

Create a Fastly service if you don't have one already. If you do, create a new draft version of the service.
Decide on a three character code to identify this traffic, e.g. if you are the New York Times, you could use NYT. This is so that if you want to you can send multiple different website traffic streams to the same globe visualization, you can identify them and colour code them.
Decide on a sample rate. A good rate to sample is 20-50 requests per second. If that represents roughly a thousandth of your traffic, the sample rate would be 1000. A sample rate of 1 is 100% of traffic. Err on the side of sampling too little and dial it up, otherwise you may trigger rate limiting on Glitch.

Add the following VCL as an INIT VCL snippet:

table globeviz_config {
  "service_name": "ABC", // Change this value
  "sample_rate": "1000"  // Change this value
}

// https://deviceatlas.com/device-data/explorer/#defined_property_values/7/2705619
table globeviz_known_browsers {
  "Chrome": "C",
  "Chrome Mobile": "C",
  "Firefox": "F",
  "Firefox for Mobile": "F",
  "Edge": "E",
  "Kindle Browser": "K",
  "Safari": "S",
  "Samsung Browser": "A"
}

// Record the following data (space separated)
//    PIO             Name of source service (3 char)
//    LCY             Name of Fastly POP (3 char)
//    51.43,-0.23     Client location (var length)
//    17234           Duration (var length)
//    EN              Normalized Accept-Language (2 char)
//    M               Cache state (1 char: H, M, P)
//    2               HTTP version (numeric, variable length)
//    1.2             TLS version (numeric, variable length)
//    C               Browser (1 char: C, F, E, K, S, A)
//
sub vcl_log {
  // Record only non-shielding, non-VPN requests at the specified sample frequency
  if (fastly.ff.visits_this_service == 0 && client.geo.proxy_type == "?" && client.geo.latitude != 0 && randombool(1,std.atoi(table.lookup(globeviz_config, "sample_rate", "100000")))) {
    log "syslog " req.service_id " fastly-globeviz :: "
      table.lookup(globeviz_config, "service_name", "-") " "
      server.datacenter " "
      client.geo.latitude "," client.geo.longitude " "
      time.elapsed.usec " "
      std.toupper(
        accept.language_lookup(
          "en:de:fr:nl:jp:es:ar:zh:gu:he:hi:id:it:ko:ms:pl:pt:ru:th:uk",
          "en",
          req.http.Accept-Language
        )
      ) " "
      substr(fastly_info.state, 0, 1) " "
      regsuball(req.proto, "[^\d.]", "") " "
      regsuball(tls.client.protocol, "[^\d.]", "") " "
      table.lookup(globeviz_known_browsers, client.browser.name, "Z")
    ;
  }
}

Create an HTTPS log endpoint called fastly-globeviz pointing to your Glitch URL (choose a content type of "text/plain" and a placement of "none"). You can do that in the UI or with this cURL command:

curl -i  -X POST "https://api.fastly.com/service/$SERVICE_ID/version/$VERSION/logging/https" -H "Fastly-Key: $YOUR_FASTLY_TOKEN" -H "Content-Type: application/x-www-form-urlencoded" -H "Accept: application/json" -d "name=fastly-globeviz&placement=none&content_type=text/plain&url=https://$GLITCH_APP_ID.glitch.me/"

Activate the new version of your service

Test the log receiver

Open the URL of your glitch app in a browser and you should start to see traffic log data streaming in.

This is the data your Fastly service is emitting. This is a good time to check that you're happy with the type of data you're sharing with the world, since the Glitch app you're sending it to is exposing it publicly.

View it on a map!

Now you can visualize your log data on a globe. You don't actually need your own version of the globe, so you don't need to remix an app for this, you can just use mine directly - and point it at your data source.

Go to https://fastly-globeviz-ui.glitch.me/ in your browser. You should see traffic animating, but that's not your traffic right now, it's the default traffic source.
Add a query parameter ?SOURCE_URL=<your log collector> to the URL, e.g. ?SOURCE_URL=https://monthly-ferret-taxi.glitch.me. You should now see your data!
Go look at the src/config.js file in the UI Glitch app. This is where you can see all the variables that are configurable. SOURCE_URL was one of them, but they are all overridable by adding more query parameters to your URL. If you prefer, you can remix the app and edit config.js directly.

Give feedback and contribute

Both the Glitch apps are remixable, so go ahead and make changes, and let us know what you come up with in the Fastly community forum!

Who kept the bots out? Stopping content being harvested by AI

Andrew Betts — Thu, 22 Jun 2023 09:57:18 +0000

AI-powered content generation has exploded in popularity recently, with bots like ChatGPT and Bard, but the giant amounts of data these bots require comes from harvesting the web. What if you don’t want your content feeding the bots? Some respect robots.txt, others notice a new ‘noai’ header tag.

An article in Vice recently drew attention to the way AI bots are harvesting the web, in some cases quite aggressively. Site owners quite reasonably want to protect the originality of their creative works, assert their copyrights, and also not have to deal with traffic surges from ill-configured or badly-behaving scraper bots.

The particular tool mentioned in the Vice article is Img2dataset, and right now, it doesn't pay attention to the robots.txt file, the normal mechanism you can use to dissuade well behaved bots from indexing your content. However, it does respect a new HTTP header directive, X-Robots-Tag: noai (and also noindex, though that's an existing and already well-known part of the robots.txt standard).

A lot of Fastly customers have multiple websites running through Fastly, and in many cases multiple backend servers feeding a single public domain. Managing the addition of HTTP metadata at the origin application can be tedious and error prone, but fortunately applying it at the edge is pretty simple.

If you have a VCL powered Fastly service, you can add a header in a deliver snippet or in the vcl_deliver subroutine of your custom VCL code:

set resp.http.x-robots-tag = "noai";

Here's a fiddle you can play with demonstrating that code.

If you are using our new Compute@Edge platform, setting a header varies from one language to another. There's a generic example in our developer hub covering all languages we support, but as an example, here's how to do it in JavaScript:

resp.headers.set("x-robots-tag", "noai");

And again, here's a fiddle with that code you can remix and try for yourself.

Of course this just fixes the issue for one bot, and while others might pick up on signals from robots.txt, some are just going to be stubbornly insistent on crawling and downloading your content regardless. These bots should be considered bad actors and blocked. Fastly has lots of ways to do this - manual logic can be written in both VCL and all Compute languages to block based on signals such as:

ASN (Autonomous system number): the block of IPs owned by a single network operator
Explicit IP addresses or ranges
HTTP headers such as User-Agent or Accept
Originating traffic location

Here's an example of banning by IP address.

You might like to use challenges to separate out bot traffic, such as a CAPTCHA or proof of work, or use DNS lookups to authenticate good crawlers.

Finally, if you want more of a magic bullet, consider enabling our Next-gen Web Application Firewall on your service, which will detect anomalous behaviour, alert and block attacks automatically.

If you want to get inspiration for what else you could be doing with edge compute technologies, check out the solutions area on the Fastly developer hub. Happy coding!

Updating Monaco broke Fastly Fiddle: here's how I solved it with useCallback in React

Andrew Betts — Wed, 19 Apr 2023 15:07:23 +0000

My colleague Dora and I recently updated Fastly Fiddle's dependencies, and we suddenly found that user input in our code editor was erratic and unusably slow. We fixed it by moving some state from the component into a local variable, persisted across renders thanks to useCallback.

This is what we saw in the UI:

Popping open the network panel, we can see a request to the server for every keystroke, 1 second after the key was pressed:

Ouch. Our debounce is clearly broken - and not only that but we're also supposed to be cancelling save requests that are already in progress if a new keystroke is made, and that clearly isn't working either.

It turns out that the bottom line is, if you read state in a function that's been memoized, you will get an out of date version of that state.

In our case, Monaco, the code editor component, was memoizing a function, invoking an outdated version of it, and as a result we were never able to tell that a save operation was already in progress, so we didn't cancel the existing operation, and cued up a new one.

The problematic code

Many performance problems and behavioural bugs in React applications seem to stem from a too-hazy understanding of what causes a component to re-render, what happens when a component re-renders, and in what context a function is being invoked. Fortunately I am friends with Ivan Akulov who is basically web performance in human form, and he helped me figure this out.

In Fastly Fiddle, we have a state variable saveOperation which is read and updated by a saveHandler function, and which is also passed down into child components:

const [saveOperation, setSaveOperation] = useState();

const saveHandler = async (userInput) => {
  if (saveOperation) saveOperation.abort();
  const debounceDelay = abortableWait(DEBOUNCE_DELAY);
  setSaveOperation(debounceDelay);
  await debounceDelay;
  ...
  setSaveOperation(null);
}

render(<Main onSave={saveHandler} />)

We're trying to wait for an idle period before saving the user's work (a 'debounce'). Every time we see new keystrokes, we abort the pending save - or any in-flight API request that is already in progress.

We're doing this using an abortable promise pattern - there's probably a more idiomatic way to do this in React but this seemed to work pretty well. saveHandler is going to get redefined every time the component renders, but that should be fine - saveOperation is component state, which will persist across renders. Right?

Memoization problem

However, there's a risk - if anything further down the component tree memoizes the reference to saveHandler, we might end up calling an out of date version of the function which has captured an out of date version of saveOperation.

Turns out, this is exactly what's happening. After our dependency update, we discovered that when keystrokes were entered into a code editor component powered by Monaco, the resulting invocation of saveHandler found saveOperation to be undefined every time.

The <Main> component eventually passes the saveHandler function to Monaco like this:

<MonacoEditor
  theme="fiddle"
  key={props.codeContext}
  options={monacoOptions}
  value={props.value}
  onChange={props.saveHandler}
/>

The new version of Monaco then memoizes the onChange callback, which means that when keystrokes are entered into the editor, the saveHandler that gets fired is one that doesn't have the latest value of saveOperation, so we don't know there's a save already in progress, so we don't cancel it, and all hell breaks loose.

Progress state is an antipattern?

So what's the solution? I could investigate why Monaco is memoizing the handler, and maybe try and convince it to not do that, but this problem highlights that using useState to store saveOperation is maybe not the best idea in the first place. Apart from making the function dependent on data in the parent context, using state also means we re-render the component every time we update it. Since saveOperation doesn't affect the component's render output, that doesn't seem right.

I've decided to call this kind of data "progress state", since it effectively carries data forward from one invocation of a function to the next, to allow a subsequent invocation to adjust its behaviour based on knowing where we left off in the previous one.

useState is the most obvious way to do this, but it's not a good one. The React-y way to do this is actually useRef:

Solution 1: useRef

const saveOperation = useRef(null);

const saveHandler = async (userInput) => {
  if (saveOperation.current) saveOperation.current.abort();
  const debounceDelay = abortableWait(DEBOUNCE_DELAY);
  saveOperation.current = debounceDelay;
  await debounceDelay;
  ...
  saveOperation.current = null;
}

Updating saveOperation now won't trigger a render, and this will work even if saveHandler gets memoized.

The reason this works is the same reason useRef variables always have that weird .current thingy hanging off them. The variable created by useRef is immutable, so capturing it into a function is fine, because we know it cannot be reassigned: the value of saveOperation (which is an object) will never change. However, the properties of that object can change, and a function that has captured a reference to the object will see those updated properties. That's why refs have a .current property.

Solution 2: useCallback closures

React's useCallback hook creates a reference to a function and then reuses it in subsequent renders of the component, rather than redefining the function every time. We figured, you could use this with an Immediately Invoked Function Expression (IIFE) to capture a value that persists between renders:

const saveHandler = useCallback((() => {
  let saveOperation;
  return async (userInput) => {
    if (saveOperation) saveOperation.abort();
    saveOperation = abortableWait(DEBOUNCE_DELAY);
    await saveOperation;
    ...
    saveOperation = null;
  };
})(), []);

Now, when the component is first rendered, React will execute the IIFE, and assign the resulting function to saveHandler. The returned function has access to saveOperation which is trapped in a closure created by the IIFE.

I like this solution because:

it leans more on fundamentals of JavaScript rather than relying on the framework to solve the problem
it creates a tighter scope for my data (saveOperation is not accessible outside of saveHandler)
It's the simplest use of saveOperation - no need to use a setter function, and no need to use a .current property.

However, React's docs on useCallback have this caveat:

In the future, React may add more features that take advantage of throwing away the cache — for example, if React adds built-in support for virtualized lists in the future, it would make sense to throw away the cache for items that scroll out of the virtualized table viewport. This should match your expectations if you rely on useCallback as a performance optimization.

So in future, my callback might get redefined more often... but for a use case like debouncing, it seems like that would still be fine most of the time.

Conclusion

There are some nice lessons in this debugging exercise: Don't overuse useState. Learn to love useRef. And it's nice that React has a solution to this and that we can also solve it using JS language primitives.

Fine, go ahead and flame me for doing React wrong. I don't care, React is weird anyway.

Filter PNGs for Acropalypse using Compute@Edge

Andrew Betts — Thu, 23 Mar 2023 10:11:26 +0000

Last week, Simon Aaarons and David Buchanan posted their discovery that images cropped using Android's Markup editor app often hid within them the original uncropped image. David went on to suggest that CDNs could transparently mitigate the vulnerability by deploying a filter at the edge.

Challenge accepted!

The easy answer

We already have an Image optimization feature, and we've verified that any image that passes through our optimizer will be stripped of trailing content. That means if you use IO on your Fastly service, you're already filtering images such that the acropalypse data will be removed.

However, not everyone uses image optimization with their Fastly service. So what if you don't?

Understanding the problem

PNG files have a well known byte sequence that identifies them, the "magic bytes", which are these:

89 50 4E 47

You can see this in a hexdump of any PNG file:

○ head -c 100 Screenshot\ 2023-03-01\ at\ 10.35.01.png | hexdump -C
00000000  89 50 4e 47 0d 0a 1a 0a  00 00 00 0d 49 48 44 52  |.PNG........IHDR|
00000010  00 00 06 cc 00 00 04 e6  08 06 00 00 00 85 ef 84  |................|
00000020  f2 00 00 0a aa 69 43 43  50 49 43 43 20 50 72 6f  |.....iCCPICC Pro|
00000030  66 69 6c 65 00 00 48 89  95 97 07 50 53 e9 16 c7  |file..H....PS...|
00000040  bf 7b d3 43 42 49 42 28  52 42 6f 82 74 02 48 09  |.{.CBIB(RBo.t.H.|
00000050  a1 85 de 9b a8 84 24 40  28 21 26 04 05 bb b2 b8  |......$@(!&.....|
00000060  82 6b 41 44                                       |.kAD|
00000064

PNG files end with a similar marker, called IEND, which is also easy to spot in the byte stream - a sequence of four null bytes and the ASCII characters IEND:

○ tail -c 100 Screenshot\ 2023-03-01\ at\ 10.35.01.png | hexdump -C
00000000  98 cf ff 9d 2f 79 c0 7b  c3 ec de 7b ef 8d 1b 96  |..../y.{...{....|
00000010  2c 0b 1e f0 24 c3 a2 32  f9 4a fd 95 c6 3a 7c 13  |,...$..2.J...:|.|
00000020  a1 1a 78 6c 47 70 c6 fb  dc 41 dd f2 8d 4b 79 d1  |..xlGp...A...Ky.|
00000030  35 f2 6b 89 ef f0 d8 3e  e8 ed 23 70 c6 3f 5e 6f  |5.k....>..#p.?^o|
00000040  6f 98 75 80 4b 7c da 01  1e 1c 72 80 93 a8 ff 37  |o.u.K|....r....7|
00000050  c8 1a 61 82 9a 74 43 f6  00 00 00 00 49 45 4e 44  |..a..tC.....IEND|
00000060  ae 42 60 82                                       |.B`.|
00000064

The problem is that some tools, notably the markup tool on Android, when cropping an image, will include the original image data in the leftover space at the the end of the file.

Simon's post helpfully illustrates the issue:

OK, so how can we stop this from leaking our user's private data?

Compute@Edge to the rescue

Let's crack open Fastly Fiddle, and write some code at the edge. To start, I'll make a basic request handler, which sends the request to a backend, and passes the response back to the client.

That looks like this:

addEventListener("fetch", event => event.respondWith(handleRequest(event)));

async function handleRequest(event) {
  const response = await fetch(event.request, {backend: 'origin_0'});
  return acropalypseFilter(response);
}

function acropalypseFilter(resp) {
  return resp;
}

And in Fiddle you'll see this:

I can configure my fiddle to use http-me.glitch.me as a backend, a server Fastly maintains to provide responses that can be handy for testing this kind of thing. The path can be set to /image-png to get HTTP-me to give me a PNG image.

Now I need to flesh out my acropalypseFilter function. Here is what I came up with:

function acropalypseFilter(response) {
  // Define the byte sequences for the PNG magic bytes and the IEND marker
  // that identifies the end of the image
  const pngMarker = new Uint8Array([0x89,0x50,0x4e,0x47]);
  const pngIEND = new Uint8Array([0x00,0x00,0x00,0x00,0x49,0x45,0x4e,0x44]);

  // Define an async function so we can use await when processing the stream
  async function processChunks(reader, writer) {
    let isPNG = false;
    let isIEND = false;
    while (true) {

      // Fetch a chunk from the input stream
      const { done, value: chunk } = await reader.read();
      if (done) break;

      // If we have not yet found a PNG marker, see if there's one
      // in this chunk.  If there is, we have a PNG that is potentially
      // vulnerable
      if (!isPNG && seqFind(chunk, pngMarker) !== -1) {
        console.log("It's a PNG");
        isPNG = true;
      }

      // If we know we're past the end of the PNG, any remaining data
      // in the file is the hidden data we want to remove.  Since we already
      // sent the Content-Length header, we'll pad the rest of the response
      // with zeroes.
      if (isIEND) { 
        writer.write(new Uint8Array(chunk.length));
        continue;

      // If it's a PNG but we're yet to get to the end of it...
      } else if (isPNG) {

        // See if this chunk contains the IEND marker
        // If so, output the data up to the marker and replace the rest of the
        // chunk with zeroes.
        const idx = seqFind(chunk, pngIEND);
        if (idx > 0) {
          console.log(`Found IEND at ${idx}`);
          isIEND = true;
          writer.write(chunk.slice(0, idx));
          writer.write(new Uint8Array(chunk.length-idx));
          continue;
        }
      }

      // Either we're not dealing with a PNG, or we're in a PNG but have not
      // reached the IEND marker yet. Either way, we can simply copy the
      // chunk directly to the output.
      writer.write(chunk);
    }

    // After the input stream ends, we should do cleanup.
    writer.close();
    reader.releaseLock();
  }

  if (response.body) {
    const {readable, writable} = new TransformStream();
    const writer = writable.getWriter();
    const reader = response.body.getReader();
    processChunks(reader, writer);
    return new Response(readable, response);
  }
  return response;
}

Let's step through this:

We set up some variables to hold the byte sequences that will tell us whether the response is a PNG and also whether we've reached the end of it.
Skipping to the bottom, the if (response.body) block creates a TransformStream, and returns a new Response that consumes the readable side of it. Now we need to read from the backend response, and push data into the writable side of the TransformStream.
Since we want to return a Response from the acropalypseFilter function, the acropalypseFilter function cannot be async (if it was declared async it would return a Promise, not a Response). So instead, I've defined a child function, processChunks, which can be async. We can also call it without awaiting the Promise, because it's OK to return the Response before the stream has ended (better, actually!)
In processChunks, we create a read loop and read chunks of data from the backend response. We are looking for the magic marker that says the file is a PNG. If we find it, we're then looking for the marker that indicates the end of the PNG. If we find that, then everything after that point is replaced by zeros. This is the key bit that defeats the data leak.

But, why not just cut off the response at that point? The problem is, we've already sent the headers of the response, and most likely, the response headers included a Content-Length, so we need to emit that much data.

One other thing. The chunk that comes out of the reader.read() method is an ArrayBuffer, and we need to search it to find the special sequences of bytes we're looking for. There's no way to do this natively in JavaScript so I invented a function to search for a sequence of elements in an array:

// Find a sequence of elements in an array
function seqFind(input, target, fromIndex = 0) {
  const index = input.indexOf(target[0], fromIndex);
  if (target.length === 1 || index === -1) return index;
  let i, j;
  for (i = index, j = 0; j < target.length && i < input.length; i++, j++) {
    if (input[i] !== target[j]) {
      return seqFind(input, target, index + 1);
    }
  }
  return (i === index + target.length) ? index : -1;
}

There's definitely ways to improve this, but for a quick exercise in what can be done with edge computing, it's really cool.

Check out my fiddle here and if you like, clone it and make your own version.

JavaScript support hits 1.0 milestone on Compute@Edge

Andrew Betts — Mon, 19 Dec 2022 16:26:51 +0000

When Compute@Edge was launched, we talked about why we didn’t fully support JavaScript at that time. Support was added in July last year, and it has been an exciting journey to see what people have already built with the SDK at scale. With confidence in its stability we are proud to announce a 1.0 release of our JavaScript SDK.

We listened to the community feedback, filled in feature gaps, and addressed many bugs in the SDK. Not only that, we’ve also overhauled the SDK reference docs making it easier for you to know what’s supported and how to implement the features. All the Fastly specific features of the JS SDK now have interactive example applications in the documentation.

We've also been building out example code on our developer hub to showcase how to use JavaScript to address common use cases. Take a look at examples like geo-tagging of requests, capturing and aggregating log data from clients, normalizing requests to improve cache efficiency, or load balancing across multiple backends.

Want to deploy a site you've built using a JavaScript-based web framework? We got you - check out our tools and instructions for Gatsby and Next.JS. RemixJS is coming soon.

What if you have a dynamic backend in NodeJS running ExpressJS? It'd be cool if Compute@Edge offered Express-like abstractions, so you can write server apps on Fastly using familiar constructs like route handlers and middleware. No problem! Check out Expressly, our Express-alike server framework at the edge!

There are blogs and tutorials to inspire you if you're after something more advanced too: learn more about using JavaScript for A/B testing, or create a waiting room to handle traffic surges, for example.

Our support for JavaScript is unique

In the serverless world, “cold starts” are the enemy. Even if a platform can run your code fast 99% of the time, an occasional 300ms delay while your app starts up can create debugging headaches, and these can easily add together in a microservices architecture and lead to much more frequent poor performance for end users. At Fastly, we don't have cold starts because we don't do warm starts: your code starts from the same state on every request, and that startup takes microseconds. My colleague Lin Clark has explored how we do this over on the Bytecode Alliance blog.

Speaking of which, as a founding member of the Bytecode Alliance, we're building an ecosystem of languages that can run on the server using WebAssembly, using the Wasmtime runtime.

For JavaScript we use SpiderMonkey on top of Wasmtime, which provides spatial and temporal separation between concurrent tasks. Spatially, each request gets its own language runtime and memory heap. Temporarily, there is no state left over from previous requests. Starting from a fresh identical snapshot on every request makes debugging significantly easier. A bug in one task cannot affect subsequent or neighboring tasks.

Future of JavaScript on Compute@Edge

We’ve solved the biggest security headaches of secure sandboxing, and have unmatched initialization performance of startup times, from cold, in the tens of microseconds. With the v1 release of our SDK we have achieved sufficient feature support in JavaScript while providing execution performance that is within a reasonable margin of modern JS JITs. But we're going to make it even faster, so stay tuned.

Not JUST JavaScript

And because our platform doesn't rely on a JavaScript engine at its core, you're not locked into one language - write JavaScript if you want to, but if you need the robustness of strict static typing and lower level memory management, you could just as well use Rust or Go instead. And developers outside of Fastly have even made SDKs for languages we don't officially support yet: we celebrate them and showcase them on our developer hub. All languages that run on Compute@Edge are equals on our platform.

We love to see what our customers build. Find us on twitter at @fastlydevs or tweet with #builtonfastly and let us know!

We won best dev portal!

Andrew Betts — Fri, 16 Dec 2022 17:08:28 +0000

Yesterday the Fastly developer hub developer.fastly.com won Best Onboarding Experience at the 2022 Dev Portal Awards!

One of the jurors said:

They really want to get you started fast and without friction. Very well done, provides a lot of helpful resources. Testing options that you can use immediately.

One of the other comments was my favourite though:

Really impressed that Fastly finally did the interactive tutorial that Stripe was known for, can try and see an API that is quite complicated, this is not easy.

We were nominated in a group of 12, some of the best developer experiences around, and we were honoured to be in such auspicious company.

It's so important to us to have respect for our developer community. The docs and tools we offer to our developers should be as good as or better than the docs we rely on ourselves. That means comprehensive coverage - if you can use it, we should document it - and tools that let you try things out as easily as possible.

Describing our abstractions is also a great opportunity to recognise rough edges and feed improvements back into our product development.

And beyond providing information and helping to shorten the path from idea to execution, we see one of the key goals of our developer hub to be to inspire. There's a balance to be struck - we want to help people understand the art of the possible but we also know the very best ideas for how to use Fastly and what to use Fastly for are going to come from the developer community. So we try to inspire but also to empower people to experiment and play with their own ideas.

In all of this we have ourselves been inspired by some spectacular sites that we use every day: Stripe, MDN, Eventbrite, Google, and Twilio have all inspired elements of the Fastly developer site.

Finally, I'm absurdly proud of the six people across three continents that work incredibly hard to make the developer hub what it is. Journie, Chris, Kats, Mark, Kailan, and Dora, have done a fantastic job, and it's really lovely to be able to recognise the people behind the webpages.

And of course thank you to our developer community, you're the people we do this for, and there's a reason we put a feedback form on every single page of the devhub, because we love talking to you and we love it when you talk to us.