DEV Community: Tim Perry

The right way to turn off your old APIs

Tim Perry — Thu, 21 Jan 2021 12:25:00 +0000

All things come to an end, even HTTP APIs. However great your API may be today, one day you'll want to release a completely new version, an improved but incompatible endpoint, a new parameter that solves the same problem better, or to shut down your API entirely. Your current API will not be live forever.

Inconveniently though, your API has clients. If you shut down endpoints, parameters, or entire APIs without properly warning them then they're going to be very unhappy.

How do you shut down your APIs safely, making it at easy as possible for your users?

There are right ways to do this, including two new draft headers being standardized by the exciting new IETF "Building Blocks for HTTP APIs" working group, designed to help with this exact process. Let's take a look.

Make a plan

First up: check if the API in question actually has any clients.

Hopefully you have some API metrics or at least logging somewhere. If you don't, add some! If you do, and you can tell for sure that nobody is using this API anymore, then you win. Turn it off right now, delete the code, skip this article and have a well-deserved nap.

The next question, if you're not napping, is to ask yourself is whether there's an alternative to shutting down this API. Everything you turn off will break somebody's code and take their time to fix it. It's good for the health of your client ecosystem and the web as a whole if APIs keep working.

In many cases, old APIs can be translated internally, to transparently transform requests into calls to a new API instead, without maintaining two completely independent versions. This is a fundamental part of the API versioning approach at Stripe who include transformations with all API changes to ensure that requests for incompatible old versions continue to work as before, automatically translating the requests and responses to use the newer code as required.

Translation like this isn't always possible, and doing so forever can entail significant extra complexity, but if you can do it, this can provide a valuable stability for your users, and avoid a lot the work required for deprecation or old version maintenance.

However, if this service/endpoint/parameter is in use in production, and it's not practical to keep supporting it, it's got to go.

To do that, you need a plan. There's three key questions to ask first:

What do you expect clients using this to do? Common answers include:
- Update to a newer still-supported version of the same thing.
- Use some other substitute endpoint/parameter/service instead.
- Use a different service, they're on their own, you don't care.
When should they start migrating away from this API? Is your proposed replacement ready to use today?
What's the deadline? I.e. when will this API stop working completely? (If you're not totally sure yet, you can delay this answer for a little bit).

Once you've got a plan, it's time to tell people about it.

Communicate

First up: tell the humans.

Email your mailing list, post it on Twitter, update your API specification if you have them (e.g. OpenAPI has a deprecated field on operations and parameters), and highlight this loudly in the relevant documentation online.

You should include all the info above: what they should do instead, when you recommend they start migrating, and the deadline when they must migrate (if you have one).

Once you've told the humans, it's time to tell the computers. This is where the new IETF headers come in.

The Deprecation Header

The Deprecation header tells clients that the requested resource still works as before, but is no longer recommended. You can state that very simply with a single HTTP header:

Deprecation: true

Alternatively, you can provide a date. This date tells users when they should start migrating elsewhere. This can be in the past (if they should start migrating immediately) or the future (usually meaning that the thing they should migrate to isn't ready yet). Like so:

Deprecation: Thu, 21 Jan 2021 23:59:59 GMT

If you're deprecating the whole endpoint or service, you can just return this with every response. If you're deprecating a specific feature, perhaps a parameter, request method, or a certain field in the request body, then you want to just return this in requests when that feature is used.

To give the client more information, you can use Link HTTP response headers to link to endpoints or human-readable documentation elsewhere. You can more than one of these in combination in the same Link header, by just comma-separating them (we'll see a full example later). The spec defines 4 links related to deprecation:

Deprecation links

You can link to a human-readable description of the deprecation like so:

Link: <https://developer.example.com/deprecation>; rel="deprecation"; type="text/html"

This is the main way to tell your users what's going on, and what they should do about it. You almost always want to use this! If you don't have the full details and a final shutdown date yet, then even a placeholder saying that will be helpful. In that case, don't forget to let users subscribe for updates, with a mailing list or RSS or similar, so they can hear about the full plan once it's ready.

Latest-Version links

If you want clients to move to the latest version of the same endpoint of your API, use this to point them there, like so:

Link: <https://api.example.com/v10/customers>; rel="latest-version"

Successor-Version links

If you have multiple versions of your API available, it's usually nicer to migrate one version forwards at a time, rather than jumping straight from the oldest now-deprecated version to the latest. To help with this, you can link to the next version of the deprecated endpoint, not just the latest, like so:

Link: <https://api.example.com/v2/customers>; rel="successor-version"

Alternate links

If there's no new equivalent version of this API, and users should migrate to a totally different resource that might be a good substitute, you can use alternate links to indicate that:

Link: <https://api.example.com/v2/users/123/clients>; rel="alternate"

The Sunset Header

Once you know when the API is going to shutdown entirely, you should add a Sunset header.

The Sunset header tells clients when this will stop working. It's a hard deadline: API clients must move elsewhere before this date, and you promise not to break anything until then.

You must provide a date here, and it should be in the future. If it's in the past, that's OK though: at that point you're effectively saying "this could turn off at any moment you need to stop using it immediately". It looks like this:

Sunset: Tue, 20 Jul 2021 23:59:59 GMT

This is super simple, and can be used for more than just API shutdowns: you can use it to signal HTTP redirects that will come in the future for URL migrations, or to indicate limited lifetimes of certain URLs (for content that's temporary by nature, or for regulatory reasons like data retention policies on certain resources). All it says is "this endpoint may stop doing what you expect after this date, be ready".

Sunset links

This spec also provides a sunset link relationship. This is designed to link to more information about your plan for shutting down this specific endpoint (probably the same documentation as your deprecation link, if you have one) or about the general sunset policy for your service. Like so:

Link: <http://developer.example.com/our-sunset-policy>;rel="sunset";type="text/html"

This is a good place to point out that a general sunset policy is a very useful thing! A sunset policy tells clients when you shut down endpoints (e.g. 1 year after a replacement goes live) how users should ensure they hear about this (mailing lists, status pages, HTTP headers, you name it), what they should usually do about it (update, check the docs, follow Link headers).

Adding one doesn't help much with doing a deprecation right now, but if you'd published one a year ago, your clients would be ready already. The second best time to publish a sunset/deprecation policy is now. Might be worth considering if you're writing deprecation docs anyway.

All together

These parts are designed to work nicely together. For example, to indicate that an API was deprecated recently, will be turned off in 6 months, link to the documentation, and provide a direct link to the next version, you should include headers like this in the response:

Deprecation: Thu, 21 Jan 2021 23:59:59 GMT
Sunset: Tue, 20 Jul 2021 23:59:59 GMT
Link: <https://api.example.com/v2/customers>; rel="successor-version",
    <https://developer.example.com/shutting-down-customers-v1>; rel="deprecation"

Progressive shutdowns

Once all that's in place, and your sunset deadline has passed, you're good to go.

That doesn't mean you need to immediately kill the API completely though. Progressive shutdowns can help ensure that any clients still using this API get a last-chance warning before it disappears completely. GitHub did this when removing some crypto support in 2018: first they disabled it for one hour, then reenabled it, then they disabled it permanently two weeks later.

There's other tricks too: Android added increasing delays to deprecated native APIs in 2015, eventually going up to a full 16 second wait, before finally turning off the API entirely. These progressive shutdowns provide a little flexibility for clients who miss your deadline, and may help clients who haven't noticed the deprecation spot and deal with the issue before the API turns off completely.

Flip the switch

Either way, once you've done the best you can to communicate the shutdown, it's time to turn off the endpoint/feature/entire service, delete the code, and finally go take that nap.

Doing deprecations and shutdowns carefully like this makes it as clear as possible to your clients how they can depend on your API, when they need to take action, and what they need to do. These kind of changes can be a big deal, and this information is important!

These new draft headers allow us to communicate not only to humans, but also to expose this information to automated systems. As these headers become widespread, I'm really excited to start seeing more tooling building on top of them. Generic HTTP clients can log useful warnings automatically based on this data, API generators themselves can handle more and more of this for you based on API specifications, and HTTP debuggers like HTTP Toolkit can highlight usages of deprecated endpoints for you in intercepted live traffic. It's an exciting time to start turning things off!

It is important to note that these headers are draft HTTP specifications. It's possible they may change before they're finalized. That said, they've been through a few rounds of revisions already, it's fairly unlikely they'll change dramatically from here, and it's time to start testing them in the wild.

This does mean there's still time for feedback though! If you have thoughts on how this works and how it could work better, get in touch with the "Building Blocks for HTTP APIs" working group. You can email the mailing list at httpapi@ietf.org, or scroll the previous mailing list discussions here.

Originally published on the HTTP Toolkit blog

Mining your CLI history for good git aliases

Tim Perry — Mon, 30 Nov 2020 13:35:00 +0000

If you use the command-line all day, CLI improvements can add a huge boost to your workflow. One of the simplest ways to improve things is to make your most used commands easier & faster to type, by creating aliases.

But which aliases? Which commands are most important for your usage? You can probably guess a couple, but it's hard to know for sure, and there's plenty you'll miss. Fortunately your CLI history already has all the answers, if you just know how to ask.

As a software developer I'm going to focus on git aliases here, but this applies equally well to any command-line tools you use heavily. I'm also assuming a bash-compatible shell, but the same concept should translate elsewhere easily enough.

The first step is to use history to do some digging and find out what commands you run most frequently. history prints every line you've run recently in your shell, in chronological order. That gives us the data we need, and with a little bash-fu we can start to get some answers.

First, what're your most run git commands?

history -n | grep git | sort | uniq -c | sort -k1,1nr -k2

That takes your history, filters for git commands, sorts them alphabetically, counts the repeated lines, and then sorts by the repeat count.

You can add a | head -n X too, if you'd like to see just the top X results. For me, with a few weeks history on a new laptop, that looks like:

    543 git status
    272 git add -p
    214 git tree
     71 git diff
     55 git commit --amend
     53 git push origin master
     32 git checkout -p
     30 git reset
     27 git stash pop
     26 git stash

That tells you a bunch about my git workflow already! These are all common commands I'm using frequently, and commands I should very seriously consider aliasing.

It doesn't tell the whole story though. How come git commit --amend is so high up, but git commit doesn't appear at all?

That's because for many commits I run git commit -m "..." to commit and pass a message inline, so each of those commands is treated as unique, and won't appear in this top list. We need to get a little smarter.

We can catch cases like that too, by limiting the input we consider for uniqueness. Like so:

history -n | grep git | cut -d' ' -f -3 | sort | uniq -c | sort -k1,1nr -k2

Here I've added cut -d' ' -f -3, which splits each line by spaces, and includes only the first 3 parts (e.g. git push origin master becomes git push origin). This isn't perfect, but it does let us find command prefixes of a given length.

With that, my results become:

    543 git status
    299 git add -p
    214 git tree
    199 git commit -m
    117 git push origin
     71 git diff
     60 git commit --amend
     34 git checkout -p
     30 git reset
     29 git diff --cached

We can now see that I'm committing a lot too, but with messages, I'm pushing with many other branches, and I'm diffing my cached (already added) changes often, but usually with an argument ('show me what I've just added to the tests').

Fiddle around with this a little, try a few different lengths of prefix, and you'll quickly find a set of commands that stand out with frequent use patterns, with or without extra arguments.

From there, it's alias time. I could make these aliases within git, but I'd actually prefer to do it at the shell level, so I can shorten them further (not just to git x but gx).

In my case, I'm doing that by adding the below to my .bashrc:

alias gs=git status
alias gap=git add -p
alias gt=git tree

alias gc=git commit
alias gcm=git commit -m
alias gca=git commit --amend

alias gpo=git push origin
alias gd=git diff
alias gdc=git diff --cached

(Don't forget to check these don't conflict with anything else you use on your machine!)

These aliases can also be a very convenient place to add any extra arguments you often want but don't always remember, like -w for diff (ignore whitespace changes) or -v for git commit (include the diff contents in the commit template, so you can see it while you write your message).

It's a quick trick, but just a little bash magic can tell you a lot about your working habits, and shine a useful light on ways you can make your life easier. Give it a go, and let me know what you think on Twitter or in the comments below.

Originally published on the HTTP Toolkit blog

Intercepting HTTPS on Android

Tim Perry — Thu, 05 Nov 2020 18:00:00 +0000

To intercept, inspect or manipulate HTTPS traffic, you need the HTTPS client to trust you.

If you want to intercept your own HTTPS on Android, perhaps to capture & rewrite traffic from your Android device for debugging or testing, how do you do that?

This isn't theoretical - HTTP Toolkit does exactly this, automatically intercepting HTTPS from real Android devices, for inspection, testing & mocking. To do so, it has to automatically ensure that it's trusted by HTTPS clients on Android devices, without breaking security on those devices completely (it would be a very bad idea to simply turn off certificate validation, for example). Here's a demo: https://www.youtube.com/watch?v=ttf8IhfI0Ao

Let's talk though how HTTPS clients in general manage this kind of trust, see how that works on Android specifically, and then look at how it's possible to get around this and intercept real HTTPS traffic.

How HTTPS trust works

An HTTPS request is an HTTP request, made over a TLS connection. Everything we're going to talk about here is really about TLS - the HTTP within is just normal GET / requests and 200 OK responses.

I'm not going to go into the lowest level details, but it is important to understand the basics of how TLS works. If you are interested in the fine details of TLS, The Illustrated TLS Connection is well worth a look, for a byte-by-byte breakdown of the whole process.

The high-level summary is this:

Every TLS client keeps track of some set of root certificate authorities (root CAs) that it trusts completely.
When any modern TLS client first connects to a server, its initial message includes a Server Name Indication (SNI), telling the server which hostname it's looking for (e.g. example.com). It expects the server's response to include a valid certificate for that hostname.
TLS certificates include a reference to the issuer of the certificate, and a signature proving that the issuer verified the certificate. The issuer's certificate in turn will have its own issuer & signature, creating a chain of certificates, up until a final self-signed root certificate.
The client must decide if it trusts the server's certificate. It does so by checking the details of the certificate (notably checking the hostname is what was expected), and then examining the issuer of the certificate, and then issuer of the issuer's certificate, and so on and so on until it reaches a certificate that it already trusts (a trusted certificate authority) or running out of issuers and deciding that it doesn't trust the certificate at all.
If the client trusts the certificate, it continues creating the encrypted connection, and then sends and receives data over that connection. If it doesn't trust the certificate, it closes the connection before sending any content, i.e. it never sends any part of its HTTPS request.

In short: every TLS client has a list of root CAs that it trusts, and to successfully receive an HTTPS request, you must be able to present a certificate for the target hostname that includes a trusted root CA somewhere in its chain.

This is a bit simplified and I'm ignoring all sorts of edge cases, but it's enough for our purposes. If you'd like to get into the nitty gritty of how the certificate validation really works, Scott Helme has written up a great guide.

So, given the above, if we want to intercept HTTPS we need to be able to present a certificate issued by a trusted certificate authority. Since nobody reading this has a globally trusted root CA to hand, in practice that means we need to create our own CA, and ensure the TLS client (in this case, Android's HTTPS clients) already trusts that CA, before we can get started.

How do Android HTTPS clients decide who they trust?

Android Certificate Stores

Each HTTPS or TLS client on Android will check certificates against the CAs in some certificate store.

There's at least 3 types of Android CA certificate store:

The OS has a 'system' certificate store, at /system/etc/security/cacerts/. This is prepopulated on the device at install time, it's impossible to add certificates to it without root access, and is used as the default list of trusted CA certificates by most apps. In practice, this store defines which CA certificate most apps on your phone will trust.
The OS also has a 'user' certificate store, usually at /data/misc/user/0/cacerts-added/, containing trusted CA certificates that were manually installed by the user of the device. Installing one of these certificates requires accepting quite a few warnings, and became even more difficult in Android 11.

Apps targeting Android API level <24, i.e. before Android 7, or applications that specifically opt in will trust CA certificates in this store. Most apps don't, but this is enabled on a few apps where it's widely useful (notably Chrome, for now) and it's easy for developers to enable for testing with a few lines of XML.
Lastly, each application can include its own CA certificates, embedding its own short list of trusted certificate authorities internally, and refusing to trust HTTPS communication with certificates signed by anybody else. Nowadays this is fairly uncommon, except for apps that are especially security conscious (banking) or very high-profile (facebook), mostly because it's complicated and the changes in Android 7 to untrust the user store make this kind of pinning unnecessary.

If you want to intercept HTTPS traffic from an app, you need to ensure your CA certificate is trusted in the app's certificate store of choice. How can you do that?

How to intercept Android HTTPS

To intercept HTTPS, you first need the TLS connections to come to you. HTTP Toolkit runs as a desktop app on your computer, acting as an HTTP(S) proxy, and does this with an Android VPN app on the device that redirects packets to that proxy. I've written quite a bit of detail about that over here, and it's fairly easy to do if you either use the VPN APIs or configure an HTTPS proxy, so let's take that as a given.

Once you have TLS connections going to our server, you need to be able to respond to the initial client handshake with a certificate that Android will trust. Typically you'll generate a self-signed CA certificate when setting up interception, and then use that to generate TLS certificates for incoming connections, generating a fresh certificate for each requested hostname.

To make that work, you need to make your Android device's HTTPS clients trust your locally generated CA.

There's two big cases here:

Non-rooted production devices. Normal phones. This includes most phones sold and used day to day, and official 'Google Play' Android emulators.
Rooted devices. More specifically: devices where root access is available via ADB (not that root access is necessarily available to apps on the device). This includes normal phones that have been manually rooted, but also both the 'Google APIs' and AOSP official Google emulators, and most other Android emulators like Genymotion.

In general, less than 1% (according to some very dubious guesstimates) of typical user's devices are rooted.

However for Android developers, testers, and security researchers, that number runs far higher. Within HTTP Toolkit's user base for example, it looks closer to 30% (whilst I don't know for sure, but I suspect that emulators make up a large percentage of that).

Injecting CA certificates into rooted devices

This is the fun case. If you have root, how do you make apps trust your CA certificate? It turns out that even with root it's not quite as easy as it could be, but it's definitely possible to inject system certificates, so that almost all apps trust your CA by default.

There's a couple of challenges:

Even as root, /system is not writable by default
Making /system writable on emulators is only possible if the emulator is always started with an extra command line argument, and so requires restarting the emulator if that's not already set. To make this worse, it's not possible to set custom command line arguments in Android Studio, making this very inconvenient for normal use.
Even if you write a valid CA certificate to the right place in the system certificate files, it won't be recognized. You need to ensure all the permissions & SELinux context labels are set correctly before Android will trust files in that directory.

To handle all this, as root, HTTP Toolkit:

Pushes the HTTP Toolkit CA certificate to the device over ADB.
Copies all system certificates out of /system/etc/security/cacerts/ to a temporary directory.
Mounts a tmpfs in-memory filesystem on top of /system/etc/security/cacerts/. This effectively places a fresh empty filesystem that is writable over the top of a small part of /system.
Moves the copied system certificates back into that mount.
Moves the HTTP Toolkit CA certificate into that mount too.
Updates the permissions to 644 & sets the system_file SELinux label on everything in the temporary mount, so it all looks like legitimate Android system files.

This is all open source of course, and the full script to do this is here: httptoolkit-server:adb-commands.ts#L200-L253.

If you have a CA certificate, you can do this for yourself on any device with root access, to temporarily add new CAs that'll be trusted like any other CA prebundled on the device.

If you are doing this for yourself though, be careful around permissions, as the default for ADB pushed files is very relaxed. If the CA you inject or the copied system certificates are globally writable, it'd be theoretically possible for another app on the device to change or add a CA during this process, and sneakily get visibility into all HTTPS traffic on the device for itself without you realizing or granting it root access.

All put together, this injects a system certificate without needing emulator startup arguments, and works 100% automatically & immediately, without even needing to reboot. As a nice bonus the tmpfs disappears on reboot, so everything is cleaned up automatically afterwards, and you only trust the inject CA temporarily (wherever possible, it's always a good idea to limit and/or avoid global developer CAs).

Intercepting HTTPS on non-rooted devices

If you don't have root access, you can't do this. Instead, the best you can do is to install the certificate into the user store. To do so:

If you're setting the device up manually:
- Download the certificate onto your device.
- Go to "Encryption & Credentials" in your device security settings.
- Select "Install a certificate", then "CA Certificate".
- Open the downloaded certificate, and follow the confirmation prompts.
If you're automating/scripting this:
- On Android up to and including Android 10, you can use the KeyChain.createInstallIntent() to prompt users to trust your CA certificate in your app. There'll be some warnings there, and they'll need to set or confirm the device pin to do so, but it's very straightforward. You can see HTTP Toolkit's code in httptoolkit-android:MainActivity.kt#L603-L606.
- On Android 11, due to recent changes you're in trouble. You can't launch the prompt to trust a CA directly, and it must be installed in the settings completely manually. If you do try to use the createInstallIntent() API to install the certificate, it just shows "Can't install CA certificates: CA certificates can put your privacy at risk and must be installed in Settings".
  
  If you download the CA certificate to the device though, it's easy enough to explain the process to users, and you can see how HTTP Toolkit does that in httptoolkit-android:MainActivity.kt#L615-L661.

With that done you can intercept HTTPS from Chrome, and other Chromium-based browsers, and you can intercept traffic from apps that explicitly opt in.

If you're debugging your own app that's fine, since it's just a few lines of XML to do so:

<?xml version="1.0" encoding="utf-8"?>
<network-security-config>
    <base-config>
        <trust-anchors>
            <certificates src="system" />
            <certificates src="user" overridePins="true" />
        </trust-anchors>
    </base-config>
</network-security-config>

That XML trusts the user CA certificates installed on the device, if saved as an XML resources like network_security_config.xml and referenced with android:networkSecurityConfig="@xml/network_security_config" in the <application> element of your app manifest.

With that and a certificate in your user store, you're done! But only if you're trying to intercept Chrome or your own apps...

Intercepting HTTPS from a 3rd party app on non-rooted devices

One last case then: what if you have a non-rooted phone, and you want to intercept HTTPS from an app that doesn't trust the user CA store, and which you can't easily edit the source code for yourself? For example, if you want to inspect HTTPS traffic from somebody else's app, or from your own existing production builds?

This is a tricky case on Android nowadays, but still it's often possible. You'll still need to edit the app, but it turns out you can do so without directly rebuilding from source.

First, you need to download the APK for the app. APKs aren't available for direct download from Google Play, but they are often available in various other alternate sites, with ApkPure.com being the most well known. You can search for most Google Play apps there to download an APK and get started.

Once you have an APK, you need to:

Decode & unpack the contents.
Patch the network config within to trust the user certificate store.
Repack an APK from that patched contents.
Sign the APK with a valid certificate so it can be installed on the device.

That can be quite complicated, but fortunately there's a tool called apk-mitm that can do all of this for you! In addition, it strips pinned certificates and can even automatically patch apps using the newer Android App Bundle format.

If you have Node.js (10+) & Java (8+) installed, installing and using this just requires:

$ npx apk-mitm ./downloaded-app.apk

That should complete the above steps and give you a patched APK. If you've already trusted your CA certificate in your device's user store, as in the previous section, just install the patched app with adb install ./patched-app.apk and you're away.

Hopefully that's a good intro into managing HTTPS trust on Android, and using & abusing it to intercept, inspect and rewrite HTTPS traffic.

Want to see this in action and see exactly what HTTPS your apps and device are sending? Give HTTP Toolkit a go now.

Want to know more about how this all works? HTTP Toolkit is 100% open-source, so feel free to check out HTTP Toolkit on GitHub, and do get in touch or comment below if you have any questions or feedback.

Originally published on the HTTP Toolkit blog

Migrating a JS project from Travis to GitHub Actions

Tim Perry — Tue, 27 Oct 2020 11:45:00 +0000

Travis has been the most popular place to build open-source code for a long time, but the world is moving on. GitHub Actions is modern, tightly integrated with the most popular code hosting platform in the world, flexible, fast, and free (for public repos).

Travis has been popular for years though, there's still a lot of projects being built there, including many of HTTP Toolkit's own repos.

Last week, I decided to bite the bullet, and start migrating. Travis was having a particularly bad build backlog day, and HTTP Toolkit is entirely open source on GitHub already, so it's super convenient. I've been looking longingly at GitHub Actions builds on other projects for a little while, and I'd already seen lots of useful extensions in the marketplace of drop-in action steps that'd make my life much easier.

Unfortunately, I knew very little about GitHub actions, and I already had some Travis configuration that worked. In this post, I want to share how I converted my JavaScript (well, TypeScript) build from Travis to GitHub, so you can do the same.

The Goal

I decided to start with the simplest Travis setup I had: the HTTP Toolkit UI repo.

Here's the previous travis.yml file:

dist: xenial
sudo: required
language: node_js
node_js:
    - '14'
install:
    - npm ci
services:
    - xvfb
before_script:
    - sudo chown root /opt/google/chrome/chrome-sandbox
    - sudo chmod 4755 /opt/google/chrome/chrome-sandbox
script:
    - npm test
addons:
    chrome: stable

There's a few notable things here:

I want to build with a specific node version.
I need Chrome & XVFB installed for testing with Puppeteer & Karma.
There's some existing workarounds (before_script) for Travis.yml in here.
The build itself is just npm ci to install dependencies and then npm test.
Although not shown here, some of the npm dependencies include native node extensions, and need a working native build environment.

One other feature I'd really like, and which I'd strongly recommend for everybody, is the option to run an equivalent CI environment locally.

Yes, you can install and run tests on my machine normally, but especially with more complicated builds you'll quickly discover that that isn't a perfect match for the cloud build environment, and you'll occasionally hit remote failures that don't reproduce in your own environment. Slightly different versions of Chrome or Node, leftover git-ignored files and build output, and other environment-specific details can cause havoc.

Being able to quickly reproduce the exact cloud build environment locally makes debugging those issues much less frustrating!

Getting Started

We'll start with GitHub's JavaScript action getting started guide.

That summarizes the options available, and with a little wrangling that quickly gets us to a basic workflow (which I've saved as .github/workflows/ci.yml) matching the essential steps of the Travis config:

name: CI
on: push
jobs:
  build:
    name: Build & test
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v2

      # Install Node 14
      - uses: actions/setup-node@v1
        with:
          node-version: 14

      # Install & build & test:
      - run: npm ci
      - run: npm test

Very clear and easy: every time code is pushed, check it out and use node 14 to install dependencies & run the tests.

Note that I've skipped the Chrome & XVFB steps here entirely - we don't need them. The GitHub base image (ubuntu-latest) includes Chrome set up for testing and a enough of a native build environment that you can immediately install native modules and get going. Great! You can see the full standard list of what's available in each image here: https://docs.github.com/en/free-pro-team@latest/actions/reference/specifications-for-github-hosted-runners#supported-software.

You may find there's one small code change required though: you need to pass no-sandbox as an option to Chrome, if you're not already using it. This ensures Chrome runs happily in containerized environments like this (I think the chrome-sandbox steps in the Travis config were actually old workarounds for this on Travis).

In my Karma config, using headless Chrome, that looks like this:

browsers: ['ChromeHeadlessNoSandbox'],
customLaunchers: {
    ChromeHeadlessNoSandbox: {
        base: 'ChromeHeadless',
        flags: ['--no-sandbox']
    }
}

For Puppeteer, my browser launch code looks like this:

puppeteer.launch({
    headless: true,
    args: ['--no-sandbox']
}),

Very easy. A quick git push and you'll see your job start running on GitHub's cloud runners straight away.

But we also wanted reproducible local builds too...

Build Like a Local

Being able to locally reproduce your CI builds is essential for a healthy CI workflow, and with GitHub Actions it's already very easy.

To run builds locally, we can use act. GitHub Actions is built on Docker, starting images specified and injecting configuration into containers to run your build. Act does the exact same thing: parsing your workflow and automating Docker on your local machine to build in the exact same way.

To try this out:

Install Docker, if you don't have it already
Install act
Run act

That will automatically find .github/workflows/*.yml files in your current directory, and attempt to run them. Unfortunately, in my project that doesn't work so well:

| > registry-js@1.12.0 install /github/workspace/node_modules/registry-js
| > prebuild-install || node-gyp rebuild
| 
| prebuild-install WARN install No prebuilt binaries found (target=14.14.0 runtime=node arch=x64 libc= platform=linux)
| gyp ERR! find Python 
| gyp ERR! find Python Python is not set from command line or npm configuration
| gyp ERR! find Python Python is not set from environment variable PYTHON
| gyp ERR! find Python checking if "python" can be used
| gyp ERR! find Python - "python" is not in PATH or produced an error
| gyp ERR! find Python checking if "python2" can be used
| gyp ERR! find Python - "python2" is not in PATH or produced an error
| gyp ERR! find Python checking if "python3" can be used
| gyp ERR! find Python - "python3" is not in PATH or produced an error
| gyp ERR! find Python 
| gyp ERR! find Python **********************************************************
| gyp ERR! find Python You need to install the latest version of Python.
| gyp ERR! find Python Node-gyp should be able to find and use Python. If not,
| gyp ERR! find Python you can try one of the following options:
| gyp ERR! find Python - Use the switch --python="/path/to/pythonexecutable"
| gyp ERR! find Python (accepted by both node-gyp and npm)
| gyp ERR! find Python - Set the environment variable PYTHON
| gyp ERR! find Python - Set the npm configuration variable python:
| gyp ERR! find Python npm config set python "/path/to/pythonexecutable"
| gyp ERR! find Python For more information consult the documentation at:
| gyp ERR! find Python https://github.com/nodejs/node-gyp#installation
| gyp ERR! find Python **********************************************************
| gyp ERR! find Python 
| gyp ERR! configure error 
| gyp ERR! stack Error: Could not find any Python installation to use
| gyp ERR! stack at PythonFinder.fail (/opt/hostedtoolcache/node/14.14.0/x64/lib/node_modules/npm/node_modules/node-gyp/lib/find-python.js:307:47)
| gyp ERR! stack at PythonFinder.runChecks (/opt/hostedtoolcache/node/14.14.0/x64/lib/node_modules/npm/node_modules/node-gyp/lib/find-python.js:136:21)
| gyp ERR! stack at PythonFinder.<anonymous> (/opt/hostedtoolcache/node/14.14.0/x64/lib/node_modules/npm/node_modules/node-gyp/lib/find-python.js:179:16)
| gyp ERR! stack at PythonFinder.execFileCallback (/opt/hostedtoolcache/node/14.14.0/x64/lib/node_modules/npm/node_modules/node-gyp/lib/find-python.js:271:16)
| gyp ERR! stack at exithandler (child_process.js:315:5)
| gyp ERR! stack at ChildProcess.errorhandler (child_process.js:327:5)
| gyp ERR! stack at ChildProcess.emit (events.js:315:20)
| gyp ERR! stack at Process.ChildProcess._handle.onexit (internal/child_process.js:275:12)
| gyp ERR! stack at onErrorNT (internal/child_process.js:465:16)
| gyp ERR! stack at processTicksAndRejections (internal/process/task_queues.js:80:21)
| gyp ERR! System Linux 4.15.0-121-generic
| gyp ERR! command "/opt/hostedtoolcache/node/14.14.0/x64/bin/node" "/opt/hostedtoolcache/node/14.14.0/x64/lib/node_modules/npm/node_modules/node-gyp/bin/node-gyp.js" "rebuild"
| gyp ERR! cwd /github/workspace/node_modules/registry-js
| gyp ERR! node -v v14.14.0
| gyp ERR! node-gyp -v v5.1.0
| gyp ERR! not ok 
| npm ERR! code ELIFECYCLE
| npm ERR! errno 1
| npm ERR! registry-js@1.12.0 install: `prebuild-install || node-gyp rebuild`
| npm ERR! Exit status 1

Whilst act runs build steps just like GitHub Actions does, it doesn't use the exact same base image (in part because the same image naively built locally would be 50GB!). There's a few options:

If you're only using basic features (normal node modules, and running node scripts), act will work out of the box and you're all good.
You can use act's own full-fat image, which includes all the standard GitHub tools in a somewhat smaller image size. This is opt-in, because it's still an up-front 6GB download (and then 18GB locally, once it's uncompressed) but it'll immediately give you everything you need from the GitHub Actions cloud environment.

To use this, you just need to map ubuntu-latest (the GitHub base runner) to the published image, with:
```
act -P ubuntu-latest=nektos/act-environments-ubuntu:18.04
```
If you're familiar with Docker, you can build your own base image including just the extra tools you need. This gives you a convenient matching environment (within the selected subset of tools) with none of the disk space & download hassle.

This is what I've done for HTTP Toolkit. The dockerfile directly runs the setup scripts from the act base image repo (in turn generated from GitHub's own setup scripts), but only runs the ones I care about: build-essentials (for native builds) and Chrome. That shrinks it down to a mere 300MB download, and below 1GB on disk.

You can do this for yourself, customizing your own image, or if you need the exact same customizations you can use the HTTP Toolkit image with:
```
act -P ubuntu-latest=httptoolkit/act-build-base
```
It is possible with this approach that your base image could diverge in behaviour from the GitHub runner. You're using the same scripts, for the scripts you include, but if you skip running a script that would affect your build then you could see differences here. To guarantee reproducibility, you can fix this by setting container: httptoolkit/act-build-base (for the HTTP Toolkit image) in the job in your GitHub workflow, thereby ensuring you use the exact same image in both places.

If you do need either of these non-default base image options, you don't have to specify the -P argument every time. You can create an .actrc file in the root of your project that sets your default arguments (HTTP Toolkit UI's is here).

With that done, we can reproduce remote GitHub Actions builds locally any time with just a quick act!

Going Further

That should give you enough to get most simple JavaScript or Node projects set up with GitHub Actions, locally and remotely. If you need a full example, feel free to take a look at the HTTP Toolkit UI repo. For me, this has dramatically sped up builds & CI feedback, mainly by them starting much faster, but also seeming to knock about 10% off the runtime itself.

Now the real fun begins though, as you can begin to extend this setup. Some more bonus steps you might want to consider:

Set up caching, to speed up slow npm install steps, with actions/cache. GitHub even have a ready-to-use example for npm.
Store build artifacts, as output attached to the workflow, using actions/upload-artifact.
Create GitHub releases from content automatically, with actions/create-release.
Deploy generated content to GitHub Pages, with peaceiris/actions-gh-pages.

Add a badge to your readme, with a sprinkle of markdown:

[![Build Status](https://github.com/$USER/$REPO/workflows/$WORKFLOW/badge.svg)](https://github.com/$USER/$REPO/actions)

Have further questions or suggestions? Get in touch on Twitter or add a comment below.

Originally posted on the HTTP Toolkit blog

How to Debug CORS Errors

Tim Perry — Wed, 07 Oct 2020 14:30:00 +0000

Your request is hitting an error due to CORS. Not all is lost! Most CORS errors are quick & easy to debug and fix, once you understand the basics. Let's sort it out.

You know you're hitting a CORS error when you see error messages like:

Access to fetch at 'https://example.com' from origin 'http://localhost:8000' has been blocked by CORS policy.

No 'Access-Control-Allow-Origin' header is present on the requested resource

Cross-Origin Request Blocked: The Same Origin Policy disallows reading the remote resource at https://example.com/

Response to preflight request doesn't pass access control check

The value of the 'Access-Control-Allow-Origin' header in the response must not be the wildcard '*' when the request's credentials mode is 'include'

Method PUT is not allowed by Access-Control-Allow-Methods in preflight response.

Request header field custom is not allowed by Access-Control-Allow-Headers in preflight response.

In each of these cases, you've asked JavaScript running in your page to send a request to a different origin, and at some stage the browser is refusing to do what you want.

What is CORS?

When you include JavaScript in a web page, you're running code on your user's computer, inside their browsing session.

That's a lot of power, and browsers are designed to protect users from the risks of this. CORS is one of these protections, aiming to protect the user and the services they use from two main attacks:

CORS stops you from using the user's existing login session (their cookies and other cached authentication details) when communicating with other servers. JavaScript on your web page shouldn't be able to send requests to the Facebook API using their existing Facebook session. Without CORS, any web page could talk to other servers as you.
CORS stops you from talking to servers that might only be accessible from their machine, but which aren't accessible publicly. Your web page should not be able to send requests to my-intranet-server.local, which might be an internal company server or your home router, and it should not be able to talk to servers that are listening only for localhost requests. Servers like these are often unauthenticated and very trusting, because they aren't connected to the public internet. Without CORS, any web page you visit could access them.

This only applies to cross origin requests, e.g. requests from https://example.com to https://google.com. The protocol, domain, and port all count as part of a URL's origin, but the path does not, so https://example.com/abc and https://example.com/def have the same origin, but http://localhost:123 and http://localhost:456 do not.

CORS protects against the above attacks by requiring the target server to opt into receiving dangerous requests from the source server, and to opt in to allowing pages from other origins to read responses. The Facebook API and your local network servers can accept requests from web pages running on other origins if they want to, but only if they agree.

Why doesn't my CORS work?

Your CORS request is failing because you're sending a request that the target server hasn't agreed to allow.

There's two classes of CORS request:

'Simple' cross-origin requests. There are basic requests that use no unsafe headers, don't stream requests or responses, and only use HEAD, GET or POST methods (with limited safe content types). Any request that's possible here would also be possible by e.g. loading an image or posting a form to the cross-origin request (and we can't stop those, for huge backwards compatibility reasons).

You can always send simple requests, but you might not be allowed to read the response.
'Preflighted' cross-origin requests. These are more complex requests, that aren't easy to send in other ways. A 'preflight' request will be sent to ask the server for permission before sending any of these requests, and if it's rejected, you won't be able to send the request at all.

If the preflight request is successful, the real request is sent, and the final response to that still has to follow the same rules as a 'simple' response for you to be allowed to read it.

When a request is preflighted, before sending the real request the browser sends an OPTIONS request with headers explaining the real request that it wants to send. It expects a response including headers that explicitly allow the real request.

There's three ways that this might hit an error:

You're sending a simple request, which is sent immediately, but the headers on the response don't allow you to read it.
You're sending a preflighted request, and the headers on the preflight response don't allow you to send the real request.
You're sending a preflighted request, the preflight went OK and the request was sent, but the headers on the final response for the real request don't allow you to read it.

The browser error message should show you which is happening for you. You can know if your request is being preflighted by looking for an OPTIONS request that's sent immediately before it.

The rules for the final (after preflight, if applicable) response are:

The response must include a Access-Control-Allow-Origin header, whose value either matches the page's origin or is *. The page's origin is sent in the request in an Origin header.
If the request included credentials (e.g. fetch(url, { credentials: 'include' })) then the response headers must include Access-Control-Allow-Credentials: true, and the Access-Control-Allow-Origin header must match exactly (i.e. * is not allowed).

If the response doesn't follow those rules, then the server hasn't opted in to your request, and you won't be allowed to read the response.

If you're in cases 1 or 3, you must be breaking one of these rules.

The rules for the preflight request are:

The preflight response must include a Access-Control-Allow-Origin header, whose value either matches the page's origin or is *. The page's origin is sent in the preflight request in an Origin header.
If the page wants to send custom headers, then it will include Access-Control-Request-Headers listing the headers in the preflight OPTIONS request, and the server must include a Access-Control-Allow-Headers header that includes all those headers in the response. * can also be used here, but it won't match an Authorization header - that must always be listed explicitly.
If the page wants to use a non-simple HTTP method, it will include Access-Control-Request-Method in the preflight OPTIONS request, and the server must include a Access-Control-Allow-Methods header that includes that method in the response.
If the page wants to send credentials (e.g. fetch(url, { credentials: 'include' })) the response must include a Access-Control-Allow-Credentials: true header, and the Access-Control-Allow-Origin header must match exactly (i.e. * is not allowed).

If your preflight OPTIONS response doesn't follow these rules, then you won't be allowed to send the real request at all.

If you're in case 2, you must be breaking one of these rules.

It's also possible that you're in case 2, but you actually don't want to read the response - you just want to send it. To do that, you'll need to simplify your request such that it's a simple request. You can use { mode: 'no-cors' } on your fetch options to enforce this (but note that this doesn't change the rules, it just enforces that it's a simple request where you can't read the result).

How can I fix my CORS error?

To know exactly why your request is failing, you need to inspect the traffic itself, find where you're breaking the rules above, and then either:

Change the request to make it a simple request
Change the server's response to follow the rules above
If all else fails, proxy the request through your own server on your own origin, so it's not a cross-origin request (proxying avoids the attacks above, because it doesn't let you use the cookies or authentication details from the user's browser, and it requires the target server to be accessible from your source server)

To inspect the traffic, you can use your browser built-in tools, but it's usually easier to use a dedicated HTTP debugger like HTTP Toolkit. Dedicated tools make it much easier to see the data, rather than (for example) Chrome's very cramped and fiddly network tab, and you can also breakpoint responses and edit the headers to test how the browser will handle changes without actually changing your server. Also, some Chrome versions don't show all CORS requests.

Hopefully, once you examine your CORS requests & responses, it's clear where you're breaking the rules above.

If not, try walking through Will It CORS. This is a self-explaining implementation of the CORS rules: you can input step by step what you're trying to do, and it'll tell you what will happen and why, and how you can change it.

There's also a few common mistakes that you should watch out for:

Trying to request content from another origin that isn't explicitly available cross-origin. If its not your server, and it doesn't actively want CORS requests, you're not going to work around most issues: you need to proxy the request, ask the owner to allow it, or do something entirely different.
Always returning * for Access-Control-Allow-Origin, and then trying to send credentials.
Adding CORS headers for preflight OPTIONS requests, but forgetting to also include CORS headers on the final request too.
Unnecessarily sending custom request headers. This will trigger a preflight request. You can often get by just using the CORS-safe request headers instead, or moving request data into the body of your request.
Incorrectnyl caching CORS response headers independent of their origin, by not using Vary: Origin. If you do this then responses for requests from one origin may be cached and returned for later requests from a different origin. That mismatched data can quickly break things.
Trying to access response headers without including an Access-Control-Expose-Headers header. In this case, all headers except the CORS-safe response headers will be unexpectedly undefined, even though they were sent by the server.
Sending cross-origin mixed-content requests (a request from https://... to http://...). These will always be blocked, regardless of the details, as insecure content like this is never allowed on HTTPS origins. There's not much you can do about this, other than changing to use HTTPS on both servers.

That covers the core of CORS, how it can go wrong, and how to fix it. Have more questions? Comment below, or get in touch on Twitter.

Originally posted on the HTTP Toolkit blog

GraphQL the Simple Way, or: Don't Use Apollo

Tim Perry — Wed, 02 Sep 2020 12:00:00 +0000

The fundamentals of GraphQL are remarkably simple. Nonetheless, a busy hype train & rocket-speed ecosystem means that building a GraphQL API in the real world can be a tricky balancing act of piling complex interacting components a mile high, none of which anybody fully understands.

About 90% of this pile is built & heavily promoted by a VC-funded company called Apollo. Apollo describe themselves as a "data graph platform" who've built the self-described "industry-standard GraphQL implementation".

Unfortunately, while I'm sure their platform is great, if you're setting up a fresh GraphQL API you should not start with Apollo. It certainly might be useful later, but on day 1 it's a trap, and you'll make your life simpler and easier if you avoid it entirely.

Let's talk about why that is, what can go wrong, and what you should do instead.

The Problem with Apollo

In practice, "industry-standard GraphQL implementation" means 169 separate npm packages, including:

44 different server packages and apollo-server-* subpackages.
7 different GraphQL 'transport layers', plus a long list of link layer extensions that build on top of this.
8 code generation packages
5 different create-* project setup packages
3 different GraphQL Babel plugins (plus a Relay un-Babel plugin, so you can avoid using Babel for some specific cases).
Much much more...

The Apollo packages required to install the base apollo-server package suggested in their Getting Started guide include the "Apollo Studio" (née Apollo Graph Manager, néeée Apollo Engine) reporting engine, which integrates your server with their cloud service, plus extra protobuf definitions on top of that to reporting to the cloud service with Protobuf. It includes request tracing for their own custom tracing format, multiple different custom caching packages, an abstraction layer that powers the many available transport link layers, an abstraction layer for connecting external data sources...

In total, installing apollo-server installs actually installs 33 direct dependencies, and 179 packages in total, pulling in about 35MB of JavaScript.

Once all put together, by itself this package creates web servers that can't do anything.

If however you also use the (official, non-Apollo) graphql package too, then you can now just about answer complex queries like:

{
  books {
    title
    author
  }
}

To be clear, that graphql package is the official GraphQL JS implementation, which takes a schema, a query, and a resolver (in effect, a data set object), and gives you a result. I.e. it does all the GraphQL heavy lifting required to process a query like this, except the HTTP.

I know I'm treating Apollo very harshly here, and that's not wholly fair. Most of their published packages do have cases where they're genuinely useful, many of the packages I'm counting are deprecated or duplicates (though published and often still well used), and as far as I'm aware everything they've released works perfectly effectively. I certainly don't think that nobody should use Apollo!

I do 100% think that Apollo shouldn't be anybody's GraphQL starting point though, and that nonetheless it's marketed as such.

They clearly want to be the entrance to the ecosystem, and of course they do! Ensuring a nascent fast-growing ecosystem depends on your free tools is a great startup play. They're the first result for "how to set up a graphql server", and either they or GraphQL-Yoga (another package on top of Apollo Server) are the suggested beginner option in most other articles on that page, from Digital Ocean's docs to howtographql.com. This isn't healthy.

If you're setting up a GraphQL API, you don't need all this. Pluggable transport layers and data sources, request tracing, cool GraphQL extensions like @defer and multi-layered caching strategies all have their place, but you don't want that complexity to start with, and they're not a requirement to make your simple API 'production ready'.

It is great that Apollo makes these available to you, but they're features you can add in later, even if you don't start off using Apollo at all. A GraphQL schema is pretty standard, and entirely portable from the standard tools to Apollo (but not always back, if you start using Apollo's own extensions...).

If I seem personally annoyed by this, it's because I am! I was burned by this myself.

HTTP Toolkit's internal APIs (e.g. for defining HTTP mocking rules and querying traffic) use GraphQL throughout. Those APIs started off built on Apollo, because that's the very widely recommended & documented option. The overly complex setup required to do so caused a long stream of serious pain:

Apollo's packages like to move fast and break things, and each often requires specific conflicting graphql peer dependencies, making updates remarkably painful all round.
The base packages include a lot of features and subdependencies, as above, which in turn means a lot of vulnerability reports. Even if vulnerabilities aren't relevant or exploitable, downstream users of my packages very reasonably don't want security warnings, making keeping everything up to date obligatory.
Some Apollo packages are effectively quietly unmaintained, meaning that conflicting dependencies there can block you from upgrading entirely, unless you fork the whole package yourself.
Once you start having multiple interacting packages in your system that use Apollo this gets even worse, as dependent packages need updating in lockstep, or your peer dependency interactions explode, scattering debris for miles.
The packages involved are huge: apollo-server alone installs 35MB of JS, before you even start doing anything (that's v2, which is 2.3x the size of the last Apollo Server v1 release, but hey the upgrade is unavoidable anyway, so who's counting?).
These problems are getting worse. apollo-server v3 is coming soon, with built-in support for GraphQL federation, non-Node backend platforms, and a new plugin API. Don't get me wrong, these features are very cool, but you don't need them all included by default in your starter project!

It's not fun. However there is an alternative:

How to Build a Simple GraphQL Server (without Apollo)

To build a GraphQL API server, you really need just 3 things:

A web server
An executable GraphQL schema (i.e. a schema and a resolver) which can together can answer GraphQL queries
A request handler that can accept GraphQL requests, hand them to the schema, and return the results or errors

I'm assuming you already have a preferred web server (if not, Express is an easy, convenient & reliable choice). The official graphql package can turn a string schema and a resolver object into an executable schema for you.

That leaves the final step, which is easily handled with express-graphql: a simple Express middleware, with just 4 dependencies that handle content negotiation & body parsing. That works for Express or Connect, and there's similar tiny packages available for most other servers.

To set up your GraphQL server, install those packages:

npm install express graphql express-graphql

And then set up a server that uses them:

const express = require('express');
const { graphqlHTTP } = require('express-graphql');
const { buildSchema } = require('graphql');

// Create a server:
const app = express();

// Create a schema and a root resolver:
const schema = buildSchema(`
    type Book {
        title: String!
        author: String!
    }

    type Query {
        books: [Book]
    }
`);

const rootValue = {
    books: [
        {
            title: "The Name of the Wind",
            author: "Patrick Rothfuss",
        },
        {
            title: "The Wise Man's Fear",
            author: "Patrick Rothfuss",
        }
    ]
};

// Use those to handle incoming requests:
app.use(graphqlHTTP({
    schema,
    rootValue
}));

// Start the server:
app.listen(8080, () => console.log("Server started on port 8080"));

Run that and you're done. This is solid, reliable, fast, and good enough for most initial use cases. It's also short, clear, and comparatively tiny: node_modules here is just over 15% of the size of the Apollo equivalent. Running 80% less code is a very good thing.

In addition, you can still add in extra features incrementally later on, to add complexity & power only where you need it.

For example, in my case, I want subscriptions. Mockttp (the internals of HTTP Toolkit's proxy) accepts GraphQL queries over websockets, so it can stream intercepted request details to clients as they come in, with a GraphQL schema like this:

type Subscription {
    requestInitiated: InitiatedRequest!
    requestReceived: Request!
    responseCompleted: Response!
    requestAborted: Request!
    failedTlsRequest: TlsRequest!
    failedClientRequest: ClientError!
}

To add this, I can just expand the basic setup above. To do so, I do actually use a couple of small Apollo modules! Most can be picked and configured independently. For this case, graphql-subscriptions provides a little bit of pubsub logic that works within resolvers, and subscriptions-transport-ws integrates that into Express to handle the websockets themselves. Super helpful

Here's a full example:

const express = require('express');
const { graphqlHTTP } = require('express-graphql');
const { buildSchema, execute, subscribe } = require('graphql');

// Pull in some specific Apollo packages:
const { PubSub } = require('graphql-subscriptions');
const { SubscriptionServer } = require('subscriptions-transport-ws');

// Create a server:
const app = express();

// Create a schema and a root resolver:
const schema = buildSchema(`
    type Book {
        title: String!
        author: String!
    }

    type Query {
        books: [Book]
    }

    type Subscription { # New: subscribe to all the latest books!
        newBooks: Book!
    }
`);

const pubsub = new PubSub();
const rootValue = {
    books: [
        {
            title: "The Name of the Wind",
            author: "Patrick Rothfuss",
        },
        {
            title: "The Wise Man's Fear",
            author: "Patrick Rothfuss",
        }
    ],
    newBooks: () => pubsub.asyncIterator("BOOKS_TOPIC")
};

// Handle incoming HTTP requests as before:
app.use(graphqlHTTP({
    schema,
    rootValue
}));

// Start the server:
const server = app.listen(8080, () => console.log("Server started on port 8080"));

// Handle incoming websocket subscriptions too:
SubscriptionServer.create({ schema, rootValue, execute, subscribe }, {
    server // Listens for 'upgrade' websocket events on the raw server
});

// ...some time later, push updates to subscribers:
pubsub.publish("BOOKS_TOPIC", {
    title: 'The Doors of Stone',
    author: 'Patrick Rothfuss',
});

My point isn't that you need subscriptions in your app, or that everybody should use all these extra packages (quite the opposite).

This does demonstrate how you can extend your setup to progressively use these kinds of features though. Moving from request/response model to also supporting subscriptions is not a trivial change, but even in this case, adding in Apollo extensions is a few simple lines on top of the existing logic here that fits nicely into a standard setup.

You can also extend with non-Apollo tools too. Here we're building primarily around the vanilla GraphQL packages and Express directly, composing Apollo components in separately, rather than basing everything on top of them. That means you could still drop in any other Express middleware or GraphQL tools you like, to add any kind of authentication, caching, logging or other cross-cutting features just using standard non-GraphQL solutions & examples, with no lock-in from the Apollo ecosystem.

Apollo do have a wide selection of interesting & useful packages, and they should be lauded for the effort and contributions they've made to the ecosystem. At the same time though, they're not a neutral actor. Don't assume that the Next Big Thing is the right choice for your project, especially if it calls itself "industry-standard".

Instead, start simple: build a system that you can fully understand & manage, avoid unnecessary complexity, and keep your project lean & flexible for as long as you can.

Using GraphQL, and want to debug, rewrite & mock live traffic? Try out HTTP Toolkit. One-click HTTP(S) interception & debugging for browsers, servers, Android & more.

Inspecting Android HTTP with a fake VPN

Tim Perry — Tue, 25 Aug 2020 12:20:00 +0000

Can you build an Android app that can inspect & rewrite the network traffic from every other app on the device?

In turns out that, yes, you can. HTTP Toolkit does exactly this, by building an app on top of the Android VPN APIs that fully simulates a fake VPN connection entirely within the device.

Here I want to talk through how that works, look at the code that makes it happen, and show you how you can do the same thing for yourself.

To be clear, this is not intended (or very effective) as a attack on the security of traffic from the device. When you actually do this Android provides clear warnings & permission prompts to the user during setup, and requires persistent UI notifications any time this is active. In addition this doesn't give you any way to read the contents of encrypted traffic, by default (in the next post, we'll talk about how HTTP Toolkit can do that).

There are some interesting & constructive use cases this opens up though for developer tooling. For example:

Inspecting & rewriting mobile traffic for testing & debugging (this is HTTP Toolkit's raison d'être).
Building a firewall for Android that blocks outgoing app connections according to your custom rules.
Recording metrics on the traffic sent & received by your device.
Simulating connection issues by adding delays or randomly injecting packet resets.

How do Android VPNs work?

The Android developer docs have a VPN guide, which is a good starting point.

These VPN APIs allow you to register a service in your app, which when activated is given a file descriptor that backs a network tunnel interface.

That tunnel interface is then used by the whole device for all network traffic. In addition, your VPN service is given the power to create protected sockets that don't use this tunnel, so the VPN app can communicate with the network without going through itself.

Once this is activated, when an app sends some data, instead of that going out to the network, each IP packet is buffered behind this file descriptor. When you read from it you're given raw network bytes directly, and when you write bytes to it they're treated as bytes received directly from the network interface.

This is designed to allow implementing a VPN connection in your app. In that case, your app would forward all the read bytes directly to a VPN provider over some protected separate connection, without any substantial processing of them on the device. The VPN provider would then forward that data on as part of the VPN's traffic, forward response packets back to your app over your connection, and you'd write the resulting packets back to the file descriptor.

That's what's this is primarily designed for, but that doesn't mean that that's all we can do with it.

When is a VPN not a VPN?

Once we have a VPN service running, our app will receive every network byte the device sends, and has the power to inject raw bytes back.

Things get interesting if rather than forwarding these bytes to a VPN provider, we examine them, and then simply put them straight back on the real network. In that case, we get to see every network byte, but we don't interfere with the network connection of the device, and we don't need an externally hosted VPN provider to do it.

Unfortunately that's easier said than done. Our file descriptor works with raw IP data, but Android doesn't actually have an API for us to send raw IP data anywhere else. Instead, we have higher level APIs for TCP and UDP, and the IP part is always done invisibly under the hood.

If we want to proxy these bytes, we need to match these two APIs up. We need to:

When we read an IP packet from our tunnel:
- Parse the raw packet bytes into an IP packet.
- Parse the TCP/UDP packet within and extract its content.
- (For TCP) Track the connection state of the overall TCP connection, and ack/fin/etc each packet in the session appropriately.
- Send the equivalent TCP/UDP content upstream, using Android's TCP/UDP APIs.
When we receive a TCP/UDP response from upstream:
- (For TCP) Match that to the tunnelled TCP connection.
- Build our own complete TCP/UDP + IP packet data around the received data.
- Write the resulting bytes back into the tunnel.
- Cleanly (or messily) close connections when the upstream socket is done.

This is quite complicated. We effectively need to reimplement UDP & TCP from scratch!

Fortunately, we're not the first people to want to do this. Most of the existing implementations are unmaintained demos, but they are open-source so we can build upon them! My own solution is based on a GitHub proof-of-concept called ToyShark (a pun on Wireshark, I assume) which was in turn based on some of the open-source network collection internals of an old AT&T project called Application Resource Optimizer (source).

The resulting HTTP Toolkit Android app implements all the above. This is 100% free & open-source (github.com/httptoolkit/httptoolkit-android) so similar open-source implementations in future can build upon it in turn.

This implementation acts as a VPN, while proxying all traffic back onto the real network, all without native code, just powered by Java NIO.

The core VPN implementation is in src/main/java/tech/httptoolkit/android/vpn, and there's a README there with an outline of the implementation details. We'll explore this a little more below, as we look at ways to extend it.

There is a performance penalty to all this of course, in both network bandwidth and latency. The impact isn't really noticeable in normal usage on any modern device though. On cellular connections it's usually dwarfed by the underlying connection performance, and even on wifi you can reach quite acceptable numbers:

This could probably be improved further by rewriting the Java code as a native code, but that entails significant extra complexity. For the HTTP Toolkit use case (targeted debugging, rather than heavy everyday usage) it's not worth it.

With that in place, we now transparently receive every network packet from the device. We can inspect it as we'd like, and even edit that traffic, through either the raw IP stream or the parsed TCP/UDP packet data. To what end?

How can we use this?

In HTTP Toolkit's case, the usage of this is very direct: we forcibly redirect all HTTP(S) traffic via the debugging proxy (which is running on your local development machine). That proxy then lets you inspect and rewrite all the traffic there as you see fit.

There's a demo video on HTTP Toolkit's Android page if you want to see this in action.

To do this, we check the target port of outgoing TCP connections, and rewrite the address if it's one of our configured HTTP ports (e.g. 80, 443, ...), by just adding the following lines into TCP session setup:

public Session createNewTCPSession(int ip, int port, int srcIp, int srcPort)
        throws IOException {
    // ...

    String ips = PacketUtil.intToIPAddress(ip);

    // Use the given target address, unless tcpPortRedirection has specified
    // a different target address for traffic on this port:
    SocketAddress socketAddress = tcpPortRedirection.get(port) != null
        ? tcpPortRedirection.get(port)
        : new InetSocketAddress(ips, port);

    channel.connect(socketAddress);

    // ...
}

That forcibly sends all the matching traffic to the proxy, and immediately gives us full visibility into HTTP traffic. On Android 10 we also set a VPN proxy configuration, which catches most traffic on further ports that aren't explicitly matched (although in that case it's advisory default configuration, rather than enforced redirection).

That's enough to redirect traffic for remote inspection & rewriting. How else could you extend this? Let's talk about the 3 other use cases I mentioned at the start:

Blocking outgoing connections

To block outgoing connections to specific addresses or on specific ports, you just need to throw away the packets after you receive them from the VPN interface, once you've parsed them to work out where they're going.

You can use this to block specific hosts you don't like, block DNS requests for certain addresses to build an on-device Pi-Hole, or allow traffic only to a short trusted list of hosts to lock down your networking entirely.

In HTTP Toolkit's implementation, SessionHandler's handlePacket is where we handle the raw packet data that the device wants to send. It looks like this:

public void handlePacket(@NonNull ByteBuffer stream)
        throws PacketHeaderException, IOException {
    final byte[] rawPacket = new byte[stream.limit()];
    stream.get(rawPacket, 0, stream.limit());
    stream.rewind();

    final IPv4Header ipHeader = IPPacketFactory.createIPv4Header(stream);

    // TODO: inspect ipHeader here, and 'return' to drop the packet

    if (ipHeader.getProtocol() == 6) {
        handleTCPPacket(stream, ipHeader);
    } else if (ipHeader.getProtocol() == 17) {
        handleUDPPacket(stream, ipHeader);
    } else if (ipHeader.getProtocol() == 1) {
        handleICMPPacket(stream, ipHeader);
    } else {
        Log.w(TAG, "Unsupported IP protocol: " + ipHeader.getProtocol());
    }
}

From that we can drop packets entirely based on the target address or port in the IP header, by simply doing nothing.

Dropping a packet here is literally packet loss, where the app sending the original request will never hear any response at all.

Alternatively, for more complex rules you can make changes within specific protocol handling, e.g. the handleTCPPacket or handleUDPPacket methods above. In both cases you can examine the parsed TCP/UDP packets, and drop them there (or in the TCP case, inject an immediate RST packet to tell the app the connection failed).

Recording traffic metrics

Want to know what your device sends and receives? Normally Android makes that more or less invisible. Within a fake VPN application like this though you have every network byte, so it's easy to examine and record data about outgoing & incoming packets.

It's simplest to do total byte metrics by address and/or port, but you could also build more complex analyses of packet data itself. E.g. tracking the duration of TCP sessions with certain hosts, recording metrics about the unencrypted data available, or looking at DNS UDP packets to examine which hostnames you're looking up.

For this codebase, we can easily capture outgoing traffic in the handlePacket method above. We have the raw IP packet data there, and the full TCP & UDP data is just a little more parsing away.

To track incoming traffic, we'd need to look at the code that handles the upstream connections. For example in readTCP from SocketChannelReader, where upstream TCP data is received:

private void readTCP(@NonNull Session session) {
    // ...

    try {
        do {
            len = channel.read(buffer);
            if (len > 0) {
                // We're received some TCP data from the external network:
                sendToRequester(buffer, len, session);
                buffer.clear();
            } else if (len == -1) {
                // The external network connection is finished:
                sendFin(session);
                session.setAbortingConnection(true);
            }
        } while (len > 0);
    }

    // ...
}

At this point, we're handling the contents of the TCP connection, before we pack it back up into the raw bytes for the VPN interface.

By examining the TCP data read here and associating it with the IP & port of the TCP session, you can quickly start to build a view into your device's network communication.

Simulating connection issues

It's possible to simulate connection issues on the device too. That's especially useful to test how applications handle low-quality internet connections and network errors.

Unfortunately you can't simulate all issues, as Android's APIs give us limited control of upstream traffic. We control the contents of upstream TCP & UDP packets, but not the raw network connection itself. That means, for example, we can't simulate our device sending the wrong upstream packet sequence number or corrupting a TCP checksum, but we can simulate the device receiving such packets.

There's still a lot of interesting things you can simulate with this:

Incoming or outgoing packet loss, where some packets simply disappear in transit.
Repeated or misordered packets.
Random connection resets (similar to tcpkill).
Delays to packets, in either direction.

In each case you'd normally do this probabilistically, so that 10% of connections fail, or packets are delayed by 500ms on average.

When you do this, you'll often see some surprising results and errors in your app. In effect we're doing on-device chaos engineering.

Adding random connection resets like this will usually result in very visible TCP connection failures, causing random HTTP requests, raw network sockets or whatever else to suddenly fail and disconnect.

Packet loss and ordering issues meanwhile are normally be handled at the TCP level, invisibly to your application code, but the process of doing so can result in unpredictable performance, and cause real issues at the application level.

During day-to-day development it's very easy to never see these issues, given the fast & reliable wifi in your office or at home, and simulating rural 2G issues like this can be eye-opening!

You do most of this at a very low level, just hooking into the places where individual raw IP packets are passed to and from the VPN. For TCP error simulation though, you'll need a lot more information about the TCP connection itself, to find packets to reorder, or to inject RSTs into active connections.

In the HTTP Toolkit app specifically:

ClientPacketWriter is where raw IP data is written back to the VPN (incoming IP packets). At this stage we can easily drop, corrupt or delay incoming packets at the IP level.
handlePacket again in SessionHandler, would allow us to drop, delay or otherwise react to outgoing packets.
SessionHandler also controls the TCP flow of each connection to process each packet, allowing us to hook into that flow directly. For example, you could extend replySynAck to schedule a connection reset (just a call to resetConnection) for 50% of new connections 2s after they're created.
SessionManager stores the state controlled by SessionHandler. Given the list of active connections there, we could select random active TCP sessions and kill them according to whatever criteria you like.

As we've seen, the Android VPN APIs are powerful, and there's a lot of potential here.

With a few tricks like this to hook into network traffic, there's a whole world of interesting tools you can build. Give it a go! Have any thoughts or feedback? Let me know on Twitter or in the comments below.

Want to take your Android debugging to the next level? HTTP Toolkit gives you one-click HTTP(S) inspection & mocking for any Android app (plus lots of other tools too).

How to Debug Node.js Segmentation Faults

Tim Perry — Wed, 12 Aug 2020 16:12:00 +0000

Oh no, your JavaScript code isn't just throwing an exception or crashing: it's segfaulting. What does that mean, and how can you fix it?

You'll know this happens because node will hard crash, exiting silently without any kind of real stack trace, perhaps printing just segmentation fault (core dumped).

(If you do get a normal JavaScript stack trace on the other hand, then you're dealing with a normal JS error, not a segfault. Lucky you! You might be more interested in the guide on How to Debug Anything)

What is a Segmentation Fault?

A segmentation fault occurs when a program attempts to access a memory location that it is not allowed to access, or attempts to access a memory location in a way that is not allowed (for example, attempting to write to a read-only location, or to overwrite part of the operating system).
- wikipedia.org/wiki/Segmentation_fault

In practice, a segfault occurs when your program breaks some fundamental rule set by the operating system. In that case, the operating system sends your process a signal (SIGSEGV on Mac & Linux, STATUS_ACCESS_VIOLATION on Windows), and typically the process shuts down immediately.

The rules that you can break to cause this include things like reading or writing to an invalid memory address (e.g. native code somewhere trying to use a null pointer as a memory address), causing a stack or buffer overflow, or reading or writing from memory that's not yours (maybe it was yours but it's now been released, maybe it's unused, or maybe it's owned by another process or the operating system).

All of these cases involve low-level concerns, like pointers & memory management. You shouldn't normally have to worry about this when writing JavaScript! The language runtime normally manages your memory, doesn't expose the kinds of APIs that could cause these issues, and enforces its own rules on the APIs that are available, to guarantee that your code behaves correctly.

That all ensures that the underlying operating system's rules are never broken, and ensures that any time you do accidentally try to take any invalid actions, you get a clear error that appears straight away, rather than random failures later.

Unfortunately, there are a few cases where you can still hit segfaults in Node:

When you use native addons (either directly, or because one of your dependencies uses them), so you're effectively running your own native code as part of your application. If that native code is either buggy or just incompatible with your version of Node, you'll often get segfaults.
If you manipulate parts of the internal private state of Node objects. This can break Node's assumptions, so that Node's built-in native code does the wrong thing, resulting in segfaults.
When Node.js itself has a bug somewhere, and segfaults all by itself.

How can I fix it?

Find the culprit

First, you need to work out which of the 3 cases above you have.

Native addons are always the most likely cause here. There's a couple of things to try straight away:

Rebuild all your native node modules with npm rebuild. This will recompile native code with your current version of node, and should resolve any issues where your native modules are compiled for the wrong node version.
Find all the native modules you have installed, by searching your node_modules folder for .node files. On Linux/Mac you can list them with:

find node_modules -iname "*.node"

If you have no native modules installed, you can rule that case out entirely. If you do have modules installed there that seem related to the crash you're seeing, then that's probably a good place to start looking.

You can also try to get more detail on the segmentation fault itself.

To do this, you can use the Segfault-Handler module. Just run npm install segfault-handler, and then add the below right at the start of your application code:

const SegfaultHandler = require('segfault-handler');
SegfaultHandler.registerHandler('crash.log');

That module listens for any SIGSEGV signal, and reports the detailed stack trace that caused it before the process shuts down. When you next hit your segmentation fault, you'll get something like this:

PID 30818 received SIGSEGV for address: 0x20
[...]/node_modules/segfault-handler/build/Release/segfault-handler.node(+0x3127)[0x7fdb5a5fb127]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x128a0)[0x7fdb735f58a0]
node(_ZN4node7TLSWrap6EncOutEv+0x170)[0xa09010]
node(_ZN4node7TLSWrap7DoWriteEPNS_9WriteWrapEP8uv_buf_tmP11uv_stream_s+0x2c7)[0xa0a6c7]
node(_ZN4node5http212Http2Session15SendPendingDataEv+0x4ce)[0x93b5ae]
node(_ZN4node5http212Http2Session5CloseEjb+0xda)[0x93c4fa]
node[0xb62a3f]
node(_ZN2v88internal21Builtin_HandleApiCallEiPPNS0_6ObjectEPNS0_7IsolateE+0xb9)[0xb635a9]
[0xcec6c2dbe1d]
[1] 30818 segmentation fault (core dumped) node ./bin/run start

That's the output from a segmentation fault I was hitting recently, where the new HTTP/2 debugging support in HTTP Toolkit occasionally crashed the Node process, after certain patterns of connections & disconnections.

A trace like this doesn't give you enough to fix the issue, but it does give a clear clue where the problem lies.

In my case, the SendPendingData method of an HTTP2Session is trying to write to a TLS stream as the session closes down, and that's then crashing the process. That gave me some clear info: it's an issue with HTTP/2 requests, and it's happening in node itself, not a native addon. From there, a quick search of the Node issue tracker led me to a reported bug, and eventually to a workaround.

Find a fix

From here, you should have some pointer towards the code that's buggy. If there's a suspicious native addon module involved then that's almost certainly the culprit, and you should start there.

Otherwise, if the trace is clearly pointing to Node internals (as above) and you're not messing around with those yourself, or using any relevant native addons, then you've probably found a bug in Node. Congratulations! Node should never segfault if you're writing normal JavaScript code, so something very wrong is going on.

From here, there's a few good next steps:

Update to the latest version of Node/the node module in question, and make sure the same bug still appears there.

In many cases just a quick update of the right thing will solve your issue, and if not then maintainers will be much happier to help you investigate if they know it's definitely a current issue.
Double-check your code is using the failing code as intended.

Check the documentation of the related properties and methods you're accessing, and make sure that they are indeed documented (i.e. you're not unexpectedly messing with internal state) and that you're following the instructions in that documentation correctly. It's often useful to look through the native module's test code too, to see some examples of how it's supposed to be accessed.
Report the issue to the addon maintainers/Node team.

GitHub is your friend here: use the details you've found to do a quick search on the relevant repo's issue tracker first. The Node issue tracker is available at github.com/nodejs/node/issues.

If you're lucky, you'll find an issue with more information, and maybe even an existing workaround. You can then add any extra details you have and an upvote there to help the maintainers. Of course, if not, it's time to file a bug for yourself.

Either way the best way to ensure these bugs actually get fixed is to provide a reliable way for other developers to reproduce the issue. The more information on how to do so, and the simpler the steps required, the better.
Use your segfault trace to find the relevant code, add detailed logging or use debugging tools, and very carefully walk through the code that's failing to try and find something that's not quite right.

If you're not familiar with the code in question, and you haven't written native addons for Node.js before this can be intimidating and difficult. It's worth a go though, and you don't need to understand the code perfectly to do this. In many cases you'll quickly spot a comment or clue for why this crash could occur, that'll lead you back to a nice clean fix in your own JavaScript.

Especially in native addons, you'll often find that they make certain assumptions (this method will never be called twice, this parameter will never be undefined) that aren't always checked everywhere. Any of these can easily mean that a minor bug in your code results in the addon's native code doing completely the wrong thing, and crashing the whole process.
Find a workaround: change how you're using the module in question, use a different module entirely for now, delete the broken feature from your product entirely, or quit your job and go live in the forest.

Hopefully that's enough to show where the issue is, and get the information to fix or workaround it so you can get your code back on track.

Have any other suggestions or advice for others in the same place? Write a comment below, or let me know on Twitter.

Originally posted on the HTTP Toolkit blog

Translating between HTTP/1 and HTTP/2

Tim Perry — Wed, 15 Jul 2020 14:30:00 +0000

Semantically, what changed in HTTP/2?

Multiplexed connections, binary frames, header compression - all the headline changes are syntactic and network format changes, rather than fundamental changes to the concept. As a developer building on top of this, you can often ignore the low-level syntax of network protocols like this, and just think about the meaning of each message (the semantics) rather than byte-by-byte how it's sent between computers.

Semantically though, while HTTP/2 is built on top of the ideas of HTTP/1.1, and the HTTP/2 spec is at pains to emphasize that it is not redefining HTTP's semantics, there are a few real-world semantic differences you need to use it effectively (and with HTTP/2 now in use on nearly 50% of the top 10 million webservers, you really do need to know how it works).

This matters when building anything non-trivial on HTTP/2, but especially matters if your application does need to translate between the two, e.g. as a proxy or from a cache, and it's important to understand if you want to reliably handle requests in both protocols using the same code.

The more things change, the more they stay the same

Let's start with what doesn't change.

Firstly, the core communication model is still a request from a client followed by a response from the server.

Requests have a method like GET, a URL, some headers, and optionally a body. Responses have a status code (200, 404...), their own headers, and their own body. Most HTTP methods and status codes still have the same fundamental meanings.

You can write code that takes a request, looks at the method & URL to decide what to do, and sends back a response with a status code and the data requested, and most frameworks will make that work for you automatically. For basic HTTP handling like that, you can just enable HTTP/2 in your server of choice, and you're golden.

Once you get into anything more complex though, you can run into some interesting issues. Let's dig into the differences:

Status messages are dead

In HTTP/1, every response has a status code, and a status message. In its raw format, that might look like this:

HTTP/1.1 200 OK
header-name: header-value
another-header: ...

OK is the default for 200, but it's not fixed. This is equally valid:

HTTP/1.1 200 Super great
header-name: header-value
another-header: ...

This status info is received in the client, and can be used to make error messages more informative, to differentiate nuance in status-only responses, and so on.

However, although it's useful in theory to have a space for a single-line summary of a response, this isn't used much in practice, and adds complexity, so HTTP/2 drops it entirely. It's just status codes now, and you'll need to put other data into the body or headers.

Everything's a header

HTTP/2 moves request and response metadata into so-called 'pseudo-headers'. These are headers, but prefixed with a colon, like :path (because in HTTP/1 that's an unparseable header name, so it's guaranteed to not have been used).

That means that although requests do still have methods and URLs and responses still have status codes, they're now part of the header data itself. Rather than building the URL from the request path plus the old Host header, we now have :scheme (http/https), :authority (the hostname) and :path (the URL path) headers. Rather than a standalone status code, we have a :status header.

This makes translation a bit more complicated, as a lot of pre-HTTP/2 code will choke on header names like these and the values need to be extracted for HTTP/1-compatible usage, but this allows these values to take advantage of header compression to be sent far more efficiently.

Hop-by-hop connection headers are no more

The Connection header, along with a few other related headers, describes specific details about how the direct TCP connection between the client and the server or proxy works. For example, in HTTP/1.1 a Connection: close header might be sent with a message to close the connection immediately afterwards, rather than keeping it alive ready for future requests.

The Connection header can also include a list of other header names, to declare them as only relevant for the current direct connection, i.e. to tell any proxies involved not to forward them to the target server.

Other hop-by-hop headers include Keep-Alive (which defines exactly how a connection should be kept alive), Proxy-Authenticate (authentication details for HTTP proxies) , Trailer (announces headers that will arrive after the message body in chunked messages), Upgrade (switches the protocol of the current connection), and Transfer-Encoding (describes data transfer formats, made unnecessary with HTTP/2's data framing).

In HTTP/2, connection-specific headers like Connection and Keep-Alive are prohibited. While some browsers will simply ignore them, Safari and other webkit-based browsers will outright reject any response that contains them. Easy to do by accident, and a big problem.

Other hop-by-hop headers aren't forbidden, but are not generally functional, and it's recommended that they not be used.

Cookie headers

In HTTP/1, according to RFC 6265:

When the user agent generates an HTTP request, the user agent MUST NOT attach more than one Cookie header field.

When a user agent (typically a browser) wants to send multiple cookies with a request, it sends them separated with a semicolon and space, like so:

Cookie: cookie1=a; cookie2=b

In HTTP/2, you can still use this form, but you can also send multiple cookie headers separately, breaking the rule above.

Splitting them up is generally better to do, because it allows HTTP/2 to more effectively compress cookie headers, so they're not sent repeatedly within the same connection. If you do send them separated like this then any change to them only requires sending the new changed header. If you concatenate them as with HTTP/1 then every change to any cookie requires resending all of them.

Unfortunately that means headers for most HTTP/2 requests are invalid in HTTP/1, potentially resulting in your server rejecting the request entirely, missing all but one of the values provided, or other undefined behaviour. It's easy to fix though: just join all Cookie header values with a '; ' (semicolon + space) separator.

The CONNECT method

A CONNECT request asks the server for a raw TCP tunnel to a different server. In HTTP/1 this is used almost exclusively for HTTP proxying, so that a client can make an HTTPS connection to a server through a proxy, without sharing the content of the connection with the proxy.

In HTTP/1, a proxy client makes a CONNECT request with the name of the target server, the proxy server returns a 200 response (if it accepts it), and then the entire TCP connection becomes a tunnel to the target server. Every byte sent is sent directly on to the target, and every byte received is sent back to the client.

In HTTP/2, a proxy client makes a CONNECT request with the name of the target server, the proxy returns a 200 response (if it accepts it), and then that one specific request stream becomes a tunnel to the target server. To send data, it must be wrapped up as an HTTP/2 DATA frame, with the id of the tunnelled stream, and the data within that is then forwarded on to the target server, and received data is similarly packed into DATA frames on that stream, whilst other HTTP/2 requests with the proxy server can keep using the same TCP connection independently.

In addition, setting up websockets and other HTTP protocol changes is now done using CONNECT requests, rather than the GET request + Upgrade header + 101 response that was used with HTTP/1. Like CONNECT tunnelling, websockets also now work over only a single stream within the HTTP/2 connection, rather than taking over the entire connection.

Server Push

Server Push allows an HTTP/2 server to proactively send content to an HTTP/2 client, without it being requested.

You can think of it semantically as an extra response given to a client, along with the metadata of a request that would have generated this response, like "if you were to send me a GET request for /abc, this is what you'd receive".

This is useful when a server can guess what you're likely to request next, for example when you request an HTML page that contains lots of images, it can be useful for the server to push the critical images back alongside the HTML rather than wait for the client to realise they need them.

HTTP/1 has no mechanism similar to this, so there's no way to translate this back for older clients, but fortunately push support is entirely optional, and clients or intermediaries are free to ignore push requests, or set the SETTINGS_ENABLE_PUSH setting to 0 to explicitly refuse them up front.

Translating one to the other

If you're translating between the two, what does that mean in practice?

If you want to safely translate an HTTP/2 message into HTTP/1, e.g. to handle an HTTP/2 request with HTTP/1.1-compatible code, you need to:

In requests, build a method, URL and 'Host' header from the pseudo-headers
In responses, read the status from the :status pseudo-header, and set the status message to the default for that status code
Strip all pseudo-headers from the header data
Join your Cookie headers with a ; separator, if you have more than one
Build tunnels from CONNECT requests like normal, but tunneling just within that specific HTTP/2 stream, not the entire connection
If you receive a server push, drop it silently (although you could cache it first), or considering signaling in the initial HTTP/2 SETTINGS frame that you don't accept pushes in the first place

To safely translate an HTTP/1.1 message into HTTP/2, e.g. to return an HTTP/1.1-compatible response to an HTTP/2 client:

In responses, drop the status message entirely, and put the status code into the :status pseudo-header
In requests, build the :scheme, :authority and:path headers from the URL and host header
Optional, but recommended: Split Cookie headers into a separate header per cookie
Strip all headers made illegal in HTTP/2 messages:
- connection (and all headers listed in the value of the connection header)
- upgrade
- host
- http2-settings
- keep-alive
- proxy-connection
- transfer-encoding
- te (unless the value is just trailers)

With all that in place, you can convert back and forth between HTTP/2 and HTTP/1 messages freely, and easily integrate HTTP/2 into your existing HTTP-powered codebase. Enjoy!

If you're interested in the finer details of implementing this, you might enjoy this thread, where I'm currently live tweeting the entire implementation of HTTP Toolkit's new HTTP/2 interception & debugging support, with links and explanation for each commit along the way.

Anything I've missed? Any thoughts? Leave a comment below or get in touch on Twitter and let me know.

Originally posted on the HTTP Toolkit blog

What's coming in TypeScript 4?

Tim Perry — Mon, 22 Jun 2020 16:00:00 +0000

TypeScript 4 is coming up fast: a first beta release is planned for this week (June 25th), with the final release aiming for mid-August.

It's important to note that TypeScript does not follow semver, so 4.0 is not as big a deal as it sounds! There can be (and often are) breaking changes between any minor TypeScript versions, and major version bumps like this happen primarily for marketing reasons, not technical ones.

This bump to 4.0 doesn't suggest that everything is going to break, and this won't be a huge world-changing release, but it does bring some nice additions, particularly on the typing side. For projects like HTTP Toolkit (written entirely in TypeScript) that means faster development & fewer bugs!

Let's dive into the details:

Variadic tuple types

Also known as 'variadic kinds', this is a complex but substantial new feature for TypeScript's type system.

It's not 100% confirmed yet (the PR remains unmerged), but it's explicitly in the 4.0 roadmap, and Anders Hejlsberg himself has called it out as planned for the coming release.

Explaining this is complicated if you don't have an strong existing grasp of type theory, but it's easy to demo. Let's try to type a concat function with tuple arguments:

function concat(
    nums: number[],
    strs: string[]
): (string | number)[] {
    return [...nums, ...strs];
}

let vals = concat([1, 2], ["hi"]);
let val = vals[1]; // infers string | number, but we *know* it's a number (2)

// TS does support accurate types for these values though:
let typedVals = concat([1, 2], ["hi"]) as [number, number, string];
let typedVal = typedVals[1] // => infers number, correctly

This is valid TypeScript code today, but it's suboptimal.

Here, concat works OK, but we're losing information in the types and we have to manually fix that later if we want to get accurate values elsewhere. Right now it's impossible to fully type such a function to avoid this.

With variadic types though, we can:

function concat<N extends number[], S extends string[]>(
    nums: [...N],
    strs: [...S]
): [...N, ...S] {
    return [...nums, ...strs];
}

let vals = concat([1, 2], ["hi"]);
let val = vals[1]; // => infers number
const val2 = vals[1]; // => infers 2, not just any number

// Go even further and accurately concat _anything_:
function concat<T extends unknown[], U extends unknown[]>(
    t: [...T],
    u: [...U]
): [...T, ...U] {
    return [...t, ...u];
}

In essence, tuple types can now include ...T as a generic placeholder for multiple types in the tuple. You can describe an unknown tuple ([...T]), or use these to describe partially known tuples ([string, ...T, boolean, ...U]).

TypeScript can infer types for these placeholders for you later, so you can describe only the overall shape of the tuple, and write code using that, without depending on the specific details.

This is neat, and applies more generally than just concatenating arrays. By combining this with existing varadic functions, like f<T extends unknown[]>(...args: [...T]), you can treat function arguments as arrays, and describe functions with far more flexible argument formats and patterns than in the past.

For example, right now rest/varadic parameters in TypeScript must always be the last param in a function. For example, f(a: number, ...b: string[], c: boolean) is invalid.

With this change, by defining the arguments of the function using a variadic tuple type like f<T extends string[]>(...args: [number, ...T, boolean]) you can do that.

That's all a bit abstract. In practice, this means you'll be able to:

Destructure array types: type head = <H extends unknown, T extends unknown[]>(list: [H, ...T]) => H
Do many of the things allowed by mapped types, but on arbitrary-length arrays of values, not just on objects.
Infer full types for functions with variadic arguments: type f = <T extends unknown[]>(...args: [...T]) => T
Infer proper types even for extra complicated partially-known variadic arguments: type f = <T extends unknown[]>(...args: [string, ...T, boolean]) => T
Fully define types for promisify.
Create accurate types for many other higher-order function definitions, like curry, apply, compose, cons, ...
Kill all sorts of workarounds where you had to separately define an overload for each possible number of arguments (I've been guilty of this myself).

Even if you're not writing a lot of higher order functions, improved typing here should allow more detailed types to spread far and wide through your code, inferring away many non-specific array types, and improving other types all over the place.

There's a lot more depth and many other use cases for this - take a look at the full GitHub discussion for more info.

Labelled Tuples

As a related but drastically simpler feature: TypeScript will allow labelling the elements of your tuples.

What does the below tell you?

function getSize(): [number, number];

How about now?

function getSize(): [min: number, max: number];

These labels disappear at runtime and don't do any extra type checking, but they do make usage of tuples like these far clearer.

These also work for rest & optional arguments too:

type MyTuple = [a: number, b?: number, ...c: number[]];

For more info, checkout the GitHub issue.

Property type inference from constructor usage

A nice clear improvement to type inference:

class X {

    private a;

    constructor(param: boolean) {
        if (param) {
            this.a = 123;
        } else {
            this.a = false;
        }
    }

}

In the above code right now, the type of a is any (triggering an error if noImplicitAny is enabled). Property types are only inferred from direct initialization, so you always need either an initializer or an explicit type definition.

In TypeScript 4.0, the type of a will be string | boolean: constructor usage is used to infer property types automatically.

If that's not sufficient, you can still explicitly define types for properties, and those will be used in preference when they exist.

Short-circuit assignment operators

Not interested in typing improvements? TypeScript 4.0 will also implement the stage 3 JS logical assignment proposal, supporting the new syntax and compiling it back to make that usable in older environments too.

That looks like this:

a ||= b
// equivalent to: a = a || b

a &&= b
// equivalent to: a = a && b

a ??= b
// equivalent to: a = a ?? b

Nowadays the last option is probably the most useful here, unless you're exclusively handling booleans. This null-coalescing assignment is perfect for default or fallback values, where a might not have a value.

The also rans

That's a few of the bigger notices, but there's a lot of other good stuff here too:

unknown now supported as a type annotation for catch clauses: try { ... } catch (e: unknown) { ... }
Support for React's new JSX internals
Editor support for @deprecated JSDoc annotations
More performance improvements, following on from the big improvements in 3.9
New editor refactorings (e.g. automatically refactoring code to use optional chaining), improved editor refactorings (better auto-import!), and semantic highlighting

None of these are individually huge, but nonetheless, cumulatively it'll improve life for TypeScript developers, with some great improvements to type safety and developer experience all round.

I should note that none of this is final yet! I've skipped a few discussed-but-not-implemented changes - from awaited T to placeholder types - and it's quite possible some of these features could suddenly appear in the next month, or equally that a new problem could cause changes in the implemented features above, so do keep your eyes peeled...

Hope that's useful! Get in touch on Twitter or stick a comment below if you've got any questions or thoughts.

Originally posted on the HTTP Toolkit blog

Bye bye Feature-Policy, hello Permissions-Policy

Tim Perry — Wed, 27 May 2020 13:30:00 +0000

Ever heard of Feature-Policy? It's a draft W3C web security standard, defining an HTTP header and iframe attribute that sets limits on the browser features a page can use.

It's useful for any site that's concerned about XSS attacks, embedded content, security risks in dependencies, or major bugs in their own software, by setting guardrails on the browser features a page can use. You can use feature policy to guarantee your page or an embedded iframe cannot access the user's microphone or camera, can't read their location or phone sensors, can't use the Payment Request API, and so on. This is an extra safeguard, in addition to the browser's own permissions system, so it only tightens existing permission restrictions further.

Feature Policy has been around a couple of years now, and got some good early press as a recommended security technique all over, from Google's web developer guide to Smashing Magazine.

Since then browser support has made steady progress, with about 75% of users globally now supporting it (that's all recent browser versions except Safari). More recently that's lead to the start of real production usage: Rails 6.1 and Node.js's popular helmet security package recently shipped built-in support, and Scott Helme's latest analysis of the top 1 million sites shows the Feature-Policy header in use by nearly 5,000 of them.

It is still just a draft though. That means it's subject to change, and it is now changing: the Feature-Policy standard & header is being renamed to Permissions-Policy.

There's some discussion of the reasoning in the spec repo. In short:

Many proposed additions don't mesh with the existing Feature-Policy behaviour, so these (along with some of the existing features) are being defined instead in a new Document-Policy header, with different semantics focused on feature configuration, rather than security.
The remaining features are a strict subset of the separately defined set of web permissions.
Renaming offers an opportunity to change the header value syntax, to align it with the new Structured Headers standard.

Any kind of migration of web standards comes with some risk. In this case, a different risk than normal: removing or renaming this header won't break anything outright, but it does silently remove a security safeguard from existing sites.

The exact migration plan is unclear, but it seems likely that browsers will include support for the existing header & syntax for a while with a warning, to ensure this is as obvious as possible for the existing sites that expect it to work. Seeing this change in the real world is still a couple of browser releases away, so we'll have to wait to find out exactly how each browser decides to handle this.

Consider this an early warning though: if you're currently using Feature-Policy, you're going to want to migrate soon, and as a community we've got a whole bunch of documentation that's going to need updating.

Want to test out Feature/Permissions policy headers right now? Fire up HTTP Toolkit, set some breakpoint rules, intercept some real web traffic, and rewrite the live headers to your heart's content.

Originally published on the HTTP Toolkit blog

How will user-agent client hints work?

Tim Perry — Wed, 06 May 2020 17:00:00 +0000

In the coming months, browsers are going to start killing the User-Agent HTTP header to replace it with user-agent client hints, a set of opt-in Sec-CH-UA-* headers.

Maybe you've heard about this already, maybe that all sounds great, but what exactly does this mean in practice?

Let's talk about how the Accept-CH and Sec-CH-UA-* headers will work, how you can test that with your own services today, and what comes next.

What's the current situation?

Right now the user agent (UA) includes your browser version, OS version and architecture, specific mobile phone manufacturer & model, and more. This creates a wide range of unique user agent header values, and that means a server & proxies can use this header (along with other data points) to fingerprint users - to recognize & track individual people without using cookies or other restricted tracking mechanisms.

In addition, many sites use UAs to decide which content to server. This UA 'sniffing' has historically been abused, blocking functional browsers from accessing services when they don't fit a whitelist of known UA formats. That in turn has resulted in UAs trying to preserve backward compatibility, and UA strings gaining more and more cruft that can never be removed. Right now, 100% of popular browsers' user agents start with Mozilla/5.0, for instance. Not great.

As a case in point, here's a user agent for Chrome on Android:

Mozilla/5.0 (Linux; Android 9; Pixel 2 XL Build/PPP3.180510.008) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/67.0.3396.87 Mobile Safari/537.36

Very specific, and very inaccurate. In reality, there's no KHTML, Gecko, Safari or Mozilla involved. All this information is sent to every service your browser communicates with in any way. This is a mess.

What's the plan?

The solution is not to remove the User-Agent header completely. For compatibility reasons it will still be sent, probably forever, but 'frozen'. The plan is to progressively reduce the number of unique UA values, by grouping more and more cases together to return the same UA.

Soon, there's likely be a single UA used by all Chrome versions on all desktop OSs, and a single UA used by all Chrome versions of all mobile OSs. That reduces the real information in the user agent down to just mobile/desktop, and the browser itself. Long term, it's very possible those will be frozen too, sharing UAs across desktop and mobile and more browsers.

This will apply to both the User-Agent header that's sent in HTTP requests, and the navigator.userAgent property accessible from client-side JavaScript.

Some services do need the information that the UA provides though. You might be serving content that depends on the specific browser version a user is using (either because the content itself is relevant to the browser, or because you need to work around behaviour in specific known versions), or you might be serving content that depends on the user's specific OS and OS version (offering a Mac download to Mac users and a Windows download to Windows users).

These cases exist, and will continue to be supported, but explicitly: the server will need to send an Accept-CH header to request this information.

The Accept-CH header

Accept-CH is an existing HTTP header, currently an active but experimental draft standard, currently in the "Last Call" phase (until the 8th of May 2020). It's been supported in Chrome on desktop & Android since 2015, and other Chromium-based browsers, though it's not yet available in Firefox or Safari.

Until now, its been used to request bonus details from browsers, such as the user's connection speed, viewport size or screen density. The idea is to allow servers to customize the content they serve, optimizing images and other content for mobile devices or users on low-bandwidth connections.

It works like so:

The client sends a request to the server with no hints, for example an initial navigation to https://example.com/index.html
The server responds with the content requested, and includes an Accept-CH header, such as:
- Accept-CH: Viewport-Width - the server wants to know the width of the client's screen
- Accept-CH: Width - the server wants to know the desired width of resources being requested (e.g. how much space is available to show an image)
- Accept-CH: DPR, Device-Memory, Downlink - the server wants to know the screen density, amount of RAM, and bandwidth of the client
For subsequent requests for pages or resources from the same origin, the client sends these hints, each as a separate header:
- Width: 123 - the size of image the device wants to show
- Device-Memory: 2 - the memory of the device, in GiB, rounded to 0.25/0.5/1/2/4/8 to resist fingerprinting
- Download: 2.5 - the bandwidth available, in Mbps, rounded to the nearest 25Kbps to resist fingerprinting

There's a few caveats to this:

First, client hints aren't always honoured. They're only supported for HTTPS connections, and only on first-party resources, so if you open https://example.com in your browser, requests to load subresources from example.com may include client hints, but requests for subresources from ads.otherdomain.com will not (although this may be configurable using a feature policy).

They're also optional. Clients might refuse to send them, or might not support them at all, and they'll likely never appear in the first request to your origin.

That said, if you do need a hint in the initial request, you could return an Accept-CH header with a 307 redirect back to the same URL to ask for the hint immediately, but you rarely want to do this. Doing so adds a redirect to your page load, and you risk putting users who can't or won't provide these hints into a redirect loop that locks them out of your site. It's better to serve a default version of your content, and treat client hints as progressive enhancement, to be used when available but not depended upon.

These client hints are then persisted for the origin. The exact lifetime is left up to the client (a previous draft included a Accept-CH-Lifetime header, but that's now been removed) but its likely to be at least the rest of the current browser session. Although this means the same hint headers are duplicated on all future requests, with HTTP/2's header compression that can be done extremely efficiently.

Lastly, if you are building a server that uses any client hints, you should be sure to include Vary: <hint name> in all responses, to ensure that they're only cached for requests that send the same hint values.

All of this is Chrome only right now, although there's some progress in other browsers, and the standardisation process is intended to encourage that. The set of opt-in hints supported in the latest stable Chrome includes:

Width
Viewport-Width
DPR
Content-DPR
Device-Memory
RTT
Downlink
ECT

Google's web fundamentals guide has more detail about using these in practice.

That's the state of things today. Let's talk about how we can use this to kill off User-Agent once and for all.

User-agent client hints

The current UA client hints draft proposes a few user-agent client hint headers to expose the information from User-Agent in a granular form:

Sec-CH-UA - basic UA info, e.g. "Google Chrome"; v="84"
Sec-CH-UA-Arch - the CPU architecture, e.g. x86_64
Sec-CH-UA-Model - the device model, e.g. Pixel 3
Sec-CH-UA-Platform - the client OS, e.g. Linux
Sec-CH-UA-Platform-Version - the client OS version, e.g. NT 6.0
Sec-CH-UA-Full-Version - the full client UA version, e.g. "84.0.4128.3"
Sec-CH-UA-Mobile - a boolean header, describing whether the client a mobile device, either ?1 (yes) or ?0 (no)

The Sec- prefix here may be unfamiliar. This is a general prefix for a forbidden header name as defined by the Fetch spec. Headers starting with Sec- can never be manually sent by JS in a web page.

Sec-CH-UA and Sec-CH-UA-Mobile are considered 'low-entropy hints', which will be sent by default. For the others, you'll need to send an Accept-CH header, with the header name without the Sec-CH- prefix. For example, if you want to know what platform the client is using, send a Accept-CH: UA-Platform response.

It's important not to ask for too much, and request only the hints you really need. In addition to potential data transfer concerns (especially for HTTP/1.1 clients or servers), requesting too much information may trip a privacy budget or otherwise trigger permission prompts in future, and implies collecting unnecessary personal information about your users.

The draft also proposes a navigator.userAgentData JavaScript API to access this hint data client-side, but that doesn't seem to be implemented anywhere yet.

How to start using these today

Right now, the only browser that supports this is Chrome, and only in the dev & canary channels, and behind a flag. It's early days! That said, testing it out now allows you to see how this might impact your application, and start to see how you can capture any hints that you need to handle this, once this does land for real.

To test this out today, you'll need an HTTPS server locally where you can log requests and play with the response headers, or you can use an HTTP debugger like HTTP Toolkit to directly inspect & inject responses to test around with. Once you have that in place:

Open Chrome (either the Dev or Canary builds)
Enable "Experimental Web Platform features" and "Freeze User-Agent request header" from chrome://flags
Load a page from your domain over HTTPS, and look at the request headers you receive - this is what'll happen soon by default: Note the frozen "84.0.0.0" version and "Windows" platform in the UA here
Load the page afresh, this time returning edited headers (directly from your server, or by adding a breakpoint from the Mock page in HTTP Toolkit) that include Accept-CH: UA-Platform
Reload once more, and you should see the client send you a new Sec-CH-UA-Platform header in the request.

Bear in mind this is still a draft, not yet released in any stable browsers, and not yet final. Don't ship code that depends on this! The full details aren't yet decided, and it's still very possible that it'll change over the coming months.

When is this happening?

In Chromium's original timeline (now disrupted by COVID-19), the goal was to freeze browser & OS versions from June 2020, eventually freezing to just 2 possible user-agent values - one for desktop & one for mobile - for all versions of Chrome from September 2020.

That's now delayed until 2021, and the specific new plan hasn't yet been announced, but it's likely to take a similar shape.

Other browsers will likely follow suite. Edge have been supportive, while Firefox are broadly supportive, and already have UA freezing implemented in place as a privacy option today. Recording Firefox's HTTP traffic with HTTP Toolkit normally shows Firefox sending a detailed UA:

But if the privacy.resistFingerprinting flag is set in Firefox's about:config, that same browser sends:

Safari haven't formally announced their position, but they've previously attempted to freeze the UA in preview builds (though that was partially rolled back), and it seems likely they'd follow suit once the rest of the ecosystem commits to this.

Watch out for more changes in the same direction too, as browsers move other fingerprintable data behind client hints in future, including the Accept-Language header, and begin investigating approaches like GREASE to mitigate sniffing risks. You can follow the detailed progress on this in the Chromium and Firefox bug trackers.

Have thoughts? Accept-CH in general is now in its last call for comments, until the 8th of May 2020, whilst the UA freezing and client hints details are still very much subject to change, with discussion happening in the WICG ua-client-hints repo on GitHub. There's still time to shape them to work for you!

Originally published on the HTTP Toolkit blog