DEV Community: Paul Elliott

AWS Verified Access preview non-review!

Paul Elliott — Mon, 02 Dec 2024 14:56:55 +0000

Yesterday AWS announced AWS Verified Access for non-HTTPs connections.

This is huge news as it opens up the possibility of getting direct access to private services without needing a VPN. This would allow for the first time, 'direct' access to internal RDS databases without needing a jump box or a proxy. Or at least that's the claim. So I was eager to give it a try. Unfortunately, I didn't get very far.

The first stumbling block is discovering that a client is required, and clients are only available for Windows and Mac, nothing for Linux. Although the contents of the installation package suggest Linux support might be coming in the future.

Still wanting to give it a try, I deployed a plain Windows 11 VM for testing. The Windows installer worked fine, but strangely, doesn't add any icons to launch the app, so I had to browse through the filesystem to launch the client. There's also no configuration options whatsoever in the app itself, instead it's configured by manually deploying a JSON file onto the filesystem, which looks something like this:

{
    "Version": "1.0",
    "VerifiedAccessInstanceId": "vai-2a7bd80dcdc3175c3",
    "Region": "eu-west-1",
    "DeviceTrustProviders": [],
    "UserTrustProvider": {
        "Type": "iam-identity-center",
        "Scopes": "verified_access:application:connect",
        "Issuer": "https://identitycenter.amazonaws.com/ssoins-6834324c3a3214a1",
        "PkceEnabled": true
    },
    "OpenVpnConfigurations": [
        {
            "Config": "Y2xpZW5***REDACTED***hbWU=",
            "Routes": [
                {
                    "Cidr": "2a07:d018:118c:3b00::/57"
                }
            ]
        }
    ]
}

Cue the soul crushing realisation that the service is just a wrapper around OpenVPN. The clue is in the OpenVpnConfigurations block which is just a base64 encoded OpenVPN configuration. 😭 WireGuard is a much better VPN technology in every way, and it could have been used here. It's faster, lighter, secure by default and much simpler to implement. A blog I wrote a while back still stands true today.

But let's carry on, because this could still be a really neat way of getting access to private databases without the overheads of running something like Client VPN.

So I copied over the configuration to the location specified on Windows, C:\ProgramData\Connectivity Client\ClientConfig1.json, and started the client. And got this..

..followed by this about a minute later..

..and that's as far as I've managed to get after following the launch blog instructions.

Given this experience, it doesn't feel like the service even warrants the 'preview' label, it's a long way from a state I would consider deploying, even for testing. Given the timing, on the first day of reInvent, I suspect commercial pressures were at play here. It's a shame, as direct access to private resources without the overhead of managing a VPN would be incredibly useful. I'll be keeping my eyes open on how it progresses and hopefully in the mid-term it will become a viable option.

Data inconsistency in AWS Amazon Aurora Postgres solved with Local Write Forwarding?

Paul Elliott — Tue, 26 Nov 2024 21:27:42 +0000

I'm a big fan of Postgres. I'm also a big fan of AWS Aurora Postgres. While working as a consultant optimising databases for clients, I witnessed first hand the amazing scalability that's possible with these two technologies. But it's not all sun and roses.

YouTube has many excellent videos on the architecture behind AWS Aurora. At a very high level, AWS takes the upstream Postgres, replaces the storage engine and makes some other modifications to the planning engine. The storage engine is the biggest change though, offloading the data into dedicated storage clusters isolated from the underlying compute. This provides a huge number of benefits, but also causes one major problem, or it did, until recently.

Aurora clusters are formed of a single writer instance and zero or more reader instances. I'm disregarding Aurora Serverless, as that's a whole other beast and a topic for another day. In the simplest setup, the cluster provides a single writer endpoint and a single reader endpoint. Clients send read only queries to the reader endpoint, which uses DNS round robin to (dumbly) route traffic to the reader instances. Clients send INSERT, UPDATE and DELETE traffic, unsurprisingly, to the writer endpoint, which will always route traffic via DNS to the current writer instance.

The writer instance and the reader instances all point to the exact same storage backend, as it's shared across all instances. This means that when the writer successfully commits a change to storage, the updated change is available through the storage layer to all the reader instances synchronously, with zero lag.

So if we commit a change via the writer and then perform a query on a reader, after successfully committing the change on the writer, we'll get our new data back, right? It depends.

Although it's true that the underlying data on disk is always consistent between the writer and reader instances, as it's the exact same blocks of storage referenced by both, there's one area that's not always consistent.

Aurora uses Linux on EC2 for the underlying compute. The Linux kernel uses a page cache to retain data from recently accessed blocks in volatile memory (RAM). And this is what can lead to inconsistent query results. Consider this scenario:

The writer instance receives a query updating a row.
The writer sends the update to the storage layer.
The storage layer commits the change and returns a success to the writer instance.
The writer instance returns to the client that the transaction was successful.
A reader instance receives a read-only query for the same row that was just updated.
The reader has a large amount of RAM, and that row is already in the page cache, so it skips going to storage and simply returns the row from the page cache.

See the problem? The reader instance skipped the lookup to the backend storage as it believed it already had the latest available data to return. Meanwhile a background process runs between the storage layer and compute layer, which invalidates the page cache blocks when the underlying block has changed. Unfortunately there's a small, but significant, latency to this process which results in this issue. Let's validate that latency.

I'll be using an Aurora Postgres cluster using engine version 16.4, a single writer instance and a single reader instance, both running db.t4g.medium instances. As a test client I'll be using a t3a.micro EC2 instance. The writer, reader and test client are all in different AZs, in the same eu-west-1 region.

For testing I'll be using a simple Python script. The script will perform the following actions:

Open database connections to the writer, reader or both, depending on the test scenario.
Update a row within a table with a counter starting at 0 and increasing sequentially up to the maximum number of repetitions.
Wait an increasing amount of time, starting with no wait and going up to 100 milliseconds.
Read the same row back and check if the counter has the new value (consistent), or still has the old value (inconsistent).
Repeat steps 2-4 until reaching the maximum number of 10,000 repetitions.

The time taken between steps 2 and 4 will also include the time it takes the script to run, which needs to be taken into consideration. So let's work out how long that will be. Let's run without any delay in step 3, with both writes and reads going to the writer instance. This will be our absolute best case scenario in terms of code latency, and in this scenario, the code takes (M=0.12795ms, SD=0.000155) measured from the successful commit of the transaction in step 2, to the issuing of the read request in step 4 across 100,000 repetitions. Pretty quick, and more than quick enough for our testing.

Now we know how quick we can read data back, we can now start to see if a read immediately after a write will return the data we expect. Here's the results when writing to the writer, and reading from the reader.

Let's walkthrough what we can see here, as it's hugely significant.

In our scenario, when we make an update through the writer instance and then read the data back from a reader instance, with less than 25 milliseconds between the queries, you'll get the wrong data back. When updating and then immediately reading back, we see a failure rate (the red line) of 99.36%, dropping to 59.96% when an artificial delay of 12.5 milliseconds is added. This is fundamentally at odds with what an ACID compliant database should be doing. (Although impossible to see on the chart, we still get a failure rate of 0.02% with a 50 millisecond delay added).

Having previously identified what the issue is likely to be here, the page cache, let's repeat the same test but sending both the updates and the reads to the writer instance.

Problem solved, and theory confirmed! We no longer see any inconsistencies when reading data back immediately after updating it.

Except, this is a really bad idea.

In Aurora Postgres, you're typically limited to a single writer instance and up to 15 reader instances. I say typically, as recently Aurora Postgres Limitless went GA, which provides horizonal autoscaling of writer instances, but that's a topic for another day as it has some significant design details which need taking into consideration. Putting the Limitless product aside, this means that you will always be limited to a single writer instance within any single Aurora Postgres cluster. So the writer instance should only be used for queries performing updates, with all other queries handled by autoscaling reader instances. But what other option do we have? If we're performing updates and then needing to read that data back with consistency, based on these findings we have to use the writer instance, or add in an artificial delay, don't we?

This was true until the general availability launch of Aurora Postgres local write forwarding, which solves this particular problem, with some caveats.

Local write forwarding is a feature of Aurora Postgres (and Aurora MySQL) which allows you to send a consistency level for writes, which are then sent to reader instances. The reader instance receiving the traffic will identify the query as an update, and forward it on to the writer instance. Depending on the consistency level requested, it will then optionally wait for the change to become consistent to the specified level, before confirming the transaction as a success to the client.

Local write forwarding is enabled at the cluster level, and once enabled clients can specify the level of consistency they need when using the feature:

OFF: Disabled, updates sent to a reader instance will fail immediately.
SESSION: The default on a cluster with local write forwarding enabled. This means that any changes made within a single session will always be consistent within that session, but may not be consistent in other sessions.
EVENTUAL: This allows for updates to be sent to reader instances for forwarding to the writer instance, but provides absolutely no guarantee that the data will be immediately consistent.
GLOBAL: The sledge hammer setting. This ensures that all updates sent through the session are replicated to all reader instances before the transaction returns.

This should solve our page cache data consistency issue, and it even allows us to set the desired consistency required on a per client basis, which is fantastic. Let's check it works by repeating our previous tests. We'll start with EVENTUAL consistency, where we should still expect to see failures.

As expected, we continue to see failures at a similar rate to before. Now let's try with SESSION consistency.

No more failures! And finally, let's try with GLOBAL consistency.

As expected, no failures. But at what cost? Let's have a look at those latency figures with the zero added delay scenario.

Let's look a bit closer at the numbers:

Consistency Level	Latency (ms)	Increase from disabled
disabled	4.461590695	0%
EVENTUAL	5.70614152	27.89%
SESSION	5.927728486	32.86%
GLOBAL	6.418921375	43.87%

A whopping 43.87% increase in latency compared to not using local write forwarding and this is on an otherwise completely empty, isolated and idle cluster, an entirely unrealistic prospect in the real world.

Now that sounds like a big increase, but the latency figures are still under 10ms across the board. How that scales with a real-life production workload, is entirely dependent on the workload in question. Using load testing tools such as locust, and some careful modelling of query patterns, it's possible to simulate such a workload. This would allow that question around scaling to be answered.

When adding local write forwarding, Amazon have included new wait states to Performance Insights. The chart below shows a reader instance actively forwarding traffic to the writer instance. These new metrics will be really useful as production workloads move onto clusters with local write forwarding enabled, helping to diagnose situations when the feature is causing unexpected bottlenecks.

Overall I would consider local write forwarding a big win, even with the latency penalty shown above. The ability to remove all traffic from the writer instance and throw everything at the readers makes life a lot simpler for developers, without having to worry about consistency issues. I highly recommend people have a play and see how it performs.

If you're interested in the raw data behind this blog post, spot any inaccuracies, or would like to add anything, please do get in touch.

Accessing GitHub Action runners using Netbird

Paul Elliott — Mon, 11 Nov 2024 16:57:21 +0000

I've written previously about using WireGuard to get remote SSH access to GitHub Actions runners, something that can be really useful for troubleshooting build issues. Have a read over that blog post for the details, but in summary, we use a GitHub Action to provision a WireGuard tunnel between our machine and the runner. It's not the most straightforward solution, and a better alternative now exists which I'll run through here.

WireGuard is a modern VPN technology built into the Linux kernel, it implements the bare bones of a VPN, just enough to get a secure tunnel established between two devices. Everything else is left up to the reader to manage. In my WireGuard SSH GitHub Action, I provided some extras required which made it possible to SSH to a GitHub Actions runner over WireGuard. But it still requires quite a few manual steps to get it working.

Recently I've been looking at VPN solutions, specifically ones built around WireGuard, that take away some of the manual steps required to manage a large scale deployment. After building proof of concept solutions with several of these offers, I settled on NetBird.

NetBird provides a management layer, with a web UI, which handles all the necessary background management tasks needed when running more complex WireGuard based VPN solutions. It's BSD licensed so you can self-host or you can use their hosted offering, which is free for small deployments and works perfectly for accessing GitHub Actions runners. Let's see how using NetBird can simplify remote access to runners.

NetBird has the ability to use an identity provider, also known as an IDP, or through a secret key known as a setup key. I want this solution to be completely hands-off and automated, so I'm going to use setup keys. These are used by the NetBird client to automatically register new peers on-demand. They can be constrained using expiry dates, number of uses, or more usefully in this use case, they can also be used to make the peer ephemeral.

Setup keys can also automatically assign peers registered with the key to a group. Groups in NetBird allow fine-grained access control lists to be created, allowing traffic flows to be restricted. In this case, we've assigned the setup key to a new group called runners.

The key is set to be reusable, limited to the maximum 365 days and to be ephemeral. Peers registered using a setup key that's set to be ephemeral means the peers will be automatically cleaned up, and removed from Netbird if there's no activity for 10 minutes. This saves us the overhead of removing old runner peers.

⚠️ Store the setup key somewhere secure, it will only be displayed once at the time of creation.

We're now ready to start registering our GitHub Action runners with NetBird. But before we do we should register a client that we will use to connect to the runners. We'll use another setup key for this, but this setup key will be far more restrictive, only allowing a single use and automatically registering with another group called clients.

Once again, store the setup key somewhere secure, it will only be displayed once at the time of creation.

Now let's register our client device which we'll be using to connect to the GitHub Action runner. Go to https://app.netbird.io/install and download the client software for your device. Now let's use the client setup key to register our client:

paul@lightoak:~$ sudo netbird up --setup-key 0CE86986-6B44-4B44-8037-D587DCE86DFC
Connected

We can now see our client peer in the Netbird UI peers dashboard:

Clicking on the peer shows us further details:

The connection status can be checked at any time by running sudo netbird status:

paul@lightoak:~$ sudo netbird status
OS: linux/amd64
Daemon version: 0.29.3
CLI version: 0.29.3
Management: Connected
Signal: Connected
Relays: 3/3 Available
Nameservers: 0/0 Available
FQDN: lightoak.netbird.cloud
NetBird IP: 100.99.96.205/16
Interface type: Kernel
Quantum resistance: false
Routes: -
Peers count: 0/0 Connected

Here we can see that we're connected to both the management and signal backends, and there's 3 relays available. The Netbird website has further details on Netbirds Architecture.

We can also see that the 'Peers count' is 0/0, meaning we have no peers available for us to connect to. Let's fix that by adding in our GitHub Action runners.

Luckily, there's no need to write my own GitHub Action this time around as Alemiz112 has already done the hard work for us with the netbird-connect action.

Let's create a minimal workflow we can use as an example:

name: Netbird demo
on: [push]
jobs:
  demo:
    runs-on: ubuntu-latest
    steps:
      - name: Checkout
        uses: actions/checkout@v4
      - name: Netbird Connect
        id: netbird
        uses: Alemiz112/netbird-connect@v1.0.1
        with:
          setup-key: ${{ secrets.NETBIRD_SETUP_KEY }}
      - name: Install public SSH key
        run: |
          mkdir ~/.ssh
          echo "${{ secrets.SSH_PUBLIC_KEY }}" > ~/.ssh/authorized_keys

The action uses the runners setup key we created earlier to authenticate to Netbird. This needs adding as a secret into the repository settings along with the public SSH key to use.

The NETBIRD_SETUP_KEY and SSH_PUBLIC_KEY values need adding as repository secrets, rather than as environment variables.

With everything in-place, and the workflow config pushed to GitHub, we can now see a successful run when viewing the workflow.

When viewing the list of peers in the Netbird console we can also see the peer for the runner.

The grey circle to the left of the peer indicates that the peer is offline. This is expected, as GitHub Action runners are ephemeral and the runner was destroyed after the workflow completed. As we used an ephemeral setup key to create the runner, the Netbird backend will automatically remove this peer after 10 minutes.

Now to validate we can actually connect to a runner we'll add a forced sleep period to the workflow:

      - name: Forced sleep
        run: sleep 30m

Now we can see that the peer is still online and available, highlighted by the green circle next to the peer name.

In the list of peers we can now get the private IP of the runner, in this case it's 100.99.231.31. All we need to do now is to use ssh to connect to the private IP over the Netbird provided tunnel:

paul@lightoak:~$ ssh -o "StrictHostKeyChecking no" runner@100.99.231.31
Welcome to Ubuntu 22.04.5 LTS (GNU/Linux 6.5.0-1025-azure x86_64)

runner@fv-az1149-775:~$

In this setup it's safe to disable strict host key checks as we're connecting peer to peer over a private encrypted and pre-authenticated tunnel.

The solution could be further refined by using the built-in SSH support that Netbird provides. This isn't something I've explored as I've always had existing SSH key infrastructures to use instead, but it could simplify the setup further and is definitely worth exploring.

Netbird does a lot more, including the ability to have network ACLs to restrict the traffic that can flow between peers. This is a killer feature above all the other solutions and will be the topic of a future blog post.