DEV Community: Ben Brazier

How to Blue-Green with DNS

Ben Brazier — Mon, 26 Sep 2022 08:52:09 +0000

The Domain Name System or DNS is the main way traffic is routed on the internet. It can be used to route traffic to one or many servers and can be chained so that the same url can lead to different locations at different points in time. This is what enables blue-green deployment of software and releasing new versions of software without downtime.

So how can we use DNS to blue-green our software? In this article we will go through the process step by step.

What is Blue Green Deployment?

First we need to understand what blue-green means. Blue-green deployment is the process of provisioning more than one production environment so that you can set up a new version before cutting over to it.

By having both environments running you can quickly switch over to the new version which minimizes downtime and allows you to switch back to the previous version if needed. This is in contrast to an in-place update that relies on stopping the production environment, updating it, and starting it again.

More information about blue-green deployment can be found here in Martin Fowlers blog post from 2010.

1. Deploy A Version of Our Software with an A Record

In order to make use of blue-green deployment via DNS we first need to deploy a running version of our software. We can deploy a single host or multiple and create a DNS Record pointing their IP addresses. This will let us consistently access our service without needing to know the hosts IP addresses.

For this DNS record we should use an A record, which is a type of DNS record that maps hostnames to IP addresses. There are multiple types of DNS records with different purposes and A records are the most basic. We need one A record for each deployment.

The best way to do this is using a version number in the A record and deploying a new set of infrastructure for each version to achieve immutable infrastructure and rolling deployments. It is also possible to just have two static environments that could be labelled blue and green.

2. Create a CNAME DNS Record for our Application

The next DNS record we need to create is a CNAME. A CNAME is an alias from one hostname to another, this means it can be used for redirects or to provide multiple names for a single service. We should provision a CNAME that our customers use for accessing our service so we have control of the routing.

This record should not be versioned as there is only one per environment and we use it as a lookup for where traffic should be routed. Here are some examples of what the initial CNAME and A record might look like:

CNAME Record -> A Record
logging.company.com -> logging-01.company.com
logging.company.com -> 01.logging.company.com
logging.company.com -> logging-blue.company.com
logging.company.com -> blue.logging.company.com

3. Deploy a New Version of our Software with an A Record

After our system is running and customers are using the CNAME for accessing our service we can deploy a new version of the software. It can be deployed right next to our current version and once deployed we should be able to access it via its own A record.

Next we should run automated integration and regression tests so that we are confident the environment is working as expected and we can move onto the cut-over. It is important to make sure these tests don't impact the other deployments as they may share databases and other resources.

4. Update our Applications CNAME DNS Record

To cut-over our application to the new version we just update our CNAME record to point at the new A record. This should be quick but depending on the time to live (TTL) of the DNS record may take some time to propagate since DNS is cached in multiple places based on the TTL provided. I usually recommend 60 seconds as a good standard for DNS TTL.

If any errors occur or we aren't happy with the new environment we can revert by redirecting the CNAME back at the old A record. This should be quick and easy to do due to the simplicity of the process.

5. Cleanup

After the cut-over we may need to clean up the previous environment by deleting it or uninstalling any software that is left running. This can happen immediately or after some time but should be done as needed based on your specific environment.

Summary

Blue-green deployments are critical to minimizing risk and downtime of software releases and DNS is the main way to do so. There are many variations of this process but it is important to keep the process simple. Let me know if this is how you release new versions of your software and if there are any other parts of DNS releases you would like me to cover.

For more content like this follow or contact me:

Twitter: @BenTorvo
Email: ben@torvo.com.au
Website: torvo.com.au

4 Reasons Not To Use Environment Variables

Ben Brazier — Tue, 02 Aug 2022 03:39:00 +0000

Environment variables are a simple way to configure software. They let you set a variable in the shell so that any processes you run can use the values provided.

This is extremely helpful for cli tools but shouldn't be used for managing configuration of complex software.

In order to understand why, we need to look at the downsides of using environment variables for configuration.

1. All Types Must be Parsed from Strings

Environment variables are always set as strings. This means any value that needs to be set as another type in the software will need to parse it from a string.

For example, if I want an integer from the environment variable "FOO" in python I would need to run the following code:

import os
foo = int(os.environ['FOO'])

This requires an understanding of how the programming language parses strings into the appropriate datatypes and requires a lot more testing than directly using native data structures. This problem is also extended to include files.

If you want to configure files like ssh keys or ssl certs then doing so with environment variables can get very messy due to newlines and formatting.

2. No Nesting or Built In Structure

Environment variables are global values, which means that they can have naming collisions. For example, if you need to set the "URL" for multiple downstream services they will need to be prefixed to prevent collisions like this:

FOO_URL=x.com
BAR_URL=y.com

This lack of structure leads to long names that can unintentionally overlap with other services or features if you are not careful.

It also prevents encapsulation and grouping of configuration values. Using structured data like JSON or YAML instead can allow you to pass through groups of keys without needing to worry about the parent structure of the configuration.

3. Global Access within a Process

Good software engineering minimises the use of global state so that code is easy to test and trace. Environment variables do not - they actively encourage the use of global state and make it very easy to pull in values from anywhere. The deeper in the code this is the harder it can be to inject configuration and run tests.

import os

def a():
    foo = int(os.environ['FOO'])
    return foo * 42

def b(foo):
    return foo * 42

In the python example above function "a" is much harder to test than function "b" because we have separated the management of configuration from the pure functionality. This doesn't go away by avoiding environment variables but in my experience it is a contributing factor.

4. Need to be Set at Runtime

Environment variables need to be set in the process before running an application. This creates additional complexity in starting and managing processes. If you execute a python application that pulls config from a file it requires no additional shell commands:

python app.py

However if you need to set environment variables it will need to be scripted into the specific shell of the environment:

FOO=BAR && python3 app.py

Summary

Environment variables are a great way to get started in managing software for configuration, however any complex software should replace the use of environment variables with file based configuration like JSON or YAML.

This makes managing the configuration simpler and enables a more mature implementation of types, testing, and process management.

For one example of how to implement configuration files refer to: How to Use Feature Flags.

For more content like this follow or contact me:

Twitter: @BenTorvo
Email: ben@torvo.com.au
Website: torvo.com.au

DevOps 101: Packaging Software

Ben Brazier — Wed, 13 Jul 2022 05:35:29 +0000

Packaging code is a key step in the software development life cycle. It is the process of taking source code, installing any dependencies that are required and converting it to code that can be executed.

Anyone who develops and maintains software will need to package a new version of their code any time the source code or its dependencies change. Doing this on a frequent basis will need to automate the process, so how do we do it?

Packaging with Shell Scripts

Firstly, we need to start with packaging our code. The most common way to package software is using shell commands (BASH or ZSH) on Linux. Most server environments run Linux and we can run them on Mac directly or on Windows with Windows Subsystem for Linux (WSL). Using these commands we need to copy our code into an empty directory, install our dependencies and run our unit tests. An example package script for a Python application could look like this:

#!/usr/bin/env bash

set -e

# Make sure our distribution directory exists
mkdir -p ./dist/

# Cleanup any old files from previous packaging
rm -rf ./dist/*

# Install any dependencies listed in requirements
python3 -m pip install -r ./requirements.txt -t ./dist/

# Copy our source code into our distribution
cp -rf ./src/* ./dist/

( cd ./dist/; python3 -m unit_test)

By automating our packaging process with a simple readable script like this we can constantly package and test our code. We can also add and modify steps within the process very easily to use any language specific tooling or testing frameworks.

Where to Run Our Script

Packaging software can be done locally or on remote build agents. It is a good idea to have the flexibility to do either so that developers can locally work on their code whilst still being able to deliver changes with central tooling for transparency and traceability.

To package our code locally we can simply execute the script from a compatible terminal. To run the script in a central tool like GitLab we can add a CICD configuration file that runs the script whenever we commit changes.

Testing Our Code

Unit testing should be done as part of our packaging process in order to maintain confidence that our software works or to get fast feedback if it doesn't. These tests will validate that our code executes and provides the expected responses based on our inputs. They don't require configuration to integrate with other systems and should be able to run without connecting to other services.

We can run our unit tests before installing dependencies or after packaging our files, depending on how much our code integrates with other libraries. An example set of unit tests that runs within a Python module might look like this:

#!/usr/bin/env python3

from . import add

print("Unit Testing Mathematics")
assert add(1, 1) == 2, "Failed addition test A001"

Storing Our Package

Once we have tested our package we can upload it to a central storage location such as AWS S3, Google Cloud Storage, Azure Blob Storage or an FTP server. This will give us access to the working software that is ready for configuration and deployment without needing to repackage it. This is highly specific to the location of the upload but the services will usually provide an appropriate cli tool that allows us to handle this upload.

Summary

Packaging code is the first step in this process of delviering software and should be automated to that we can quickly and effectively deliver working software to our customers on a frequent basis. By choosing flexible tools and including testing as part of the process we can speed up software development whilst keeping the process as simple as clicking a button.

After packaging our code we need to configure it for deployment and release. In an upcoming article we will discuss the process of automating configuration and how it can be done without relying on complex tools or repetitive manual work.

Follow me here for more content or contact me:

Twitter: @BenTorvo
Email: ben@torvo.com.au
Website: torvo.com.au

How to Use Feature Flags

Ben Brazier — Tue, 05 Jul 2022 05:26:32 +0000

Feature flags or feature toggles are settings that control how software runs and which features are enabled or disabled. Feature flags should be configured on the same host our software runs on using structured data formats to ensure that they are performant, stable and easy to manage.

Why Use Feature Flags

By using feature flags it is possible to integrate incomplete functionality into our software without breaking the whole system. Feature flags also allow us to test in production environments and roll out functionality to small groups of users at a time. This can be helpful when aiming to continuously integrate and deliver new code on a frequent basis.

Feature flags can be as simple as enabling features based on the environment, or as complex as modifying what functionality to use for different users at different points in time.

Avoid Network Requests

When we configure feature flags we can do so on the host that is running our software or somewhere else over the network. However, we should avoid making network requests to get our feature flags because it can increase outages, slow down our systems and make changes harder to manage.

If feature flags need to be accessed over a network, any time a host handles a request it can cause failures due to throttling. This can be common for processes running in serverless environments or environments that scale up and down frequently. The problem of throttling can be mitigated by getting the values and caching them locally or in a data store, but doing so can increase code complexity whilst also introducing more issues related to caching.

When feature flag configuration is accessed via a network call the latency of any request increases based on the response time of that system. This can cause long and unpredictable latency compared to the consistency of making a request to the local environment.

If feature flags are stored externally the values and system that manages the configuration may change separately to the host that needs the flags. This can lead to configuration changes being modified in production with potential outages if things are misconfigured. Instead it is better to change the feature flag configuration with the service and test them before releasing the software.

Avoid Environment Variables

Environment variables are often used to pass values to command-line interface software that needs configuration. This is a quick and easy way to change settings but we should avoid using them to configure feature flags.

Environment variables are limited to string only, so any software that needs to write or read environment variables needs to parse the strings into appropriate data types. This complexity is extra work on both the software and any tooling that manages the software, which shouldn't be necessary.

Environment variables are not structured and are globally scoped within a process, which means that any structured keys need to be flattened. As an example, to set the property C of group B within software A to the value 9090 we would need to flatten the environment variable as:

A_B_C="9090"

This extra parsing and global scope can lead to even more complex handling of the environment variables.

It is hard to package and include environment variables with software as they are loaded in at run time. To set the values we need to run commands or a script setting the values on the host for the shell that the software runs in. One option is to load them from a file by exporting everything in it just before running the software. This works, but there is a better way ...

The Solution

We can configure the feature flags for our software by including a structured data file such as JSON/YAML within our software directory as config.json/config.yaml. We can then load this file when our software starts and parse it with built in tooling. We can use built in types as well as structured config for different flags.

config.json

{
    "features": {
        "customer_icon": {
            "enabled": true
        },
        "billing_alerts": {
            "release_date": "2022-08-06"
        },
        "date_format": "%Y-%m-%d",
        "console": {
            "version": 1.0
        }
    }
}

config.yaml

---
features:
  customer_icon:
    enabled: true # We can also write comments in yaml
  billing_alerts:
    release_date: '2022-08-06'
  date_format: "%Y-%m-%d"
  console:
    version: 1.0 # Should be updated to 2.0 soon

We can avoid network issues by storing the feature flag configuration on the host running the software. By including it as a file on the host we reduce the latency to almost 0 and avoid outages caused by network failures.

We can also avoid the limitations of environment variables by using structured data formats such as JSON and YAML. This lets us structure our data appropriately and use built in types other than strings.

To use these feature flags we just load them in and switch functionality on or off depending on the values. Here is a Python example:

import datetime
import json

# Loading JSON File
configuration = json.loads(open('./config.json', 'r').read())

# Date Format
date_format = configuration['features']['date_format']
print("Using Date Format:", date_format)

# Todays Date
date_today = datetime.datetime.today()
print("Todays Date:", date_today.strftime(date_format))

# Billing Alerts
billing_alerts_release_date = datetime.datetime.strptime(
    configuration['features']['billing_alerts']['release_date'],
    date_format
)
print(
    "Billing Alerts Release Date", 
    billing_alerts_release_date.strftime(date_format)
)
if billing_alerts_release_date < date_today:
    print("Billing Alerts Are Enabled")
else:
    print("Billing Alerts Aren't Enabled Yet")

# Customer Icon
print(
    "Showing Customer Icon", 
    configuration['features']['customer_icon']['enabled']
)

# Console Version
print(
    "Using Console Version", 
    configuration['features']['console']['version']
)

This is how we can enable or disable features, release functionality on a specified date, and use built in types to manage our configuration.

Summary

Structured configuration files are the best way to manage feature flags without worrying about network issues or parsing everything in and out of environment variables. This can also be used to manage any environment specific settings within our system as a central point of control.

There are many more positive side effects of configuring software with structured files that we haven't covered yet. Let us know if this solution works for you or if you have any further questions.

Follow me here for more content or contact me on:

Twitter: @BenTorvo
Email: ben@torvo.com.au
Website: torvo.com.au

The Truth About Continuous Delivery

Ben Brazier — Mon, 04 Jul 2022 04:55:29 +0000

Continuous delivery is hard. There is no simple way to learn all of the moving parts required to automate packaging, configuring, deploying, testing and releasing code. It isn’t something you can implement in a week and it requires time and effort to work towards.

If every step in the process is done with a different tool, by a different team, or in a different location, then it can feel impossible to continuously deliver on a daily basis. But it isn’t. You can continuously deliver code without needing to know 15 different tools and talk to 5 different teams within your organisation.

What is Continuous Delivery?

Continuous delivery is about having the ability to go from source code to production quickly and in a fully automated process. It enables small, frequent updates that minimise risk and maximise feedback. The benefits are generally well understood across the industry but the implementations vary. Some teams deliver changes to production every few minutes, while others can take weeks or months to release new code.

It should be possible for most teams to deliver changes daily, but in order to understand why we need to know what issues can get in the way.

Why is it so Difficult?

Dev and Ops culture separation can be a strong contributor to how well teams can continuously deliver. A significant factor in the culture of a team is resourcing.

It is easy to find developers who want to write code on their laptop but hard to find developers who want to manage infrastructure.

It is easy to find operations people who want to manage infrastructure but hard to find operations people who want to write code.

It is especially hard to find team members that want to write, deploy, test, release and support services when a large part of the industry prefers to split these tasks into different teams and even different sections of an organisation.

Tools and vendors are a dime a dozen. For any part of the continuous delivery process, you can find a handful of tools and vendors trying to get you to use them. This makes it extremely easy to pick up and add complexity to the stack, which simultaneously raises the bar of knowledge required and often hides the underlying fundamentals of how software is delivered.

Even when implementing a tool for each step in continuous delivery it is common to find gaps and integration points that aren’t supported and require an understanding of the whole stack to resolve. This is why we should focus on a minimal stack and treat every extra tool or vendor as a debt in both the technical and business sense.

Guidance and training is easy to find when it comes to using one of the vendor provided tools, but is hard to find if you just want to understand the fundamentals. But where and how can we learn the fundamentals?

I have been asked a lot in my work to provide reference for how to implement continuous delivery and in my experience there is a lot of theory available but not much implementation. I believe this is a problem that needs to be resolved in the industry right now, which is what I am aiming to achieve.

How Can We Make it Easier?

When it comes to complex problems we need to think long and hard about our options. In the words of Kent Beck, “make the change easy, then make the easy change”. But how do we make the change easy?

Kent Beck

@kentbeck

for each desired change, make the change easy (warning: this may be hard), then make the easy change

23:07 PM - 25 Sep 2012

Firstly, we must reduce the number of different tools we use and focus our effort on a few key solutions. The more different tools you use, the more knowledge is split up and harder to share.

Tools used across the industry range from code, to DSLs, to cli tools, and more . However, the one thing that we can always rely on is code. Instead of using declarative languages or cli tools we should focus our efforts on code and writing re-usable packages that can be integrated with other tasks. There is no good reason why we can’t package, configure, deploy, test and release code within a single programming language after getting it’s dependencies.

Secondly, we can invest in understanding the implementation. Instead of learning many different tools we can focus on the fundamentals of both the steps required and automation in general.

This is significantly harder initially but results in far better long term goals because tools come and go but understanding is retained.

Thirdly, we need to iterate on your implementation. No matter how simple the system or how much knowledge you have there is no way to improve without iteration because each environment is different. Each change we make may or may not improve things, and the only way to know is to try.

Summary

Continuous delivery is hard, but it doesn’t have to be. By simplifying what is required, focusing on our understanding and investing in learning we can continuously improve our ability to deliver software.

I recently posted a tutorial demonstrating an MVP for Continuous Delivery in the simplest way I could. It only requires the use of Bash, Python and AWS, which should make it approachable for any developer who can code.

I want to help improve the quality of software delivery across the industry so please let me know if this is valuable to you or if you have any suggestions.

Follow me here for more content or contact me on:

Twitter: @BenTorvo
Email: ben@torvo.com.au
Website: torvo.com.au

6 Symptoms of a Distributed Monolith

Ben Brazier — Fri, 01 Jul 2022 04:47:23 +0000

Distributed monoliths occur when the structure of an organisation and its code is split up whilst still being tightly coupled. This becomes a problem because the scale of the system increases but all of its parts need to be managed together, slowing down and increasing the risk of any changes.

It is important to be able to identify when you are dealing with a distributed monolith but how can we? Lets look at 6 symptoms that your software architecture is a distributed monolith.

1. Lock Step Deployment

When two services need to be deployed together, they are coupled. If they are within a monolith this can be easy to manage. However, deployment can become slow and painful if they are separate services that should actually be managed independently. We can resolve this problem by identifying the cause of the coupling and refactoring our solution.

If the two services use shared code, the solution could be restructuring the code to share a library that is managed separately. If they use shared infrastructure, this could mean splitting the services onto separate hosts.

2. Version Coupling

If Service A directly calls a version of Service B, every time Service B is updated we need to update the configuration of Service A. This coupling of changes can make it complex to manage the lifecycles of both applications but it is possible.

Ideally, Service A should call an endpoint for Service B that isn't a specific version and is also managed by Service B. This can usually be done with DNS by using CNAMES like this:

A1.domain.com > calls > B.domain.com > B1.domain.com

instead of this:

A1.domain.com > calls > B1.domain.com

3. Bi-directional Dependencies

If Service A calls Service B and Service B calls Service A, it can be difficult to test changes to either system. When this happens we have the option of changing the whole environment simultaneously, testing events being sent to through both the old and new versions of software, or significant extra complexity in how to route messages.

4. Shared Database Access

When two services share the same database access it becomes easy to pull data from another domain without calling the service directly. This can lead to coupled services that are connected at the database layer. When a service modifies it's own database structure it shouldn't impact any other services. If it does, then the services are coupled.

There is nothing wrong with sharing a database to save on administrative effort but this should be tightly controlled with security to prevent coupling. A common way to do this with Postgres is by using separate users and schemas per service. If a service wants access to another service's data it should call the downstream service directly.

5. Shared Queues

In the same way that shared database access causes coupling, so too do shared queues. If two services talk to a shared queue directly, it isn't possible for either service to safely modify, change or redirect the messages without risk of breaking both services.

Sharing queues should be avoided as much as possible and any queue should be owned by a single service and not exposed to others.

6. The Enterprise Service Bus

The enterprise service bus (ESB) is another common data storage pattern similar to shared databases and shared queues but on a larger scale. ESBs aim to allow any message to be sent and received via a central bus. This means that once a service sends a message to the bus it is not clear who is consuming the message, what is being done with it, and if it succeeded. These issues can't be easily resolved and mean that our system becomes coupled with not only one other system but more than we may be aware of.

Summary

Regardless of your system architecture you should be able to clearly identify if your system is a monolith, microservices, or a distributed monolith. Identifying coupling to other services through shared deployments, shared data, and bidirectional dependencies is critical in this process. Doing so will allow us to structure our architecture so that we can avoid ending up with a distributed monolith.

Follow me here for more content or contact me:

Twitter: @BenTorvo
Email: ben@torvo.com.au
Website: torvo.com.au

The Pros and Cons of AWS Lambda Function URLs

Ben Brazier — Thu, 30 Jun 2022 04:17:04 +0000

AWS Lambda Function URLs were released on the 6th of April 2022 as an infrastructure configuration that gives you public URLs to directly trigger AWS Lambda Functions.

This service provides a faster and simpler way to trigger AWS Lambda Functions than AWS API Gateway or AWS Application Load Balancer (ALB). The URLs follow a standard structure as follows:

https://**<url-id>**.lambda-url.**<region>**.on.aws

Pro: Fast Implementation & Deployment

AWS Lambda Function URLs are very quick to configure and deploy whilst requiring less set up than the other alternatives. When I set up the endpoint it is almost instantly available for use and only really requires a few configuration values.

AWS API Gateway takes longer to deploy and requires a more complex set of resources to set up. The API, Resources, Methods, Deployment, Stage, and Lambda Permissions are all part of setting this up.

AWS ALB requires additional underlying resources to be configured and has an upfront cost to using it. There is quite a bit that goes into designing AWS networks but at a minimum they contain a VPC, Subnets, IGW, and Security Groups. AWS ALBs also require multiple different resources to make up a working system.

Pro: IAM for Access Control

The options for AWS Lambda Function URLs authentication are None or using IAM. When you use None any additional security should be within the code. However, when you use IAM you can reference IAM resources with significant granularity. This means that if you and your customers both use AWS there is a strong and simple integration point for security by just referring to their AWS IAM resources directly.

Pro: AWS Lambda Alias Support

AWS Lambda Aliases are also supported by the new URLs which means that you can keep a consistent public facing URL when replacing the underlying infrastructure. This isn’t as practical as if it was done with DNS but can work if you don’t mind the AWS Lambda alias and versioning system.

Con: No Custom DNS Support

As mentioned previously DNS can’t be used for AWS Lambda Function URLs which can be an issue for both releases and the presentation of the service to customers.

The lack of custom DNS is similar to the Private AWS API Gateway and can cause issues if you use DNS automation for release processes.

It is possible to work around the custom DNS issue by deploying AWS CloudFront and pointing at the URL on the backend. This is about the same effort as deploying AWS API Gateway, which negates the benefits of using the new feature.

Con: AWS Account ID Exposure

According to AWS the new URLs expose the AWS Account ID used for the service which can provide information to any malicious actors. AWS Account IDs are not considered a credential for security purposes but publishing this information should probably be avoided where possible.

Con: Limited Network Controls

Due to the implementation of AWS Lambda Function URLs there is no way to implement network controls based on IP or any lower level than the prior mentioned authorisation. This is a limitation that could be an issue depending on your environment. For large enterprises like I have worked for in the past it is not acceptable but if you are running a start up it may be worth sacrificing for the speed to implementation.

Summary

It is always good to see more functionality being added to AWS Lambda and serverless solutions. However, the new AWS Lambda Function URLs leave a lot to be desired in their access controls and DNS implementation. Hopefully we will get custom DNS as a feature in the near future so that we don’t have to use AWS Aliases and the AWS provided URLs.

More Information

Lambda function URLs

Follow me here for more content or contact me on:

Twitter: @BenTorvo
Email: ben@torvo.com.au
Website: torvo.com.au

Why Cache Invalidation Doesn't Work

Ben Brazier — Wed, 29 Jun 2022 00:04:58 +0000

Using cache invalidation to release a new version of a website doesn’t work.

When developing a small website it is easy to skip the process of managing caching and just serve everything directly. However, once a website scales up and starts leveraging a Content Distribution Network (CDN) it is important to implement a proper caching strategy. When developers skip this step it can result in customers receiving out of date content and even broken websites.

Why CDNs Use Caches

Content Distribution Networks (CDN) are commonly used to cache responses to web requests for performance. When a CDN caches a website it will serve content without returning to the origin. This saves time and provides a better customer experience. However, if the CDN has cached content it can serve information that is out of date. One of the ways developers can roll out new content is by invalidating the cache with tools provided by the CDN.

Why Cache Invalidation Doesn’t Work

Invalidating the cache at the CDN level doesn’t have any impact on the users browser or Internet Service Provider (ISP) caching. This means that even if you invalidate the cache there is no way to know what is happening from the customer’s perspective.

Browsers cache headers using the same instructions the CDN would. This means that if a user accesses your site, you subsequently invalidate the cache, and the user comes back to your site they can still be receiving the old versions.

Invalidation is intended for use in exceptional circumstances, not as part of your normal workflow. Invalidations don’t affect cached copies in web browser caches or caches operated by third-party internet service providers. — Google

It is also common for ISPs to cache the data of commonly used websites to save them networking costs and reduce the load on the servers they own.

Once a user’s browser or an ISP caches your content you cannot remove it from the cache. This is why cache invalidation doesn’t work and alternative solutions should be used.

What Are The Alternatives?

Instead of invalidating caches when making changes to websites we should use appropriate cache headers or cache busting. It is common for websites to set caching headers by default but it is important to use them properly to avoid the need for cache invalidation.

Expires

We could use the expires header to set a time for when a specific object should be cleared from the cache. However, it can be difficult to manage this header and it has some technical issues:

the time format is difficult to parse, many implementation bugs were found, and it is possible to induce problems by intentionally shifting the system clock — Mozilla

Expires: Tue, 28 Feb 2022 22:22:22 GMT

Cache-Control: Max-Age

Generally we should use the max-age header to provide a duration that an object should be cached. A good starting point is to set this to 86400, which is equal to one day as the header is set in seconds. This means that if you rolled out a new version of a site it would be guaranteed to show up to all users after 24 hours.

Cache-Control: max-age=86400

Cache Busting

If we want to use caching but also roll out instant updates to our customers we can use a more complex solution known as Cache Busting. By setting a unique URL or query string value to our requests caches will treat each version as a new object. This means if we deploy new links with every change and never cache the initial index.html we can load new files without waiting for the cache to clear the old ones.

# Initial link  
/css/main.css?1653714985

# New link  
/css/main.css?1653715024</pre>

Summary

Next time you need to invalidate a cache to deploy a website change try using max-age or cache busting instead. It will save you time and effort every deployment whilst providing confidence that the correct files are being served to your customers.

More Information

For more content follow me here or contact me via:

Twitter: @BenTorvo
Email: ben@torvo.com.au
Website: torvo.com.au

5 AWS Lambda Pitfalls Most Developers Don't Know About

Ben Brazier — Tue, 28 Jun 2022 03:37:03 +0000

Using AWS Lambda can save a lot of time in getting services up and running but has some major drawbacks when used extensively. It is important to understand these pitfalls as they don’t usually matter until you run into them in production when it is too late to avoid.

1. Private API Gateway is Incompatible with Custom DNS

It is common in large scale software development to use private networking to increase security of internal services. This helps prevent unauthorized access to services that should never be available for public consumption. AWS Lambda supports this by using AWS VPC when configured appropriately.

It is also standard in most software development to use DNS to decouple services. DNS lets you deploy a new version of a service and cutover to it without modifying the existing service, this works because DNS records are just a pointer for where the service is hosted.

One of the common configurations of AWS Lambda uses AWS API Gateway to make the lambda accessible as a HTTP endpoint. However if you use API Gateway in a private network configuration you cannot leverage custom hostnames. This means you can only use the AWS API Gateway DNS record that is tightly coupled to the infrastructure and therefore need to modify the infrastructure to release a new version of the service.

Modifying the existing service infrastructure to release a new version increases the risk of the change and makes it harder to roll back.

2. Application Load Balancer to Lambda is Expensive

Another frontend for AWS Lambda is AWS Application Load Balancer which fixes the prior mentioned DNS and networking issue. However, compared to the extremely low costs AWS API Gateway, this is an expensive alternative.

In my region AWS ALB costs approximately $18 per month as a base (before the cost of compute power). This might not seem like much but with the prominence of microservices it is common for organisations to provision hundreds of different AWS Lambdas, which can lead to huge additional costs.

3. AWS Lambda Environment Variables Exposed

AWS Lambda can be configured with environment variables for things like database URLs, usernames, and passwords. This configuration shows up in the console when viewed with read only privileges. This means that giving people read only access in production can lead to exposed credentials.

Depending on your control of AWS IAM and your business environment this may or may not impact you but it is worth thinking about when configuring your services.

An alternative solution to this is embed your configuration into your application zip, I will cover this in a future article so follow me here to get updates about that.

4. API Gateway & Lambda Timeout Limits

AWS Lambda has a hard timeout of 15 minutes and when used in conjunction with AWS API Gateway a maximum timeout of 29 seconds. This means that any request that goes longer than one of these, depending on the context, will throw an error and fail.

It is important to handle these exceptions gracefully and return a response to the user but in order to do so we need to timeout before the timeout limit. This requires additional code to track the current duration of the execution, and cancelling it when getting too close to the timeout limit.

AWS Lambda timeouts also cause extra work to manage long running processes. To run a permanent background process you will need to handle starting and stopping constantly to fit within timeouts. This is usually far more difficult than an EC2 based solution that keeps a single process open.

5. SQS & Lambda Throughput Limit Errors

When using the AWS SQS as a direct trigger for AWS Lambda and limiting the throughput of the AWS Lambda Function any additional requests will throw errors. These errors can wreak havoc on alerting systems because each of these errors will show in your metrics.

It can also be a problem if you leverage dead letter queues as these failures contribute to the maximum retries per task without even invoking the lambda. Which can cause good messages to be pushed into the dead letter queue due to throughput limitations.

In Conclusion

AWS Lambda is a great service that enables software development with minimal operational overhead, but like all tools it has limits and it is helpful to understand them. I often use AWS Lambda to get services running but find that moving to AWS EC2 based services is often more cost effective and simple in the long term.

Follow me here for more content or contact me on:

Twitter: @BenTorvo
Email: ben@torvo.com.au
Website: torvo.com.au

Git the Parts to Avoid

Ben Brazier — Mon, 27 Jun 2022 06:45:40 +0000

Git is a great tool for managing code changes over time and allowing code to be integrated by teams of developers. But even the most helpful tools can become problematic when misused, and we need to understand these drawbacks in order to avoid them. The most common problems with Git are caused by rewriting history and undoing changes that have already been made.

Lets look at the parts of Git that we should avoid to keep our life as developers simple and easy.

Squash

The standard process for using Git is to clone a repository, make changes locally, then integrate them with the remote server. When a developer has committed multiple changes to a repository and pushes the code to the remote server there will be multiple commits showing in the version history. This is beneficial so that the changes can be tracked but sometimes is seen as confusing or hard to read.

Sometimes developers will use the git squash functionality to convert the multiple commits into a single commit so that a whole change is in one commit. However, this should be avoided for a few reasons.

Firstly, it removes intermediary states, which means that when something goes wrong we are not able to access the commits between the start and the end of the change.

Secondly, it rewrites the history of the changes and we lose track of when the commits were made. This results in less information being available to help identify when changes from different branches were made and when conflicts arose.

Thirdly, it increases the size of changes, which in turn increases risk. It is far easier to deal with frequent small changes as opposed to infrequent large changes.

Instead of using squash it is better to keep the commit history unmodified and present each commit as it was when made. This lets us keep the history of changes, review all of the commits made within a change, and manage the risk by making smaller changes.

Force Push

Force push is a way to overwrite changes on the remote server with our version of the code. But doing this can cause issues for other team members and cause us to lose code changes.

When we force push our changes to the remote server, other developers working on the same code base may end up working off old code. If they pull the latest copy and you subsequently force push over the top of that, their clients will be out of sync. The problem is compounded if they respond by force pushing their own changes, causing the same issue for you.

Forcing the overwrite of remote code can also lose changes that were made since your last pull. This can lead to old bugs coming back, failed builds, and the loss of work.

Summary

Git is a great tool but when misused can cause a lot of issues. As part of using Git we should avoid rewriting history or undoing changes with functionality such as squash and force push. Doing so will help us keep our work simple and efficient.

Follow me here for more content or contact me on:

Twitter: @BenTorvo
Email: ben@torvo.com.au
Website: torvo.com.au

Deadlines Without Estimates

Ben Brazier — Fri, 24 Jun 2022 06:45:39 +0000

In my last article Why Estimates Are Waste we covered some problems with estimates but this raised some questions about managing deadlines. Most companies think that in order to meet deadlines we need to estimate and plan out all of our work. This isn’t the case and today we will cover why.

Large Releases Are Higher Risk

Software delivery in large organisations is often managed by identifying a deadline far into the future, determining what is required for the deadline, then aiming to deliver on or just before that date.

// Detect dark theme var iframe = document.getElementById('tweet-1451360776233623557-665'); if (document.body.className.includes('dark-theme')) { iframe.src = "https://platform.twitter.com/embed/Tweet.html?id=1451360776233623557&theme=dark" }

This leads to a large amount of time spent estimating and planning what can be done and in what order. The risk involved in this planning work should be taken seriously because the further in the future we plan the harder it is to be accurate. And as mentioned previously, estimates are only as valuable as they are accurate.

“there’s no value in wasting time estimating something that you know is going to change, and making long-term plans based on those estimates is foolhardy” — Allen Holub

Having a large amount of time between identifying customers needs and delivering value to them also increases the risk that the solution doesn’t work or isn’t what the customer wants.

So how can we reduce the risk of large releases?

Deliver Frequently

We may have a deadline set for 12 months from now, but what stops us delivering sooner?

Instead of delivering a large solution in 12 months we can deliver a small one in 3 months. This gives us faster feedback and if we are going in the wrong direction we have saved 9 months of development. If things are going well we can continue to deliver every 3 months whilst incorporating feedback we wouldn’t otherwise have.

What if instead of delivering in 3 months we deliver today?

We might not even be able to get a working version today, but how close can we get? If we try and deliver today it becomes clear that we need to only focus on the most important piece of work. This mentally lets us work with priorities instead of estimates.

Prioritisation Over Planning

We don’t need plans, we need priorities. Priorities let us define what is important and trying to deliver a solution today can help guide us.

We need priorities to identify what to work on but we only need enough prioritisation to continue working. Any effort spent not trying to deliver value to our customers as soon as possible is waste and slows us down.

This is why in order to deliver more value faster all we need to do is:

Identify the most valuable task.
Work on that task.
Repeat.

If everyone in your organisation follows this process when delivering software I believe it will be a lot more effective than any other process you’ve seen.

Contact me on:

Twitter: @BenTorvo
Email: ben@torvo.com.au
Website: torvo.com.au

Why Estimates Are Waste

Ben Brazier — Thu, 23 Jun 2022 06:58:27 +0000

Software development is unpredictable but people waste hours every week trying to guess how long tasks will take.

Estimation can provide confidence in future changes, but this doesn’t apply to software development because:

If you can predict work then you should automate it instead of repeating it.
If you can’t predict work then attempting to do so is a waste of time.

Automation Instead of Accurate Estimation

Repetition increases predictability. It isn’t possible to accurately estimate how long a task will take if you have never done it. In other words, the more you repeat a task the more predictable it becomes.

Repeated work should be automated. In software development the ability to re-use libraries, frameworks, and APIs means that repeatable work can be automated. The more you repeat it the more effort would be saved with code re-use and automation.

Accurate Estimates Correlate with Wasted Effort. As repetitions increase the value of automating and the predictability both go up. Therefore …

Inaccurate Estimation is Harmful

If the purpose of estimation is to plan around the future then inaccurate estimates can only lead to problems. This is compounded by the cost of estimating work in time and effort.

// Detect dark theme var iframe = document.getElementById('tweet-1466892889439494147-224'); if (document.body.className.includes('dark-theme')) { iframe.src = "https://platform.twitter.com/embed/Tweet.html?id=1466892889439494147&theme=dark" }

Instead of spending time and effort on estimating work that can’t be predicted or should be automated we should find alternatives. One of the most agile approaches is to work without estimates.

Working Without Estimates

Working without estimates is critical to good software development and goes hand in hand with continuous delivery. The simplest process to work without estimates is:

Identify the most valuable task.
Work on that task.
Repeat.

It is important to consider the time spent on estimating work and the value it provides. This minimal process may be too strict for you but next time you are estimating your work consider the value it provides.

If you would like to discuss this further please contact me on Twitter @BenTorvo or Email ben@torvo.com.au