DEV Community: Paul Mowat

How we moved from Artifactory and saved $200k p.a. Part 5 of 5 - Reaching our goal

Paul Mowat — Wed, 28 Sep 2022 15:39:54 +0000

Introduction

Welcome back to the final part of our 5-part series on 'How we moved from Artifactory and saved $200k p.a'.

If you are just joining we recommend jumping back to the beginning and starting from there.

The deadline

On 19th August 2022 at 21:56 the final migration request was completed, a proud moment. The migration of approximately 1.5 million artefacts had been planned, transferred and verified. We had met our tight deadline.

completing the final migration request

A short while after, on the 31st of August 2022, our subscription with JFrog expired and in turn, all access was revoked. This was the point of no return and the milestone at which we were most apprehensive. The team were ready for the inevitable influx of tickets. Largely (perhaps surprisingly) though, the noise was relatively quiet. Yes, we had some issues, we expected these and had readied ourselves but nothing much arrived. A positive sign!

A pleasant surprise

This, perhaps, provides a good opportunity to let us recall the goal we set for ourselves:

To migrate all requested artefacts from Artifactory without losing any, writing custom tooling as necessary

Approaching 4 weeks since our Artifactory subscription ended and the Advanced Artefacts service went live, post-migration support has consistently been quiet, with zero packages lost to date.

In summary some interesting facts and stats…

To conclude we thought it would be nice to summarise some of the key points from the process:

1.5 million artefacts analysed
24.5 TB of data migrated without losing a file.
222 tickets processed. 215 completed, 7 for phase 2.
18 clinics
- The first clinic had over 130 participants
772 users interacted with through our internal Ms Teams Channel:
- 109 posts
- 590 replies
- 67 mentions
- 115 reactions
Provided dedicated support channels with fast response (typically less than an hour during UK hours)
Days sleeping, thinking and dreaming of artefacts - 90+
14+ hour days - many many many

The numbers

1,200,624 objects in S3
25 TB storage consumed
21.5 MB average object size
15,307.69 EC2 usage hours
2,540 migration jobs run with SSM
888 EC2 spot instances used
56 CloudFormation stacks created
10,539 ECR Authorization tokens requested
5,027 ECR Image pushes
7,968 CloudWatch log streams created
680,263 CloudWatch PutLogEvents called
100% Spot instance utilisation

The costs and savings

The 3-Month spend over the project, including the migration workers = $8400
- Our original budget estimate = $6000
The 3-Month S3 storage costs = $407.05
The 3-Month CodeArtifact costs = $161.19
The 3-Month ECR costs = $115.94
Using that as a basis of calculating our costs for the upcoming year, we estimate savings of $200,000 per annum vs our Artifactory subscription

That’s all folks

We are hugely proud of our efforts and what we have delivered. We would like to thank our engineering teams for their support and engagement. This was a tremendously challenging project, involving complexities, long hours and hard work, but overall it was incredibly fun and rewarding in equal measures.

Thanks for reading! Please return your chairs to an upright position and thanks for flying with Air Advanced Artefacts ;)

How we moved from Artifactory and saved $200k p.a. Part 4 of 5 - Migration

Paul Mowat — Wed, 28 Sep 2022 15:38:55 +0000

Introduction

Welcome back to Part 4 of our 5-part series on 'How we moved from Artifactory and saved $200k p.a'.

If you are just joining we recommend jumping back to the beginning and starting from there.

Migration preparation

The following steps were performed in readiness:

Publish a checklist of steps that teams could follow which would help identify product assets that need migrating
Research how we could append custom metadata to existing artefacts
- tagging
Study the Artifactory REST API and AQL documentation
Research metrics that could be queried for comparison and verification purposes:
- storage summary
- size
- number of artefacts
- number of files
- number of folders
- creation timestamps
- deployment ordering
Analyse the repository path structure and flag outliers
Research publishing options for native CLIs through a lens of using containers to perform the workload
- discovered requirements which ruled out AWS Batch and ECS
- e.g. windows/windows containers
Build a categorised data source to map existing repositories, including:
- dev or release repository
- Advanced Artefacts repository name using new dev/release convention
- flag large repository
- flag empty repository
- flag large numbers of artefacts in the repository
- flag unsupported/problematic repository package types
Consider failure scenarios and recovery option features:
- dry-run
- ability to replay
- debug levels
- resume (using offsets)
- clear progress counters
Log decisions using Architecture Design Records
Review Architecture Design Records

We spent a considerable amount of our time budget on the planning and preparation stages. This served us well.

Checklist

To facilitate the process, we set about creating a preparation checklist that was intended to give all impacted products and teams a clear and concise checklist of preparational steps that would (hopefully) heighten awareness in the appropriate areas. The main points this covered were:

Begin ASAP!
Determine a complete list of impacted builds/artefacts which needed to be supported in production
Review all CI/CD pipelines
Review all runbooks (automated and manual)
Review all production deployments
Review all disaster recovery processes
Identify all artefacts that will need to be migrated through property sets tagging in Artifactory

Tagging

However, before we begin looking at these, we need to cover tagging. Artifactory has a useful feature called Property Sets. We decided early (decision log - check) in the process to make wide use of custom properties through Property Sets. Our requirements were principally:

Artefact hygiene - remove unused or unsupported packages, waste
Docker images underlying OS type - differentiate between Linux and Windows containers

Note: That whilst possible to read a manifest for each Docker image and determine the existence of foreign layers, we were focused on stability and repeatability as well as speed. We wanted to utilise the native tooling as much as possible and felt that determining supported schema/configuration around custom tooling would negatively impact complexity.

Failure recovery

The focus for migrations was specific to our end goals and where appropriate we tried to reuse our tools across different package types. Ultimately, the project as a whole needed to be successful once, with many smaller eventual successes along the way. Building effective tooling was critical to the success of the project. We needed to use software to automate as much as we could whilst also allowing us to clearly aggregate the activities we had performed which in turn could be used as means of verification.

Key features such as dry-run mode, debug levels, and the ability to replay/resume from an offset was essential. The sheer volume of different files, paths, conventions/structures meant that we could only analyse so far before we needed to begin. We fully expected the need to debug activities during the migration phase and that these tools would really support us.

Migration tooling

Large Network bandwidth and scalable compute resources were top of our platform requirements. The choices taken would drip down into the tooling we created.

As an organisation, we place a strong emphasis on Infrastructure as Code (IaC) so creating processes and workflows around AWS Systems Manager using the AWS CDK is expected. We created stacks that enabled us to scale our migration efforts on demand which included:

Migration stack
EC2 Spot Fleet worker node stacks
- Windows
- Linux
SSM Documents library stacks
Custom command line interface tools
- archive runner
- migrator
- migraterunner

Concurrency was an interesting feature. The default was to plan for concurrency, if we run tasks concurrently across our cloud resources then this would accelerate the process, surely? Frustratingly, this was not the case. In the details of AWS CodeArtifact were areas where the order that packages were uploaded is important, particularly npm and Maven. Furthermore, package managers have different ways to determine the latest version, but support for this in AWS CodeArtifact was not universal during our project. We also discovered that there were places where Artifactory supported deprecated features which we happily used, yet AWS CodeArtefact did not. This resulted in us essentially serially migrating packages but utilising the cursor-like features of offset and limits to optimise our efforts.

Migration workers

The migration workers performed the brunt of the migration tasks and were orchestrated through AWS Systems Manager. The (awesome) network throughput enabled by AWS meant we were able to migrate hundreds of GiB of data per hour. This was crucial given that the order that most packages were migrated in was important.

Using property sets that were laboriously set by our engineering teams, the project was able to leverage the excellent Artifactory API as a catalogue/pseudo-state-machine through which, workers could page their way whilst advancing the migration.

In (often long-running) batches, workers would pull packages using the native package managers to the worker’s local storage, then the documented AWS CodeArtifact/ECR/S3 tool was used to push packages to their new location. This is where container images became tricky because pushing and pulling containers needs to be performed on a host running the relevant operating system. Whilst it is documented in the AWS SDK that you pull and push layers as mere blobs, the guidance was that this was only a feature to be used internally which was enough to ward us off.

Command line interfaces (CLIs)

A handful of CLI tools were authored in Go. These tools provided the brunt of the work where validation and custom logic were applied to transfer the different artefact types under the conditions of the project. AWS SSM documents were used in orchestrating the tools to transition the artefacts to the correct destination repositories via the appropriate worker node fleet. Much credit needs to go to the excellent spf13/cobra commander Go module that was used in these CLIs.

Migrating 1.5 million artefacts

Migration process

We wrote our tooling to support a ticket-driven approach where teams could raise tickets to work with the implementation team and migrate their product packages in batches, working together to verify successful completion.

This was important in allowing us to progressively work through the migration rather than a whole big-bang event occurring on an agreed date. Such an approach would be simply unworkable, requiring far too much alignment (thousands of pipelines updated in anticipation). Finding a date that would perfectly fit with hundreds of teams would be virtually impossible and crucially what rollback options would we have? Thus we decided this progressive approach would give us greater flexibility and the reassurance to engage team by team, product by product while checking off milestones on the route to completion.

As things panned out, we ended up with a large backlog of migration activities ranging from a few packages to entire release repositories containing hundreds of thousands of artefacts and terabytes of data. The backlog was continually refined and tickets were resolved step by step with the owning teams. The team had overall responsibility for verifying the migration objective had been met before marking the work as complete.

Resolving all migration tickets in the backlog would culminate in disabling write access for all standard users at a key, predetermined date, before swiftly proceeding, after a reasonable holding period, to remove complete access for all users, except system administrators. Any issues experienced along the way would be addressed in isolation, verifying with the team and making updates to our tooling as required.

Verification

This was tricky. Reporting on the AWS CodeArtifact side is currently poor (09/2022). CloudWatch metrics are entirely missing but ultimately it was not at all straightforward to aggregate key metrics and statistics in the same way you can with Artifactory.

We had to get creative and had to combine metrics queried from the Artifactory REST API with internal counters residing within our tooling to cross-reference our progress. Through deploying webhooks and using DynamoDB to record the dates when batches of work were undertaken we could replay activities using the DateTime offset to determine any deltas where pipelines were not fully updated and artefacts were still being deployed to Artifactory after the migration activity for the solution artefacts had begun. This worked out well and coupled with a reduction of write access permissions, enabled us to verify the process had worked to a point in time whilst giving us another window to replay the same batches but for a much smaller delta of packages. Significantly, as we advanced through the project, confidence grew as we were replaying actions over and over with consistent results. Testing our tooling was tricky so having the ability to utilise dry-run mode was invaluable.

Next up

The migration has now been completed.

Next up, we’ll take a look at whether we’ve reached our goal.

How we moved from Artifactory and saved $200k p.a. Part 3 of 5 - The future is Advanced Artefacts

Paul Mowat — Wed, 28 Sep 2022 15:36:42 +0000

Introduction

Welcome back to Part 3 of our 5-part series on 'How we moved from Artifactory and saved $200k p.a'.

If you are just joining we recommend jumping back to the beginning and starting from there.

Approach

Having identified that we wanted to create a structured service we had to determine the best way to approach it.

Our earlier analysis helped us identify the artefact types that we needed to support. Yet a remaining challenge was to identify how to support these and empower our development teams across the technologies and tools we use on a daily basis.

Architecture

The following architecture gives a high-level overview of the service components.

Repository convention

Something that became apparent was that our Artifactory configuration was a disordered muddle which has since provided us with a harsh lesson about paying particular attention to the rollout of such platform tooling. It had never been implemented in a controlled or consistent way.

Determined to avoid this at all costs we decided to build naming conventions for each of our artefact types into our service. This would be implicit, removing disambiguation and preference from any decisions.

Our products commonly have both development and production environments so it was decided that the service should mirror this and have just two conforming types of repository.

Development repositories would be where teams push build artefacts continually via Continuous Integration (CI). Then, when the appropriate levels of testing had been passed, the artefacts could be promoted into the corresponding release repository.

A benefit of this approach was ensuring a clear separation between development and release dependencies so that we could start to look at implementing automated housekeeping rules in the future. We do not need hundreds of development packages so why bother keeping them?

This helps with our goals of enforcing convention and consistency, which in turn makes it easier to automate and roll out changes in the future.

Infrastructure

We needed to start building out our infrastructure to support the service.

As we were utilising several AWS services, using AWS CDK was the obvious choice. It allowed us to build the service quickly and was also easy to change when required.

Going back to enforcing convention and consistency we leveraged AWS Service Catalog with several custom templates to help us create new repositories.

Delivery

Providing a service that worked successfully meant that we had to look at how we delivered our software and consider what an exemplary software lifecycle looks like, as well as the platforms we needed to support.

The following key areas were identified:

Local Development
Application Configuration
Authorisation
Continuous Integration (CI)
- GitHub Actions
- Jenkins
Continuous Delivery (CD)
- Harness

We also were aware of some products using other technologies such as Azure DevOps and TeamCity that we would not directly support, but still had to take into consideration how they could access and use the service.

Tooling

As developers, we are used to using tools to help make our day-to-day easier. If you look at any good service you will see that they typically have a range of tools to make interacting with them easy.

Command Line Interface (CLI)

We determined that creating a CLI would provide us with a centralised entry point for all of our delivery mechanisms and be flexible enough to allow it to work for any that we didn’t support.

The CLI had to support multiple operating systems (Windows, Linux & macOS) and be easy to update and use as required.

We started analysing what functionality the CLI would require and quickly identified potential overlaps between native package manager and docker commands. There is little point in us trying to write, maintain and support any tooling that mirrored these. Everyone knows how they work, they are industry standard tools.

It was decided that our CLI would work complementary to these. It would bridge the gaps and provide the functionality we needed for our service to work.

We determined our key functional requirements were:

Authorisation
Packages - get and promote
Generic Artefacts - get, list, publish and promote
Container Images - promote

The most important feature of the CLI was authorisation into the service. Every developer and delivery mechanism must authorise into the service before they can use it.

The CLI had to make authorisation easy and limit the impact on our engineering teams on a day-to-day basis.

We looked at other open-source CLIs for inspiration and took the time to understand how this could be done effectively for multiple operating systems, shells and from a user or service perspective.

In the end, we went with a multi-pronged approach and created mechanisms that allowed authorisation in several ways i.e. user, role and service level.

Our security is of the utmost importance and having authorisation tokens written to files was an absolute no-go. Everything by default was going to be applied to the running shell process in order for it to be used and thrown away when finished. That was implemented across several different shells such as bash, Powershell and Windows command prompt.

With our authorisation mechanism now in place, working and flexible enough to handle different operating systems and shells, the other features were much more straightforward.

Generic Artefacts proved the most labour intensive, only due to having to implement an entire set of commands to allow complete artefact management.

Other

With the CLI now in place we used it to power any other tooling that would help accelerate our development teams.

Our core Continuous Integration (CI) platform is GitHub Actions. We decided it was worth the effort to create a custom action that automatically downloaded the latest CLI, installed it and performed the required authorisation. This meant that teams could drop that action straight into their workflows and it would just work. Minimal change, maximum satisfaction.

Next, we looked at Jenkins. Although we are moving away from it, we still have some products still using it, therefore we spent a bit of time putting together some example pipelines on how the CLI could be used and included that in our documentation for teams to follow.

Now we have covered our Continuous Integration (CI) tools, we needed to look at our Continuous Delivery (CD) ones.

Harness is our Continuous Delivery (CD) tool of choice. It provides a flexible template engine, that we were able to utilise to create templates that could be reused across our teams.

Next up

With our new Advanced Artefacts service in place, we were ready to get on with the actual migration from Artifactory.

Next up, we’ll walk through how we built our migration tooling, defined our process and performed the migration.

How we moved from Artifactory and saved $200k p.a. Part 2 of 5 - Design

Paul Mowat — Wed, 28 Sep 2022 15:33:26 +0000

Introduction

Welcome back to Part 2 of our 5-part series on 'How we moved from Artifactory and saved $200k p.a'.

If you are just joining we recommend jumping back to the beginning and starting from there.

Decision making

The nature of larger projects such as these requires plenty of discussion and decision-making around temporary and permanent processes. We had lots of data to migrate and we needed to be efficient in our decision-making process. We decided upon using Architecture Decision Records to log the key implementation decisions which significantly helped us deliver consistency throughout our support and guidance.

As it turned out, undertaking this method of logging was not onerous and we ended up with records for around a dozen key strategic choices that we made; an example of one being the choice to utilise a spot fleet of EC2 workers to perform the migration versus something like AWS Batch or ECS. At first glance, we expected to go with a solution based on AWS Batch or AWS ECS but we had requirements to move resources such as Windows container images and it was so helpful to be able to easily recover the decision steps when we moved to create tooling to support this.

Workshopping

Workshopping commenced on the 10th of June 2022 and we had until the 4th of July 2022 to perform the required analysis, design and implement our solution.

Analysis of requirements

One of the first items of business was to determine which artefact types it was essential to support, those that would be unsupported and any transitions from these to corresponding supported types. Then we would need to determine the options to migrate to, whilst fulfilling the necessary obligations to supported packages and platforms.

Over the past few years, engineering at Advanced has been consolidating its toolchain and programming languages adopted by default. In no way intent on dissuading reviews of new or emerging options, but rather adding consistency in those used and bringing a larger collective intelligence to engineering as a whole.

From analysing our usage within Artifactory we settled upon support for npm, NuGet, generic artefacts (zip, exe, dll etc), Docker images and Maven. We quickly determined that our biggest challenge would be Docker images, accounting for greater than 50% of our consumed storage, with several repositories holding more than 1 TB of image data. Latterly, Maven would also prove challenging.

From this analysis, we were acutely (and financially) aware that we were also wastefully holding onto obsolete build artefacts. We decided to use this as an opportunity to leverage our engineering teams to review and select the versions of artefacts that our products needed to retain. This would help reduce the scale of the migration ahead somewhat and perform some well-overdue housekeeping. After all, there is no point in migrating and paying for artefacts that are no longer required.

Solution analysis

Having gathered an understanding of what needed support and delivery, we had to identify where we were going to migrate to.

AWS is our preferred Cloud Provider and platform, as well as a key technical partner. It was a natural choice to look at their services for our solution. From investigation, we found that AWS CodeArtifact was a decent fit for supporting npm, NuGet, Maven and Python (if required in the future), however, it was not a complete match for all our requirements. Favourably, S3 is an excellent fit for generic artefacts, and Elastic Container Registry (ECR) is perfectly appropriate for Docker images (even leading us to correct misunderstandings between images and repositories internally!).

We now had the artefact types we needed to support at a high level and where they were going to migrate to.

Solution design

Now we firmly knew our direction, we needed to decide how we get there.

Initially, we considered publishing guidance around best practices for various AWS services to satisfy our artefact requirements but ultimately that was deemed unmaintainable.

We wanted to finish the project with our artefact management strategy in a much better position than it started. Significant to us was ensuring we had the ability to define convention, consistency, clear guidance and expectations. We aimed to provide a maintainable solution that continues to build upon the best practices as it matures.

This led us to agree that it was important for the culmination of the migration to result in a new, custom service that any engineering team within Advanced could consume. Advanced Artefacts was born.

We now had two streams we needed to complete within the project:

The Advanced Artefacts service
The Migration

We will get into the detail around these in future posts.

Support channels

As mentioned previously Advanced has over seven hundred engineers from across the globe working on many projects and we needed to identify a strategy for how we could support them in the best way possible.

We came up with the following three-pronged approach.

Documentation

We decided early that we needed to document all parts of the project to allow our engineering teams to self-serve where possible. Without good documentation, there is no way a team of four can support over seven hundred developers.

We focused on providing some getting-started documentation that walked teams through the process in an end-to-end fashion. Then proving the appropriate reference documentation for each step.

This covered items such as the support channels available, each team's responsibilities, the migration preparation and also information on how to use our new Advanced Artefacts service both locally and from our CI/CD pipelines.

A great deal of time was spent pouring over this, it was however crucial to the success of the project.

Clinics

A technique that has worked fairly well for our organisation is the idea of online clinics. We held clinics twice a week for the duration of the project.

We used the first two clinics to kick off the project with our engineering teams. This helped us set timelines around key milestones and clear expectations on what was being delivered.

After that, they were reserved for anyone to drop into, receive updates and ask for assistance directly.

Microsoft teams channel

Microsoft Teams is our internal communication tool, therefore, we created a dedicated channel that we would use for communicating any important updates to the engineering teams.

They could also ask us questions or get further clarification as required outside clinic sessions. The artefacts team committed to replying to the questions as soon as possible ensuring teams were unblocked and able to progress quickly.

Next up

Now we have our design in place we need to start implementing it.

Next up, we will cover the creation of the Advanced Artefacts service.

How we moved from Artifactory and saved $200k p.a. Part 1 of 5 - Planning

Paul Mowat — Wed, 28 Sep 2022 15:31:35 +0000

Introduction

A 5-part blog post by Alex Harrington and Paul Mowat covering the migration of 25 TB of artefacts from JFrog Artifactory to a custom solution we created for Advanced, achieving significant cost efficiency.

A journey

Early in 2022, we decided that Artifactory had become an expensive option for us. Whilst a good product, Artifactory wasn't without difficulties surrounding our subscription. To provide a little more detail, specifically, you are either all in or all out with the JFrog platform, you can only subscribe to every component which is not too desirable at the enterprise level.

In retrospect, we came to learn of significant portions of the JFrog platform (Xray for example) from which we were not getting any real value (for us) and this made the overall service costly. Moreover, we were serious about doubling down on the security of our software supply chain and researching a wider (custom) array of best-in-class solutions.

Still, this was no easy decision as we were a large user of Artifactory with over 1.5 million artefacts published and 25 TB of data storage consumed. Many of our CI/CD pipelines and developer settings were all configured to use Artifactory, so the scale of the task was somewhat sizeable. Still, we proceeded to assess our options and planning the task(s) at hand was critical.

And so it begins

Let’s start by declaring our initial goal:

To migrate all requested artefacts from Artifactory without losing any, writing custom tooling as necessary

The possibility of migrating away from Artifactory was first aired at the beginning of January.

Artifactory comes under a category of services that could be described as quite "sticky". A SaaS solution where the impact of migrating away would reach far and wide within many an organisation.

Advanced has over 150 products active product suites covering different market areas. Some examples are delivering care to 40 million people throughout the UK, sending 10 million sporting fans through turnstiles and supporting 1.2 billion passengers to arrive at their destinations on time. Our solutions are engineered by hundreds of colleagues from across the globe, built using multiple technologies, living in more than 2600 GitHub repositories and powered by thousands of CI/CD pipelines. All deploying to numerous cloud/hybrid-cloud platforms. Not withstanding backup, disaster recovery, escrow and many other internal and market-driven requirements.

We needed to plan, but plan in a way that would allow us to scale.

Team

We formed a small dedicated artefacts team with four members that had to support more than seven hundred engineers through the process. The artefacts team initially needed to design and implement the migration machine, followed by educating and guiding our engineering teams through the project. This had to be as efficient as possible, in order for it to scale.

The artefacts team structure was as follows:

1 x Principal DevOps Architect (Paul Mowat)
1 x Principal DevOps Engineer (Alex Harrington)
1 x Senior DevOps Engineer (Karthik Holikatti)
1 x DevOps Engineer (Likhith Kotian)

Milestones

Our next task was looking at the wider project and breaking it down into key milestones so we could share these. This was a critically important area. Accuracy and clarity in our communication were paramount.

Starting with setting a hard immovable deadline and taking heed from previous hard-learned lessons with sliding deadlines that run and run, we chose to set this date in stone and declared this at the outset of our engagement with our wider engineering community.

We felt this offered a powerful message, which clearly illustrated that urgent engagement was necessary from all sides.

The key milestones were:

10-06-2022 - Project Kick-off - First implementation team workshop held to begin formulating the plan
04-07-2022 - Deadline for our design and implementation complete ready for a wider rollout
05-07-2022 - First Advanced Artefacts support clinic held with over 100 participants
06-07-2022 - Migration Period Start
19-08-2022 - Migration Period End
22-08-2022 - Engineering teams would lose access to Artifactory
31-08-2022 - Project End - Our Artifactory subscription would end

Next up

We have our team and plan.

Next up, we get to work designing our solution.

CloudWatch RUM with Cognito Identity Pool for SAM/Cloudformation

Paul Mowat — Thu, 03 Feb 2022 17:53:52 +0000

Overview

I was asked to look into Amazon CloudWatch RUM and how to implement it for some of our applications at Advanced.

CloudWatch RUM is a service that was released at AWS reInvent 2021. It allows you to configure your web application to perform real-user monitoring. It can also be configured to collect additional data, such as performance, errors and HTTP requests. It also works with AWS XRay, which allows it to track client to server traces. Overall it looks like a helpful tool that will aid analysis and debugging.

The cost is $1 for every 100k of events that are collected. Every data item collected by the RUM web client is considered an event e.g. page view, JavaScript or HTTP errors. See AWS CloudWatch Pricing for detailed information.

The guide will take you through getting CloudWatch RUM deployed and hooked into your application.

Requirements

To follow along, you will need.

AWS Account
AWS SAM CLI
Web application you want to do real-user monitoring on

CloudWatch RUM & Cognito Deployment

The SAM/Cloudformation template will deploy the CloudWatch RUM application monitor along with a Cognito Identity Pool. The Identity Pool is configured to allow unauthorized access to the CloudWatch RUM web client. This is required for the CloudWatch RUM web client to send the events back to the CloudWatch RUM service.

To deploy, follow the below steps.

Copy the below template into a file called template.yml

AWSTemplateFormatVersion: "2010-09-09"
Transform: AWS::Serverless-2016-10-31
Description: >
  Setup Cloudwatch RUM using Cognito IdentityPool for specified application and domain
Parameters:
  ApplicationName:
    Description: The name of the service
    Type: String
  ApplicationDomain:
    Description: The top-level internet domain name
    Type: String

Resources:
  CWRumIdentityPool:
    Type: AWS::Cognito::IdentityPool
    Properties: 
      IdentityPoolName: !Ref ApplicationName
      AllowUnauthenticatedIdentities: true

  CWRumIdentityPoolRoles:
    Type: AWS::Cognito::IdentityPoolRoleAttachment
    Properties:
      IdentityPoolId: !Ref CWRumIdentityPool
      Roles:
        unauthenticated: !GetAtt CWRumClientRole.Arn

  CWRumClientRole:
    Type: AWS::IAM::Role
    Properties:
      AssumeRolePolicyDocument:
        Version: 2012-10-17
        Statement:
          - Effect: Allow
            Principal:
              Federated:
                - cognito-identity.amazonaws.com
            Action:
              - sts:AssumeRoleWithWebIdentity
            Condition:
              StringEquals:
                cognito-identity.amazonaws.com:aud: !Ref CWRumIdentityPool
              ForAnyValue:StringLike: 
                cognito-identity.amazonaws.com:amr: unauthenticated
      Description: Unauthenticated Role for AWS RUM Clients
      Path: /
      Policies:
        - PolicyName: AWSRumClientPut
          PolicyDocument:
            Version: "2012-10-17"
            Statement:
              - Effect: Allow
                Action:
                  - "rum:PutRumEvents"
                Resource: !Sub arn:aws:rum:${AWS::Region}:${AWS::AccountId}:appmonitor/${ApplicationName}

  CWRumAppMonitor:
    Type: AWS::RUM::AppMonitor
    Properties:
      AppMonitorConfiguration:
        AllowCookies: false
        EnableXRay: false
        IdentityPoolId: !Ref CWRumIdentityPool
        GuestRoleArn: !GetAtt CWRumClientRole.Arn
        SessionSampleRate: 0.1
        Telemetries:
          - errors
          - performance
          - http
      Domain: !Ref ApplicationDomain
      Name: !Ref ApplicationName

Outputs:
  CWRumAppMonitor:
    Description: The Cloud Watch RUM App Monitor Name
    Value: !Ref CWRumAppMonitor

Open up the command line and navigate to the folder you saved the above template.yml into
From the command line, use AWS SAM to deploy the template.yml file

sam deploy --guided

During the prompts

Enter a stack name
Enter the desired AWS Region
Enter the Application Name
Enter the Application Domain
Allow SAM CLI to create IAM roles with the required permissions

Once the deployment has finished, you will have a CloudWatch RUM application monitor.

Application Configuration

The next step is to configure your application to connect to the CloudWatch RUM application monitor.

Get JavaScript Snippet from AWS Console

The JavaScript snippet is specific to the CloudWatch RUM application monitor and is used by your application to inject in the CloudWatch RUM web client. This is what captures and sends back events to the service.

Login to the AWS Console
Navigate to CloudWatch RUM

Identify your application monitor and click the View JavaScript snippet link to show the snippet

Click Copy to clipboard to copy the Javascript snippet

Modify Snippet Configuration

If required, you can modify the code snippet to configure the CloudWatch RUM web client with additional options.

See the CloudWatch RUM Modify Snippet documentation for further information on the additional options.

From those options, I've decided to configure CloudWatch RUM as follows.

Enables cookies to track user and session details
Enables XRay traces
Sample all sessions
Collect telemetry events for errors, performance and HTTP requests
- Adds the X-Amzn-Trace-Id header to the HTTP requests to allow client to server tracing
- Record all requests i.e. not just errors
- Only track HTTP requests including my domain using absolute and relative path

The snippet below shows the above configuration in action as an example.

Note: The body of the snippet's function has been omitted for readability.

<script>
  (function(n,i,v,r,s,c,u,x,z){...})(
    'cwr',
    '00000000-0000-0000-0000-000000000000',
    '1.0.0',
    'us-west-2',
    'https://client.rum.us-east-1.amazonaws.com/1.0.2/cwr.js',
      {
      allowCookies: true,
      enableXRay: true,
      sessionSampleRate: 1,
      guestRoleArn:'arn:aws:iam::000000000000:role/RUM-Monitor-us-west-2-000000000000-00xx-Unauth',
      identityPoolId:'us-west-2:00000000-0000-0000-0000-000000000000',
      endpoint: "https://dataplane.rum.us-east-1.amazonaws.com",
      telemetries: [
        "errors",
        "performance",
        [
          'http',
          {
            addXRayTraceIdHeader: true,
            recordAllRequests: true,
            urlsToInclude: [
              /^https:\/\/www\.paulmowat\.co\.uk\/.*/,
              /^(?!www\.|(?:http|ftp)s?:\/\/|[A-Za-z]:\\|\/\/).*/
            ]
          }
        ]
      ]
    }
  );
</script>

Insert & Deploy Snippet

Now that we have the snippet configured, we can insert it into the web application code. It needs to be inserted within the <head> element, above any other <script> tags.

The below example shows where to add it to an HTML page.

<!doctype html>
<html lang="en">
<head>
  <script>
    // snippet goes here
  </script>
  ...
</head>
<body>
  ...
</body>

See the below page on the CloudWatch RUM web client documentation for specific frameworks.

Now that we've got the CloudWatch RUM web client inserted. Let's get the application redeployed.

Verifying it works

Everything should now be configured and deployed. Let's verify it's all working as expected.

Navigate to your application
Interact with your application to generate events e.g. change pages
Look at the AWS CloudWatch RUM console for the application monitor. You should be able to see the last updated time and also see data appear as below

Wrap-up

You should have CloudWatch RUM deployed, your web application configured and now be able to see the data on the AWS console.

CloudWatch RUM looks like a good addition to the CloudWatch ecosystem and is very easy to get up and running with minimal changes to your application required.

You can find the code from this blog at the following locations.

Additional Resources

How to configure SonarLint to connect to SonarQube for VS Code

Paul Mowat — Sun, 23 Jan 2022 15:55:12 +0000

Overview

At Advanced we use SonarQube as our static code analysis tool.

SonarQube is an open-source tool that can scan and identify problems with code quality across several different technology stacks.

SonarQube analysis is typically built directly into our CI/CD processes and provides feedback to the teams as the build is happening.

This provides good information however, the feedback loop can be too slow for rapid development.

One of our current focuses is to improve the feedback loop for developers.

Fortunately, when it comes to SonarQube, they also provide SonarLint, which can be configured directly into your IDE to give that true shift-left mentality.

We use VS Code as our editor of choice across several projects now, so will cover how to get up and running below.

Requirements

To get started you will need

A SonarQube server
VS Code installed

Installation & Global Configuration

Open up VS Code and under extensions install SonarLint

Once installed, restart or reload VS Code to ensure it's taken effect.

If SonarLint can't detect a JAVA JRE on your system, it will prompt you to download one. Let it download if required.

Once VS Code is up and running again, hit Ctrl + Shift + P to open the command palette. Then enter Preferences: Open Settings (JSON) and select to open up your settings.

To get SonarLint working, you need to specify the following settings.

"sonarlint.connectedMode.connections.sonarqube": [
    { 
        "connectionId": "sonar", // id of your sonarqube server
        "serverUrl": "https://sonarqube-server.com/", // url of your sonarqube server
        "token": "XXXX" // token to authenticate with sonarqube
    }

],
"sonarlint.pathToNodeExecutable": "C:\\\\Program Files\\\\nodejs\\\\node.exe", // path to your node.js installation if analyzing Javascript/Typescript

To generate the token, you will need to:

Login to your SonarQube server
Click on your profile picture on the top right-hand side of the page and select My Account

Next select Security, specify a token name and hit generate

Your token will be displayed

Copy your token and paste this into the above token setting in VS Code

SonarLint is now configured globally within VS Code to access SonarQube via the specified connectionId.

Workspace Configuration

Next, we need to configure your project workspace to allow it to scan the appropriate SonarQube project.

Go back onto your SonarQube server and grab the project key.

This can be found on the project page on the bottom right of the page.

In VSCode hit Ctrl + Shift + P to open the command palette. Then enter Preferences: Open Workspace Settings (JSON) and select to open up your workspace settings.

To get SonarLint working, you need to specify the following settings.

{
  "sonarlint.connectedMode.project": {
    "connectionId": "sonar", // should be same connectionId you defined above
    "projectKey": "XXXX" // Replace with project key you grabbed from SonarQube server
  }
}

We now need to update the SonarLint bindings for the workspace to ensure the rules are in-sync locally and on the server.

Again, hit Ctrl + Shift + P to open the command palette. Then enter SonarLint: Update all bindings to SonarQube/SonarCloud and select. You should see the following message on the bottom right of VS Code once complete.

The project should now be connected and configured.

Verifying

You can verify this by opening up a file that has some problems.

These will now be highlighted within your code:

With an underline that shows a popup of the issue when hovered over

Within the VS Code problems panel

You have achieved true shift-left and now have a fast feedback loop allowing you to resolve any SonarQube issues as you code.

How to resolve GH006 Protected Branch Update Failed

Paul Mowat — Wed, 03 Nov 2021 19:03:46 +0000

Overview

At work, we've used Github protected branches for quite a while now. They allow us to enforce rules that developers must follow before their branches can be committed.

We typically set up the following:

Require a pull request before merging.
Require status checks to pass before merging.
Require the branch to be up to date before merging.
Require linear history.
Restrict who can push to matching branches.

These rules help us maintain quality by ensuring all branches meet the required criteria.

We started to use Rush as our monorepo management tool a while back. It's a fantastic tool and helps us in many ways to make our development process simpler.

As part of its usage, we use the rush change functionality, which allows our developers to create change logs for their PRs. These are then identified by the build pipeline and automatically increment the package versions.

To do this Rush creates a version bump branch and merges it back in as part of the build pipeline. This process works fine when branch protection is disabled.

The Problem

However, when branch protection is enabled, you get a lovely error like the following.

error: GH006: Protected branch update failed for refs/heads/main
error: Required status check "Build" is expected. At least 1 approving review is required by reviewers with write access

This is a problem! We want to use protected branches and want to be able to use rush change to control versions. So how can we fix it?

I assumed this would be an easy fix and just a case of giving the Github action extra permissions to push to the protected branch. Wrong!

After spending a bit of time Googling, I came across a Github community post @ https://github.community/t/allowing-github-actions-bot-to-push-to-protected-branch/16536. The TLDR is that Github can't make the change to fix it for security reasons.

At that point in time, we were busy with deadlines and had to decide quickly on how to progress. That decision was to temporarily disable branch protection and look at it later, even though it felt wrong. The issue was put in the backlog to be looked at later.

Solution

Fast forward to last week, and I finally got some time to investigate further. There is, unfortunately, no movement from GitHub's side, but I managed to track down a workaround.

The workaround is as follows:

Create a new Github user specifically for building.
Create a new personal access token for that user with access to repo.
Add the personal access token as a Github secret e.g. BUILD_SVC_PAT.
Update your branch protection and add your new build user to 'Restrict who can push to matching branches'.
Update your Github action to check out the code using the Github secret.

jobs:
  build:
    runs-on: ubuntu-latest
    steps:
      - name: Checking out...
        uses: actions/checkout@v2
        with:
          token: ${{ secrets.BUILD_SVC_PAT }}

This will let you configure branch protection and let the rush change version bump branch be committed.

Now, this workaround isn't perfect! You are creating an admin user who can do the build and allowing that user to push to the protected branch.

This means you can't set the Include Administrators branch protection rule, therefore your other admins can still push directly and bypass branch protection. However, it does stop any other non-admin developers from being able to push directly to the branch. For our needs, it was a suitable solution.

Move your AWS Lambdas to Graviton2 easily with Cloudformation/SAM

Paul Mowat — Mon, 04 Oct 2021 18:53:03 +0000

Yesterday, Amazon Web Services (AWS) shared AWS Lambda Functions Powered By Graviton2 Processor.

At a high-level AWS claim:

Up to 19% better performance
At 20% lower cost
Also applies to functions using provisioned concurrency.

All supported Lambda runtimes, including the custom runtime are supported. You just need to be careful with any binaries or functions packaged as containers to ensure they are built to support the architecture.

To help with performance testing, you can create two versions of a function, one in x86 and one for Arm. You can then use an alias with appropriate weights to distribute traffic between them. Once your test is complete, you can compare the performance difference within CloudWatch.

When migrating from x86 to Arm in production, you can use the same function version and weighted alias approach. This will allow you to slowly start ramping up, e.g. from 1%, gradually up to 100%. If something looks wrong, or you are experiencing errors, then adjust the weights down to zero to force traffic back to the x86 function.

Everything I deploy to AWS is via Cloudformation or AWS SAM. Below we will look at how to configure those to use the new architecture.

Update CLIs

If you use a CLI tool to deploy, it is probably a good idea to update to the latest one for your setup. That will ensure it's compatible with the new changes.

Template Changes

The example below shows the changes required for an AWS::Lambda::Function and an AWS::Lambda::LayerVersion. The changes are the same if you are using an AWS::Serverless::Function.

exampleLambdaFunction:
  Type: AWS::Lambda::Function
  Properties:
    # OTHER ITEMS REMOVED FOR EXAMPLE
    Layers:
      - !Ref exampleLibrary
    Architectures:
      - arm64

exampleLibrary:
  Type: AWS::Lambda::LayerVersion
  Properties:
    # OTHER ITEMS REMOVED FOR EXAMPLE
    CompatibleArchitectures:
      - arm64

Now the change is made, you are ready to deploy using your preferred mechanism, whether thats CLI or via the console.

Summary

There is really no reason not to try out the changes. They perform better, cost less, and the changes are minimal. There are also mechanisms in place by using the function versions and alias to do a gradual rollout until you have the confidence to switch over fully.

My thoughts on working from home

Paul Mowat — Mon, 04 Oct 2021 17:59:18 +0000

Four years ago, I got the opportunity to work from home full time. I had never worked from home more than the odd day here or there before and wasn't sure what to expect. This was before the world had to do it during the Coronavirus pandemic, and information on how to be effective was a bit more scarce. I decided to go for it and jump in, below are the key takeaways.

Work environment

I had just recently moved house and was lucky to have a spare room that could be turned into an office.

I created a comfortable setup with the following:

Large Desk
Comfortable chair
Whiteboard
Three monitor setup
Webcam
Headset
All in one Printer/Scanner

This was everything I needed to get going. I know that not everyone can set up a dedicated space. However, I felt it was essential for me if this was going to work.

Takeaway - I'm glad I spent some time and money getting myself setup to be comfortable and effective work environment. My office lets me focus fully on my work. I've recently upgraded a few items to make it even better, more storage, a new webcam, microphone and better speakers. I'm now able to have video calls without the need of a headset, which is great.

Effective Communication

At work, we used several different communication channels. Sending emails was always the primary one and is used heavily. We also had Skype for Business for speaking to each other and screen sharing when required.

To begin with, there were challenges being the remote member of a team. During calls where everyone else was in an office, it was hard to hear, follow along and interact as well as possible.

At the beginning of the pandemic, the entire company moved to use Microsoft Teams. I noticed a significant improvement instantly. The sound and video quality were better, and since everyone was working from home there was more acknowledgement around better communication.

We've evolved along the way and now have an accepted meeting etiquette. Video calls are also used much more than before to try and add that personal element.

Takeaway - This has been a difficult one. We all need to stay in touch and it's went from few meetings to lots of meetings due to the pandemic. It's now starting to calm down which is great. A good suggestion is to look at how you can do async communication more. Always think, do you need that extra meeting? Could I just have a group chat instead?

Focus

Being able to focus on work while at home is something that concerns a lot of people. For me, though this was never a concern. I love what I do and enjoy coding and building new applications.

As someone who happily spends hours coding, I found my productivity levels increased drastically due to fewer interruptions and background office noise. I could turn on Spotify and get on with it.

During my first two years working from home, I was the lead developer for two brand new applications. These were built using entirely new technology stacks to me. I loved the challenge of it!

Once the pandemic started, things began to change as everyone was trying, to understand how to work effectively. There was a large increase in meetings being scheduled. All the meetings were starting to impact our developers getting into the Zone and being as productive as possible. We've since introduced blocked focus time which has made a significant difference.

Takeaway - It is ok to indicate to others that you need time to do your work. Block that time in your calendar as Focus Time. Outlook has a great feature to help automate this now. Constant distractions have a large impact on productivity but also on morale/stress levels.

Work/Life Balance

If you work where you live, how do you separate them? It's a difficult question.

The first few years working at home and learning new technologies, I struggled with the balance. There were deadlines to meet and a lot of learning to do, which meant quite a few midnight sessions when I got stuck to learn how to unblock myself. Not ideal, but needs must.

Over the last two years, I've been much better. I tend to work from 8 am to 5 pm and stop. My son, coming in every day at 5 pm and shouting "It's dinner time, Daddy", signals the end of the day for me.

Takeaway - If you focus on one, more than the other you are going to have a problem in the long run. Stay in control of your work by planning your day. Create a to-do list, book out your lunchtime in your calendar. Don't answer calls or emails after work hours. You need that separation to maintain your health.

Family

The single best benefit of working at home is getting to spend extra time with your family. I'm married and have two sons, one five years old and the other eight months old.

I started to work from home when my firstborn was six months old. I'm lucky in that I've got to see all the key milestones. Today, for example, I got to see my eight-month-old clap his hands for the first time.

I also get to have breakfast, lunch and dinner with them every day. I don't have to waste hours in a commute back and forth to the office.

Working at home also gives you that bit of flexibility that's handy when you have kids.

Takeaway - Spend as much time with your family as possible. They appreciate it. It's the little things like being able to go for lunch together at the local cafe or go for a walk together at lunchtime. Make the best of it!

DEV Community: Paul Mowat

How we moved from Artifactory and saved $200k p.a. Part 5 of 5 - Reaching our goal

Introduction

The deadline

A pleasant surprise

In summary some interesting facts and stats…

The numbers

The costs and savings

That’s all folks

How we moved from Artifactory and saved $200k p.a. Part 4 of 5 - Migration

Introduction

Migration preparation

Checklist

Tagging

Failure recovery

Migration tooling

Migration workers

Command line interfaces (CLIs)

Migrating 1.5 million artefacts

Migration process

Verification

Next up

How we moved from Artifactory and saved $200k p.a. Part 3 of 5 - The future is Advanced Artefacts

Introduction

Approach

Architecture

Repository convention

Infrastructure

Delivery

Tooling

Command Line Interface (CLI)

Other

Next up

How we moved from Artifactory and saved $200k p.a. Part 2 of 5 - Design

Introduction

Decision making

Workshopping

Analysis of requirements

Solution analysis

Solution design

Support channels

Documentation

Clinics

Microsoft teams channel

Next up

How we moved from Artifactory and saved $200k p.a. Part 1 of 5 - Planning

Introduction

A journey

And so it begins

Team

Milestones

Next up

CloudWatch RUM with Cognito Identity Pool for SAM/Cloudformation

Overview

Requirements

CloudWatch RUM & Cognito Deployment

Application Configuration

Get JavaScript Snippet from AWS Console

Modify Snippet Configuration

Insert & Deploy Snippet

Verifying it works

Wrap-up

Additional Resources

How to configure SonarLint to connect to SonarQube for VS Code

Overview

Requirements

Installation & Global Configuration

Workspace Configuration

Verifying

How to resolve GH006 Protected Branch Update Failed

Overview

The Problem

Solution

Move your AWS Lambdas to Graviton2 easily with Cloudformation/SAM

Update CLIs

Template Changes

Summary

Further reading

My thoughts on working from home

Work environment