DEV Community: OneAdvanced

API provider contract testing for all with Portman, OpenAPI and Postman

Alex Savage — Wed, 10 May 2023 15:08:00 +0000

Introduction

APIs are everywhere and power almost everything we use today.
OpenAPI has revolutionized the way APIs are designed, documented, deployed... Its kind of a big thing.

Making a great OpenAPI definition is helped with amazing tools such as Stoplight Spectral which can help you ensure that your definition is not only valid against the OpenAPI specification but also your own API standards through custom linting rules.

How do you ensure that what you create and deploy matches your definition? Do you write all your tests by hand? Do you probe with Postman and hope for the best?

Maybe you do today but this blog will help you try something new... Welcome to the world of contract testing.

By the end of this blog you will have completed your first contract test (against my mocked API) and you should be free to use this knowledge to find gaps in your or any API out there subject to permission!

Getting started

Rather than spending too long explaining what contract testers are, where they came from etc... Lets use one! (It's free so what is stopping you?)

We will be using apideck Portman. You are welcome to follow their readme.MD and come back later or follow the instructions here.

Installing Portman

Install Portman

$ npm install -g @apideck/portman

Initialize your CLI

portman --init

Creating a contract test to run in Postman

Make a new directory and download this example portman configuration file and store it there
https://api.oneadvanced.com/portman/configurations/portman-config.adv.json

Download this openAPI definition from SwaggerHub (the unresolved YAML works fine)
Link to SwaggerHub: https://app.swaggerhub.com/apis/AdvancedComputerSoft/Contract-Breaker/1.0.0
Direct link to YAML: https://api.swaggerhub.com/apis/AdvancedComputerSoft/Contract-Breaker/1.0.0

Open the directory that has both files and run the following:

portman -c portman-config.adv.json -l AdvancedComputerSoft-Contract-Breaker-1.0.0-swagger.yaml

Portman will use the configuration file to port the openAPI definition to a Postman collection you can now import into Postman and start running tests!

Run the test

Import the Postman collection file from the tmp/converted folder that Portman created as part of the process.

Open the List Countries request and press send!

Congratulations! You ran a test and you got some failed tests. (Hopefully)
Don't worry, that was supposed to happen.

What happened?

Lets go back and look at the collection.
At first glance, it looks like any other collection. Check the tests tab and you should see something new.

This is where Portman has done its magic and created contract tests for you to run against the API. The OpenAPI definition was used to generate these tests.

You can see Postman will now test to ensure:

The status code matches 200
The response is application/json
The response has a body which is an object
The response body validates against the schema from the OpenAPI
Bonus - The response is received within 2000ms (This is not taken from the OpenAPI definition but from the Portman configuration

All of this and you did not need to write anything. No human error, no missed tests. All of it was auto-generated.

Interpreting the responses

Postman has reported that 3 out of 5 tests have failed:

1) [GET]::/countries - Response status code is 200 | AssertionError: expected 201 to equal 200

This error was raised by the status code check as the API incorrectly returned a 201 response code instead of a 200 which it was expecting

2) [GET]::/countries - Schema is valid | AssertionError: expected data to satisfy schema but found following errors: data.data[0] should have required property 'countryId', data.data[0].population should be integer, data.data[1] should NOT have additional properties, data.data[1].countryId should be string, data.data[1].population should be integer, data.pagination should NOT have additional properties, data.pagination.offset should be integer

This error was contains all the json schema validation issues. There are missing required properties, wrong types and additionProperties (these are caused by misspelled properties or those that were not documented)

What you should find is this isn't an API that you would want to release! Contract testing lets you find where your deployed API (or your one that you are developing) doesn't match the OpenAPI definition. Catch things now before you release!

More tests

Portman has lots of other functionality to explore.

I would recommend to first look at overrides to try manipulating the collection. I started by making tests that removed the security and tried to call the API. You can stipulate which response from the OpenAPI you wish to receive. I was expecting a 401 to be returned. The Portman documentation has details of all of the functionality on offer.

Summary

APIs need to match their documentation. Whether or not you start with Design in OpenAPI or start with code, contract testing can ensure the 2 are in sync so your consumers don't get surprises and raise issues.

Standard visual modelling languages

Joe Wallace — Fri, 18 Nov 2022 17:35:08 +0000

Within the profession of Business Analysis, a key function is for the Analyst to be able to communicate ideas effectively, in a way that an intended audience is easily able to understand.

One way of doing this is visually modelling a process or system; a large benefit of this is to be able to distil complex concepts down into one easy-to-digest visual. Additionally, it’s easy to iterate on a model based on feedback, and new requirements coming to light.

In order to help Business Analysts, various standardised modelling languages have been created. In this article, we’ll focus on two of the most common ones: Unified Modelling Language (UML) and Business Process Modelling Notation (BPMN). When should you use them, and what do they consist of?

Unified Modelling Language 🌐

UML is an object-oriented modelling language, intended for use in a broad range of different domains, architectures and coding languages. It is focussed on the design of IT systems specifically, and as such can be used for software or architectural designs.

What is UML good for?

Interactions within an IT system or between IT systems.

Determining objects used by a system, and the data items within them.

Communicating architectural designs to stakeholders with technical expertise.

Automated generation of code and/or test cases.

There are two different types of UML diagrams:

Structure diagrams

These demonstrate the objects and components within an IT system and how they relate to each other. Various types of structure diagrams are available, including:

Class diagrams
Denote object classes within the system, and the relationships between them.

Object diagrams
Similar to class diagrams, but show how the user of a system might see the objects within it.

Composite structure diagrams
Also similar to class diagrams, but also show the composite parts within classes.

Component diagrams
Depict how system components hang together and depend on each other.

Deployment diagrams
Denote the distribution order and configuration required when deploying an IT system.

Package diagrams
Model how packages in the system fit into the overall application model and their dependencies.

Profile diagrams
Allow you to document prototype data structures using “stereotypes” - domain-specific data models which can be used for more formal class or object diagrams later.

Example

Example class diagram on Diagrams.net

Behaviour diagrams

These demonstrate dynamic changes which may occur within an IT system as a result of user or automated triggers. Many different behaviour diagrams exist, including:

Use case diagrams
Describe how users and systems functionally interact with an IT system, in a series of steps.

Activity diagrams
Graphical representations of the workflow through a system, organised into swim lanes.

Interaction overview diagrams
Similar to activity diagrams, but with a focus on interactions over workflow.

Sequence diagrams
Models dependencies in the flow of an IT system in sequence.

Communication diagrams
Similar to sequence diagrams, but more focussed on how processes interact than on the time sequence.

Timing diagrams
Also similar to sequence diagrams, but with an emphasis on the amount of time each process takes.

State diagrams
Show which state transitions are permitted and how they are triggered.

Example

Example use case diagram on Diagrams.net

Business Process Modelling Notation 🔁

BPMN is a process-oriented modelling language, intended to represent the full business process supporting an IT system. It is based on flow charts, and likely to cover areas outside of the scope of a software or architectural change being investigated or proposed.

What is BPMN good for?

Mapping out business processes underpinned by an IT system.

Determining the different scenarios that may occur, and how a system should behave in each scenario.

Communicating system behaviour to all business stakeholders, regardless of technical knowledge.

BPMN has four main components:

Flow objects
These can be events (triggers for the beginning or end of a flow), activities (processes or sub-processes undertaken by a user or system within the flow), or gateways (decision points where the flow can branch off in multiple directions).
Connecting objects
These can be flows of activities or messages (denoted by a solid or dashed arrow, respectively) or associations (links between an artefact and a flow object).
Swimlanes
These can exist on two levels: Pools (groupings of roles such as a function or organisation) and, within them, lanes (individual roles within the process).
Artefacts
These can be data objects (information required for a process), groups (which link together activities in a process), or annotations (textual descriptions of to give additional context).

Example

Example business process flow diagram on Diagrams.net

Epics: Purpose, structure, and content

Joe Wallace — Mon, 07 Nov 2022 16:36:29 +0000

✅ Epic User Stories, often called “Epics”, are simply User Stories which are too large to fit into an individual development timebox (“Sprint”, in the Scrum methodology).

❌ Epic User Stories are not themes which group together individual User Stories, new product features such a modules, or buckets of time to collect recorded time against.

The purpose of an Epic 🎞

The purpose of an Epic is similar to a typical User Story - an informal description of a user’s requirement, written in natural language relevant to the business domain. However, an Epic is less constrained by development capacity or process, so may describe a requirement a little broader and looser than required in a typical User Story.

The structure and content of an Epic 🧬

Epic User Stories, much like any other User Story, typically begin with a card describing the user’s problem, such as the “Connextra” template:

As a user persona
I want capability
So that business benefit

They will also have acceptance criteria like a typical User Story, to inform the test cases that are run to verify that the conditions to fulfil the user story have been met, as in the “Gherkin” template:

Given pre-condition
When user action
Then outcome

User Story mapping 📌

A process known as User Story mapping is often used by software development teams to sketch out a user’s journey through the system, break that down into Epics, and then break the (must have) Epics within it down into User Stories (of any priority).

See an example below for a holiday story map. Note that this might differ based on the scenario at hand and the needs of the customer. For example, if the customer had experience operating a boat, they might rent their own, and then a captain is not a must have!

Example story map on Diagrams.net

GitHub Reusable Workflows and Custom Actions

Saurabh Shah — Thu, 06 Oct 2022 09:47:42 +0000

Liquid syntax error: Unknown tag 'endraw'

How we moved from Artifactory and saved $200k p.a. Part 5 of 5 - Reaching our goal

Paul Mowat — Wed, 28 Sep 2022 15:39:54 +0000

Introduction

Welcome back to the final part of our 5-part series on 'How we moved from Artifactory and saved $200k p.a'.

If you are just joining we recommend jumping back to the beginning and starting from there.

The deadline

On 19th August 2022 at 21:56 the final migration request was completed, a proud moment. The migration of approximately 1.5 million artefacts had been planned, transferred and verified. We had met our tight deadline.

completing the final migration request

A short while after, on the 31st of August 2022, our subscription with JFrog expired and in turn, all access was revoked. This was the point of no return and the milestone at which we were most apprehensive. The team were ready for the inevitable influx of tickets. Largely (perhaps surprisingly) though, the noise was relatively quiet. Yes, we had some issues, we expected these and had readied ourselves but nothing much arrived. A positive sign!

A pleasant surprise

This, perhaps, provides a good opportunity to let us recall the goal we set for ourselves:

To migrate all requested artefacts from Artifactory without losing any, writing custom tooling as necessary

Approaching 4 weeks since our Artifactory subscription ended and the Advanced Artefacts service went live, post-migration support has consistently been quiet, with zero packages lost to date.

In summary some interesting facts and stats…

To conclude we thought it would be nice to summarise some of the key points from the process:

1.5 million artefacts analysed
24.5 TB of data migrated without losing a file.
222 tickets processed. 215 completed, 7 for phase 2.
18 clinics
- The first clinic had over 130 participants
772 users interacted with through our internal Ms Teams Channel:
- 109 posts
- 590 replies
- 67 mentions
- 115 reactions
Provided dedicated support channels with fast response (typically less than an hour during UK hours)
Days sleeping, thinking and dreaming of artefacts - 90+
14+ hour days - many many many

The numbers

1,200,624 objects in S3
25 TB storage consumed
21.5 MB average object size
15,307.69 EC2 usage hours
2,540 migration jobs run with SSM
888 EC2 spot instances used
56 CloudFormation stacks created
10,539 ECR Authorization tokens requested
5,027 ECR Image pushes
7,968 CloudWatch log streams created
680,263 CloudWatch PutLogEvents called
100% Spot instance utilisation

The costs and savings

The 3-Month spend over the project, including the migration workers = $8400
- Our original budget estimate = $6000
The 3-Month S3 storage costs = $407.05
The 3-Month CodeArtifact costs = $161.19
The 3-Month ECR costs = $115.94
Using that as a basis of calculating our costs for the upcoming year, we estimate savings of $200,000 per annum vs our Artifactory subscription

That’s all folks

We are hugely proud of our efforts and what we have delivered. We would like to thank our engineering teams for their support and engagement. This was a tremendously challenging project, involving complexities, long hours and hard work, but overall it was incredibly fun and rewarding in equal measures.

Thanks for reading! Please return your chairs to an upright position and thanks for flying with Air Advanced Artefacts ;)

How we moved from Artifactory and saved $200k p.a. Part 4 of 5 - Migration

Paul Mowat — Wed, 28 Sep 2022 15:38:55 +0000

Introduction

Welcome back to Part 4 of our 5-part series on 'How we moved from Artifactory and saved $200k p.a'.

If you are just joining we recommend jumping back to the beginning and starting from there.

Migration preparation

The following steps were performed in readiness:

Publish a checklist of steps that teams could follow which would help identify product assets that need migrating
Research how we could append custom metadata to existing artefacts
- tagging
Study the Artifactory REST API and AQL documentation
Research metrics that could be queried for comparison and verification purposes:
- storage summary
- size
- number of artefacts
- number of files
- number of folders
- creation timestamps
- deployment ordering
Analyse the repository path structure and flag outliers
Research publishing options for native CLIs through a lens of using containers to perform the workload
- discovered requirements which ruled out AWS Batch and ECS
- e.g. windows/windows containers
Build a categorised data source to map existing repositories, including:
- dev or release repository
- Advanced Artefacts repository name using new dev/release convention
- flag large repository
- flag empty repository
- flag large numbers of artefacts in the repository
- flag unsupported/problematic repository package types
Consider failure scenarios and recovery option features:
- dry-run
- ability to replay
- debug levels
- resume (using offsets)
- clear progress counters
Log decisions using Architecture Design Records
Review Architecture Design Records

We spent a considerable amount of our time budget on the planning and preparation stages. This served us well.

Checklist

To facilitate the process, we set about creating a preparation checklist that was intended to give all impacted products and teams a clear and concise checklist of preparational steps that would (hopefully) heighten awareness in the appropriate areas. The main points this covered were:

Begin ASAP!
Determine a complete list of impacted builds/artefacts which needed to be supported in production
Review all CI/CD pipelines
Review all runbooks (automated and manual)
Review all production deployments
Review all disaster recovery processes
Identify all artefacts that will need to be migrated through property sets tagging in Artifactory

Tagging

However, before we begin looking at these, we need to cover tagging. Artifactory has a useful feature called Property Sets. We decided early (decision log - check) in the process to make wide use of custom properties through Property Sets. Our requirements were principally:

Artefact hygiene - remove unused or unsupported packages, waste
Docker images underlying OS type - differentiate between Linux and Windows containers

Note: That whilst possible to read a manifest for each Docker image and determine the existence of foreign layers, we were focused on stability and repeatability as well as speed. We wanted to utilise the native tooling as much as possible and felt that determining supported schema/configuration around custom tooling would negatively impact complexity.

Failure recovery

The focus for migrations was specific to our end goals and where appropriate we tried to reuse our tools across different package types. Ultimately, the project as a whole needed to be successful once, with many smaller eventual successes along the way. Building effective tooling was critical to the success of the project. We needed to use software to automate as much as we could whilst also allowing us to clearly aggregate the activities we had performed which in turn could be used as means of verification.

Key features such as dry-run mode, debug levels, and the ability to replay/resume from an offset was essential. The sheer volume of different files, paths, conventions/structures meant that we could only analyse so far before we needed to begin. We fully expected the need to debug activities during the migration phase and that these tools would really support us.

Migration tooling

Large Network bandwidth and scalable compute resources were top of our platform requirements. The choices taken would drip down into the tooling we created.

As an organisation, we place a strong emphasis on Infrastructure as Code (IaC) so creating processes and workflows around AWS Systems Manager using the AWS CDK is expected. We created stacks that enabled us to scale our migration efforts on demand which included:

Migration stack
EC2 Spot Fleet worker node stacks
- Windows
- Linux
SSM Documents library stacks
Custom command line interface tools
- archive runner
- migrator
- migraterunner

Concurrency was an interesting feature. The default was to plan for concurrency, if we run tasks concurrently across our cloud resources then this would accelerate the process, surely? Frustratingly, this was not the case. In the details of AWS CodeArtifact were areas where the order that packages were uploaded is important, particularly npm and Maven. Furthermore, package managers have different ways to determine the latest version, but support for this in AWS CodeArtifact was not universal during our project. We also discovered that there were places where Artifactory supported deprecated features which we happily used, yet AWS CodeArtefact did not. This resulted in us essentially serially migrating packages but utilising the cursor-like features of offset and limits to optimise our efforts.

Migration workers

The migration workers performed the brunt of the migration tasks and were orchestrated through AWS Systems Manager. The (awesome) network throughput enabled by AWS meant we were able to migrate hundreds of GiB of data per hour. This was crucial given that the order that most packages were migrated in was important.

Using property sets that were laboriously set by our engineering teams, the project was able to leverage the excellent Artifactory API as a catalogue/pseudo-state-machine through which, workers could page their way whilst advancing the migration.

In (often long-running) batches, workers would pull packages using the native package managers to the worker’s local storage, then the documented AWS CodeArtifact/ECR/S3 tool was used to push packages to their new location. This is where container images became tricky because pushing and pulling containers needs to be performed on a host running the relevant operating system. Whilst it is documented in the AWS SDK that you pull and push layers as mere blobs, the guidance was that this was only a feature to be used internally which was enough to ward us off.

Command line interfaces (CLIs)

A handful of CLI tools were authored in Go. These tools provided the brunt of the work where validation and custom logic were applied to transfer the different artefact types under the conditions of the project. AWS SSM documents were used in orchestrating the tools to transition the artefacts to the correct destination repositories via the appropriate worker node fleet. Much credit needs to go to the excellent spf13/cobra commander Go module that was used in these CLIs.

Migrating 1.5 million artefacts

Migration process

We wrote our tooling to support a ticket-driven approach where teams could raise tickets to work with the implementation team and migrate their product packages in batches, working together to verify successful completion.

This was important in allowing us to progressively work through the migration rather than a whole big-bang event occurring on an agreed date. Such an approach would be simply unworkable, requiring far too much alignment (thousands of pipelines updated in anticipation). Finding a date that would perfectly fit with hundreds of teams would be virtually impossible and crucially what rollback options would we have? Thus we decided this progressive approach would give us greater flexibility and the reassurance to engage team by team, product by product while checking off milestones on the route to completion.

As things panned out, we ended up with a large backlog of migration activities ranging from a few packages to entire release repositories containing hundreds of thousands of artefacts and terabytes of data. The backlog was continually refined and tickets were resolved step by step with the owning teams. The team had overall responsibility for verifying the migration objective had been met before marking the work as complete.

Resolving all migration tickets in the backlog would culminate in disabling write access for all standard users at a key, predetermined date, before swiftly proceeding, after a reasonable holding period, to remove complete access for all users, except system administrators. Any issues experienced along the way would be addressed in isolation, verifying with the team and making updates to our tooling as required.

Verification

This was tricky. Reporting on the AWS CodeArtifact side is currently poor (09/2022). CloudWatch metrics are entirely missing but ultimately it was not at all straightforward to aggregate key metrics and statistics in the same way you can with Artifactory.

We had to get creative and had to combine metrics queried from the Artifactory REST API with internal counters residing within our tooling to cross-reference our progress. Through deploying webhooks and using DynamoDB to record the dates when batches of work were undertaken we could replay activities using the DateTime offset to determine any deltas where pipelines were not fully updated and artefacts were still being deployed to Artifactory after the migration activity for the solution artefacts had begun. This worked out well and coupled with a reduction of write access permissions, enabled us to verify the process had worked to a point in time whilst giving us another window to replay the same batches but for a much smaller delta of packages. Significantly, as we advanced through the project, confidence grew as we were replaying actions over and over with consistent results. Testing our tooling was tricky so having the ability to utilise dry-run mode was invaluable.

Next up

The migration has now been completed.

Next up, we’ll take a look at whether we’ve reached our goal.

How we moved from Artifactory and saved $200k p.a. Part 3 of 5 - The future is Advanced Artefacts

Paul Mowat — Wed, 28 Sep 2022 15:36:42 +0000

Introduction

Welcome back to Part 3 of our 5-part series on 'How we moved from Artifactory and saved $200k p.a'.

If you are just joining we recommend jumping back to the beginning and starting from there.

Approach

Having identified that we wanted to create a structured service we had to determine the best way to approach it.

Our earlier analysis helped us identify the artefact types that we needed to support. Yet a remaining challenge was to identify how to support these and empower our development teams across the technologies and tools we use on a daily basis.

Architecture

The following architecture gives a high-level overview of the service components.

Repository convention

Something that became apparent was that our Artifactory configuration was a disordered muddle which has since provided us with a harsh lesson about paying particular attention to the rollout of such platform tooling. It had never been implemented in a controlled or consistent way.

Determined to avoid this at all costs we decided to build naming conventions for each of our artefact types into our service. This would be implicit, removing disambiguation and preference from any decisions.

Our products commonly have both development and production environments so it was decided that the service should mirror this and have just two conforming types of repository.

Development repositories would be where teams push build artefacts continually via Continuous Integration (CI). Then, when the appropriate levels of testing had been passed, the artefacts could be promoted into the corresponding release repository.

A benefit of this approach was ensuring a clear separation between development and release dependencies so that we could start to look at implementing automated housekeeping rules in the future. We do not need hundreds of development packages so why bother keeping them?

This helps with our goals of enforcing convention and consistency, which in turn makes it easier to automate and roll out changes in the future.

Infrastructure

We needed to start building out our infrastructure to support the service.

As we were utilising several AWS services, using AWS CDK was the obvious choice. It allowed us to build the service quickly and was also easy to change when required.

Going back to enforcing convention and consistency we leveraged AWS Service Catalog with several custom templates to help us create new repositories.

Delivery

Providing a service that worked successfully meant that we had to look at how we delivered our software and consider what an exemplary software lifecycle looks like, as well as the platforms we needed to support.

The following key areas were identified:

Local Development
Application Configuration
Authorisation
Continuous Integration (CI)
- GitHub Actions
- Jenkins
Continuous Delivery (CD)
- Harness

We also were aware of some products using other technologies such as Azure DevOps and TeamCity that we would not directly support, but still had to take into consideration how they could access and use the service.

Tooling

As developers, we are used to using tools to help make our day-to-day easier. If you look at any good service you will see that they typically have a range of tools to make interacting with them easy.

Command Line Interface (CLI)

We determined that creating a CLI would provide us with a centralised entry point for all of our delivery mechanisms and be flexible enough to allow it to work for any that we didn’t support.

The CLI had to support multiple operating systems (Windows, Linux & macOS) and be easy to update and use as required.

We started analysing what functionality the CLI would require and quickly identified potential overlaps between native package manager and docker commands. There is little point in us trying to write, maintain and support any tooling that mirrored these. Everyone knows how they work, they are industry standard tools.

It was decided that our CLI would work complementary to these. It would bridge the gaps and provide the functionality we needed for our service to work.

We determined our key functional requirements were:

Authorisation
Packages - get and promote
Generic Artefacts - get, list, publish and promote
Container Images - promote

The most important feature of the CLI was authorisation into the service. Every developer and delivery mechanism must authorise into the service before they can use it.

The CLI had to make authorisation easy and limit the impact on our engineering teams on a day-to-day basis.

We looked at other open-source CLIs for inspiration and took the time to understand how this could be done effectively for multiple operating systems, shells and from a user or service perspective.

In the end, we went with a multi-pronged approach and created mechanisms that allowed authorisation in several ways i.e. user, role and service level.

Our security is of the utmost importance and having authorisation tokens written to files was an absolute no-go. Everything by default was going to be applied to the running shell process in order for it to be used and thrown away when finished. That was implemented across several different shells such as bash, Powershell and Windows command prompt.

With our authorisation mechanism now in place, working and flexible enough to handle different operating systems and shells, the other features were much more straightforward.

Generic Artefacts proved the most labour intensive, only due to having to implement an entire set of commands to allow complete artefact management.

Other

With the CLI now in place we used it to power any other tooling that would help accelerate our development teams.

Our core Continuous Integration (CI) platform is GitHub Actions. We decided it was worth the effort to create a custom action that automatically downloaded the latest CLI, installed it and performed the required authorisation. This meant that teams could drop that action straight into their workflows and it would just work. Minimal change, maximum satisfaction.

Next, we looked at Jenkins. Although we are moving away from it, we still have some products still using it, therefore we spent a bit of time putting together some example pipelines on how the CLI could be used and included that in our documentation for teams to follow.

Now we have covered our Continuous Integration (CI) tools, we needed to look at our Continuous Delivery (CD) ones.

Harness is our Continuous Delivery (CD) tool of choice. It provides a flexible template engine, that we were able to utilise to create templates that could be reused across our teams.

Next up

With our new Advanced Artefacts service in place, we were ready to get on with the actual migration from Artifactory.

Next up, we’ll walk through how we built our migration tooling, defined our process and performed the migration.

How we moved from Artifactory and saved $200k p.a. Part 2 of 5 - Design

Paul Mowat — Wed, 28 Sep 2022 15:33:26 +0000

Introduction

Welcome back to Part 2 of our 5-part series on 'How we moved from Artifactory and saved $200k p.a'.

If you are just joining we recommend jumping back to the beginning and starting from there.

Decision making

The nature of larger projects such as these requires plenty of discussion and decision-making around temporary and permanent processes. We had lots of data to migrate and we needed to be efficient in our decision-making process. We decided upon using Architecture Decision Records to log the key implementation decisions which significantly helped us deliver consistency throughout our support and guidance.

As it turned out, undertaking this method of logging was not onerous and we ended up with records for around a dozen key strategic choices that we made; an example of one being the choice to utilise a spot fleet of EC2 workers to perform the migration versus something like AWS Batch or ECS. At first glance, we expected to go with a solution based on AWS Batch or AWS ECS but we had requirements to move resources such as Windows container images and it was so helpful to be able to easily recover the decision steps when we moved to create tooling to support this.

Workshopping

Workshopping commenced on the 10th of June 2022 and we had until the 4th of July 2022 to perform the required analysis, design and implement our solution.

Analysis of requirements

One of the first items of business was to determine which artefact types it was essential to support, those that would be unsupported and any transitions from these to corresponding supported types. Then we would need to determine the options to migrate to, whilst fulfilling the necessary obligations to supported packages and platforms.

Over the past few years, engineering at Advanced has been consolidating its toolchain and programming languages adopted by default. In no way intent on dissuading reviews of new or emerging options, but rather adding consistency in those used and bringing a larger collective intelligence to engineering as a whole.

From analysing our usage within Artifactory we settled upon support for npm, NuGet, generic artefacts (zip, exe, dll etc), Docker images and Maven. We quickly determined that our biggest challenge would be Docker images, accounting for greater than 50% of our consumed storage, with several repositories holding more than 1 TB of image data. Latterly, Maven would also prove challenging.

From this analysis, we were acutely (and financially) aware that we were also wastefully holding onto obsolete build artefacts. We decided to use this as an opportunity to leverage our engineering teams to review and select the versions of artefacts that our products needed to retain. This would help reduce the scale of the migration ahead somewhat and perform some well-overdue housekeeping. After all, there is no point in migrating and paying for artefacts that are no longer required.

Solution analysis

Having gathered an understanding of what needed support and delivery, we had to identify where we were going to migrate to.

AWS is our preferred Cloud Provider and platform, as well as a key technical partner. It was a natural choice to look at their services for our solution. From investigation, we found that AWS CodeArtifact was a decent fit for supporting npm, NuGet, Maven and Python (if required in the future), however, it was not a complete match for all our requirements. Favourably, S3 is an excellent fit for generic artefacts, and Elastic Container Registry (ECR) is perfectly appropriate for Docker images (even leading us to correct misunderstandings between images and repositories internally!).

We now had the artefact types we needed to support at a high level and where they were going to migrate to.

Solution design

Now we firmly knew our direction, we needed to decide how we get there.

Initially, we considered publishing guidance around best practices for various AWS services to satisfy our artefact requirements but ultimately that was deemed unmaintainable.

We wanted to finish the project with our artefact management strategy in a much better position than it started. Significant to us was ensuring we had the ability to define convention, consistency, clear guidance and expectations. We aimed to provide a maintainable solution that continues to build upon the best practices as it matures.

This led us to agree that it was important for the culmination of the migration to result in a new, custom service that any engineering team within Advanced could consume. Advanced Artefacts was born.

We now had two streams we needed to complete within the project:

The Advanced Artefacts service
The Migration

We will get into the detail around these in future posts.

Support channels

As mentioned previously Advanced has over seven hundred engineers from across the globe working on many projects and we needed to identify a strategy for how we could support them in the best way possible.

We came up with the following three-pronged approach.

Documentation

We decided early that we needed to document all parts of the project to allow our engineering teams to self-serve where possible. Without good documentation, there is no way a team of four can support over seven hundred developers.

We focused on providing some getting-started documentation that walked teams through the process in an end-to-end fashion. Then proving the appropriate reference documentation for each step.

This covered items such as the support channels available, each team's responsibilities, the migration preparation and also information on how to use our new Advanced Artefacts service both locally and from our CI/CD pipelines.

A great deal of time was spent pouring over this, it was however crucial to the success of the project.

Clinics

A technique that has worked fairly well for our organisation is the idea of online clinics. We held clinics twice a week for the duration of the project.

We used the first two clinics to kick off the project with our engineering teams. This helped us set timelines around key milestones and clear expectations on what was being delivered.

After that, they were reserved for anyone to drop into, receive updates and ask for assistance directly.

Microsoft teams channel

Microsoft Teams is our internal communication tool, therefore, we created a dedicated channel that we would use for communicating any important updates to the engineering teams.

They could also ask us questions or get further clarification as required outside clinic sessions. The artefacts team committed to replying to the questions as soon as possible ensuring teams were unblocked and able to progress quickly.

Next up

Now we have our design in place we need to start implementing it.

Next up, we will cover the creation of the Advanced Artefacts service.

How we moved from Artifactory and saved $200k p.a. Part 1 of 5 - Planning

Paul Mowat — Wed, 28 Sep 2022 15:31:35 +0000

Introduction

A 5-part blog post by Alex Harrington and Paul Mowat covering the migration of 25 TB of artefacts from JFrog Artifactory to a custom solution we created for Advanced, achieving significant cost efficiency.

A journey

Early in 2022, we decided that Artifactory had become an expensive option for us. Whilst a good product, Artifactory wasn't without difficulties surrounding our subscription. To provide a little more detail, specifically, you are either all in or all out with the JFrog platform, you can only subscribe to every component which is not too desirable at the enterprise level.

In retrospect, we came to learn of significant portions of the JFrog platform (Xray for example) from which we were not getting any real value (for us) and this made the overall service costly. Moreover, we were serious about doubling down on the security of our software supply chain and researching a wider (custom) array of best-in-class solutions.

Still, this was no easy decision as we were a large user of Artifactory with over 1.5 million artefacts published and 25 TB of data storage consumed. Many of our CI/CD pipelines and developer settings were all configured to use Artifactory, so the scale of the task was somewhat sizeable. Still, we proceeded to assess our options and planning the task(s) at hand was critical.

And so it begins

Let’s start by declaring our initial goal:

To migrate all requested artefacts from Artifactory without losing any, writing custom tooling as necessary

The possibility of migrating away from Artifactory was first aired at the beginning of January.

Artifactory comes under a category of services that could be described as quite "sticky". A SaaS solution where the impact of migrating away would reach far and wide within many an organisation.

Advanced has over 150 products active product suites covering different market areas. Some examples are delivering care to 40 million people throughout the UK, sending 10 million sporting fans through turnstiles and supporting 1.2 billion passengers to arrive at their destinations on time. Our solutions are engineered by hundreds of colleagues from across the globe, built using multiple technologies, living in more than 2600 GitHub repositories and powered by thousands of CI/CD pipelines. All deploying to numerous cloud/hybrid-cloud platforms. Not withstanding backup, disaster recovery, escrow and many other internal and market-driven requirements.

We needed to plan, but plan in a way that would allow us to scale.

Team

We formed a small dedicated artefacts team with four members that had to support more than seven hundred engineers through the process. The artefacts team initially needed to design and implement the migration machine, followed by educating and guiding our engineering teams through the project. This had to be as efficient as possible, in order for it to scale.

The artefacts team structure was as follows:

1 x Principal DevOps Architect (Paul Mowat)
1 x Principal DevOps Engineer (Alex Harrington)
1 x Senior DevOps Engineer (Karthik Holikatti)
1 x DevOps Engineer (Likhith Kotian)

Milestones

Our next task was looking at the wider project and breaking it down into key milestones so we could share these. This was a critically important area. Accuracy and clarity in our communication were paramount.

Starting with setting a hard immovable deadline and taking heed from previous hard-learned lessons with sliding deadlines that run and run, we chose to set this date in stone and declared this at the outset of our engagement with our wider engineering community.

We felt this offered a powerful message, which clearly illustrated that urgent engagement was necessary from all sides.

The key milestones were:

10-06-2022 - Project Kick-off - First implementation team workshop held to begin formulating the plan
04-07-2022 - Deadline for our design and implementation complete ready for a wider rollout
05-07-2022 - First Advanced Artefacts support clinic held with over 100 participants
06-07-2022 - Migration Period Start
19-08-2022 - Migration Period End
22-08-2022 - Engineering teams would lose access to Artifactory
31-08-2022 - Project End - Our Artifactory subscription would end

Next up

We have our team and plan.

Next up, we get to work designing our solution.

User Stories 101

Joe Wallace — Tue, 27 Sep 2022 10:27:24 +0000

What is a User Story? 📖

A “user story” in software development is an informal description of a user’s requirement, written in natural language relevant to the business domain, preferably without reference to the software in use (although sometimes this may be necessary), and worked on by a software development team.

It should also ideally be written by (or at least in collaboration with) a user, though they are also often typically written by a Business Analyst or Product Owner, and really can be written by anyone, if they have the right information on the business context.

User stories are conversation starters and anchor what a solution needs to facilitate, but they don’t define it. You may even be able to achieve what you wanted to in your user story without software development!

User Story Formats 🃏

User stories are often produced in a formatted fashion, largely to drive consistency or to help people to structure their thoughts, but they don’t have to be.

Typically at Advanced (and in many other software companies), user stories follow the “Connextra” template:

As a user persona
I want capability
So that business benefit

The lightweight nature of this format makes it useful in different contexts; on a project management tool like Jira, or physically, on a post-it note or index card.

The key part of any user story is the business benefit (or the “So that” on the Connextra template); if you can’t define why a user wants a capability in your software or process, you probably don’t need it at all!

User Story Splitting 🕷

User stories should be as small and self-contained as possible in their scope. This is both to ensure a focussed solution and to plan work in manageable chunks.

There are multiple methods of splitting user stories, but the most commonly used is SPIDR:

Spikes
Investigations and prototyping to better understand the ideal solution for a user story.
Paths
Different routes a user may take through a process, e.g. reschedule an appointment, or cancel it without rescheduling.
Interfaces
Different devices, browsers, or alternate interfaces for different types of users of the system.
Data
Different categories of information being managed or browsed, e.g. displaying someone’s allergies, or displaying their medication.
Rules
Constraints on the system, e.g. rules on the maximum number of appointments someone can have booked at once, or a minimum gap between appointments.

Acceptance Criteria ✅

Acceptance criteria are often added to user stories to determine conditions which need to be fulfilled when implementing the story such as to make it acceptable to the customer. These are used to inform the test cases that are run to verify that the conditions to fulfil the user story have been met.

These may take the format of simple bullet points, or use the “Given, When, Then” template (also known as Gherkin):

Given pre-condition
When user action
Then outcome

Example User Story 🔍

As a District Nurse
I want to be able to view a patient’s allergies when visiting them in their home
So that I don’t prepare them food or medication that may be dangerous to them

Acceptance Criteria
Given a patient has any currently active allergies in their clinical record
When I open the patient’s visit on my tablet or phone
Then I will see them listed, with the causative agent, reaction severity and description shown

Technical Debt and Technical Efficiency

Joe Wallace — Wed, 14 Sep 2022 14:12:56 +0000

Technical Debt: “The Loan” 💰

Within agile software development, one of our key aims is to get working software out as early as we can. This allows us to enable the value for our users quickly (and therefore bring in revenue as a business) and get early feedback to improve and iterate the product in line with customer needs, without wasting time on things customers might not ultimately want.

While we may refactor our code as we go along, this tends to mean that though the software we deploy works as intended, it may not be as efficient as we’d like. These inefficiencies are often called technical debt.

Technical Efficiency: “The Repayment” 💳

It is important to address inefficiencies in our code over time, for a multitude of reasons: To reduce the likelihood of bugs occurring, to make the code base more manageable, and to improve performance, among other things.

Therefore, its recommended that each product at Advanced sets aside a bucket of time in each release period for technical efficiency - refactoring and improving code to “pay back” the technical debt, like a loan we took out in order to deliver the software early. When American developer Ward Cunningham coined the term “technical debt”, this was exactly what he meant - repaying technical debt is essential, much like paying back a loan.

Technical Product Initiatives and Technical Efficiency 👩‍💻

There are some scenarios where initiatives we want to complete at Advanced are technical in nature, for example:

Replacing a third party component or platform.
Upgrading to a new version of a framework or integration.
Changing our authentication methods.

These initiatives are often erroneously categorised as “technical efficiency”, as the end user does not notice a difference in their day-to-day work. This is problematic, however, as these types of initiatives are not related to technical efficiency by nature, and due to their complexity and business benefit, need to be prioritised against other initiatives.

All of these initiatives should be considered like any customer-facing one, and supported by an appropriate business case that clearly quantifies the value of the initiative, and justifies why it should be done. For example, we wouldn’t be replacing a third party component for an abstract reason; we might be doing this because it causes unacceptable performance issues which are affecting the abilities for users to do their work, or the component might be going out of support, putting our entire service at risk.

It is vital that these items are prioritised and analysed in the same way that customer-facing initiatives are, with just as much care put into the business case. Prioritisation is a wider topic, the scope of which extends far beyond this article, but the following should be considered for more technical initiatives, as well as user-facing ones:

The business risk incurred in not completing the initiative.
Potential cost saving.
Potential revenue add, in terms of additional module sales or selling into new markets.
Customer satisfaction (and therefore retention).

Paying off technical debt is important, and freeing up space in our development teams' capacity by categorising these initiatives correctly helps us to recognise this.

What size is my T-shirt? Can someone look at the label?

AprilBauer — Fri, 17 Jun 2022 13:37:09 +0000

Introduction
Imagine you are standing in the dressing room of your favorite shop trying on T-shirts. Did you grab the right size, does it have extra detail, is the price too high? What does mom think?

You look at the construction of the T-shirt, discussing the color and stitching, does it need to be dry cleaned? Can it be mass produced or is it haute couture? This would affect the cost to create the T-shirt. And “mom” is certainly going to be thinking about price, as well as how well it will wash and wear (in our world, think how many customers it’s being sold to).

T-shirt sizing isn’t just for clothing, it’s also used to measure the effort need to develop a software initiative. Just like the T-shirt in the dressing room, you need to understand the size of T-shirt to get the right fit for the software initiative. You accomplish this by sitting down with Engineering and Product Management teams (just think of them as mom).

What does this mean for software development, then?

Now if you are reading this, thinking “how can software fit into a T-shirt”, and “I would never care about the color and stitching”, you do care about the what the software does; the user experience, if extra security is needed, or if it needs to be available on mobile devices or a portal. Does it have certain requirements needing new UI, will it involve ancillary teams such as CoEs or the C4Es? The T-shirt size can represent task, scope, effort, complexity work hours, time estimates, or often all of the above.

After everyone has looked the T-shirt over, maybe you decide you want the T-shirt for your niece Joan, and she only needs an XS (0-29 days of development), or maybe it will be a gift for Uncle Joe, and he is one big dude, so you will definitely need an XXXL (1500+ days); or maybe, just want one for yourself and a L (200-399 days) will do for you.

Great, you know the T-shirt sizes you need – but they not in stock, so you are going to need to order them!

Imagine your order arrives at the warehouse (Engineering and Product Management teams), but the staff are busy, so they start looking through the orders to see what sizes are needed, and what the value of those T-shirts will be against the time they have available to get them. Once they decide what T-shirts they have room for, you will get a “Go” or “No Go” decision regarding whether your T-shirt can be delivered.

So, if it’s your lucky day, your brand new T-shirt will arrive!