DEV Community: Toby Bellwood

Lagoon v2.21 released

Toby Bellwood — Tue, 17 Sep 2024 05:28:24 +0000

We are excited to announce the release of Lagoon v2.21.

What’s New in v2.21?

This release includes a series of new features and enhancements to improve user flexibility and security.

Deprecating Unsupported and Unmaintained Images

Unsupported or unmaintained images may have unresolved vulnerabilities, creating security risks for customers. Deprecation ensures users migrate to more secure, maintained versions.

There are now warnings in the Lagoon UI and Docker Desktop. We will continue to post the warning and while at this time we do not intend to remove out-of-date images, we could potentially block builds in the future.

Lagoon users take heed: we cannot guarantee compatibility with potentially broken builds forever. It is in your best interest to keep your code up to date, and action any build warnings that Lagoon detects.

Warnings can be viewed in Docker Desktop, via a docker inspect command or in future releases of Lagoon, highlighted in a build.
If the image has a suggested replacement, it will be mentioned in the warning.

Example:

To learn more please read further here in the Lagoon documentation.

Multiple Volume Mounts

By supporting multiple volume mounts per pod, a single pod can access multiple persistent storage locations, which is essential for enabling more applications, frameworks, and languages to run on Lagoon. This feature boosts Lagoon’s ability to run diverse and complex setups within a single pod architecture.

Path Based Routing

Path-based routing within an environment means that an ingress can direct traffic to different services based on the request’s URL path. This enhancement, which allows routing multiple paths to different services within the same ingress or namespace ultimately improves flexibility, scalability, and simplicity in how traffic is managed across multiple services in an environment.

Base Image Refresh

For projects that use custom upstream images, there is now additional functionality that forces Lagoon to check for a new image automatically on every build, just by adding the lagoon.base.image: label to any image in the docker-compose.yml file and setting the value to the upstream image you want to check.

Upgrade to Keycloak 24

This upgrade ensures better security, performance, and usability for managing identities and access. It will also give Lagoon greater support for authentication methods, and you can expect to hear more from us on this in the coming months.

Conclusion

Our latest release is a commitment to enhancing and improving Lagoon and the user experience. These updates are designed to help our customers manage their environments more effectively and securely.

Additionally, we are working with users to understand the importance of maintaining your code base and fixing errors. Together, we're building a more robust, efficient, and user-friendly platform that empowers your development process.

Upcoming Lagoon CLI release v0.30.0 - there's a lot of changes!

Toby Bellwood — Thu, 15 Aug 2024 20:55:47 +0000

Lagoon CLI v0.30.0: Breaking Changes and New Features

We are excited to announce the upcoming release of Lagoon CLI v0.30.0 next week. This is a significant release that introduces several breaking changes, as well as some powerful new features. We want to give our users advance notice of what to expect, so you can prepare your tooling and workflows for a smooth transition.

Pinning CLI Versions for Stability

As Lagoon CLI has not yet reached a stable v1.0.0 release, occasional breaking changes are necessary to improve the tool. With v0.30.0 introducing multiple breaking changes, it has the potential to disrupt scripts and automation using the CLI.

We strongly recommend pinning a specific version of the CLI in your tooling. This allows you to control when you update and adapt to any breaking changes. If you want to test the latest lagoon-cli version locally before updating, you can always build it from the source: https://www.github.com/uselagoon/lagoon-cli.

Overview of Changes

The release notes will provide a full list of changes in v0.30.0, and the docs will be updated when released.

Here are some key areas to be aware of in advance - we've tried to cover most of the main items that have changed:

Configuration Updates

amazee.io customers using Lagoon CLI should update some configuration values to use the latest DNS/hostnames:

graphql: https://api.amazeeio.cloud/graphql
hostname: token.amazeeio.cloud
port: "22"

SSH Host Key Verification

SSH connections will now attempt verification of the host keys of the ssh endpoints, you may see messages like this in your output (it is logged to stderr)

Warning: Permanently added 'ssh.us2.amazee.io:22' to the list of known hosts

You may also have an old host key saved in your known hosts file, if you encounter an error that contains the following

WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!

You’ll have to remove the old host key from your known hosts file manually (http://ssh-known_hosts-ignore-temporarily)

Flags Standardization

All command flags have been changed from camelCase to kebab-case for consistency. The short code/aliases have stayed the same

--autoIdle is now --auto-idle
--gitUrl is now --git-url
--productionEnvironment is now --production-environment
and more...

Flags related to OpenShift have been generalized:

--openshift options now use --deploytarget instead.

Binary 0|1 flags for enabling/disabling features now use true|false: - --autoIdle 0 becomes --auto-idle=false

--deploymentsDisabled 1 becomes --deployments-disabled=true
and more...

Streamlined Output

Some commands (get environment, get project and list deploytargets) have had their output tables streamlined. In these cases, a new --wide flag will display additional fields in the output.

Command Structure Changes

The structure of some commands has changed, particularly around organization-related functionality. Refer to the documentation for an overview of the new command syntax.

Described below are some of the changes to be aware of if you're using organizations,

lagoon add organization organization becomes lagoon add organization
lagoon add organization project becomes --organization-name flag on lagoon add project > Don't forget to add this flag if you work with organizations
lagoon add organization group becomes --organization-name flag on lagoon add group > Don't forget to add this flag if you work with organizations
lagoon add organization deploytarget becomes lagoon add organization-deploytarget
lagoon add organization user becomes lagoon add organization-administrator
lagoon delete organization organization becomes lagoon delete organization
lagoon delete organization project becomes lagoon delete organization-project
lagoon delete organization deploytarget becomes lagoon delete organization-deploytarget
lagoon delete organization user becomes lagoon delete organization-administrator
lagoon list organization organizations becomes lagoon list organizations
lagoon list organization projects becomes lagoon list organization-projects
lagoon list organization groups becomes lagoon list organization-groups
lagoon list organization deploytargets becomes lagoon list organization-deploytargets
lagoon list organization users becomes lagoon list organization-administrators

Log Streaming Support

Lagoon CLI v0.30.0 introduces support for retrieving and streaming container logs. It allows for truncating and following logs for services and containers, similar to SSH commands.

This feature requires support from the remote cluster and will be progressively rolled out to amazee.io clusters in the coming weeks and months.

Embrace the Change, Get in Touch

We're thrilled to bring you this powerful new version of Lagoon CLI. While the breaking changes may require some adjustment, we believe the improvements in consistency, capability, and performance are well worth it.

If you have any questions or need assistance with this transition, don't hesitate to reach out to us - either in the Lagoon support channels or via your existing support arrangements. We're here to ensure your Lagoon experience remains smooth and productive.

Get ready to take your workflow to the next level with Lagoon CLI v0.30.0!

Managing upstream security fixes in uselagoon docker images

Toby Bellwood — Tue, 24 Jan 2023 22:21:55 +0000

This week, the Git project announced a maintenance release to patch two CVEs (CVE-2022-41903 and CVE-2022-23521).

I thought this would be a good opportunity to cover how the Lagoon team handles ensuring these updates make it through to our uselagoon-produced images.

The majority of our images are based on upstream images, created by the application, framework, or language maintainers.

Using The Node.js runtime as an example:

Lagoon publishes a range of node images on Docker Hub (e.g. https://hub.docker.com/r/uselagoon/node-18/tags)
This image is built using a Dockerfile
We inherit the upstream image FROM node:18.13-alpine3.17 and then we add a number of customisations.
This node image is just one of a range published by the Node.js team (https://hub.docker.com/_/node), and they also have the Dockerfile for their build available
The node image itself is built from the upstream image FROM alpine:3.17 and customised to add the nodejs runtime and supporting libraries.
The Dockerfile used to build the Alpine image is also available - this just unzips the relevant Alpine tarball into an empty "scratch" image and sets the shell

I can digress slightly to explain the image naming/tagging we use:

uselagoon/node-18:22.12.0 is the latest release of our Node.js version 18 image (December 2022 - we use CalVer for our tag convention, and release monthly)
node:18.13-alpine3.17 in the uselagoon image is an alias tag for the latest version 18.13 Node.js image, built using the Alpine Linux distribution, version 3.17. There are at least 10 aliases for this same image - in our case, we want to pin it to a specific minor release (18.13) using a specific Alpine version (3.17) - we could tighten or relax these, but we feel this gives us more control over automated updates
alpine:3.17 in the Node.js image is the alias for the latest Alpine 3.17 release (currently 3.17.1). Alpine is minor version updated twice a year https://alpinelinux.org/posts/Alpine-3.17.0-released.html

I know this sounds really tortuous and convoluted, but bear with me!

Lagoon also publishes a version of our Node.js image, with additional tools used to "build" Node.js projects - we call it node-18-builder - see the Dockerfile. We add a range of packages to the standard image, as they are not included out of the box.

Because the Lagoon team maintains the images here, we are responsible for ensuring that any components added at this stage are secure, and have any available updates. The two packages I'm going to focus on here are curl and git, as they have some of the best documented vulnerability announcement and management strategies.

Alpine-based packages (apks) are managed in the Alpine package registry, and updates are usually tied to the Alpine Linux release cycle:
e.g. Git on Alpine 3.17 is currently at version 2.38.3-r1 - which is the correct version to address the above CVEs!

In the Git repository linked from the actual package there is also an APKBUILD file that outlines a bit more information - and from this, we can see that those CVEs were actually addressed in the previous 2.38.3-r0 release.

There may even be occasions where the Alpine Linux version of a package includes the CVE fix before the source package - you can see this clearly on curl on Alpine 3.16 - the 7.83.1 version of curl has been patched over 10 times now - https://curl.se/docs/vuln-7.83.1.html - the updated 7.87 version is only available via Alpine 3.17

Reading these dockerfiles, Git repositories and package logs is a bit of a "dark art", but rest assured that the Lagoon team secretly loves it, and it's all part of our role in ensuring the safety and security of the applications running on our platforms!

Keeping pace with Kubernetes

Toby Bellwood — Tue, 13 Dec 2022 21:48:03 +0000

As people are generally aware, most evolution in tech follows a rapid pace, and none more so than the Kubernetes project (and the surrounding ecosystem).

In mid-2021, the Kubernetes Release Team announced that they will be releasing a new version of Kubernetes every 4 months, and will be supporting each release of Kubernetes for 12-14 months from release. This effectively means that Kubernetes has an N-2 support cycle (current release, and the two immediately prior to it.

Each release of Kubernetes comes with an associated raft of API deprecations, releases, and additions (see https://kubernetes.io/docs/reference/using-api/deprecation-guide/). These can range from relatively minor API version changes (e.g. a beta API becomes GA - usually available in parallel for 3 releases/12 months) to pretty substantial code or process changes (which require re-writes/re-architecting and have been available for 6+ releases).

One such milestone release was Kubernetes 1.22 - which took significant work for the Lagoon team to achieve compliance with (see the blog at https://dev.to/uselagoon/lagoon-kubernetes-122-ek8). In this release, announcing compatibility with 1.22, we also added a “minimum supported version” of 1.19, in order to be able to utilize the parallel API releases.

With the release of 1.24, 1.25 (and shortly 1.26), there come similar (if not as impacting) challenges with Lagoon components. For this reason, we will be implementing similar “minimum supported version” constraints on the Lagoon releases supporting these Kubernetes releases.

When timing our Lagoon releases, however, we have some ability to define our own schedule (within certain bounds!).

Looking at the timing for Kubernetes 1.24 alone:

It was released on May 3rd, 2022
It became “Generally Available” in Azure AKS in July 2022
It became available in the “Regular Channel” in Google GKE in November 2022
It became available on AWS EKS in November 2022
Official EOL is July 28th, 2023
Azure AKS will end support in July 2023
Google GKE will end support in September 2023
Amazon EKS will end support in January 2024

While Kubernetes has no officially published API deprecations for 1.24, there is actually one thing that does catch us out, in that service accounts no longer get secrets automatically defined for them (which we use to control builds, tasks etc) - so we will have to rewrite some of our controller code to handle the generation of time-scoped tokens instead.

Given the only recent availability of 1.24 across the clusters we support and manage - this hasn’t been a pressing issue for us. But together with the EOL of 1.20 support in all the major providers, it also presents an opportunity for us to be more agile in adopting the parallel API availability and reducing the upgrade burden for new releases.

The most recent version of Lagoon (v2.11.0 and the additional lagoon-remote and lagoon-logging components) already supports Kubernetes 1.24.

The Lagoon team is proposing to release a version of Lagoon in early January that:

Fully supports Kubernetes 1.24 (as it is now GA on all platforms)

Supports a minimum Kubernetes 1.21 (even though it’s almost EOL)

Introduces the API upgrades available in 1.21 (CronJob and PodDisruptionBudget)

Generally, going forwards:

We will make Lagoon available for use on the Kubernetes versions available on the leading managed Kubernetes platforms that (we know) are running Lagoon in production (EKS, GKE, AKS).

Lagoon components will have a minimum Kubernetes version to allow us to use the more modern features. We'll try to keep it close to N-3 to allow the lag from providers.

Users of Lagoon should pay close attention to the Lagoon release notes, and ensure that not only Lagoon but also the underlying Kubernetes installations are regularly updated.

We’ll be publishing more comprehensive helmchart-specific changelogs via our repository on artifacthub - https://artifacthub.io/packages/search?org=uselagoon&sort=relevance&page=1

Please reach out to the team if you want us to cover any of this in more detail.

Support/Release periods backgrounds:

Kubernetes
Amazon EKS
Azure AKS
Google GKE
Oracle OCI
IBM CKS
Alibaba ACK
Openshift 4 has varied release calendars depending on which variant you are running, and on which provider - note that Openshift support in Lagoon isn’t always guaranteed, owing to some inconsistencies with the Kubernetes implementation, and the difficulties in testing.

build-deploy the future, a behind the scenes look at Lagoon

Toby Bellwood — Tue, 20 Sep 2022 13:35:51 +0000

When we announced Lagoon 2.0, we discussed how we'd reformed the architecture to better suit a distributed global platform. This unveiled the "separation" of lagoon-core and lagoon-remote, and the gradual migration of services needed for running sites on Lagoon towards the lagoon-remote.

Background

Over the last few months, we've really pushed ahead with this.

The Lagoon <> Harbor integration has been moved from a single core-affiliated Harbor towards a lagoon-remote being able to define and manage it's own, or a shared, Harbor install.
The lagoon-core auto-idler service has been deprecated in favour of running independent idlers in lagoon-remotes (using Aergia, which has the added ability to unidle)
A number of other features, such as backups, restores and logging configurations can now be set at the lagoon-remote level.

The primary way, though, that lagoon-core provides lagoon-remote with the instructions is via a "Lagoon Build" sent as a message to the build-deploy controller in the lagoon-remote. This build contains most of the information needed for the controller to progress a build (the rest of the information about that specific cluster is stored against the build-deploy controller).

The build itself is handled by a service image called "kubectl-build-deploy-dind" - catchy, huh? This service is essentially a collection of Bash scripts, across thousands of lines of code, that knows all the ins and outs of Lagoon - how to use the various variables, settings etc that Lagoon can control when deploying a site.

With the increased capabilities of Lagoon over the last couple of years, these files are becoming increasingly more complex to maintain, and the logic is getting harder to follow. As a team, we've also evaluated the cloud-native landscape a little more closely, and looked at what other projects, tools or specifications there may be that we could utilize. Lagoon has always prided itself in its developer focus, and the predictable relationship between production and local has been key in ensuring minimal friction in deploying workloads. The other critical thought is that the majority of the fast-release work we do is on this service, and curating a full Lagoon release is a lot of effort for a simple script change.

So what's happening?

With these considerations in mind, we've decided to take the following steps:

The current "kubectl-build-deploy-dind" Bash scripts will be progressively rewritten, rearchitected and replaced by a series of Go modules, built into a new build-deploy-tool.
Development of the build-deploy-tool (and the generation of the Docker image to drive the service) will be split from the main Lagoon repository, into its own repository, in order that it may have its own development lifecycle, allowing for targeted testing, faster releases and easier rollbacks.
The parts of the tool that need to interact with the user repositories will utilize standards-compliant methods where possible. This means that the docker-compose.yml file will be read in by the compose-spec reference library compose-go.
We will create a schema for the .lagoon.yml to allow easier parsing and error-detection.

What does this mean for users, administrators etc?

Good question! Hopefully nothing. However, as we have started on this journey, migrating the first few components (see the full list here across to go & compose-go, we've observed some things:

Not all "working" docker-compose.yml and .lagoon.yml files are actually valid. It's possible to make breaking errors in your file, and still have it deploy locally. Reading it in via a standards-compliant parser, however...
Not all users actually test their code locally before deploying to Lagoon - otherwise they would have known that their file was badly broken...
Even if those errors are in a section unrelated to the section being processed, the inability to parse the file will cause the build-deploy-tool to error out.

However, to counter this, we've (in Lagoon release v2.8.4) added a new feature into the existing builds that will create a user-facing warning when the build-deploy-tool encounters an error that would ordinarily cause the process to fail, and we've generated a message in the build log.

We're also working very closely with the wider team at amazee.io (as the largest installed base of Lagoons) to trawl through thousands of build logs to try and capture any of these errors to be sure that we're causing no adverse harm to running builds, and where we encounter a fatal error that we could handle, we create a test case and a workaround for it. Some are easy (such as handling the presence of local-only env files that are not pushed to Git), some are harder (like handling anchor and alias edge cases in the YAML).

Next Steps

We will also be implementing (as of Lagoon v2.10.0) the switch across to uselagoon/build-deploy-image as the main source of the builder. However - fear not, we will still be re-publishing it as the kubectl-build-deploy-dind image to match the Lagoon release. We'll build a fair amount of flexibility into Lagoon to support this - as well as allowing Lagoon admins to specify build images per remote in the API, and provide access to more cutting-edge builds outside of the Lagoon release cycle.

In the meantime, keep an eye out here and we'll keep you updated on what we're up to!

Lagoon + Kubernetes 1.22 🥳

Toby Bellwood — Fri, 17 Jun 2022 06:34:11 +0000

For longtime followers of Lagoon, you'll know we've been working towards Kubernetes 1.22 support ever since it was released 9 months ago.

Whilst Kubernetes sets a fairly aggressive development and release pace (three releases per year), the major cloud providers make these releases available on a different schedule, which often lags behind the upstream release by a few months.

See a few here:

What is most important is noting when your provider will auto-upgrade your cluster to a new version of Kubernetes though. Most Kubernetes releases are relatively smooth to upgrade between, but occasionally there are more significant Kubernetes releases that are always tricky to navigate as they usually come with a raft of deprecations. These are usually communicated well in advance, but can still cause issues, especially in a code base with as many moving parts as Lagoon.

Thankfully, the Kubernetes API is extremely well documented, and the team publishes a Deprecated API Migration Guide for us to follow. When an API is deprecated, it is usually because the version of that API has incremented, either to a beta, or stable release. Occasionally, though, an API that we were using may be removed from Kubernetes.

For every API that we use, we have to assess whether the deprecation impacts us:

is it a simple name change? (i.e. coordination.k8s.io/v1beta1 renames to coordination.k8s.io/v1)
is it a name change with some spec changes? (i.e. in authorization.k8s.io/v1 API version, spec.group was renamed to spec.groups
has that API (or part of it) been removed from Kubernetes? (i.e. PodSecurityPolicy in Kubernetes 1.25)
is it a larger change that requires a rewrite (i.e. most of the deprecated APIs that Lagoon uses 🤦)

We've gone through all the various components of Lagoon with a fine-toothed comb (and the aid of the built-in deprecation warnings) to make sure that all references to these deprecated APIs have been updated for the replacements. This includes all the Helm charts used to deploy Lagoon, Lagoon projects running on Lagoon, and the third-party services that Lagoon utilizes. It also includes some code rewrites in the controllers and operators that Lagoon deploys.

As of lagoon-core release 1.0.0, this work has been completed! All the latest releases of the Lagoon Helm charts are fully Kubernetes 1.22 (and also 1.23!) compatible. We have taken some additional steps to ensure future compatibility, such as matrix testing versions 1.21,1.22,1.23 and 1.24 for releases, and running a range of test clusters prior to releases, and marking the Lagoon components as not supporting Kubernetes <1.19 - as that is when a number of the replacement APIs that we have migrated to were first introduced.

To anybody running a pre-1.0.0 version of Lagoon on their <1.21 cluster, we would encourage you to upgrade to 1.22 ahead of time.

Kubernetes has introduced a number of webhooks that will automatically update any resources to be compatible with Kubernetes 1.22, and the cluster will upgrade safely to 1.22. However, subsequent Helm chart upgrades may fail, as they are operating on the pre-updated state of those resources, which are no longer valid Kubernetes objects (confusing, huh?).

There is a fix to this, thankfully, if you've already upgraded, or been upgraded and see this error:

  Error: UPGRADE FAILED: current release manifest contains removed kubernetes api(s) for this kubernetes version and it is therefore unable to build the kubernetes objects for performing the diff. error from kubernetes: unable to recognize "": no matches for kind "PriorityClass" in version "scheduling.k8s.io/v1beta1"

The Helm plugin https://github.com/helm/helm-mapkubeapis was designed to resolve exactly this issue, and comes preloaded with the necessary resource mappings to update for any version of Kubernetes! We can testify that it works perfectly :cough:

As always, feel free to reach out to the Lagoon team if you've got any questions - you can even find us in our shiny new Discord Server

November release update

Toby Bellwood — Tue, 30 Nov 2021 22:50:44 +0000

Lagoon-core

In the last month, we released a series of updates to Lagoon 2.2.0 (2.2.1, 2.2.2, 2.2.3 and 2.2.4

These were all to address unforeseen issues with the new features introduced in 2.2.0 - namely Kubernetes Network Policy support, Rootless workloads and Ingress annotation linting.

See the full 2.2 changelog

Lagoon-cli

November saw the first release of lagoon-cli under the uselagoon namespace 0.12.0 - complete with updated homebrew taps, docker images and M1-compatible binaries

See the full changelog

Lagoon-images

November saw the first time we've used a patch release for the Lagoon images. The 21.11.0 release saw updates to a number of packages that had been held back in October.

We used the 21.11.1 release to update the Alpine versions in use for all the images to address a couple of vulnerabilities

See the full changelog

Lagoon-sync

As Lagoon-sync is still in active development, there was a couple of small hotfix-releases in November.

Lagoon-examples

There were the usual updates across the Lagoon examples, reflecting the Drupal core update cycles.

What's up next month?

In December, we will be debuting a couple of new things:

a reorganisation of our Lagoon example repositories (one repo per example, instead of the multi-branch Drupal example)
The ability to publish and consume "experimental" images from Lagoon-images ahead of their general release
A new documentation site, built using Github Pages
A new Lagoon website - be still my beating heart!

Lagoon 2.0.0 released

Toby Bellwood — Wed, 06 Oct 2021 23:04:40 +0000

After what seems like an eternity, Lagoon has finally had its 2.0.0 stable release.

When we first put the 2.0.0-alpha.1 tag into the world, we'd already made over 350 updates to the Lagoon 1.x codebase, and since that release, we've made almost 800 more updates across 150 pull requests to get to this point.

This 2.0.0 release is a significant moment for us - even though we foreshadowed it slightly in our late June post. Most importantly, the legacy Lagoon codebase (that runs only on OpenShift) is no longer supported, and that repository has now been archived.

From now on we'll follow a usual major/minor/patch scheme for Lagoon, So up next is Lagoon 2.1 - which we're already working on, as well as a vision for what Lagoon 3 could look like!

We took the opportunity in this release to trial a new security advisory process, using the built-in GitHub tooling (it's pretty neat!).

Of note is that the lagoon-charts are still in the 0.x major version range - this is because there is a significant breaking change on the horizon to make them compatible with Kubernetes. The release for that will be the first 1.x release of the charts.

We've also done a heap of work in the other Lagoon components:

the examples are starting to develop nicely - and with more tests than ever!
the images now have a settled monthly release cadence, with automated updates and tests
the other tools lagoon-cli and lagoon-sync are also being updated and developed, expect to hear more from us soon on them too!

Thanks to all the team that have brought this together, and to all our users for their patience, support and willingness to help us test and build together!

Improving image-specific testing with actual examples

Toby Bellwood — Thu, 23 Sep 2021 23:41:22 +0000

In Lagoon 1.x, Lagoon ran a full battery of tests for every pull request submitted to the GitHub repository. Not only was this inefficient, but unrelated errors also added false negatives to the testing process. In addition, running these tests required a full local instance of Lagoon to be created, which presented a significant technical and resource hurdle for contributions. With the addition of Kubernetes, this hurdle was reduced, but still significant.

At the same time as completing the work on Lagoon 2, we have been working hard on a set of example projects for Lagoon – covering a wider range of frameworks, content management systems, and situations than previously. We have gathered these examples over at https://github.com/uselagoon/lagoon-examples as Git submodules to the source repositories and branches.

We will be taking advantage of these example projects to test our images. We have built a CI process that will build our images, clone, build, and test the relevant example projects using the images being modified. With the introduction of automated dependency resolution triggering CI builds, we have automated a large portion of the image production process.

The tests we will be using for the process have been written in Markdown, and are parsed into Mocha tests by the excellent Leia parser from the Lando team. These tests are human-readable and easy to contribute to. You can see an example at https://github.com/amazeeio/drupal-example-simple/blob/9.x-advanced/TESTING_dockercompose.md.

Likewise, we will be running the same set of tests inside these example projects to ensure the projects work with the latest and edge releases of the images prior to any changes being merged.

Releasing more versions of more images more often

Toby Bellwood — Tue, 03 Aug 2021 00:12:48 +0000

Whilst we have traditionally offered multiple versions of some of our images (PHP, Node.js, Solr, and Python), we have started to encounter situations where broadening the range of versions (and the images available with versioning) would be beneficial to our users.

The most notable examples here are:

PostgreSQL (we’ve expanded our images available to include v12 alongside v11)
Python (adding new Python 3.x releases)
Redis (adding v6 alongside v5)
Varnish (adding v6 alongside v5)
MariaDB (adding 10.5 alongside 10.4)

As with our other images, we will follow the upstream EOL policies closely.

When we select whether to name images after a major version (i.e. Node.js 14) versus a minor version (i.e. PHP 7.4), we will look at the release frequency, semantic version policies, and the possibility for breaking changes.

We will also be able to generate pre-release images more easily – whilst they’re still in alpha/beta/RC status. Obviously, these images shouldn’t be used in production, but making them available for testing should be of benefit to us, our users, and the wider community.

This will allow our users to take advantage of the functionality available in the newer releases of these images. In order to configure this, the dockerfiles in your projects may need modification to point to the correct image for your purpose.

We have already written functionality into the release process to ensure that these new images are also published back to the existing images, under the same versions (i.e uselagoon/postgres-11:21.5.0 is the same image as uselagoon/postgres:21.5.0).

A complete list of these mappings is in the (makefile)[https://github.com/uselagoon/lagoon-images/blob/main/Makefile#L188]

Automating Lagoon Image updates from Upstream Source images

Toby Bellwood — Wed, 14 Jul 2021 23:58:04 +0000

As Lagoon images ordinarily extend “Docker Official” images with tooling and configuration necessary for Lagoon, we have a dependency on these upstream images. To date, we haven’t been able to exert much control over which versions of these images we use to build our versions. As we were refactoring the release process anyway, it made sense to look more closely at which exact versions of upstream images we consume, and how we could avoid additional burden in tracking them.

We identified that to specify exact versions in our Dockerfiles, and then to have an automated dependency resolution system watch these, and submit pull requests to us for any updates would be the most automated solution to this. We selected Whitesource Renovate for this purpose – the level of control over how these updates are selected was by far the most customizable for our needs. We didn’t want a situation where a Node.js 14 update was submitted for a Node.js 12 image, or where the PHP base version for two different images wasn’t simultaneously updated. We also wanted to be able to control whether a particular image got updates based on patch or minor releases.

We are always actively working on the configuration for Renovate and will continue to fine-tune it. Having pull requests created for every upstream version change will also optimise the generation of automated changelogs (using the excellent Release Drafter GitHub Action we were already using on Lagoon).

In early 2020, we made the first steps in aligning all our base images on Alpine 3.11, and because of this work, it has been much easier to upgrade to Alpine 3.12 and 3.13 along the way. Now that Alpine 3.14 has been released we will continue to monitor our upstreams, and once the balance of our images is available, we will commence the testing process.

We've also integrated vulnerability scanning into the CI process - using Aqua Security's Trivy scanner to scan each image as it's created and saving the results in the CI run. We'll cover securities and vulnerabilities in a later post!

Lagoon 2 release candidate available

Toby Bellwood — Mon, 28 Jun 2021 23:31:18 +0000

Last week, the Lagoon team released their first release candidate for Lagoon 2. In this post we'll cover a few of the headline features that make it slightly different to Lagoon 1.

Updated architecture

In Lagoon 1.x, Lagoon could support a local and (under certain conditions) remote Openshift 3 clusters. In Lagoon 1.4, we added additional support for remote Kubernetes clusters.

Having reviewed this architecture, it became clear to the team that supporting remote workloads should be the norm, giving a logical separation to the services needed to power Lagoon and the client workloads running on it. In addition, Lagoon required a high level of administrative access to the remote clusters, which could have presented an issue with higher-security installations. Ben has developed a great controller-based solution that effectively delegates the job of deploying projects and environments, running tasks, and monitoring builds to these target remote clusters, and only connecting to the main Lagoon to update status, or collect new builds or tasks.

Going forward, we will be referring to the main Lagoon install (where the API, authentication, notification and other operational services live) as “Lagoon Core”. The Lagoon component that is used to run client projects (where the environments, database services, etc, run) is now referred to as “Lagoon Remote.” This allows Remote to just be an application installed into a cluster managed by a different team. Of course, for smaller installs, Lagoon Core and Remote can be colocated.

To fully support the Kubernetes ecosystem, both Lagoon Core and Lagoon Remote (as well as the other allied Lagoon tools) can all be deployed using Helm charts.

Find more about the architecture in our Git Repository

Node 16, Alpine 3.13 and the updated MariaDB connector

Lagoon was built on Node.js 10, and has been pinned there for the last few years, because of a complicated web of npm dependencies. In preparation for Lagoon 2, Brandon untangled this web as part of upgrading our MariaDB/MySQL connector to the latest version of the official npm package (https://www.npmjs.com/package/mariadb). This gives us some of the newly-available performance benefits, as well as unblocking the Node.js 16 upgrade for the rest of the application. We’ve also updated all the system images to the latest version available of Alpine 3.13. The one exception is the MariaDB-based api-db and keycloak-db images are still on MariaDB 10.4 (while we work out the upgrade path to 10.5).

New logs2webhook service

More and more of our users are tying their Lagoon deployments into other systems. They are used to retrieve backups or database dumps, check post-deploy UI consistency, and perform other tasks. In order to integrate these systems more closely together, Blaize has added a logs2webhooks service that will allow users to create another notification endpoint (to join the existing Slack, email, Microsoft Teams and RocketChat versions). We’ve got a broader overhaul of the notification system in our sights, but this will help our users now, so we’ve added it.

API Audit logging

One of the most important aspects of running systems that are placed into highly secure environments is the presence of audit logging for privileged users. While Lagoon already had very verbose logs, pinpointing some API-driven requests could be fairly taxing. Tim has added a logging service into the API that collects the requests, and generates a searchable log entry for it. This will return the user, source, command and other necessary information. It makes it much easier to find out who actioned what, and where it came from. Again, it’ll be refined over time, but we thought it important enough to release now.

Lagoon 2 supports per-cluster Harbor 2 installations

Lagoon’s traditional architecture has closely coupled the installation of Harbor (the image registry) with the Lagoon component. Now that we have moved to a Lagoon Core and Lagoon Remote architecture, it makes sense to locate the registry as physically close to the destination projects as possible, to minimize transfer times, cost, and latency, as well as tightening the security perimeter. Administrators have the option of defining which image registry is in use for a particular cluster. This allows for multi-cluster installations to share a common image registry.

Adding Gitea support to Lagoon

Lagoon has long supported all versions of GitHub, GitLab and Bitbucket as source code management options. As part of the local development workflow we undertook for Lagoon 2, we were looking for a full-featured local Git server to enable more rapid testing and prototyping. Gitea is a lightweight self-hosted Git service, available as a Helm chart and deployable anywhere. We have added the necessary webhook handlers into Lagoon to support it (they differ ever so slightly from GitHub). For anyone looking for a completely in-cluster (either in-cloud or local) workflow, Gitea now works with Lagoon!

Develop locally on Kubernetes with an expanded suite of tests

Lagoon had a lot of tests – covering core operations (variables, promotions, active/standby), integrations (Gitlab, GitHub and BitBucket, DBaaS) and project types (Drupal, Python, Node.js etc). These test processes always ran sequentially, and could not be easily selected. Scott has done a fantastic job in rebuilding Lagoon to run locally via Helm charts, but also in configuring Helm’s chart-testing facility to control test runs. Toby has done a lot of work to refactor these tests to be more self-sufficient – meaning that they can provision and remove their own test projects, and also to overhaul the entire suite – adding new tests (mongoDB, Elasticsearch), upgrading old ones, and adding new versions.

And there will be more coming!

As we write this, the next collection of new features to Lagoon 2 are nearing completion. Now that the team can solely focus on Lagoon 2 development, expect to see some great stuff!