DEV Community: Mark Harrison

Better multi-tenancy with Octopus Deploy

Mark Harrison — Mon, 26 Jul 2021 10:15:10 +0000

Most people using Octopus will deploy projects to one or more environment. For customers providing Software as a Service (SaaS) applications, they typically need to deploy multiple instances of the application for each of their customers.

Fortunately, there's a feature that's been around since Octopus 3.4 designed exactly for these types of deployment, multi-tenancy.

In this post, I look at two approaches to deploying applications without tenants, and discuss the benefits of using the multi-tenancy feature.

In this post

Introduction
Deploying without tenants
- Using multiple projects
  - Multiple projects pros
  - Multiple projects cons
- Using multiple environments
  - Multiple environment pros
  - Multiple environment cons
Deploying with tenants
- Tenants
- Tenant tags
- Tenant variables
  - Project variable templates
  - Common variables
  - Variable snapshot
  - Missing variables
- Tenanted deployment targets
Conclusion
Learn more
Register for the webinar: Better multi-tenancy deployments using Octopus Deploy

Introduction

This post assumes knowledge of some key Octopus concepts, including:

Projects
Environments
Variables
Lifecycles

If you're new to Octopus, I recommend reading our Getting started with Octopus guide.

To demonstrate how you can model deployments of multiple instances of an application with Octopus, I use a fictitious company called Vet Clinic, deploying the Java application, Pet Clinic.

Deploying without tenants

There are two main implementations we see when deploying multiple instances of the same application for each customer:

Using multiple projects
Using multiple environments

While easy to set up, they don't scale well and can result in duplication.

Using multiple projects

In this scenario, you configure Octopus with multiple projects, each one representing one of your customers.

Onboarding a new customer typically requires creating all of the resources in Octopus needed for a successful deployment for the customer, including:

A new set of deployment targets
Common project variables
Any customer specific "paid-for" environments

In addition, any common steps across the application's deployment process need to be duplicated in the new project. These are usually manual intervention and notification steps.

Multiple projects pros

So why choose multiple Octopus projects to deploy instances of an application to each customer?

Clear customer release dashboard overview

This approach allows you to see which release has been deployed to which environment, for each customer on the dashboard overview.
Variable and Deployment process isolation

Multiple projects allow for complete isolation of variables, and deployment process for a customer. For example, making a change to one project's process only affects that one customer. You can also tailor the deployment process for the customer depending on the features they've signed up for.

In the below example, only Capital Animal Hospital has a step for applying custom branding:
Simpler environment and variable scoping

Environments don't need to be duplicated per customer, resulting in simpler lifecycle configuration. Variables can also be scoped to each environment without risk of choosing the wrong "customer" scoping.

Multiple projects cons

While multiple projects can be used to deploy customer instances separately, there are a number of problems with this approach.

Duplicated project configuration

With every customer project, you end up duplicating project configuration. These include variables, deployment process steps, runbooks, channels and lifecycles. With duplication comes inconsistency and the overhead of managing that. For example, if you want to amend the deployment process for all of your customers, you need to change multiple projects.
Different deployment target roles per customer

If your customers have isolated infrastructure, you need a unique way for Octopus to know which deployment targets belong to the customer you're deploying to. This results in each customers' deployment targets needing target roles that include a differentiator per customer. This is typically a customer name, code or ID.
Multiple projects per customer doesn't scale

The per-project approach doesn't scale well when you have more than a handful of customers. This can be problematic if you deploy many more applications per customer. Each application needs to be modeled n times, where n is the number of customers you have.
No guard rails to ensure variables provided

Using multiple projects, there are no guard rails to ensure all project configuration is set correctly. For example, if a variable is not added (or has an incorrect value), you might not find out about the issue until the deployment of the customer's instance.

Using multiple environments

An alternative to deploying without tenants is using a single Octopus project per application, and modeling each customer with a set of environments they deploy to.

Onboarding a new customer typically involves:

Creating a new set of environments, named after the customer.
Creating a new set of deployment targets, or re-using existing ones and tagging them with the associated customer environments.
Adding new environment-scoped variables.
Updating the project lifecycle to include the new customer environments.

Multiple environment pros

So why choose one or more environments to represent your customers in Octopus?

Single set of project resources to manage

In contrast to multiple projects, with this approach there's just one project, one deployment process, one set of variables, and one lifecycle to manage. When a change is required, it can be made once instead of per project.

For example, if you want to add a step that's required for all customers, such as a Manual intervention step before deployments to production, it can be added quickly and easily.
Customer environments are modeled explicitly

With this approach customer environments have to be modeled explicitly. When a new customer is added, the environment they deploy to has to be created to allow deployment for that customer. It's also possible to see in a single row on the dashboard overview which environment a customer can deploy to.

Multiple environment cons

Although you can use multiple customer environments, there are usually many problems associated with this approach:

Multiple environments created per customer

For each customer, you need to create a new environment record for each customer environment, which doesn't scale. For example, if you have 10 customers, and 4 environments (Development, Test, Staging and Production), you need to create 40 customer environments.
Complicated variable scoping

As there's a single project per application, handling multiple variable values per customer needs to be achieved using different environment scopes. This can quickly become overwhelming. Adding new values, or editing existing ones has to be done carefully to ensure correct scopes are applied to each value. There's a high risk of cross tenant communication in this model.
Rigid deployment process

Customizing the deployment process in a project using multiple customers requires you to add environment run conditions for each step that needs to be run for specific customers. This isn't flexible and doesn't scale as you need to modify these conditions per step, when you add a new customer or change your environments.
Unclear customer release dashboard overview

In contrast to the per-project customer model, it's difficult to see at a glance which release has been deployed to which customer on the dashboard and project overview screens, without endless scrolling.
Complicated, unwieldy Lifecycle

As you onboard new customers, you typically add new customer environments to the project's lifecycle. You then have to define awkward lifecycle phases that fit the environments that are required for each customer (both existing and new). Typically this is handled with a phase that allows any 1 customer environment to be deployed to before progressing to the next phase. As the number of customers grows, so does the complexity of the lifecycle and its phases.

Deploying with tenants

Using tenants in Octopus allows you to easily create customer specific deployment pipelines without duplicating project configuration. You can manage separate instances of your application in multiple environments in a single Octopus project.

Using our Vet Clinic company, here's what the dashboard overview might look like using tenants to model each customer:

This gives us a concise overview showing which release is in which environment. Instead of multiple rows per customer project, this is replaced with a discrete count of tenants who have been deployed to each environment.

If we navigate to the project, we see a more granular overview, this time showing which tenant has what release in each environment:

Sample Octopus project
You can see an example of the Vet Clinic tenanted project in our samples instance.

There are a number of multi-tenancy features working together to make this happen:

Tenants
Tenant tags
Tenant variables
Tenanted deployment targets

Tenants

Tenants in Octopus are the backbone of the multi-tenancy feature. They usually represent the customers of your application, especially when it comes to SaaS products.

Although we discuss the use of tenants to model customers in this post, we designed tenants to be generic so that they can satisfy multiple use cases. Tenants can also represent:

Geographical regions or data centers
Developers, testers, or teams
Feature branches

Learn more about types of tenancy in our documentation.

The tenant overview provides a central place to manage which projects are connected to a tenant, and also which environment.

Note that each project can control its interaction with tenants. By default the multi-tenant deployment features are disabled. You can allow deployments with/without a tenant, which is a hybrid mode that's useful when you're transitioning to a fully multi-tenant project.

There's also a mode where you can require a tenant for all deployments, which disables untenanted deployments for that project.

Onboarding a new customer as a tenant in Octopus can be as simple as creating your tenant, connecting your project to each applicable environment and entering your variable values, then deploying.

Tenant tags

In Octopus, tenant tags help you classify your tenants using custom tags, and tailor tenanted deployments for your projects and environments.

Tenant tags also make it easier to work with tenants as groups instead of individuals. As tenant tags are fully customizable, you can apply meaningful metadata to tenants. This allows you to describe them using your own terminology, and tailor the deployment process to their needs.

In the tenant overview below, Capital Animal Hospital has the Branding tag included:

This indicates that they've opted-in for customized branding of their instance of the Vet Clinic application.

When you build out the deployment process, you can include a tenant tag as a run condition to customize the process for your customers. By applying tags to steps, you are able to specify steps that should only run for customers that match selected tenant tags.

You can associate multiple tenants with the same tag. This automatically groups these tenants together and enables any tenant with the Branding tag to have the custom branding step included as part of any deployment for that tenant.

Tenant tags can also be used to associate multiple tenants with deployment targets and channels, and even choosing which tenants to deploy to. They're a powerful way to help you simplify and scale your deployments.

Tenant variables

You often want to define variable values that are different for each customer. For example:

A database server name or connection string
A tenant-specific URL
Contact details for a tenant

Using an untenanted project, you would define these values in the project itself. With a tenanted project, you can set these values directly on the tenant for any connected projects.

With tenants, there are two types of variable you can specify:

Project variable templates
Common variables.

These both use the variable templates feature.

Project variable templates

Project variables allow you to specify a variable which a tenant can change. A perfect example is a connection string or a database server. With project variables you define them at the project level using project templates.

You can specify the variable type for the project template, just like regular variables. You can also provide a default value which the tenant can overwrite.

Then, on the tenant variable screen, you can set those variables.

Common variables

Common variables are similar to project variables. The main difference is that common variables can be used across multiple projects, and they aren't scoped to environments. Common variables are defined using Library variable set templates

For example, to define an abbreviation for the tenant to use in a deployment or runbook, you can configure a variable template for the library set.

To include common variables for a tenant, you must add the library variable set in the tenant connected project.

Just like project variables, common variable values are supplied at the tenant level.

Variable snapshot

When you create a release Octopus takes a snapshot of the deployment process and the current state of the project variables.

However, tenant variables aren't included in any snapshot. This is helpful as you can add new tenants at any time and deploy to them without creating a new release.

This also means any changes you make to tenant variables will take immediate effect.

Missing variables

One of the great things about tenant variables is the guard rails they put in place for your deployments. Defining either a project template or common variable without a default value means any tenant must provide a value for that variable. Octopus won't allow a deployment to occur without one:

But those guard rails don't start just at deployment. Octopus will also warn you about any missing values in the tenant's variable overview too:

This level of safety reduces the chance of a failed deployment for a tenant due to a missing or incorrect variable value.

Tenanted deployment targets

The way you host your infrastructure for multiple instances of the same project usually varies depending on your application and customers. Two common implementations we see are:

Dedicated hosting: You have dedicated deployment targets for each customer.
Shared hosting: You create farms or pools of servers to host all of your customers, achieving higher density.

In this post, we focus on dedicated hosting, but you can design and implement both dedicated and shared multi-tenant hosting models in Octopus using environments, deployment targets, and tenant tags.

If your customers have isolated infrastructure, in an untenanted configuration, you need to define unique target roles in Octopus to ensure there's no cross-customer communication in the form of deploying one customer's application to another customer's infrastructure.

Using tenants, customer-specific target roles aren't required. You can choose deployments that a target can be involved in:

Exclude from tenanted deployments (default) - The deployment target will never be included in tenanted deployments.
Include only in tenanted deployments - the deployment target will only be included in deployments to the associated tenants. It will be excluded from untenanted deployments.
Include in both tenanted and untenanted deployments - The deployment target will be included in untenanted deployments, and deployments to the associated tenants.

To choose which tenants to associate with a deployment target:

Navigate to the Restrictions ➜ Associated Tenants section of the deployment target.
Select one or more tenants to allow to deploy to individually, or choose from any of the configured tenant tags.

We recommend keeping tenanted and untenanted deployment targets separate, particularly in Production. You could use the same deployment targets for other environments but it's generally better to avoid this situation.

Conclusion

This post covers common approaches when customers deploy multiple instances of the same application for each of their customers without tenants. It also details how you can use the multi-tenancy feature to model this too.

I hope you can see how the Octopus multi-tenancy feature solves some of the problems presented when deploying without tenants, and how it can be leveraged for scalable, reusable, simplified deployments.

Learn more

Register for the webinar: Better multi-tenancy deployments using Octopus Deploy

Join Senior Solutions Architect Mark Harrison and Solutions Architect Adam Close from Octopus Deploy to learn how the multi-tenancy feature in Octopus can be leveraged for scalable, reusable, simplified deployments.

We're running three sessions of the webinar:

Wed 28 Jul, 2021
Thurs 29 Jul, 2021
Mon 2 Aug, 2021

Happy deployments!

This post was originally published at octopus.com.

Using Azure Key Vault with Octopus

Mark Harrison — Wed, 23 Jun 2021 11:31:12 +0000

I recently wrote about extending the functionality of Octopus to integrate with HashiCorp Vault using step templates. Afterwards, several people asked if I plan to create step templates to integrate with other secret managers.

In this post, I walk through a new step template, Azure Key Vault - Retrieve Secrets, which is designed to retrieve secrets from an Azure Key Vault for use in your deployments or runbooks.

Introduction

This post assumes some familiarity with custom step templates and the Octopus Community Library.

In addition, this post doesn't go into great detail about Azure Key Vault concepts or how to set it up. You can learn more by reading the Azure Key Vault basic concepts guide from Microsoft.

The step template in this post retrieves secrets from an Azure Key Vault using the Az.KeyVault PowerShell module. The module must be downloaded and installed on the deployment target or worker before the step can retrieve secrets successfully. The step template has been tested on both Windows and Linux (with PowerShell Core installed).

Authentication

Before you can retrieve secrets from Azure Key Vault, you must authenticate with Azure. In their authentication concepts documentation, Microsoft note:

Authentication with Key Vault works in conjunction with Azure Active Directory (Azure AD), which is responsible for authenticating the identity of any given security principal.

In Octopus, authentication with Azure Key Vault can be achieved with an Azure Account, using a service principal.

In addition to accessing resources in Azure, your service principal may need further permissions configured to access and retrieve secrets stored in Azure Key Vault. To learn more, read the Azure Key Vault RBAC guide on how to provide access to keys, certificates, and secrets with an Azure role-based access control.

Retrieving secrets

The Azure Key Vault - Retrieve Secrets step template retrieves one or more secrets from an Azure Key Vault and creates sensitive output variables for each one retrieved.

For each secret, you can optionally choose to retrieve a specific version, and provide a custom output variable name.

Retrieving a single secret requires:

An Azure account with permission to access the secret.
The name of the Azure Key Vault to retrieve the secret from.
The name of the secret to retrieve.

An advanced feature of the step template offers support for retrieving multiple secrets at once. This requires entering each secret on a new line.

For each secret retrieved, a sensitive output variable is created for use in subsequent steps. By default, only a count of the number of variables created will be shown in the task log. To see the names of the variables in the task log, change the Print output variable names parameter to True.

Step template parameters

The step template uses the following parameters:

Azure Account: An Azure account with permissions to retrieve secrets from the Azure Key Vault.
Vault Name: The name of the Azure Key Vault to retrieve secrets from.
Vault Secrets to retrieve: Specify the names of the Secrets to be returned from Azure Key Vault, in the format: SecretName SecretVersion | OutputVariableName where:
- SecretName is the name of the Secret to retrieve.
- SecretVersion is the optional version of the Secret to retrieve. If this value isn't specified, the latest version will be retrieved.
- OutputVariableName is the optional Octopus output variable name to store the secret's value in. If this value isn't specified, an output name will be generated dynamically.
Note: Multiple fields can be retrieved by entering each one on a new line.
Print output variable names: Write out the Octopus output variable names to the task log. Default: False.
Az PowerShell Module version (optional): If you wish to use a specific version of the Az PowerShell module (rather than the default), enter the version number here. e.g. 5.9.0.

Note: The version specified must exist on the machine.

Az PowerShell Install Location (optional): If you wish to provide a custom path to the Az PowerShell module (rather than the default), enter the value here.

Note: The Module must exist at the specified location on the machine. This step template will not download the Module.

Using the step

The Azure Key Vault - Retrieve Secrets step is added to deployment and runbook processes in the same way as other steps.

After you've added the step to your process, fill out the parameters in the step:

After you've filled in the parameters, you can execute the step in a runbook or deployment process. On successful execution, any matching secrets will be stored as sensitive output variables. If you've configured your step to print the variable names, they'll appear in the task log:

In subsequent steps, output variables created from matching secrets can be used in your deployment or runbook.

Tip: Remember to replace Azure Key Vault - Retrieve Secrets with the name of your step for any output variable names.

Conclusion

The step template covered in this post demonstrates that it's easy to integrate with Azure Key Vault, and make use of secrets stored there with your Octopus deployments or runbooks.

Happy deployments!

This post was originally published at octopus.com.

Using HashiCorp Vault with Octopus Deploy

Mark Harrison — Mon, 24 May 2021 10:19:57 +0000

Storing sensitive values in Octopus Deploy solves many problems. If your organization has standardized on a secrets manager though, that might mean storing sensitive values twice, making secrets management more complicated.

Octopus has supported the concept of sensitive variables since Octopus 2.0, but customers often ask about support for secret managers. One in particular is HashiCorp Vault.

In this post, I walk through a number of new HashiCorp Vault step templates designed to retrieve secrets from Vault for use in your deployments or runbooks.

In this post

Introduction
Authentication
- LDAP login step
  - LDAP login parameters
  - Using the LDAP login step
- AppRole login step
  - AppRole login parameters
  - Using the AppRole login step
- AppRole best practises
- AppRole Get Wrapped SecretID step
  - AppRole Get Wrapped SecretID parameters
  - Using the AppRole Get Wrapped SecretID step
- AppRole Unwrap SecretID step
  - AppRole Unwrap SecretID parameters
  - Using the Unwrap SecretID step
- AppRole Unwrap SecretID and Login step
  - AppRole Unwrap SecretID parameters
  - Using the Unwrap SecretID step
Retrieving secrets
- Retrieve KV (v1) secrets step
  - Retrieve KV (v1) secrets parameters
  - Using Retrieve KV (v1) secrets step
- Retrieve KV (v2) secrets step
  - Retrieve KV (v2) secrets parameters
  - Using Retrieve KV (v2) secrets step
Conclusion
Learn more

Introduction

This post assumes some familiarity with custom step templates and the Octopus Community Library. To learn more about these, you can read Ryan Rousseau's two-part series on creating your own step template and publishing it to the library.

In addition, this post doesn't go into great detail about Vault server concepts or how to configure a Vault server.

The step templates covered in this post perform both Vault authentication and secret retrieval for both versions 1 and 2 of the Key-Value (kv) Secrets Engine.

All of the step templates make use of the Vault HTTP API so there are no additional dependencies required to use them, except being able to connect to your Vault server. They've all been tested using Vault version 1.7.1 and can run on both Windows and Linux (with Powershell Core installed).

Authentication

Before interacting with Vault, you must authenticate against an auth method. Vault offers a number of different authentication options. The following step templates have been created to support Vault authentication:

LDAP login
AppRole login
AppRole Get wrapped SecretID
AppRole Unwrap SecretID
AppRole Unwrap SecretID and Login

The AppRole method is the recommended way to authenticate with Vault for servers.

Upon authentication with Vault, a token is generated that can be used in further interactions with Vault.

LDAP login step

The HashiCorp Vault - Login with LDAP step template authenticates with a Vault Server using the LDAP authentication method. This allows Vault integration without having to duplicate username or password configuration.

You might choose to authenticate using LDAP if you already have an LDAP server available and use service accounts to control access to sensitive information.

Microsoft's Active Directory supports LDAP using Active Directory Lightweight Directory Services.

After authentication, the client_token from the Vault response will be made available as a sensitive output variable named LDAPAuthToken for use in other steps.

LDAP login parameters

The step template has the following parameters:

Vault Server URL: The URL of the Vault instance you are connecting to, including the port (The default is 8200).
API version: Choose the API version to use from a drop-down list. Currently, there is only one option: v1.
LDAP Auth Login path: The path that the LDAP method is mounted at. The default is /auth/ldap.
Username: The LDAP username.
Password: The LDAP password.

Using the LDAP login step

The LDAP login step is added to deployment and runbook processes in the same way as other steps.

After you've added the step to your process, fill out the parameters in the step:

You can then execute the step in a runbook or deployment process. On successful execution, the sensitive output variable name containing the token is displayed in the task log:

In subsequent steps, the output variable #{Octopus.Action[HashiCorp Vault - Login with LDAP].Output.LDAPAuthToken} can be used to authenticate and retrieve secrets.

Tip: Remember to replace HashiCorp Vault - Login with LDAP with the name of your step for any output variable names.

AppRole login step

The HashiCorp Vault - Login with AppRole step template authenticates with a Vault Server using the AppRole authentication method. This is perfect for use with Octopus. HashiCorp themselves recommend it for machines or apps:

This auth method is oriented to automated workflows (machines and services), and is less useful for human operators.

With an AppRole, a machine can log in with:

RoleID, think of this as the username in an authentication pair.
SecretID, think of this as the password in an authentication pair.

Don't store the SecretID:
Storing the RoleID in Octopus as a sensitive variable is a good way to ensure it remains encrypted until required.

However, the same is not recommended for the SecretID.

A SecretID, just like a password is designed to expire. Storing the SecretID could also provide the capability to retrieve all secrets as both the RoleID and SecretID would be available.

We recommend you use the more secure Get wrapped SecretID and Unwrap SecretID and Login step templates, as they use one of the best practices response wrapping.

If you use the AppRole login step template, we recommend you provide the SecretID at execution time using a sensitive prompted variable.

Once authenticated, the client_token from the Vault response will be made available as a sensitive output variable named AppRoleAuthToken for use in other steps.

AppRole login parameters

The step template has the following parameters:

Vault Server URL: The URL of the Vault instance you are connecting to, including the port (The default is 8200).
API version: Choose the API version to use from a drop-down list. Currently, there is only one option: v1.
App Role Path: The path where the approle auth method is mounted.
Role ID: The RoleID of the AppRole.
Secret ID: The SecretID of the AppRole.

Using the AppRole login step

The AppRole login step is added to deployment and runbook processes in the same way as other steps.

After you've added the step to your process, fill out the parameters in the step:

You can then execute the step in a runbook or deployment process. On successful execution, the sensitive output variable name containing the token is displayed in the task log:

In subsequent steps, the output variable #{Octopus.Action[HashiCorp Vault - Login with AppRole].Output.AppRoleAuthToken} can be used to authenticate and retrieve secrets.

Tip: Remember to replace HashiCorp Vault - Login with AppRole with the name of your step for any output variable names.

AppRole best practices

The AppRole authentication method is considered a trusted-broker method. This means that the onus of trust rests in the system acting as the authentication intermediary (the broker) between the client (typically an Octopus deployment target) and Vault.

An important best practice is to avoid storing an AppRole SecretID. Instead, use response wrapping to provide a wrapping token that will provide an access mechanism to retrieve a SecretID when required. This method of obtaining a SecretID is also known as a Pull mode as it requires the SecretID to be fetched or pulled from the AppRole.

The Vault documentation contains recommended patterns when using AppRole authentication.

Here's a summary of the recommendations:

Use a secured system to act as the broker for retrieving a wrapped SecretID.
Use response wrapping to obtain a SecretID.
Limit the number of uses and Time-to-live (TTL) value for a SecretID to prevent overuse.
Avoid anti-patterns such as having the broker retrieve secrets.

Secure the broker:
Since the trust rests on the broker, we strongly recommend using the Octopus Server's built-in worker, or a highly-secured external worker to act as the broker. It would be responsible for retrieving a wrapped SecretID and passing that value to the machine (the client) that authenticates with Vault.

To support these recommended practices, three additional AppRole step templates have been created:

AppRole Get Wrapped SecretID
AppRole Unwrap SecretID
AppRole Unwrap SecretID and Login

AppRole Get Wrapped SecretID step

The HashiCorp Vault - AppRole Get Wrapped Secret ID step template generates a response-wrapped SecretID for the AppRole authentication method.

The step template authenticates with a Vault Server using a token to retrieve a wrapped SecretID. The response contains a wrapping token and other metadata such as the creation path for the token.

This value can be used to validate no malfeasance has occurred. The wrapping token can then be used to retrieve the actual SecretID value.

The token used to authenticate to retrieve a wrapped SecretID should be of limited scope and should only be allowed to retrieve wrapped SecretIDs. Consider creating a long-lived Vault token as this presents only a minor risk.

After the response has been received from the Vault server, two sensitive output variables are created for use in other steps:

WrappedToken This is the wrapped token from the response, used to retrieve the actual SecretID value.
WrappedTokenCreationPath This is the creation path for the token. It allows you to validate no malfeasance has occurred.

AppRole Get Wrapped SecretID parameters ### AppRole Get Wrapped SecretID step

The step template uses the following parameters:

Vault Server URL: The URL of the Vault instance you are connecting to, including the port (The default is 8200).
API version: Choose the API version from a drop-down list. Currently, there is only one option: v1.
App Role Path: The path where the AppRole auth method is mounted.
Role Name: The role name of the AppRole.
Time-to-live (TTL): The TTL in seconds of the response-wrapping token itself. The default is: 120s
Auth Token: The token used to authenticate with Vault to generate a response-wrapped SecretID.

Using the AppRole Get Wrapped SecretID step

The Get Wrapped SecretID step is added to deployment and runbook processes in the same way as other steps.

After you've added the step to your process, fill out the parameters in the step:

You can then execute the step in a runbook or deployment process. On successful execution, the sensitive output variable names are displayed in the task log:

In subsequent steps, the output variables can be used to validate and retrieve the actual SecretID value:

#{Octopus.Action[HashiCorp Vault - AppRole Get Wrapped Secret ID].Output.WrappedToken}
#{Octopus.Action[HashiCorp Vault - AppRole Get Wrapped Secret ID].Output.WrappedTokenCreationPath}

Tip: Remember to replace HashiCorp Vault - AppRole Get Wrapped Secret ID with the name of your step for any output variable names.

AppRole Unwrap SecretID step

The HashiCorp Vault - AppRole Unwrap Secret ID step template retrieves and unwraps a SecretID for an AppRole using a wrapping token.

The secret_id from the Vault response will be made available as a sensitive output variable named UnwrappedSecretID for use in other steps.

AppRole Unwrap SecretID parameters

The step template uses the following parameters:

Vault Server URL: The URL of the Vault instance you are connecting to, including the port (The default is 8200).
API version: Choose the API version to use from a drop-down list. Currently, there is only one option: v1.
Wrapped Token: The wrapping token used to retrieve the actual Secret ID from Vault.
Token Creation Path: Optional The creation path for the wrapped token. If this value is provided, the step template will perform a wrapping lookup to validate no malfeasance has occurred.

Using the Unwrap SecretID step

The Unwrap SecretID step is added to deployment and runbook processes in the same way as other steps.

After you've added the step to your process, fill out the parameters in the step:

Note the use of sensitive output variables in the step parameters. In this example, the values are created using the Get Wrapped SecretID step template named HashiCorp Vault - AppRole Get Wrapped Secret ID.

You can then execute the step in a runbook or deployment process. On successful execution, the sensitive output variable names are displayed in the task log:

In subsequent steps, the output variable #{Octopus.Action[HashiCorp Vault - AppRole Unwrap Secret ID].Output.UnwrappedSecretID} can be used to authenticate with Vault and receive a token that can then be used to retrieve secrets.

Tip: Remember to replace HashiCorp Vault - AppRole Unwrap Secret ID with the name of your step for any output variable names.

AppRole Unwrap SecretID and Login step

The HashiCorp Vault - AppRole Unwrap Secret ID and Login step template is provided as a convenient way to combine two step templates used with Vault into one:

AppRole Unwrap SecretID: It retrieves and unwraps a SecretID for an AppRole using a wrapping token.
AppRole login: It authenticates with Vault using a supplied RoleID and the unwrapped SecretID.

It's designed as the second part of a two-step workflow with Vault:

Get a wrapped SecretID using the AppRole Get wrapped SecretID step template.
Provide the wrapped SecretID stored in a sensitive output variable from the first step to this step template to unwrap and authenticate.

Once authenticated, the client_token from the Vault response will be made available as a sensitive output variable named AppRoleAuthToken for use in other steps.

AppRole Unwrap SecretID and Login parameters

The step template uses the following parameters:

Vault Server URL: The URL of the Vault instance you are connecting to, including the port (The default is 8200).
API version: Choose the API version to use from a drop-down list. Currently, there is only one option: v1.
App Role Path: The path where the AppRole auth method is mounted.
Role ID: The RoleID of the AppRole.
Wrapped Token: The wrapping token used to retrieve the actual Secret ID from Vault.
Token Creation Path: Optional The creation path for the wrapped token. If this value is provided, the step template will perform a wrapping lookup to validate no malfeasance has occurred.

Using the Unwrap SecretID and Login step

The Unwrap SecretID and Login step is added to deployment and runbook processes in the same way as other steps.

After you've added the step to your process, fill out the parameters in the step:

Note the use of sensitive output variables in the step parameters. In this example, the values were created using the Get Wrapped SecretID step template named HashiCorp Vault - AppRole Get Wrapped Secret ID.

You can then execute the step in a runbook or deployment process. On successful execution, the sensitive output variable names are displayed in the task log:

In subsequent steps, the output variable #{Octopus.Action[HashiCorp Vault - AppRole Unwrap Secret ID and Login].Output.AppRoleAuthToken} can be used to authenticate, and retrieve secrets.

Tip: Remember to replace HashiCorp Vault - AppRole Unwrap Secret ID and Login with the name of your step for any output variable names.

Retrieving secrets

After you've authenticated with Vault, you receive an authentication token that can be used to retrieve secrets. Secrets in Vault are stored in a secrets engine, of which there are many different types.

The step templates created to support retrieving secrets focus on the Key-Value (kv) Secrets Engine as it's a generic Key-Value store used to store arbitrary secrets:

Retrieve KV (v1) secrets step
Retrieve KV (v2) secrets step

Retrieve KV (v1) secrets step

The HashiCorp Vault - Key Value (v1) retrieve secrets step template retrieves one or more secrets stored in a v1 Key-Value secrets engine.

Retrieving a single secret requires:

The path to the secret.
An authentication token with permission to access the secret.
Optionally, a list of field names to retrieve.

An advanced feature of the step template offers support for retrieving multiple secrets at once. This requires changing the Secrets retrieval method parameter to Multiple vault keys.

It's also possible to recursively retrieve secrets. This is useful when you want to retrieve all secrets for a given path.

Retrieve KV (v1) secrets parameters

The step template uses the following parameters:

Vault Server URL: The URL of the Vault instance you are connecting to, including the port (The default is 8200).
API version: Choose the API version to use from a drop-down list. Currently, there is only one option: v1.
Auth Token: The token used to authenticate to retrieve secrets.
Secrets Path: The full path to the secret(s) you want to retrieve. The value should include both the path where the secrets engine is mounted, as well as the path to the secret itself.
Secrets retrieval method: Choose between retrieving a single secret or multiple secrets. Retrieving a single secret is the equivalent of a vault kv get command using the Get method. Retrieving multiple secrets is the equivalent of the combination of both a vault kv list command using the List method and then subsequent vault kv get commands for each secret.
Recursive retrieval: If multiple secrets are being retrieved, should any sub-folders also be enumerated and retrieved? The default is: False.
Field names: Choose specific fields to be retrieved from identified secrets. This is useful when you only want to retrieve specific fields from one or more secrets. You can optionally include a name for the output variable.
Print output variable names: Write out the Octopus output variable names to the task log. The default is: False.

Using Retrieve KV (v1) secrets step

The Key Value (v1) retrieve secrets step is added to deployment and runbook processes in the same way as other steps.

After you've added the step to your process, fill out the parameters in the step:

Note the use of the sensitive output variable in the Auth Token parameter. In this example, the value was created using the Unwrap SecretID and Login step template named HashiCorp Vault - AppRole Unwrap Secret ID and Login.

In subsequent steps, output variables created from matching secrets can be used in your deployment or runbook.

Tip: Remember to replace HashiCorp Vault - Key Value (v1) retrieve secrets with the name of your step for any output variable names.

Retrieve KV (v2) secrets step

The HashiCorp Vault - Key Value (v2) retrieve secrets step template retrieves one or more secrets stored in a v2 Key-Value secrets engine.

One of the key advantages of the v2 Key-Value secrets engine is its support for versioned secrets. This is useful if you need to roll back secrets in the event of unintentional data loss.

Retrieving a single secret requires:

The path to the secret,
An authentication token with permission to access the secret.
Optionally, a list of field names to retrieve.

This step template offers advanced features:

Support for retrieving multiple secrets at once. This requires changing the Secrets retrieval method parameter to Multiple vault keys. It's also possible to recursively retrieve secrets. This is useful to retrieve all secrets for a given path.
Support for retrieving a specific version of a secret. This is only supported when retrieving a single secret.

For each secret retrieved, a sensitive output variable is created for use in subsequent steps. By default, only a count of the number of variables created will be shown in the task log. To see the names of the variables in the Task log, change the Print output variable names parameter to True.

Retrieve KV (v2) secrets parameters

The step template uses the following parameters:

Vault Server URL: The URL of the Vault instance you are connecting to, including the port (The default is 8200).
API version: Choose the API version to use from a drop-down list. Currently, there is only one option: v1.
Auth Token: The token used to authenticate to retrieve secrets.
Secrets Path: The full path to the secret(s) you want to retrieve. The value should include both the path where the secrets engine is mounted, as well as the path to the secret itself.
Secrets retrieval method: Choose between retrieving a single secret or multiple secrets. Retrieving a single secret is the equivalent of a vault kv get command using the Get method. Retrieving multiple secrets is the equivalent of the combination of both a vault kv list command using the List method and then subsequent vault kv get commands for each secret.
Recursive retrieval: If multiple secrets are being retrieved, should any sub-folders also be enumerated and retrieved? The default is: False.
Secret Version: Optional When retrieving a single secret, choose the version of the secret to retrieve. For example, if you want version 2 of all field values in a secret, enter the value 2.
Field names: Choose specific fields to be retrieved from identified secrets. This is useful when you only want to retrieve specific fields from one or more secret(s). You can optionally include a name for the output variable.
Print output variable names: Write out the Octopus output variable names to the task log. The default is: False.

Using Retrieve KV (v2) secrets step

The Key Value (v2) retrieve secrets step is added to deployment and runbook processes in the same way as other steps.

After you've added the step to your process, fill out the parameters in the step:

Note the use of the sensitive output variable in the Auth Token parameter. In this example, the value was created using the Unwrap SecretID and Login step template named HashiCorp Vault - AppRole Unwrap Secret ID and Login.

In subsequent steps, the output variables created from matching secrets can be used in your deployment or runbook.

Tip: Remember to replace HashiCorp Vault - Key Value (v2) retrieve secrets with the name of your step for any output variable names.

Conclusion

The templates covered in this post show how it's possible to extend the functionality of Octopus and retrieve secrets from Vault, or any other secrets manager, and use them in your deployments or runbooks.

Happy deployments!

Learn more

For further information, you can read:

The AppRole Pull Authentication tutorial showing how to retrieve SecretIDs securely.
HashiCorp Vault documentation for the K/V v1 Secrets Engine.
HashiCorp Vault documentation for the K/V v2 Secrets Engine.

This post was originally published at octopus.com.

Octopus Pipe for Bitbucket: octopus-cli-run

Mark Harrison — Thu, 06 Aug 2020 10:20:11 +0000

In a previous post, I wrote how to create a Bitbucket Pipe and integrate it with Octopus Deploy. If you’re starting out with Pipes for the first time, it’s worth a read.

In this post, I’ll give you an overview of the new experimental Bitbucket Pipe for Octopus - octopus-cli-run.
If you’re interested in trying the experimental pipe, you can use it to run commands from the Octopus CLI, allowing you to further integrate your Atlassian Bitbucket Pipeline with Octopus to manage your packages, releases, and deployments.

In this post

Pipe YAML Definition
- Pipe variable definitions
Supported commands
- Pack
- Push
- Build information
- Create release
- Deploy release
Using the Pipe
Conclusion

Pipe YAML Definition

The base definition of the Pipe includes the reference to its repository hosted on Bitbucket. It has also been published as octopipes/octopus-cli-run on Docker Hub.

It has one required CLI_COMMAND variable. This is the CLI command to run.

To use the Pipe in your bitbucket-pipelines.yml file, add the following YAML snippet to the script section:

- pipe: octopusdeploy/octopus-cli-run:0.13.0
  variables:
    CLI_COMMAND: "<string>"
    # EXTRA_ARGS: ['<string>','<string>' ..] # Optional
    # DEBUG: "<boolean>" # Optional

The Pipe also provides an optional array variable called EXTRA_ARGS that you can use to include any additional command line arguments for the specified command.

Pipe variable definitions

Variables in Bitbucket Pipelines and Pipes are configured as Environment variables. As the octopus-cli-run Pipe contains a number of commands, the specific variables that are required depend on which command you are using. See the README for further details of the variables that are required for each command.

Supported commands

The octopus-cli-run Pipe was written with the most commonly used CLI commands in mind, and it’s actually built on top of the Octopus CLI Docker image. This includes the ability to:

Package your files or build artifacts using pack.
Send packages to the Octopus built-in repository using push.
Push build information to Octopus using build-information.
Automate creation of releases using create-release.
Deploy releases that have already been created using deploy-release.

Next, we’ll explore what the Pipeline steps look like for each of the commands using the PetClinic sample application available on Bitbucket. To keep it simple, the steps have been reduced to the minimum definition that is needed.

Pack

The pack command allows you to create packages (either as zip or nupkg) from files on disk, without the need for a .nuspec or .csproj file.

To create a package, define a step like this:

- step:
    name: octo pack mysql-flyway
    script:
      - pipe: octopusdeploy/octopus-cli-run:0.13.0
        variables:
          CLI_COMMAND: 'pack'
          ID: 'petclinic.mysql.flyway'
          FORMAT: 'Zip'
          VERSION: '1.0.0.0'
          SOURCE_PATH: 'flyway'
          OUTPUT_PATH: 'flyway'
    artifacts:
      - "flyway/*.zip"

This packages the flyway folder and creates a zip file named petclinic.mysql.flyway.1.0.0.0.zip in the same folder.

Push

The push command enables you to push packages (.zip, .nupkg, .war, etc) to the Octopus built-in repository

It also supports pushing multiple packages at the same time. To perform a multi-package push, define a step like this:

- step:
    name: octo push
    script:
      - pipe: octopusdeploy/octopus-cli-run:0.13.0
        variables:
          CLI_COMMAND: 'push'
          OCTOPUS_SERVER: $OCTOPUS_SERVER
          OCTOPUS_APIKEY: $OCTOPUS_API_KEY
          OCTOPUS_SPACE: $OCTOPUS_SPACE
          PACKAGES: [ "./flyway/petclinic.mysql.flyway.1.0.0.0.zip", "target/petclinic.web.1.0.0.0.war" ]

This pushes both the petclinic.mysql.flyway.1.0.0.0.zip and the petclinic.web.1.0.0.0.war packages to Octopus.

Build information

The build-information command helps you to pass information about your build (number, URL, commits) to Octopus. This information can be viewed within Octopus, and can also be used in both release notes and deployment notes.

If you have already created a build-information file, you can supply this to the command using the FILE variable. If the variable isn’t provided, the Pipe will generate its own build information file and send it to Octopus.

To push an auto-generated build info file, define a step like this:

- step:
    name: octo build-information
    script:
      - pipe: octopusdeploy/octopus-cli-run:0.13.0
        variables:
          CLI_COMMAND: 'build-information'
          OCTOPUS_SERVER: $OCTOPUS_SERVER
          OCTOPUS_APIKEY: $OCTOPUS_API_KEY
          OCTOPUS_SPACE: $OCTOPUS_SPACE
          VERSION: '1.0.0.0'
          PACKAGE_IDS: ['petclinic.web']

This creates build information, associates it with version 1.0.0.0 of the petclinic.web package, and pushes it to Octopus.

Create release

The create-release command allows you to create a release in Octopus. You specify the project to create the release for using the PROJECT variable.

Optionally, you can also deploy the release to one or more environments. To achieve this, you should use the global EXTRA_ARGS array variable and provide the appropriate options. For example:

EXTRA_ARGS: ['--deployTo', 'Development', '--guidedFailure', 'True']

To create a release, and let Octopus choose the version to use, create a step like this:

- step:
    name: octo create-release
    script:
      - pipe: octopusdeploy/octopus-cli-run:0.13.0
        variables:
          CLI_COMMAND: 'create-release'
          OCTOPUS_SERVER: $OCTOPUS_SERVER
          OCTOPUS_APIKEY: $OCTOPUS_API_KEY
          OCTOPUS_SPACE: $OCTOPUS_SPACE
          PROJECT: $OCTOPUS_PROJECT

Deploy release

The deploy-release command lets you deploy releases that have already been created. You specify the project and the release number to deploy the release for using the PROJECT and RELEASE_NUMBER variables.

Choose the environment(s) to deploy to by specifying them in the DEPLOY_TO variable using either the Name or ID, like so:

DEPLOY_TO: ['Environments-1', 'Development', 'Staging', 'Test']

To deploy the latest release to Development for a project, create a step like this:

- step:
    name: octo deploy-release
    script:
      - pipe: octopusdeploy/octopus-cli-run:0.13.0
        variables:
          CLI_COMMAND: 'deploy-release'
          OCTOPUS_SERVER: $OCTOPUS_SERVER
          OCTOPUS_APIKEY: $OCTOPUS_API_KEY
          OCTOPUS_SPACE: $OCTOPUS_SPACE
          PROJECT: $OCTOPUS_PROJECT
          RELEASE_NUMBER: 'latest'
          DEPLOY_TO: ['Development']

Using the Pipe

Finally, let’s see the use of the Pipe in multiple steps to make up the complete Bitbucket Pipeline:

image: maven:3.6.1

pipelines:
  branches:
    master:
      - step:
          name: build petclinic
          caches:
            - maven
          script:
            - mvn -B verify -DskipTests -Dproject.versionNumber=1.0.0.0 -DdatabaseUserName=$DatabaseUserName -DdatabaseUserPassword=$DatabaseUserPassword -DdatabaseServerName=$DatabaseServerName -DdatabaseName=$DatabaseName
          artifacts:
          - "target/*.war"
      - step:
          name: octo pack mysql-flyway
          script:
            - pipe: octopusdeploy/octopus-cli-run:0.13.0
              variables:
                CLI_COMMAND: 'pack'
                ID: 'petclinic.mysql.flyway'
                FORMAT: 'Zip'
                VERSION: '1.0.0.0'
                SOURCE_PATH: 'flyway'
                OUTPUT_PATH: './flyway'
          artifacts:
            - "flyway/*.zip"
      - step:
          name: octo push
          script:
            - pipe: octopusdeploy/octopus-cli-run:0.13.0
              variables:
                CLI_COMMAND: 'push'
                OCTOPUS_SERVER: $OCTOPUS_SERVER
                OCTOPUS_APIKEY: $OCTOPUS_API_KEY
                OCTOPUS_SPACE: $OCTOPUS_SPACE
                PACKAGES: [ "./flyway/petclinic.mysql.flyway.1.0.0.0.zip", "target/petclinic.web.1.0.0.0.war" ]
      - step:
          name: octo build-information
          script:
            - pipe: octopusdeploy/octopus-cli-run:0.13.0
              variables:
                CLI_COMMAND: 'build-information'
                OCTOPUS_SERVER: $OCTOPUS_SERVER
                OCTOPUS_APIKEY: $OCTOPUS_API_KEY
                OCTOPUS_SPACE: $OCTOPUS_SPACE
                VERSION: '1.0.0.0'
                PACKAGE_IDS: ['petclinic.web']
      - step:
          name: octo create-release
          script:
            - pipe: octopusdeploy/octopus-cli-run:0.13.0
              variables:
                CLI_COMMAND: 'create-release'
                OCTOPUS_SERVER: $OCTOPUS_SERVER
                OCTOPUS_APIKEY: $OCTOPUS_API_KEY
                OCTOPUS_SPACE: $OCTOPUS_SPACE
                PROJECT: $OCTOPUS_PROJECT
      - step:
          name: octo deploy-release
          script:
            - pipe: octopusdeploy/octopus-cli-run:0.13.0
              variables:
                CLI_COMMAND: 'deploy-release'
                OCTOPUS_SERVER: $OCTOPUS_SERVER
                OCTOPUS_APIKEY: $OCTOPUS_API_KEY
                OCTOPUS_SPACE: $OCTOPUS_SPACE
                PROJECT: $OCTOPUS_PROJECT
                RELEASE_NUMBER: 'latest'
                DEPLOY_TO: ['Development']

And that’s it!

You can view the complete PetClinic bitbucket-pipelines.yml file on Bitbucket.

Sample Octopus project
You can see the PetClinic Octopus project in our samples instance.

Conclusion

Using a Bitbucket Pipe really helps to simplify the configuration in your Bitbucket Pipeline, and as an author helps to promote re-use of your actions. Check out Bitbucket Pipelines for more information and the experimental Octopus Pipe for more details on how you can use Bitbucket and Octopus together.

This post was originally published at octopus.com.

Bitbucket Pipelines: Pipes and integrating with Octopus Deploy

Mark Harrison — Tue, 04 Aug 2020 08:32:31 +0000

Atlassian’s Bitbucket Pipelines is a lightweight cloud continuous integration server that uses pre-configured Docker containers, allowing you to define your infrastructure as code. Pipes let you add configuration to your Pipelines and are particularly useful for third-party tools.

In this post, I create an experimental Pipe for an Octopus CLI command, use it in a Bitbucket Pipeline for our sample node.js application RandomQuotes-Js, and finally, integrate the Pipeline with Octopus.

In this post

What are Bitbucket Pipes?
Example Pipe usage
- Refer to a Pipe in a Pipeline step
Why are Pipes useful?
Create a Bitbucket Pipe
Choose a candidate for a Pipe
- Create the Pipe repository
- Create the Pipe skeleton
- Create the Pipe’s metadata
- Create the Pipe script
- Mandatory Pipe variables
- Optional Pipe variables
- Run the Pack Pipe
- Complete Pipe script
- Create the Pipe Dockerfile
- Create the Pipe’s own pipeline
- Create the Pipe README
Run the Pipe
Test the Pipe
Publish the Pipe
Integrating the Pipe into a Pipeline
- Use the Pipe
Integrate the Pipeline with Octopus
- Push the package to Octopus
- Push build information to Octopus
Conclusion
Learn more

What are Bitbucket Pipes?

Atlassian says:

Pipes provide a simple way to configure a pipeline. They are especially powerful when you want to work with third-party tools. Just paste the pipe into the YAML file, supply a few key pieces of information, and the rest is done for you. You can add as many pipes as you like to your steps, so the possibilities are endless!

Pipes build on the core concept of Pipelines, containers. A Pipe makes use of a script that lives inside of a Docker container, and it typically has the commands you’d write in your pipeline YAML file before pipes were available.

Example Pipe usage

This is what the Atlassian bitbucket-upload-file Pipe looks like in your Pipeline YAML file:

- pipe: atlassian/bitbucket-upload-file:0.1.3
  variables:
    BITBUCKET_USERNAME: '<string>'
    BITBUCKET_APP_PASSWORD: '<string>'
    FILENAME: '<string>'
    # ACCOUNT: '<string>' # Optional
    # REPOSITORY: '<string>' # Optional
    # DEBUG: '<boolean>' # Optional

atlassian/bitbucket-upload-file:0.1.3 is the name of the Docker image containing the Pipe to run.
BITBUCKET_USERNAME is an example of a variable you need to provide that has the value for the Pipe to use when executing inside of the container.

Refer to a Pipe in a Pipeline step

There are two ways you can refer to a Pipe in a step within a pipeline:

Refer to the Docker image directly:

pipe: docker://<Docker_Account_Name>/<Image_Name>:<tag>

Refer to a Pipe repository hosted on Bitbucket:

pipe: <Bitbucket_account>/<Bitbucket_repo>:<tag>

This method looks for the location of the Docker image from the pipe.yml file within the referenced <Bitbucket_account>/<Bitbucket_repo> Pipe repository.

Why are Pipes useful?

Why go to the trouble of writing a Pipe at all?

Pipes are all about re-use. They allow you to repeat the same action in multiple steps of your Pipeline. By centralizing the core of your action into a Pipe, you end up with a simpler Pipeline configuration. Another key feature of a Pipe versus directly scripting in your Pipeline is the ability to include dependencies that your main Pipeline doesn’t require.

Create a Bitbucket Pipe

A Pipe consists of a bunch of files that make up a Docker image. The Pipe I created has its image based on the pre-existing octopusdeploy/octo image. The finished Pipe has been published as octopipes/pack on Docker Hub.

At first, creating a Pipe might seem quite daunting, but Atlassian provide a helpful step-by-step guide.

I created this Pipe on an Ubuntu machine using a Bash terminal. If you are using a different platform, you may need to tweak the commands you use.

Choose a candidate for a Pipe

I’ve often heard people say that naming something is the hardest thing when it comes to software, and the same is true for choosing a command to wrap in a Pipe. However, in most CI/CD pipelines, after you have built and run any tests on your code, you probably want to package your applications. So it felt natural to choose the Octopus CLI pack command to create a Pipe for.

The added bonus was that the pack command only has a few required parameters, and the optional ones can be tackled with some pipeline magic (more on that later).

There are two types of Pipe that you can author:

Simple
Complete

I opted for a Complete Pipe so that I can publish it and make use of it in other repositories.

Create the Pipe repository

Next up, I needed to create a new octopusdeploy/pack Git repository in Bitbucket, and clone it locally.

For further information on creating a new Git repository, please see the Atlassian documentation.

Create the Pipe skeleton

Atlassian provides a method to generate a skeleton of a Pipe repository using Yeoman. When you have all the pre-requisites (nodejs and npm) installed, you can run the generator using the yo command from a terminal:

yo bitbucket-pipe

This prompts you to select the Pipe you want to create. I chose New Advanced Pipe (Bash).

You are prompted with some questions to help fill in the metadata and other useful information for consumers of the Pipe. Once complete, it will generate the required files you need to get started:

At a minimum, you need to edit the following files to suit your Pipe requirements:

pipe.yml
pipe/pipe.sh
Dockerfile
bitbucket-pipelines.yml
README.md

Tip: Check other repositories to see how they have written their Pipe!

One of the great things about every Bitbucket Pipe is that the code is public, so you can browse it. For example, you can view the source code for the bitbucket-upload-file Pipe on Bitbucket.

This is a really great way to see how other authors have structured their Pipe.

Create the Pipe’s metadata

When you create a Complete Pipe, Atlassian requires you to create a pipe.yml file. This document contains metadata about your Pipe and includes things like:

A name for the Pipe.
The Docker Hub image for your Pipe in the format: account/repo:tag.
A list of Pipe variables where you can specify default values.

If you chose one of the Advanced Pipes using the Pipe generator, the pipe.yml file will be created for you with all of the relevant information already added. Here are the contents of my auto-generated pipe.yml file:

name: Octo Pack
image: octopipes/pack:0.0.0
description: Creates a package (.nupkg or .zip) from files on disk, without needing a .nuspec or .csproj
repository: https://bitbucket.org/octopusdeploy/pack
maintainer: support@octopus.com
tags:
    - octopus
    - package
    - deployment

Create the Pipe script

The main part of your Pipe is the script or binary that will run when it’s executed within a container. It will include all of the logic needed to execute the Pipe task. You can choose any language you are familiar with. When I created the skeleton of our Pipe earlier, I used Bash, and a sample pipe/pipe.sh file was created for me to finish.

TL;DR

If you want to see the complete Pipe script, skip straight to the end or view the source code. If not, read on!

The general structure of a Pipe script file follows this convention:

Mandatory Pipe variables

The pack command has five parameters I want the Pipe to handle:

The --id of the package to create.
The --format for the package, e.g. NuPkg or Zip.
The --version for the package (SemVer).
The --basePath to specify the root folder containing files and folders to pack.
The --outFolder to specify the folder where the generated package will be written.

To support these parameters, I mapped each one to a Bitbucket Pipeline variable.

I also followed the validation of variables, as shown in the Atlassian demo-pipe-bash script:

NAME=${NAME:?'NAME variable missing.'}

This checks for a $NAME variable value and errors with a message when the variable isn’t present.

For the five variables I created, my variable validation looks like this:

# mandatory parameters
ID=${ID:?'ID variable missing.'}
FORMAT=${FORMAT:?'FORMAT variable missing.'}
VERSION=${VERSION:?'VERSION variable missing.'}
SOURCE_PATH=${SOURCE_PATH:?'SOURCE_PATH variable missing.'}
OUTPUT_PATH=${OUTPUT_PATH:?'OUTPUT_PATH variable missing.'}

Optional Pipe variables

Next up were some optional variables consumers of the Pipe could choose to supply if they wish.

I included an EXTRA_ARGS array variable to include multiple additional arguments for the pack command. You can specify this variable by using a special Bitbucket array type in your pipeline:

variables:
  EXTRA_ARGS: ['--description', 'text containing spaces', '--verbose']

The array type is really useful, as it provides an easy way to supply any other arguments to the Pipe. In my case, this allows consumers of the pack Pipe the ability to provide any of the additional argument options that I haven’t explicitly handled.

Advanced Pipe writing techniques:

To learn more about the array type and how its values are passed to the Docker container, please see Atlassian’s documentation.

Lastly, I included a boolean DEBUG variable to include additional debugging information. You specify its value in your Pipe like this:

variables:
  DEBUG: 'true'

Run the Pack Pipe

The pack Pipe’s sole purpose is to package a set of files, to achieve that, we make use of our reference to the octopusdeploy/octo Docker image I used as a base for our Pipe Docker image. This gives us access to the full Octopus CLI in our Pipe.

To package the files in the Bitbucket Pipeline, our script needs to run the pack command and pass in our variables:

run octo pack --id "$ID" --version "$VERSION" --format "$FORMAT" --basePath "$SOURCE_PATH" --outFolder "$OUTPUT_PATH" "${EXTRA_ARGS[@]}"

Lastly, we check if the command was successful. If it was, we display a success message and set a variable with the filename of the package that was created. If not, we display an error message and halt execution.

The run command is a helper function specified in a separate common.sh file. It looks like this:
run() {
 output_file="/var/tmp/pipe-$(date +%s)-$RANDOM"

 echo "$@"
 set +e
 "$@" | tee "$output_file"
 status=$?
 set -e
}
The function wraps the call to the supplied command, in this case octo pack, logs the output to a temporary file, and captures the exit status.

Complete Pipe script

And that’s all there is to our script. Here is the finished pipe.sh file:

#!/usr/bin/env bash

# Creates a package (.nupkg or .zip) from files on disk, without needing a .nuspec or .csproj
#
# Required globals:
#   ID
#   FORMAT
#   VERSION
#   SOURCE_PATH
#   OUTPUT_PATH
#
# Optional globals:
#   EXTRA_ARGS
#   DEBUG

source "$(dirname "$0")/common.sh"

# mandatory parameters
ID=${ID:?'ID variable missing.'}
FORMAT=${FORMAT:?'FORMAT variable missing.'}
VERSION=${VERSION:?'VERSION variable missing.'}
SOURCE_PATH=${SOURCE_PATH:?'SOURCE_PATH variable missing.'}
OUTPUT_PATH=${OUTPUT_PATH:?'OUTPUT_PATH variable missing.'}

FORMAT=$(echo "$FORMAT" | tr '[:upper:]' '[:lower:]')

# Default parameters
EXTRA_ARGS_COUNT=${EXTRA_ARGS_COUNT:="0"}
DEBUG=${DEBUG:="false"}

enable_debug

if [ "${EXTRA_ARGS_COUNT}" -eq 0 ]; then
  # Flatten array of extra args
  debug "Setting EXTRA_ARGS to empty array"
  EXTRA_ARGS=
fi

debug "Flattening EXTRA_ARGS"
init_array_var 'EXTRA_ARGS'

debug ID: "${ID}"
debug FORMAT: "${FORMAT}"
debug VERSION: "${VERSION}"
debug SOURCE_PATH: "${SOURCE_PATH}"
debug OUTPUT_PATH: "${OUTPUT_PATH}"
debug EXTRA_ARGS_COUNT: "${EXTRA_ARGS_COUNT}"
debug EXTRA_ARGS: "${EXTRA_ARGS}"

run octo pack --id "$ID" --version "$VERSION" --format "$FORMAT" --basePath "$SOURCE_PATH" --outFolder "$OUTPUT_PATH" "${EXTRA_ARGS[@]}"

if [ "${status}" -eq 0 ]; then
  OCTO_PACK_FILENAME="$ID.$VERSION.$FORMAT"
  success "Packaging successful. Created package $OUTPUT_PATH/$OCTO_PACK_FILENAME."

else
  fail "Packaging failed."
fi

Create the Pipe Dockerfile

Now that we have our main script to run, we need to create our image using a Dockerfile. If you ran the Pipe generator, you already have the Dockerfile ready to edit to suit your Pipe.

For the pack Pipe, the Dockerfile looks like this:

FROM octopusdeploy/octo:7.3.2

RUN apk add --update --no-cache bash

COPY pipe /
RUN chmod a+x /*.sh

ENTRYPOINT ["/pipe.sh"]

The Dockerfile takes the octopusdeploy/octo as its base, and then adds bash to the image. It then copies the contents of the pipe folder and grants permissions for all users to execute the .sh files present. Lastly, it sets the ENTRYPOINT for the container to the pipe.sh file we created earlier.

Create the Pipe’s own pipeline

When you’ve completed your Pipe, you could deploy your Docker image manually to Docker Hub. However, it’s also possible to get Bitbucket Pipelines to do the heavy lifting for you automatically when you push changes to your Bitbucket repository with your own bitbucket-pipelines.yml file.

For the pack Pipe, in the auto-generated file, I only modified the push step:


push: &push
  step:
    name: Push and Tag
    image: python:3.7
    script:
    - pip install semversioner==0.7.0
    - chmod a+x ./ci-scripts/*.sh
    - ./ci-scripts/bump-version.sh
    - ./ci-scripts/docker-release.sh octopipes/$BITBUCKET_REPO_SLUG
    - ./ci-scripts/git-push.sh
    services:
    - docker

The step installs semversioner, which is a python tool to help automatically generate release notes and version your Pipe according to SemVer. After this, it increments the version of the Pipe, creates a new Docker image, and pushes it to Docker Hub. Finally, it tags the new version and pushes that back to the Bitbucket repository.

You can view the complete bitbucket-pipelines.yml file for the pack Pipe on Bitbucket.

No double Docker push:
The push step doesn’t trigger two Docker images to be pushed to Docker Hub when the step is doing its own commit and push back to the Bitbucket repository. The reason for this is that Bitbucket Pipelines support the option to skip a Pipeline run if [skip ci] or [ci skip] is included anywhere in the commit message.

Create the Pipe README

You might be thinking, "Why spend time creating a readme file?" Well, Atlassian themselves recommend it:

Your readme is how your users know how to use your pipe. We can display this in Bitbucket, so it needs to be written with markdown, in a specific format.

Including an informative README increases the chances of your users being successful with your Pipe.

One of the more important parts of the README is the YAML Definition. This tells users what to add to their bitbucket-pipeline.yml file.

Here’s what the pack one looks like:

script:
  - pipe: octopipes/pack:0.6.0
    variables:
      ID: "<string>"
      FORMAT: "<string>"
      VERSION: "<string>"
      SOURCE_PATH: "<string>"
      OUTPUT_PATH: "<string>"
      # EXTRA_ARGS: "['<string>','<string>' ..]" # Optional
      # DEBUG: "<boolean>" # Optional

You can view the README for the octopipes/pack in full here.

Run the Pipe

Since the Pipe is just a Docker image, after you have built the image, you can execute the Pipe using docker run, passing in any required parameters as environment variables.

Here’s what the command looks like to run the pack Pipe to package the root directory from our RandomQuotes-JS application:

sudo docker run \
   -e ID="randomquotes-js" \
   -e FORMAT="Zip" \
   -e VERSION="1.0.0.0" \
   -e SOURCE_PATH="." \
   -e OUTPUT_PATH="./out" \
   -e DEBUG="false" \
   -v $(pwd):$(pwd) \
   -w $(pwd) \
 octopipes/pack:0.6.0

The output shows the successful packaging of the randomquotes-js.1.0.0.0.zip file:

Test the Pipe

To make sure your Pipe does what you expect it to, it’s a good idea to write tests for it. Following Atlassian’s lead, I opted to use BATS (Bash Automated Testing System).

Just like a lot of testing frameworks, a test file (which typically ends in .bats) contains the following constructs:

A setup method to set up any required variables or shared resources for your tests.
A number of individual @test declarations; these are your test cases.
A teardown method to remove any resources you used.

Here is my test.bats file:

#!/usr/bin/env bats

setup() {
  DOCKER_IMAGE=${DOCKER_IMAGE:="test/pack"}

  echo "Building image..."
  run docker build -t ${DOCKER_IMAGE}:test .

  # generated
  RANDOM_NUMBER=$RANDOM

  # locals
  ID="MyCompany.MyApp"
  FORMAT="zip"
  VERSION="1.0.0.0"
  SOURCE_PATH="test/code"
  OUTPUT_PATH="test/out"

  echo "$FORMAT"

  EXPECTED_FILE="$ID.$VERSION.$FORMAT"

  # Create test output dir
  rm -rf test/out
  mkdir test/out/extract -p

  echo "Create file with random content"
  echo $RANDOM_NUMBER > test/code/test-content.txt
}

teardown() {
  echo "Cleaning up files"
  chmod -R a+rwx test/out/
  rm -rf test/out
}

@test "Create Zip package using Octo pack command" {

    echo "Run test"
    run docker run \
        -e ID="${ID}" \
        -e FORMAT="${FORMAT}" \
        -e VERSION="${VERSION}" \
        -e SOURCE_PATH="${SOURCE_PATH}" \
        -e OUTPUT_PATH="${OUTPUT_PATH}" \
        -e DEBUG="false" \
        -v $(pwd):$(pwd) \
        -w $(pwd) \
        ${DOCKER_IMAGE}:test

    echo "Status: $status"
    echo "Output: $output"
    [ "$status" -eq 0 ]

    # Verify
    unzip "test/out/$EXPECTED_FILE" -d test/out/extract
    run cat "test/out/extract/test-content.txt"
    echo "Output: $output"
    [[ "${status}" -eq 0 ]]
    [[ "${output}" == *"$RANDOM_NUMBER"* ]]

}

In my test.bats file, the setup builds a local Docker image of the Pipe called test/pack. Next, it creates a file with a random number for its contents.

My test then executes docker run (just as I did in my local test) and verifies the container ran to completion, and finally extracts the files from the package and checks the content matches the random number I generated in the setup.

The tests run as part of my automated CI/CD pipeline configured in my bitbucket-pipeline.yml. You can also run the tests locally if you have bats installed.

Here is the output from a test run of my test.bats file:

And that’s all there is to testing your Pipe!

Publish the Pipe

When you are happy your Pipe is working, you can publish your Docker image directly by running docker push.

If you have set up an automated pipeline like we did earlier, then you can make use of semversioner to create a changeset:

semversioner add-change --type patch --description "Initial Docker release"

After you have committed your changeset, push them to Bitbucket and your pipeline should handle the rest. It will also automatically update the version number in:

The CHANGELOG.md file.
The README.md file.
The metadata pipe.yml.

You can see an example of the pack Bitbucket pipeline creating the 0.5.1 release here:

Integrating the Pipe into a Pipeline

If you’ve got this far, you have your Pipe published to Docker Hub, but how do you integrate that Pipe in another repository’s Pipeline?

To find out, I imported one of our existing node.js samples RandomQuotes-JS, which is hosted on GitHub, into a new Bitbucket repository with the same name.

Use the Pipe

Next, I created a bitbucket-pipelines.yml file and set up my Pipeline. After the build and testing step, I inserted a package step like this:

- step:
    name: Pack for Octopus
    script:
      - export VERSION=1.0.0.$BITBUCKET_BUILD_NUMBER
      - pipe: octopusdeploy/pack:0.6.0
        variables:
          ID: ${BITBUCKET_REPO_SLUG}
          FORMAT: 'Zip'
          VERSION: ${VERSION}
          SOURCE_PATH: '.'
          OUTPUT_PATH: './out'
          DEBUG: 'false'
    artifacts:
      - out/*.zip

The step looks similar to the examples in the Pipe’s README file. I’ve added a line just before the pipe instruction to set a variable:

- export VERSION=1.0.0.$BITBUCKET_BUILD_NUMBER

The export command tells Bitbucket to create the VERSION variable and use it in the Pipe.

I also make use of Bitbucket artifacts, where I specify a glob pattern to choose any zip files present in the out folder:

artifacts:
    - out/*.zip

This allows the package created by the Pipe to be used by future steps in the Pipeline.

Integrate the Pipeline with Octopus

After I have created a package, I want to complete the Bitbucket Pipeline by integrating Bitbucket with Octopus; specifically, I want to push the package I created and the related commit information to Octopus.

Push the package to Octopus

After the packaging step I’d created earlier, I added another step to push the package to the Octopus built-in repository.

This step makes use of a feature in Bitbucket that allows you to specify a container image, which can be different to the default image used elsewhere in the Pipeline. In this case, I chose the octopusdeploy/octo:7.3.2 Docker image.

This means I can run the octo push command and specify the package I created in the previous Pack for Octopus step like so:

octo push --package ./out/$BITBUCKET_REPO_SLUG.$VERSION.zip  --server $OCTOPUS_SERVER --space $OCTOPUS_SPACE --apiKey $OCTOPUS_APIKEY

You can see the minimum YAML required to achieve the push to Octopus below:

- step:
    name: Push to Octopus
    image: octopusdeploy/octo:7.3.2
    script:
      - export VERSION=1.0.0.$BITBUCKET_BUILD_NUMBER
      - octo push --package ./out/$BITBUCKET_REPO_SLUG.$VERSION.zip  --server $OCTOPUS_SERVER --space $OCTOPUS_SPACE --apiKey $OCTOPUS_APIKEY

Push build information to Octopus

To round off the integration, I want to have the build information available within Octopus.

For me, one of the best things about Octopus is that it’s built API-first. This allows us to build a first-class CLI on top of it. Pushing build information turned out to be pretty easy. Once I knew the format of the JSON payload, I created a Bash script to do just that using the build-information command.

Tip: Build information payload
To help demystify some of the complexities of the build information payload, I followed my colleague Shawn’s excellent piece on its structure.

To add to our previous Push to Octopus step, we can include that script to push build information as well. The complete step looks like this:

- step:
    name: Push to Octopus
    image: octopusdeploy/octo:7.3.2
    script:
      - apk update && apk upgrade && apk add --no-cache git curl jq
      - export VERSION=1.0.0.$BITBUCKET_BUILD_NUMBER
      - octo push --package ./out/$BITBUCKET_REPO_SLUG.$VERSION.zip  --server $OCTOPUS_SERVER --space $OCTOPUS_SPACE --apiKey $OCTOPUS_APIKEY
      - /bin/sh create-build-info.sh $BITBUCKET_REPO_OWNER $BITBUCKET_REPO_SLUG $BITBUCKET_BUILD_NUMBER $BITBUCKET_COMMIT $BITBUCKET_BRANCH $BITBUCKET_GIT_HTTP_ORIGIN
      - octo build-information --package-id $BITBUCKET_REPO_SLUG --version $VERSION --file=octopus.buildinfo --server $OCTOPUS_SERVER --space $OCTOPUS_SPACE --apiKey $OCTOPUS_APIKEY

The create-build-info.sh script creates a JSON payload output file called octopus.buildinfo, and then we use that in the build-information command to push commit information.

After the commits have been pushed to Octopus, you can see them in the Packages section of the Library:

And that’s it!

You can view the complete bitbucket-pipelines.yml file on Bitbucket.

Sample Octopus project
You can view the RandomQuotes-JS Octopus project setup in our samples instance.

Conclusion

Creating my first Bitbucket Pipe was pretty straightforward. I can definitely see the advantages of creating a Pipe in Bitbucket. That said, I found that when making changes to the pipe, the feedback cycle took a while to get right. My advice is to write only the minimum functionality you require, and then build on that over time. Integrating a Bitbucket Pipeline with Octopus is a breeze with the Octopus CLI, and for anything more complex, you always have the API at your disposal.

Learn more

Take a peek at the experimental Pipe - pack
Guides: Octopus CI/CD pipeline guides

This post was originally published at octopus.com.

Farmer: Simpler ARM deployments with Octopus Deploy

Mark Harrison — Tue, 21 Jul 2020 08:32:51 +0000

Having worked with Azure for around a year, it’s clear to see why ARM templates are popular. They provide a declarative model to generate entire environments at the touch of a button.

However, if like me, you’ve ever tried to author an ARM template file, you might have come across one of my biggest gripes with them; they rely on strings and are prone to human error. There’s no compiler to help me out when I have a typo in a template (and there have been plenty of those!).

I’ve used C# as my primary development language since 2012, however, since then, its functional counterpart, F# has become increasingly popular. As I found out recently, it has some useful features that can help out with my ARM template dilemma. One area in particular where F# excels is its built-in type-safety.

In this post, I’ll demonstrate the type-safety in F# in action by using Farmer to generate a simple Azure WebApp ARM template, and then I’ll walk through how you can use its deployment capabilities through Octopus to deploy different WebApps to Azure directly.

In this post

What is Farmer?
Why is Farmer needed?
Create the Farmer template
Package the Farmer template
Deploy the Farmer template
Conclusion

What is Farmer?

The authors of Farmer says:

Farmer is an open source, free to use .NET domain-specific-language (DSL) for rapidly generating non-complex Azure Resource Manager (ARM) templates.

To use Farmer, you create a Farmer template. These are .NET Core applications that reference Farmer via a NuGet package, and they define your Azure resources that you wish to create.

Why is Farmer needed?

Rather than repeating what is already there, I encourage you to read the About section of the Farmer documentation for more details about the motivations to create a DSL for ARM templates.

For me, the highlights are:

It provides a set of types that you can use to create Azure resources, and it eliminates the chances of creating an invalid template as they are strongly-typed.
It can generate simple ARM templates in a very concise manner and optionally, deploy them.

Create the Farmer template

To create a Farmer template, we first need to create a .NET Core application. You can do this in your IDE of choice, or if you prefer the command-line, you can use the dotnet new command, passing the template of the type of application you require.

It’s typical to use a console application for a Farmer template, and you can create one with the dotnet new console command:

dotnet new console -lang "F#" -f "netcoreapp3.1" -n "SimpleAzureWebApp"

This creates a new F# .NET Core 3.1 application with the name SimpleAzureWebApp, using the -n parameter we supplied.

Next, we need to add Farmer to the project by running the add package command:

dotnet add package Farmer

Now we have our dependencies, we can go ahead and edit the Program.fs file which was auto-generated for us when we created the new console application.

TL;DR

If you want to see the complete program, skip straight to the end or view the source code. If you’d like more details, read on!

Template parameters

To make the Farmer template flexible, we’ll add some parameters to the application. This will allow us to supply different values and Farmer will create our resources in Azure based on those values.

The first three we need are related to authenticating with Azure. These values can be obtained by creating an Azure Service Principal.

AppID: The application identifier used for the Service Principal.
Secret: The password used for the Service Principal.
TenantID: The ClientID used for the Service Principal.

Securing credentials:
Store the credentials you use to log in to Azure in a secure location, such as a Password Manager, or your Octopus Deploy instance using, preferably, an Azure account or sensitive variables. You should also avoid committing them into source control.

To run the application, we’ll also supply:

Resource Group Name: Which resource group to add the Azure WebApp to.
WebApp Name: The name to give to the Azure WebApp.
WebApp SKU: What type of App Service plan to use for the WebApp.
WebApp Location: The data center location where you’d like to host the Azure WebApp.

To add our required parameters, the code looks like this:

let azAppId = argv.[0]
let azSecret = argv.[1]
let azTenantId = argv.[2]
let azResourceGroupName = argv.[3]
let azWebAppName = argv.[4]
let azWebAppSku = argv.[5]
let azWebAppLocation = argv.[6]

This assigns the parameters from the argument collection supplied to the program when it runs, based on their position from the command-line.

Parameter validation:
I don’t show parameter validation in this example, but you may want to consider adding it to your Farmer template to ensure they have acceptable values.

Define Azure Resources

After we have our parameter values, we can define our Azure WebApp in F#:

let webAppSku = WebApp.Sku.FromString(azWebAppSku)
let webApp = webApp {
    name azWebAppName
    sku webAppSku
}

Here we assign the WebApp SKU to a variable named webAppSku. This is done by a helper function to return a strongly typed Sku. Then we create our webApp variable using the Farmer Web App builder.

Next, we create our ARM deployment using the Farmer ARM deployment builder, which in this example, consists of the location to deploy to, and the Azure WebApp as previously defined:

let deployLocation = Location.FromString(azWebAppLocation)
let deployment = arm {
    location deployLocation
    add_resource webApp
}

Built-in type safety

In both of the previous code examples, the power of the F# type-system comes into its own. It’s not possible to create a value that is invalid according to its type.

Let’s see an example. Suppose I wanted to create our Azure WebApp with a Sku which had a value of VeryFree. If I try to create that in our application, the compiler will give me a warning, and it won’t build:

This is because the compiler knows the string value VeryFree is the wrong type, and instead should be a Sku type.

This is where Farmer really excels over crafting your own ARM template by hand. Its use of F# provides you with type safety to ensure you have valid templates from the outset.

Farmer and ARM:
There is a more detailed comparison between Farmer and ARM templates here.

Generate ARM template

When you have your Azure resources modeled, Farmer supports different ways to generate the ARM template. One way is to write it out to a file directly:

deployment |> Writer.quickWrite "output"

You can then take this file and deploy to Azure using your preferred method.

Deployment to Azure

In addition to generating the ARM template, you can also, optionally, have Farmer execute the deployment to Azure when the application runs.

Azure CLI required
If you use the Integrated deployment to Azure feature, you will need the Azure CLI installed on the machine where you run your application.

In our example SimpleAzureWebApp application, we’ll take advantage of this feature.

Before we can execute the deployment, we need to authenticate with Azure. Farmer comes with a Deploy.authenticate command, and you call it by passing in the credentials you supplied to the application previously, like this:

Deploy.authenticate azAppId azSecret azTenantId
|> ignore

When the authenticate call finishes, it returns a list of the Azure subscriptions associated with the Service Principal. In this example, these results are piped to the ignore function.

If there are any errors authenticating with Azure, an error will be raised. If the login succeeds, we then need to get Farmer to execute our deployment, using the Deploy.execute command:

deployment
|> Deploy.execute azResourceGroupName Deploy.NoParameters
|> ignore

You can query the results of the ARM deployment, but as with the authenticate call, we ignore them. Similarly, any errors on deployment will be surfaced as an exception.

Complete Farmer template

And that’s all there is to our application. Here is the finished Program.fs file:

open Farmer
open Farmer.Builders
open SimpleAzureWebApp.SkuExtension

[<EntryPoint>]
let main argv =

    let azAppId = argv.[0]
    let azSecret = argv.[1]
    let azTenantId = argv.[2]
    let azResourceGroupName = argv.[3]
    let azWebAppName = argv.[4]
    let azWebAppSku = argv.[5]
    let azWebAppLocation = argv.[6]

    let webAppSku = WebApp.Sku.FromString(azWebAppSku)
    let webApp = webApp {
        name azWebAppName
        sku webAppSku
    }

    let deployLocation = Location.FromString(azWebAppLocation)
    let deployment = arm {
        location deployLocation
        add_resource webApp
    }

    printf "Authenticating with Azure\n"
    Deploy.authenticate azAppId azSecret azTenantId
    |> ignore

    printf "Deploying Azure WebApp %s (%s) into %s using Farmer\n" azWebAppName azResourceGroupName azWebAppLocation

    deployment
    |> Deploy.execute azResourceGroupName Deploy.NoParameters
    |> ignore

    printf "Deployment of Azure WebApp %s (%s) complete!\n" azWebAppName azResourceGroupName

    0 // return an integer exit code

Package the Farmer template

Now we have the application written, the next step is to package it for use with Octopus. For sake of simplicity, I build and package the application using command-line tools, but I recommended automating this as part of a full CI/CD pipeline.

If you are new to building .NET Core applications, we have a number of guides that include step-by-step instructions to setup a CI/CD pipeline using various tools.

To build the SimpleAzureWebApp application, we run a dotnet publish command in the application directory:

dotnet publish -o output

This builds and publishes the console application and places the binaries in the output folder, as specified by the use of the -o parameter.

Next, we need to package the application, and this time we use the Octopus CLI pack command:

octo pack --id SimpleAzureWebApp --format Zip --version 1.0.0.0 --basePath output

This generates a file named SimpleAzureWebApp.1.0.0.0.zip which can either be uploaded to the Octopus built-in repository or an external package repository.

You can push to the Octopus built-in repository using the Octopus CLI command, push:

octo push --package SimpleAzureWebApp.1.0.0.0.zip --server https://my.octopus.url --apiKey API-XXXXXXXXXXXXXXXX

After the package has been uploaded, we can set-up Octopus to run our application to deploy to Azure.

Deploy the Farmer template

One of the cool things about Octopus is that you get to choose how to deploy. With the introduction of Operations Runbooks last year, that flexibility has been extended even further to operations tasks, for example managing your infrastructure.

Create the runbook

To execute our Farmer template, we’ll create a runbook that deploys it to Azure. To do that:

Create a new project in Octopus.
Go to the runbook process from the Operations ➜ Runbooks section.
Click ADD RUNBOOK.
From the Overview, Click DEFINE YOUR RUNBOOK PROCESS.
Click ADD STEP.

On the step selection, choose the Run a script step and give it a name. By using the script step, we can use the reference package feature to include our package as part of the script execution.

To include our package, in the Referenced Packages section, click ADD and add the SimpleAzureWebApp package we uploaded earlier:

Keep all of the defaults and click OK.

Add the runbook script

Next, we need to add the inline script which will execute our Farmer template. We start by adding the required Azure credentials:

$appId = $OctopusParameters["Project.Azure.Account.Client"]
$secret = $OctopusParameters["Project.Azure.Account.Password"]
$tenantId = $OctopusParameters["Project.Azure.Account.TenantId"]

The script references a number of expanded Azure account variable properties such as Client and TenantId from the project variable named Project.Azure.Account. This is handy as we don’t need to specify separate variables for each property.

After we have the credentials, we want to specify the Azure WebApp parameters, including the Resource Group and WebApp name that will be passed to our SimpleAzureWebApp .NET Core application:

$resourceGroupName = $OctopusParameters["Project.Azure.ResourceGroupName"]
$webAppName = $OctopusParameters["Project.Azure.WebAppName"]
$webAppSku = $OctopusParameters["Project.Azure.WebAppSku"]
$webAppLocation = $OctopusParameters["Project.Azure.WebAppLocation"]

Lastly, we get the path of the extracted Farmer template package using a package variable called Octopus.Action.Package[SimpleAzureWebApp].ExtractedPath and then set the working directory to that path and call the dotnet run command passing in all of our parameters:

$farmerPackagePath = $OctopusParameters["Octopus.Action.Package[SimpleAzureWebApp].ExtractedPath"]
Set-Location $farmerPackagePath

dotnet SimpleAzureWebApp.dll $appId $secret $tenantId $resourceGroupName $webAppName $webAppSku $webAppLocation

.NET Core runtime pre-requisite
In order for this script step to execute, it requires the .NET Core runtime to be installed on the deployment target or worker where the step is configured to execute.

Add the variables

We also need to add the variables referenced in the script above:

The Project.Azure.Account variable is an Azure account variable, and the rest are text variables.

Run the runbook

If you’ve got this far, the last part is to bring it all together and run our runbook in Octopus and deploy the Farmer template to Azure.

You can see an example runbook run to development creating the Azure WebApp called farmer-webapp-dev:

After the runbook has run to completion, you can check your WebApp has been created using the Azure portal. Here is the corresponding WebApp that was created in Azure as a result of the runbook run to development:

Conclusion

The great thing about this technique of using Farmer to generate and deploy your resources to Azure is that you can version control your templates. The code that defines your infrastructure can live alongside the code that runs on it. Plus, no more hassle manually editing JSON files, and who doesn’t want that!

Until next time, Happy Deployments!

Learn more

This post was originally published at octopus.com.

Convert an existing application to use rolling deployments

Mark Harrison — Mon, 20 Jul 2020 08:29:56 +0000

In a previous post, I wrote about the benefits of the rolling deployments pattern as a way to reduce application downtime at deployment time. Designing an application to fit this deployment pattern is arguably much easier when you’re first creating it, but where do you start with an existing application, and how do you take the application and convert it to use the rolling deployments pattern?

In this post, I show you how to convert an existing application to use the rolling deployments pattern in Octopus with the help of child steps.

In this post

The application
Sequential deployment process
Convert to a rolling deployment process
- Scale up the servers
- Choosing a load balancer
- Load balancer target pools
- Create a new project
- Convert the PetClinic deployment process
- Switch over to new infrastructure
- Clean-up
Conclusion

The application

I am going to use PetClinic as an example and convert the deployment process for the application from one that runs deployment steps sequentially in Octopus to a rolling deployment process. PetClinic is a sample Spring Boot application written in Java that has two main components:

A web front-end.
A database.

I don’t explain how to build the PetClinic application in this post. If you are new to building Java applications, we have a number of guides that include step-by-step instructions to setup CI/CD pipelines for various tools.

For both the sequential and rolling deployment processes, the PetClinic application and MySQL database are hosted in Google Cloud. All of the infrastructure used in these examples, including the servers, load balancer, and databases are re-created regularly using Operations Runbooks.

Some caveats

It’s important to highlight that this post won’t cover every element required for a zero-downtime deployment. It makes some assumptions about the application set-up:

The database is already deployed in a highly available configuration. For more information on MySQL high availability, refer to the documentation.
Changes to the database are made in a backward and forward compatible way using Flyway.
Any required session state is persisted for when an individual server is being deployed to.

Sequential deployment process

For deployments where you aren’t concerned about application downtime, Octopus caters for this by running steps sequentially one after the other, by default.

Start trigger
It’s also possible to configure your deployment process to run steps in parallel. However, care should be taken to avoid situations where steps running in parallel depend on one another.

The existing PetClinic application is modeled using this sequential deployment technique. The deployment process consists of a number of key steps:

A manual intervention approval step for the Production environment only.
Flyway DB migration community step template steps to display the migration state and apply any new database changes.
A deploy to WildFly step for the PetClinic web front-end.

In addition, the deployment process includes steps that will post messages to a Slack channel with deployment progression updates.

The complete deployment process can be seen here:

After creating a release in Octopus, you can see an example of the sequential deployment in action to the Development environment:

The steps are run one at a time until the deployment is complete. When the Deploy PetClinic web app step is run, the application becomes unavailable to serve requests to users.

Perhaps unsurprisingly, the infrastructure used in these deployments (per environment) looks like this:

It includes:

A single Ubuntu virtual machine hosting the Wildfly application server.
A MySQL database hosted in a Google Cloud SQL service.

Sample Octopus project
You can see the PetClinic sequential deployment process before the conversion to a rolling deployment process in our samples instance.

Convert to a rolling deployment process

Now that we have seen the deployment process for the existing application, we first need to decide on what our infrastructure will look like for the rolling deployment process.

Scale up the servers

In order to reduce downtime and still serve requests for users, we need to scale up the number of servers we use. We also need a load-balancer that we can use to control which servers are available.

In the previous sequential deployment example, we had a single virtual machine per environment. To keep things simple we’ll keep the infrastructure for the Development environment the same as before. For the Test and Production environments, however, the infrastructure will look like this:

This includes a shared load balancer, and this time, two application servers in each environment, connecting to the MySQL database as before.

After you have created your new servers, you also need to add them in Octopus as new deployment targets and tag them with any appropriate target roles.

Choosing a load balancer

There are many different types of load balancer, but a key requirement for this rolling deployments example is the ability to control which servers are available to serve traffic. For this reason, and as this example is running from Google Cloud, we’ll use a network load balancer. This provides a way to add and remove our servers from the load balancer as part of the deployment process, which we’ll see a little later on. For more information about setting up a network load balancer, please refer to the Google documentation.

In this example, the load balancer is shared between the Test and Production environments. To route traffic to the correct place, a different TCP port is used at the load balancer to identify the intended environment.

Port 8080 is used for traffic destined for the Test environment.

Port 80 is used for traffic destined for the Production environment.

Load balancer target pools

Previously, users accessed the PetClinic web front-end directly on a single virtual machine. Here, we use dedicated target pools for the Test and Production environments. A target pool is the name given to a group of virtual machine instances hosted in Google Cloud.

Create a new project

In order to change our deployment process, and keep the ability to deploy PetClinic sequentially, we need to create a new project. One way to achieve this is by cloning the existing project.

In the existing project, under Settings, use the overflow menu (...) and select Clone:

Give the new project you are cloning from the original project a name, review the settings, and when you are satisfied, click SAVE:

Convert the PetClinic deployment process

Next, we’ll convert the deployment process for the project itself. Not all of the PetClinic deployment process lends itself naturally to the rolling deployments pattern, and for this reason, we are focusing on the web front-end of PetClinic.

Choosing what to convert
It’s important to decide for yourself what elements of your project’s deployment process should be converted to use a rolling deployments pattern. In certain situations, it could make things harder to deploy.

Configure a rolling deployment

To convert the Deploy PetCinic web app step to a rolling deployment we perform the following actions:

Open the step in the deployment process editor, expand the section On Targets in Roles, and click CONFIGURE A ROLLING DEPLOYMENT:

In the Rolling Deployment option that appears, choose a Window size. I’ve chosen a Window Size of 1 since we will only deploy to a maximum of two servers per environment:

Click SAVE to update the deployment step.

Add the child steps

We’ve configured a rolling deployment, but it’s not very intelligent yet. Currently, the process deploys PetClinic to each server one at a time, taking each instance of the application offline as it deploys.

We need to add new steps to our rolling deployment to safely deploy new versions of the PetClinic application to each virtual machine and continue serving traffic to users on the other server.

These steps will:

Retrieve the virtual machine name.
Remove the virtual machine from the load balancer before the PetClinic deployment.
Add the virtual machine into the load balancer after the PetClinic deployment.
Test the PetClinic web front-end is available.

In Octopus, adding multiple steps to a rolling deployment process is done with child steps.

gcloud CLI and authorization
Most of the commands used for interacting with Google in this next section make use of the Google Cloud CLI. To use the gcloud CLI you usually need to authorize it. For further information on gcloud authorization, please refer to the documentation.

Add a new child step

To add a child step, we open the overflow menu (...) for the existing Deploy PetClinic web app step and select Add child step:

We are presented with the Choose Step Template selector where we choose the required step type.

Next, I’ll walk through the new child steps needed to complete the rolling deployment process. Some of the script examples have been reduced to the minimum needed to highlight the key parts.

Retrieve the instance name

This script step is required so that we can identify the name of the virtual machine hosted in Google Cloud for use when removing and adding to the load balancer. We do this by querying Google with the gcloud instances describe command:

$machineName = $OctopusParameters["Octopus.Machine.Name"]

$instanceName=(& gcloud compute instances describe $machineName --project=$projectName --zone=$zone --format="get(name)" --quiet) -join ", "

If a match is found using the machine identified by the Octopus system variable Octopus.Machine.Name, the script sets an output variable with the name as recorded in Google Cloud:

Set-OctopusVariable -name "InstanceName" -value $instanceName

Remove the machine from the load balancer

When we have the name of the machine we are deploying to, we need to remove it from the load balancer target pool. However, to prevent an attempt to remove the virtual machine from the target pool when it’s not present to start with, we can run the gcloud target-pools describe command to check:

$instances=(& gcloud compute target-pools describe $targetPoolName --format="flattened(instances[])" --region=$region --project=$projectName --quiet)

If the instance is found in the target pool, we run the gcloud target-pools remove-instances command supplying the instance name with the --instances parameter:

$instanceName = $OctopusParameters["Octopus.Action[Retrieve machine instance name].Output.InstanceName"]

$response=(& gcloud compute target-pools remove-instances $targetPoolName --instances=$instanceName --instances-zone=$zone --project=$projectName --quiet)

After the virtual machine has been removed from the load balancer, we can proceed to deploy the PetClinic application.

We don’t need to add a new child step to deploy the PetClinic application, as it already exists. Instead, we will place that step in the right place after we have added the necessary child steps.

Testing the PetClinic application

After the PetClinic front-end has been deployed, we can test if it’s responding to requests by adding a community step template called HTTP - Test URL as a child step.

We use a variable of #{Project.Wildfly.Url} to test that it returns an HTTP 200 OK response. This will tell us if the application is running. Any other HTTP response will result in a failure.

Add the machine to the load balancer

Lastly, when the PetClinic application has been verified as online, we can add it back to the load balancer target pool. We do this by running the gcloud target-pools add-instances command and supplying the instance name (as before) with the --instances parameter:

$instanceName = $OctopusParameters["Octopus.Action[Retrieve machine instance name].Output.InstanceName"]

$response=(& gcloud compute target-pools add-instances $targetPoolName --instances=$instanceName --instances-zone=$zone --project=$projectName --quiet)

Re-order the child steps

After all of the child steps have been added, you can reorder them if necessary. In our case, we need to move the original Deploy PetClinic web app step to the middle so that we don’t deploy the application to the virtual machine until it has been removed from the load balancer.

To reorder the child steps:

Use the overflow menu (...) and select Reorder child steps.
Rearrange the steps in the desired order.
Click SAVE when done.

The rolling deployment process

Some of the steps in the rolling deployment process aren’t required for the Development environment. This is because we’re not using a load balancer in that environment. To skip the steps that don’t need to run, we use an environment run condition. This will skip the steps applicable to the load-balanced environments when deploying to Development.

The complete rolling deployment process is shown here:

You can see an example deployment to Production using the new rolling deployment process:

And that’s it! We’ve successfully converted our deployment process from a sequential one to a rolling deployment process.

Sample Octopus project
You can see the complete PetClinic deployment process after the conversion to a rolling deployment process, in our samples instance.

Switch over to new infrastructure

In order to use our new infrastructure, we need to direct our users to our application via the load balancer. The simplest way to do that is to adjust your DNS records. In the case of PetClinic, I just need to adjust the DNS A record to point to the load balancer and wait for the DNS changes to update.

Changing any DNS records may result in a period of time where users are still connecting directly to the virtual machines. This is usually no more than 24 hours, but this will depend on your DNS provider and how long the DNS changes take to propagate.

Clean-up

At this point, you no longer need your old Octopus project. You can clean it up by either disabling it (and effectively archiving it) or delete it if you no longer need it for any auditing requirements.

Conclusion

As you can hopefully see from this post, with a few steps, you can switch from a sequential deployment process in Octopus to one using the rolling deployments feature. This allows you to benefit from reduced downtime, safe in the knowledge that your application can remain online to serve requests to your users.

Until next time, Happy Deployments!

Learn more

This post was originally published at octopus.com.

The ultimate guide to rolling deployments

Mark Harrison — Mon, 13 Jul 2020 16:46:19 +0000

When deploying new versions of modern software, like web applications and services, it’s still common for some teams to take their entire website down while they do deployments.

If the majority of your customers only use your application during business hours, then this big-bang approach is probably acceptable, but what happens if your customers are using your applications 24-7?

Today, users expect applications to be available all the time, and there are a few deployment patterns you can use to achieve zero-downtime. In this post, I’ll discuss one of these patterns in more depth; rolling deployments. I’ll also provide you with some practical examples of how to implement rolling deployments with different tooling.

In this post

What are rolling deployments?
Why are they useful?
Rolling deployment patterns in practice
A word on databases
Conclusion

What are rolling deployments?

A rolling deployment is a deployment pattern (also known as an incremental deployments, batched deployments, or ramped deployment) where new software is delivered, usually to one or more deployment targets at a time, until all of the targets have the updated version of the software rolled out.

A typical process looks something like this:

With two nodes running v1.0 of your application, drain the first node to be updated, take it out of the load-balancer pool, and leave the remaining node online to serve traffic:
Stop the v1.0 application from running on the drained node, then deploy the new v1.1 version. Optionally, verify the deployment was successful by running tests on your newly deployed application. All the while, maintaining at least one node running v1.0 of your application:
After the first node has updated successfully, proceed with draining the remaining node still running v1.0 of your application, while your new v1.1 version is now online serving traffic:
Stop the v1.0 application on the remaining node from running, deploy the new v1.1 version. Again, optionally verify the deployment was successful:
Finally, after v1.1 of your application has been deployed successfully to all of your nodes, your rolling deployment is complete!

If you want to ramp up your rolling deployment and deliver a new version to more than one node simultaneously, say two for example, then it would look like this:

This incremental approach is often favored in web applications which sit behind a load balancer, as most load balancers support a concept known as Connection draining. This allows connections to a service to finish naturally and prevents new connections from being established.

By performing this action, instances which are selected to be updated, can be removed from the available pool after they have finished their work, while others remain online serving traffic.

Although the scenario above describes a web application rolling deployment, it’s possible to achieve rolling deployments for other types of applications, providing they are built in a way that supports ending their process safely.

For example, Octopus Deploy’s High Availability configuration also has a drain option, which prevents any new tasks from executing, and finishes up any tasks it’s currently executing until idle. Features like draining, allow for the safe termination of a process, which can then be updated and brought back online.

Why are they useful?

So why use rolling deployments over other patterns such as canary or blue/green? Well, rolling deployments offer the following benefits:

Incremental update

New versions of your application are rolled-out incrementally. This allows you to verify it’s working, for example, by running health checks or tests before moving on to the next batch of updates.

In the event that you need to initiate a rollback, you can also do this in a safe and controlled manner.

Keeping the lights on

While you go about updating a small number of your application instances, the rest continue to serve requests. This means there is no downtime for your application, and it’s available for your users throughout the deployment.

Parallelism

You can usually control the number of concurrent instances that are deployed to at any one time. Further deployments won’t start until a previous deployment has finished.

You can use the Window size option within an Octopus rolling deployment to control how many deployment targets can be deployed to at once.

Rolling deployment patterns in practice

To demonstrate the different approaches to rolling deployments, we have a very simple .NET Core 3.1 application which will display a web page.

The HTML for the section I’m interested in is shown below:

<div class="text-center">
    <h1 class="display-4">Welcome</h1>
    <p>If you are seeing this, then <strong>Congratulations!</strong> <br/> You've got the example application running. </p>

    @if(Settings.Value.AppVersion == "0.0.2") {
        <p>v0.0.2 of the application comes with this text </p>
    }
    @if(Settings.Value.AppVersion == "0.0.3") {
        <p>But don't miss out on v0.0.3 of the application which comes with this text! </p>
    }
</div>

The code for the application is available on GitHub and has a Tag corresponding to the three different AppVersion values. A Docker image has also been published as octopusdeploy/rolling-deploy-web-example.

I wanted to see just how easy it would be to perform a rolling deploy of this application using some popular technologies and tools, so I’ll demonstrate rolling deployments with:

Docker
Kubernetes
Octopus

Docker rolling application updates

Docker has become the de facto container technology to use in the last few years. It will come as no surprise therefore, that it natively supports rolling deployments with its concept of a Docker service. Typically, a service is a small piece of a much larger architectural picture and is popular with microservices.

Services support a number of different options, including a rolling update policy as well as the ability to rollback.

Docker containerized application

I’m running Docker on an Ubuntu server and using our pre-built container image. There are a couple of ways to install Docker on Ubuntu:

Use the Ubuntu repository by running: sudo apt-get install docker.io.
Use the Official Docker guide.

I opted for the Ubuntu repository as it seemed quicker and easier, but your mileage may vary. Whichever method you choose, it’s worth ensuring you meet the installation prerequisites.

For the sake of simplicity, I’ll be interacting with Docker in an SSH terminal session to my Linux box. There are production-ready setups to automate this, which feature the definition of your services in a Docker Compose file, including sections to control automatic updates and rollback settings.

Permissions requirement:
Most of the commands in this demonstration make use of sudo. By default, the Docker daemon runs as the root user and requires elevated permissions to execute commands. If you prefer not to use sudo when executing your commands, be sure to follow the Docker post-install instructions.

Firstly, to see the Docker image of this running standalone, we’ll run it with the following command:

markh@ubuntu01:~$ sudo docker run -d -p 5001:5001 octopusdeploy/rolling-deploy-web-example:0.0.1

Unsurprisingly, running this Docker image displays the web page:

I don’t explain how to build a container image in this post. If you are new to Docker, my colleague Shawn has written an excellent series on how to containerize a real world application, instead of another “Hello World” example.

Container clean-up

A quick-tidy up is needed next. To delete the container we created using the run command above, we need to stop it, and then remove it using the rm command. We can do this in a condensed one-liner:

sudo docker rm $(sudo docker stop $(sudo docker ps -a -q --filter ancestor=octopusdeploy/rolling-deploy-web-example:0.0.1 --format="{{.ID}}"))

This locates our container by the image name octopusdeploy/rolling-deploy-web-example:0.0.1 and passes that to the stop command, and finally passes that to the rm command.

To deploy more than one instance of our container, we need to create our Docker service. This uses Docker Swarm as its orchestrator under the hood.

Docker Kubernetes orchestrator
Docker also supports Kubernetes as an orchestrator when deploying containers using the Docker stack command, but it’s not possible to specify the orchestrator when using service create.

So let’s see what our command to create a service looks like:

markh@ubuntu01:~$ sudo docker service create --name rolling-deploy-svc --replicas 3 --publish published=5001,target=5001 --update-delay 10s --update-parallelism 1 octopusdeploy/rolling-deploy-web-example:0.0.1

There’s quite a lot going on in that command, so let’s unpick what we are asking of Docker here:

The --name is self explanatory.
The --replicas flag controls the number of containers we want (3).
The --publish published=5001,target=5001 specifies the service to be accessed on port 5001, using Swarm’s routing mesh which acts essentially like a software load-balancer.
The --update-delay configures the time delay (10s) between updates to a service task.
The --update-parallelism controls the maximum number of tasks that Docker will schedule simultaneously (1).
Lastly, we specify the image to use: octopusdeploy/rolling-deploy-web-example:0.0.1.

Hint:
When running service create for the first time, you may receive a warning, just as I did: This node is not a swarm manager. To fix this, run either of the following commands:

sudo docker swarm init: This will initialize your current node as a swarm manager.

sudo docker swarm join: This will connect your local node to swarm.

Executing this results in our service being deployed to Docker Swarm with three instances:

wxi1w4m7crknaz1f800kr9ztt
overall progress: 3 out of 3 tasks
1/3: running   [==================================================>]
2/3: running   [==================================================>]
3/3: running   [==================================================>]
verify: Service converged

We can also check our service has the correct update configuration by running the service inspect command:

markh@ubuntu01:~$ sudo docker service inspect rolling-deploy-svc --pretty

ID:             bh03s0yjzkevzkkwvu8q2h0jj
Name:           rolling-deploy-svc
Service Mode:   Replicated
 Replicas:      3
Placement:
UpdateConfig:
 Parallelism:   1
 Delay:         10s
 On failure:    pause
 Monitoring Period: 5s
 Max failure ratio: 0
 Update order:      stop-first
RollbackConfig:
 Parallelism:   1
 On failure:    pause
 Monitoring Period: 5s
 Max failure ratio: 0
 Rollback order:    stop-first
ContainerSpec:
 Image:         octopusdeploy/rolling-deploy-web-example:0.0.1@sha256:4da10d630025bf268b855b0b4afafa7334769ab6d0b3e75e11a3f11949708552
 Init:          false
Resources:
Endpoint Mode:  vip
Ports:
 PublishedPort = 5001
  Protocol = tcp
  TargetPort = 5001
  PublishMode = ingress

The result of this shows we have our desired UpdateConfig which will update one task at a time.

Docker service update

Now we can update the container image for octopusdeploy/rolling-deploy-web-example to v0.0.2 by running the service update command:

markh@ubuntu01:~$ sudo docker service update rolling-deploy-svc --image octopusdeploy/rolling-deploy-web-example:0.0.2

Docker runs the update to each container, one task at a time just as we have configured it to:

overall progress: 0 out of 3 tasks
1/3: running   [=============================================>     ]
2/3:
3/3:

After the first task is complete, it moves onto task two:

overall progress: 1 out of 3 tasks
1/3: starting  [==================================================>]
2/3: ready     [=====================================>             ]
3/3:

Until all the tasks to update the containers to v0.0.2 are complete:

overall progress: 3 out of 3 tasks
1/3: running   [==================================================>]
2/3: running   [==================================================>]
3/3: running   [==================================================>]
verify: Service converged

Browsing to the website now shows the text which applies for v0.0.2:

Docker service rollback

Just as it’s straight-forward to roll-out, it’s also possible to manually rollback with a simple command in Docker.

Firstly, we will update to our final version v0.0.3 of the application by running:

markh@ubuntu01:~$ sudo docker service update rolling-deploy-svc --image octopusdeploy/rolling-deploy-web-example:0.0.3

We verify the new v0.0.3 version by running the service inspect command, passing in a --format parameter for only the output we want to see:

markh@ubuntu01:~$ sudo docker service inspect --format='{.Spec.TaskTemplate.ContainerSpec.Image}}' rolling-deploy-svc

octopusdeploy/rolling-deploy-web-example:0.0.3@sha256:151a8f2aaed0192bf9f22eaeff487d546e6ff8fec4d0691e6697dede743b187c

Because Docker Swarm knows the versions we deployed, we can revert to the previous one (v0.0.2) using the rollback command:

markh@ubuntu01:~$ sudo docker service rollback rolling-deploy-svc

rolling-deploy-svc
rollback: manually requested rollback
overall progress: rolling back update: 3 out of 3 tasks
1/3: running   [>                                                  ]
2/3: running   [>                                                  ]
3/3: running   [>                                                  ]
verify: Service converged

Once successfully rolled back, it has confirmed the service is running.

Hint:
As I didn’t specify any parameters to the rollback command, Docker will by default, rollback one task at a time with no delays between each one. You can specify different values by passing the following to the command:

--rollback-parallelism

--rollback-delay

The Docker documentation has a full list of parameters you can use.

We can verify the rollback was successful using the same command to inspect the service as before:

markh@ubuntu01:~$ sudo docker service inspect --format='{.Spec.TaskTemplate.ContainerSpec.Image}}' rolling-deploy-svc

octopusdeploy/rolling-deploy-web-example:0.0.2@sha256:4843a91ba84ace97cb11a6e3f68827a8c28a628d509159910c868f9ad02c3053

This results in the expected v0.0.2 version in the output.

Database rollbacks
When a service in Docker utilizes a database for storage, it’s important to have a strategy in place to deal with a service rollback, particularly if the database is within a container. Docker won’t rollback database changes for you automatically, and that could leave your database and application in an incompatible state. The Docker storage documentation provides some guidance on the different options for storage in a container.

Docker service clean-up

Finally, to remove our Docker service we just run the rm command:

markh@ubuntu01:~$ sudo docker service rm rolling-deploy-svc

Docker summary

As you can see, it doesn’t take much setup to get rolling deployments working in Docker. Coupled with its support for rollbacks makes it an attractive option to consider.

Kubernetes rolling updates

Rolling deployments in Kubernetes is called rolling updates.

A pod’s instances will be updated incrementally with new ones. It supports both a max number or percentage of pods to be unavailable during an update, as well as a max number of new pods that can be created. In addition to this, Kubernetes has a handy built-in feature to allow updates to be reverted to a previous version.

To find out more about Kubernetes, my colleague Shawn continued his container series focussing on Kubernetes.

The Kubernetes tutorial on updates includes a nice diagram showing how it works:

Kubernetes cluster setup

Just as before, I’ll re-use our pre-built container image for this demonstration, this time using MicroK8s. I’ll also interact with it primarily in an SSH terminal session.

Canonical, the authors, describe MicroK8s as:

a single package of k8s for 42 flavours of Linux. Made for developers, and great for edge, IoT and appliances.

It’s useful for people like me who want to try out Kubernetes or do some development with it.

One of the benefits of MicroK8s is that it doesn’t require any type of Virtual Machine as with other Kubernetes choices (such as Minikube).

Before you install MicroK8s, it’s worth noting that there are some prerequisites:

Ubuntu 18.04 LTS or 16.04 LTS (or other OS that supports snapd).
At least 20G free disk space.
4GB of RAM.
An internet connection.

To install MicroK8s, we run the snap install command:

markh@ubuntu01:~$ sudo snap install microk8s --classic

microk8s v1.17.0 from Canonical✓ installed

Full install output
If you want to see the full install output from snap, run the snap changes command:
markh@ubuntu01:~$ snap changes

ID   Status  Spawn               Ready               Summary
1    Done    today at 10:17 UTC  today at 10:17 UTC 
Initialize system state
2    Done    today at 10:17 UTC  today at 10:17 UTC  Initialize device
3    Done    today at 14:38 UTC  today at 14:38 UTC  Install "microk8s" snap
From this, you can then run the command snap change 3, where 3 is the value from the ID column above for the installation of MicroK8s. This will give you a line breakdown of the install steps.

Kubernetes deployments

Now we have MicroK8s installed and running, let’s go ahead and create a Kubernetes deployment using our existing image rolling-deploy-web-example and set it to listen on port 5001.

Google describes Kubernetes deployments as items which:

represent a set of multiple, identical Pods with no unique identities. A Deployment runs multiple replicas of your application and automatically replaces any instances that fail or become unresponsive. In this way, Deployments help ensure that one or more instances of your application are available to serve user requests. Deployments are managed by the Kubernetes Deployment controller.

This sounds perfect for a rolling deployment.

Kubernetes containerized application setup

To set up our deployment for our application, we will use the kubectl binary which is packaged with MicroK8s. This is the command-line interface (CLI) for managing Kubernetes. It’s specially prefixed with microk8s. to avoid any naming conflicts with any other instances of Kubernetes you may have running. If we run the create deployment command:

markh@ubuntu01:~$ sudo microk8s.kubectl create deployment rollingdeploy-microk8s --image=octopusdeploy/rolling-deploy-web-example:0.0.1

deployment.apps/rollingdeploy-microk8s created

MicroK8s will create our deployment and confirm it has been successfully created.

Next, we’ll set the application pods to listen on port 5001. To do that, we run the expose command:

markh@ubuntu01:~$ sudo microk8s.kubectl expose deployment rollingdeploy-microk8s --type=NodePort --port=5001

service/rollingdeploy-microk8s exposed

Kubernetes dashboard

Although the rollingdeploy-microk8s pod has been created, it might not be available immediately. We can check its status by looking at our service using the Kubernetes dashboard, which is included as an add-on in MicroK8s.

Attempting to access the dashboard remotely requires you to jump through a few hoops. After the add-on was enabled, I found the simplest way was to create a proxy from my machine to the server by running the kubectl proxy command:

markh@ubuntu01:~$ sudo microk8s.kubectl proxy --accept-hosts=.* --address=0.0.0.0

Starting to serve on [::]:8001

From there you can access the dashboard on port 8001, but you’ll need either a Kubeconfig file or Token to login. See the MicroK8s Dashboard add-on for further details.

Note: You can skip the login by setting the --enable-skip-login argument for the dashboard container, but this isn’t advised as it goes against security best practices.

Once open, you can use the dashboard to deploy containerized applications, manage and interact with your cluster resources.

In order to perform a rolling update, first we need more than one replica of our application. We can scale our deployment directly from the dashboard by clicking on the three ellipsis on the right hand side of the Deployments section:

For our Kubernetes deployment, I updated the Desired Replicas to 3 so I can perform a rolling update and then hit Scale:

Equivalent kubectl commands
You may have noticed the dashboard provides the equivalent command to run for our action. For scaling our resource, that is:
sudo microk8s.kubectl scale -n default deployment rollingdeploy-microk8s --replicas=3

After the pods have been provisioned, we can confirm this by querying the pod’s status directly by running the get pod command (names may vary):

markh@ubuntu01:~$ sudo microk8s.kubectl get pod

NAME                                      READY   STATUS    RESTARTS   AGE
rollingdeploy-microk8s-794bdc64c4-fv7zt   1/1     Running   0          76s
rollingdeploy-microk8s-794bdc64c4-t6mh5   1/1     Running   0          76s
rollingdeploy-microk8s-794bdc64c4-trr6f   1/1     Running   0          76s

To verify our application is working, we need to find the port that has been exposed by Kubernetes to the deployment we created at the start by running get service:

markh@ubuntu01:~$ sudo microk8s.kubectl get service rollingdeploy-microk8s

NAME                     TYPE       CLUSTER-IP      EXTERNAL-IP   PORT(S)          AGE
rollingdeploy-microk8s   NodePort   10.152.183.39   <none>        5001:32334/TCP   1m

In my case, the port is 32334 so my URL to access my service is:

http://ubuntu01.octopusdemos.com:32334

Note:
The port may be different when running this on your own machine. A random port, in the range 30000-32767 (by default) will be assigned by Kubernetes as we chose a NodePort type when we ran the expose command earlier.

Open the URL in a browser, and we can see that we have v0.0.1 for our application running in Microk8s:

Kubernetes rolling update

Let’s go ahead and instruct Kubernetes to update our three pods with v0.0.2 of our image octopusdeploy/rolling-deploy-web-example by running the set image command:

markh@ubuntu01:~$ sudo microk8s.kubectl set image deployment/rollingdeploy-microk8s rolling-deploy-web-example=octopusdeploy/rolling-deploy-web-example:0.0.2 --record

deployment.apps/rollingdeploy-microk8s image updated

Next, we can watch the live progress of our rollout until its complete by running the rollout status command:

markh@ubuntu01:~$ sudo microk8s.kubectl rollout status deployment.v1.apps/rollingdeploy-microk8s

Waiting for deployment "rollingdeploy-microk8s" rollout to finish: 1 out of 3 new replicas have been updated...
Waiting for deployment "rollingdeploy-microk8s" rollout to finish: 1 out of 3 new replicas have been updated...
Waiting for deployment "rollingdeploy-microk8s" rollout to finish: 1 out of 3 new replicas have been updated...
Waiting for deployment "rollingdeploy-microk8s" rollout to finish: 2 out of 3 new replicas have been updated...
Waiting for deployment "rollingdeploy-microk8s" rollout to finish: 2 out of 3 new replicas have been updated...
Waiting for deployment "rollingdeploy-microk8s" rollout to finish: 2 out of 3 new replicas have been updated...
Waiting for deployment "rollingdeploy-microk8s" rollout to finish: 1 old replicas are pending termination...
Waiting for deployment "rollingdeploy-microk8s" rollout to finish: 1 old replicas are pending termination...
deployment "rollingdeploy-microk8s" successfully rolled out

You can see it indicated it was updating one Pod at a time.

Kubernetes deployments ensure that only a certain number of pods are down while they are being updated. It does this by creating a new pod and destroying the old ones after it has completed.

Default pod update control

By default, Kubernetes ensures there are at least 75% of the desired number of pods available.
In addition, another default is to create no more than 25% of the overall desired number of pods.

Open the URL in a browser, and we can see we have v0.0.2 of our application running in Microk8s:

The deployment’s rollout was triggered here, as set image caused an update to the underlying deployment pod’s template. A template is a specification document that describes the way a replication controller should create an actual pod.

We can see what the template looks like for our application by running edit:

markh@ubuntu01:~$ sudo microk8s.kubectl edit deployment.v1.apps/rollingdeploy-microk8s

This opens the template file in a text editor. For me, that’s in the terminal itself. You can edit this file interactively. Changing the deployment pod’s template (the section within .spec.template) will result in triggering the deployment’s rollout:

Hint:
Other updates to a deployment, like the scaling we did earlier, don’t result in a rollout being triggered.

Kubernetes deployment rollback

A successful rolling deployment is obviously what we all hope for, but it’s inevitable that at some point, you’ll need to initiate a rollback, either part of the way through a rollout itself or some time after.

With Kubernetes, all of a deployment’s rollout history is kept in the system by default. That means, you can rollback anytime you want.

Note:
It’s possible to change the amount of history that’s stored for a deployment’s rollout (by modifying the revision history limit), but it’s not generally recommended as this limits your ability to rollback a deployment.

To see the rollout history for our deployment we run the rollout history command:

markh@ubuntu01:~$ sudo microk8s.kubectl rollout history deployment.v1.apps/rollingdeploy-microk8s

deployment.apps/rollingdeploy-microk8s
REVISION  CHANGE-CAUSE
1         <none>
2         kubectl set image deployment/rollingdeploy-microk8s rolling-deploy-web-example=octopusdeploy/rolling-deploy-web-example:0.0.2 --kubeconfig=/var/snap/microk8s/1107/credentials/client.config --record=true

We can choose to revert back to the previously deployed v0.0.1 version by running rollout undo:

markh@ubuntu01:~$ sudo microk8s.kubectl rollout undo deployment.v1.apps/rollingdeploy-microk8s

deployment.apps/rollingdeploy-microk8s rolled back

Hint:
You can also choose to revert to a specific revision of your application by running:
markh@ubuntu01:~$ sudo microk8s.kubectl rollout undo deployment.v1.apps/rollingdeploy-microk8s --to-revision=1

Where the --to-revision parameter has the revision you wish to go back to.

The Kubernetes documentation has a full list of parameters you can use.

We can confirm we have rolled back, either by looking back in the dashboard, viewing the application in a browser, or by running the describe command:

markh@ubuntu01:~$ sudo microk8s.kubectl describe deployment

Name:                   rollingdeploy-microk8s
Namespace:              default
CreationTimestamp:      Wed, 22 Jan 2020 13:19:30 +0000
Labels:                 app=rollingdeploy-microk8s
Annotations:            deployment.kubernetes.io/revision: 3
Selector:               app=rollingdeploy-microk8s
Replicas:               3 desired | 3 updated | 3 total | 3 available | 0 unavailable
StrategyType:           RollingUpdate
MinReadySeconds:        0
RollingUpdateStrategy:  25% max unavailable, 25% max surge
Pod Template:
  Labels:  app=rollingdeploy-microk8s
  Containers:
   rolling-deploy-web-example:
    Image:        octopusdeploy/rolling-deploy-web-example:0.0.1
    Port:         <none>
    Host Port:    <none>
    Environment:  <none>
    Mounts:       <none>
  Volumes:        <none>
Conditions:
  Type           Status  Reason
  ----           ------  ------
  Available      True    MinimumReplicasAvailable
  Progressing    True    NewReplicaSetAvailable
OldReplicaSets:  <none>
NewReplicaSet:   rollingdeploy-microk8s-794bdc64c4 (3/3 replicas created)
Events:

This shows us the Image is set to octopusdeploy/rolling-deploy-web-example:0.0.1 as we expected.

Database rollbacks
As with Docker, it’s important to understand what will happen with any databases you have when you initiate a Kubernetes rollback. Google has a great article discussing running databases on Kubernetes in more depth.

Kubernetes deployment clean-up

To remove all of the resources associated with our Kubernetes deployment, we use the delete command:

markh@ubuntu01:~$ sudo microk8s.kubectl delete services,deployment rollingdeploy-microk8s -n default

service "rollingdeploy-microk8s" deleted
deployment.apps "rollingdeploy-microk8s" deleted

Kubernetes summary

It felt like there was a little more to the setup for a rolling deployment with Kubernetes than with Docker, particularly to gain access to the dashboard, but after it was all configured, it worked excellently.

Rolling deployments with Octopus

Octopus has supported the concept of rolling deployments since Octopus 2.0.

With the use of child steps, we can set-up our deployment process for the rolling-deploy-web-example application in Octopus.

After creating a new project, we configure a rolling deployment with three steps:

A script to remove the node from the load balancer.
Deployment of the web application.
A script to add the node back into the load balancer.

To achieve an incremental release in Octopus, we need to make our Window size lower than the total number of deployment targets. In my example, I set this to 1, as you can see below:

I have two deployment targets configured with the target role: rolling-deploy-webapp.

When I deploy the release to the Test environment, Octopus deploys to one deployment target at a time, as I configured in my deployment process earlier:

And that’s all there is to it! Check out our docs for a complete reference on rolling deployments in Octopus.

Sample Octopus project
You can view this Octopus project setup in our samples instance.

A word on databases

Usually one of the main sticking points with rolling deployments is the database. Performing rolling deployments which involve some kind of persistent storage can be tricky, though not impossible. The devil is always in the detail.

If you want to perform rolling deployments which includes database changes, I recommend deploying the database first. You also want to ensure any changes you make to your database are backward compatible with previous versions of code you have deployed.

Lastly, it’s definitely a good idea to test your rollback strategy frequently.

We have an excellent series of posts on database deployments that discuss this topic and more.

Conclusion

No matter which tooling you use, rolling deployments is just one pattern available to optimize deployment of your software. But with an incremental approach, it allows you to keep your applications online while rolling out newer versions of your software in a controlled manner, often with native support for rollbacks, making it a firm favorite of mine for minimal disruption.

CI/CD pipeline guides
If you need any additional help configuring your CI/CD pipeline, we have created the Octopus Guides which include step-by-step instructions to setup CI/CD pipelines for various technology stacks, including Docker and Kubernetes.

This post was originally published at octopus.com.