DEV Community: Octopus Deploy

A tip for debugging Octopus Deploy variables

Ryan Rousseau — Tue, 29 Mar 2022 21:49:21 +0000

I sometimes need to debug issues related to incorrect variable substitutions or filters in my deployments.

The Octopus docs have some tips on debugging problems with variables.

The third option, writing variables to the deployment log, is the one I use the most.

Writing variables to the deployment log

Octopus provides two system variables, OctopusPrintVariables and OctopusPrintEvaluatedVariables. Setting these variables to True will write all variable values to the deployment log.

OctopusPrintVariables will write the unevaluated variable values to the log. OctopusPrintEvaluatedVariables will write the fully evaluated variable values to the log.

I find these useful enough to keep them in the project. Before, I have left the values set to False when not using them. I change them to True when I need to debug other variables. The doc page mentions that you can update the variable snapshot on a release instead of creating a new release when adding or updating these variables.

I added a little extra to my configuration to avoid switching the values back and forth - prompted variables.

Conditionally writing debug logs with Prompted Variables

I configure OctopusPrintEvaluatedVariables to also prompt with the label Print Evaluated Variables?.

The values will default to False. I can check the boxes during a deployment to turn on the debug logging only for that deployment.

Wrap-up

Combining prompted variables with OctopusPrintVariables and OctopusPrintEvaluatedVariables provides an easy way to debug your deployments' variable values.

This post was originally published at blog.rousseau.dev. Cover photo by Timothy Dykes on Unsplash.

Better multi-tenancy with Octopus Deploy

Mark Harrison — Mon, 26 Jul 2021 10:15:10 +0000

Most people using Octopus will deploy projects to one or more environment. For customers providing Software as a Service (SaaS) applications, they typically need to deploy multiple instances of the application for each of their customers.

Fortunately, there's a feature that's been around since Octopus 3.4 designed exactly for these types of deployment, multi-tenancy.

In this post, I look at two approaches to deploying applications without tenants, and discuss the benefits of using the multi-tenancy feature.

In this post

Introduction
Deploying without tenants
- Using multiple projects
  - Multiple projects pros
  - Multiple projects cons
- Using multiple environments
  - Multiple environment pros
  - Multiple environment cons
Deploying with tenants
- Tenants
- Tenant tags
- Tenant variables
  - Project variable templates
  - Common variables
  - Variable snapshot
  - Missing variables
- Tenanted deployment targets
Conclusion
Learn more
Register for the webinar: Better multi-tenancy deployments using Octopus Deploy

Introduction

This post assumes knowledge of some key Octopus concepts, including:

Projects
Environments
Variables
Lifecycles

If you're new to Octopus, I recommend reading our Getting started with Octopus guide.

To demonstrate how you can model deployments of multiple instances of an application with Octopus, I use a fictitious company called Vet Clinic, deploying the Java application, Pet Clinic.

Deploying without tenants

There are two main implementations we see when deploying multiple instances of the same application for each customer:

Using multiple projects
Using multiple environments

While easy to set up, they don't scale well and can result in duplication.

Using multiple projects

In this scenario, you configure Octopus with multiple projects, each one representing one of your customers.

Onboarding a new customer typically requires creating all of the resources in Octopus needed for a successful deployment for the customer, including:

A new set of deployment targets
Common project variables
Any customer specific "paid-for" environments

In addition, any common steps across the application's deployment process need to be duplicated in the new project. These are usually manual intervention and notification steps.

Multiple projects pros

So why choose multiple Octopus projects to deploy instances of an application to each customer?

Clear customer release dashboard overview

This approach allows you to see which release has been deployed to which environment, for each customer on the dashboard overview.
Variable and Deployment process isolation

Multiple projects allow for complete isolation of variables, and deployment process for a customer. For example, making a change to one project's process only affects that one customer. You can also tailor the deployment process for the customer depending on the features they've signed up for.

In the below example, only Capital Animal Hospital has a step for applying custom branding:
Simpler environment and variable scoping

Environments don't need to be duplicated per customer, resulting in simpler lifecycle configuration. Variables can also be scoped to each environment without risk of choosing the wrong "customer" scoping.

Multiple projects cons

While multiple projects can be used to deploy customer instances separately, there are a number of problems with this approach.

Duplicated project configuration

With every customer project, you end up duplicating project configuration. These include variables, deployment process steps, runbooks, channels and lifecycles. With duplication comes inconsistency and the overhead of managing that. For example, if you want to amend the deployment process for all of your customers, you need to change multiple projects.
Different deployment target roles per customer

If your customers have isolated infrastructure, you need a unique way for Octopus to know which deployment targets belong to the customer you're deploying to. This results in each customers' deployment targets needing target roles that include a differentiator per customer. This is typically a customer name, code or ID.
Multiple projects per customer doesn't scale

The per-project approach doesn't scale well when you have more than a handful of customers. This can be problematic if you deploy many more applications per customer. Each application needs to be modeled n times, where n is the number of customers you have.
No guard rails to ensure variables provided

Using multiple projects, there are no guard rails to ensure all project configuration is set correctly. For example, if a variable is not added (or has an incorrect value), you might not find out about the issue until the deployment of the customer's instance.

Using multiple environments

An alternative to deploying without tenants is using a single Octopus project per application, and modeling each customer with a set of environments they deploy to.

Onboarding a new customer typically involves:

Creating a new set of environments, named after the customer.
Creating a new set of deployment targets, or re-using existing ones and tagging them with the associated customer environments.
Adding new environment-scoped variables.
Updating the project lifecycle to include the new customer environments.

Multiple environment pros

So why choose one or more environments to represent your customers in Octopus?

Single set of project resources to manage

In contrast to multiple projects, with this approach there's just one project, one deployment process, one set of variables, and one lifecycle to manage. When a change is required, it can be made once instead of per project.

For example, if you want to add a step that's required for all customers, such as a Manual intervention step before deployments to production, it can be added quickly and easily.
Customer environments are modeled explicitly

With this approach customer environments have to be modeled explicitly. When a new customer is added, the environment they deploy to has to be created to allow deployment for that customer. It's also possible to see in a single row on the dashboard overview which environment a customer can deploy to.

Multiple environment cons

Although you can use multiple customer environments, there are usually many problems associated with this approach:

Multiple environments created per customer

For each customer, you need to create a new environment record for each customer environment, which doesn't scale. For example, if you have 10 customers, and 4 environments (Development, Test, Staging and Production), you need to create 40 customer environments.
Complicated variable scoping

As there's a single project per application, handling multiple variable values per customer needs to be achieved using different environment scopes. This can quickly become overwhelming. Adding new values, or editing existing ones has to be done carefully to ensure correct scopes are applied to each value. There's a high risk of cross tenant communication in this model.
Rigid deployment process

Customizing the deployment process in a project using multiple customers requires you to add environment run conditions for each step that needs to be run for specific customers. This isn't flexible and doesn't scale as you need to modify these conditions per step, when you add a new customer or change your environments.
Unclear customer release dashboard overview

In contrast to the per-project customer model, it's difficult to see at a glance which release has been deployed to which customer on the dashboard and project overview screens, without endless scrolling.
Complicated, unwieldy Lifecycle

As you onboard new customers, you typically add new customer environments to the project's lifecycle. You then have to define awkward lifecycle phases that fit the environments that are required for each customer (both existing and new). Typically this is handled with a phase that allows any 1 customer environment to be deployed to before progressing to the next phase. As the number of customers grows, so does the complexity of the lifecycle and its phases.

Deploying with tenants

Using tenants in Octopus allows you to easily create customer specific deployment pipelines without duplicating project configuration. You can manage separate instances of your application in multiple environments in a single Octopus project.

Using our Vet Clinic company, here's what the dashboard overview might look like using tenants to model each customer:

This gives us a concise overview showing which release is in which environment. Instead of multiple rows per customer project, this is replaced with a discrete count of tenants who have been deployed to each environment.

If we navigate to the project, we see a more granular overview, this time showing which tenant has what release in each environment:

Sample Octopus project
You can see an example of the Vet Clinic tenanted project in our samples instance.

There are a number of multi-tenancy features working together to make this happen:

Tenants
Tenant tags
Tenant variables
Tenanted deployment targets

Tenants

Tenants in Octopus are the backbone of the multi-tenancy feature. They usually represent the customers of your application, especially when it comes to SaaS products.

Although we discuss the use of tenants to model customers in this post, we designed tenants to be generic so that they can satisfy multiple use cases. Tenants can also represent:

Geographical regions or data centers
Developers, testers, or teams
Feature branches

Learn more about types of tenancy in our documentation.

The tenant overview provides a central place to manage which projects are connected to a tenant, and also which environment.

Note that each project can control its interaction with tenants. By default the multi-tenant deployment features are disabled. You can allow deployments with/without a tenant, which is a hybrid mode that's useful when you're transitioning to a fully multi-tenant project.

There's also a mode where you can require a tenant for all deployments, which disables untenanted deployments for that project.

Onboarding a new customer as a tenant in Octopus can be as simple as creating your tenant, connecting your project to each applicable environment and entering your variable values, then deploying.

Tenant tags

In Octopus, tenant tags help you classify your tenants using custom tags, and tailor tenanted deployments for your projects and environments.

Tenant tags also make it easier to work with tenants as groups instead of individuals. As tenant tags are fully customizable, you can apply meaningful metadata to tenants. This allows you to describe them using your own terminology, and tailor the deployment process to their needs.

In the tenant overview below, Capital Animal Hospital has the Branding tag included:

This indicates that they've opted-in for customized branding of their instance of the Vet Clinic application.

When you build out the deployment process, you can include a tenant tag as a run condition to customize the process for your customers. By applying tags to steps, you are able to specify steps that should only run for customers that match selected tenant tags.

You can associate multiple tenants with the same tag. This automatically groups these tenants together and enables any tenant with the Branding tag to have the custom branding step included as part of any deployment for that tenant.

Tenant tags can also be used to associate multiple tenants with deployment targets and channels, and even choosing which tenants to deploy to. They're a powerful way to help you simplify and scale your deployments.

Tenant variables

You often want to define variable values that are different for each customer. For example:

A database server name or connection string
A tenant-specific URL
Contact details for a tenant

Using an untenanted project, you would define these values in the project itself. With a tenanted project, you can set these values directly on the tenant for any connected projects.

With tenants, there are two types of variable you can specify:

Project variable templates
Common variables.

These both use the variable templates feature.

Project variable templates

Project variables allow you to specify a variable which a tenant can change. A perfect example is a connection string or a database server. With project variables you define them at the project level using project templates.

You can specify the variable type for the project template, just like regular variables. You can also provide a default value which the tenant can overwrite.

Then, on the tenant variable screen, you can set those variables.

Common variables

Common variables are similar to project variables. The main difference is that common variables can be used across multiple projects, and they aren't scoped to environments. Common variables are defined using Library variable set templates

For example, to define an abbreviation for the tenant to use in a deployment or runbook, you can configure a variable template for the library set.

To include common variables for a tenant, you must add the library variable set in the tenant connected project.

Just like project variables, common variable values are supplied at the tenant level.

Variable snapshot

When you create a release Octopus takes a snapshot of the deployment process and the current state of the project variables.

However, tenant variables aren't included in any snapshot. This is helpful as you can add new tenants at any time and deploy to them without creating a new release.

This also means any changes you make to tenant variables will take immediate effect.

Missing variables

One of the great things about tenant variables is the guard rails they put in place for your deployments. Defining either a project template or common variable without a default value means any tenant must provide a value for that variable. Octopus won't allow a deployment to occur without one:

But those guard rails don't start just at deployment. Octopus will also warn you about any missing values in the tenant's variable overview too:

This level of safety reduces the chance of a failed deployment for a tenant due to a missing or incorrect variable value.

Tenanted deployment targets

The way you host your infrastructure for multiple instances of the same project usually varies depending on your application and customers. Two common implementations we see are:

Dedicated hosting: You have dedicated deployment targets for each customer.
Shared hosting: You create farms or pools of servers to host all of your customers, achieving higher density.

In this post, we focus on dedicated hosting, but you can design and implement both dedicated and shared multi-tenant hosting models in Octopus using environments, deployment targets, and tenant tags.

If your customers have isolated infrastructure, in an untenanted configuration, you need to define unique target roles in Octopus to ensure there's no cross-customer communication in the form of deploying one customer's application to another customer's infrastructure.

Using tenants, customer-specific target roles aren't required. You can choose deployments that a target can be involved in:

Exclude from tenanted deployments (default) - The deployment target will never be included in tenanted deployments.
Include only in tenanted deployments - the deployment target will only be included in deployments to the associated tenants. It will be excluded from untenanted deployments.
Include in both tenanted and untenanted deployments - The deployment target will be included in untenanted deployments, and deployments to the associated tenants.

To choose which tenants to associate with a deployment target:

Navigate to the Restrictions ➜ Associated Tenants section of the deployment target.
Select one or more tenants to allow to deploy to individually, or choose from any of the configured tenant tags.

We recommend keeping tenanted and untenanted deployment targets separate, particularly in Production. You could use the same deployment targets for other environments but it's generally better to avoid this situation.

Conclusion

This post covers common approaches when customers deploy multiple instances of the same application for each of their customers without tenants. It also details how you can use the multi-tenancy feature to model this too.

I hope you can see how the Octopus multi-tenancy feature solves some of the problems presented when deploying without tenants, and how it can be leveraged for scalable, reusable, simplified deployments.

Learn more

Register for the webinar: Better multi-tenancy deployments using Octopus Deploy

Join Senior Solutions Architect Mark Harrison and Solutions Architect Adam Close from Octopus Deploy to learn how the multi-tenancy feature in Octopus can be leveraged for scalable, reusable, simplified deployments.

We're running three sessions of the webinar:

Wed 28 Jul, 2021
Thurs 29 Jul, 2021
Mon 2 Aug, 2021

Happy deployments!

This post was originally published at octopus.com.

Using Azure Key Vault with Octopus

Mark Harrison — Wed, 23 Jun 2021 11:31:12 +0000

I recently wrote about extending the functionality of Octopus to integrate with HashiCorp Vault using step templates. Afterwards, several people asked if I plan to create step templates to integrate with other secret managers.

In this post, I walk through a new step template, Azure Key Vault - Retrieve Secrets, which is designed to retrieve secrets from an Azure Key Vault for use in your deployments or runbooks.

Introduction

This post assumes some familiarity with custom step templates and the Octopus Community Library.

In addition, this post doesn't go into great detail about Azure Key Vault concepts or how to set it up. You can learn more by reading the Azure Key Vault basic concepts guide from Microsoft.

The step template in this post retrieves secrets from an Azure Key Vault using the Az.KeyVault PowerShell module. The module must be downloaded and installed on the deployment target or worker before the step can retrieve secrets successfully. The step template has been tested on both Windows and Linux (with PowerShell Core installed).

Authentication

Before you can retrieve secrets from Azure Key Vault, you must authenticate with Azure. In their authentication concepts documentation, Microsoft note:

Authentication with Key Vault works in conjunction with Azure Active Directory (Azure AD), which is responsible for authenticating the identity of any given security principal.

In Octopus, authentication with Azure Key Vault can be achieved with an Azure Account, using a service principal.

In addition to accessing resources in Azure, your service principal may need further permissions configured to access and retrieve secrets stored in Azure Key Vault. To learn more, read the Azure Key Vault RBAC guide on how to provide access to keys, certificates, and secrets with an Azure role-based access control.

Retrieving secrets

The Azure Key Vault - Retrieve Secrets step template retrieves one or more secrets from an Azure Key Vault and creates sensitive output variables for each one retrieved.

For each secret, you can optionally choose to retrieve a specific version, and provide a custom output variable name.

Retrieving a single secret requires:

An Azure account with permission to access the secret.
The name of the Azure Key Vault to retrieve the secret from.
The name of the secret to retrieve.

An advanced feature of the step template offers support for retrieving multiple secrets at once. This requires entering each secret on a new line.

For each secret retrieved, a sensitive output variable is created for use in subsequent steps. By default, only a count of the number of variables created will be shown in the task log. To see the names of the variables in the task log, change the Print output variable names parameter to True.

Step template parameters

The step template uses the following parameters:

Azure Account: An Azure account with permissions to retrieve secrets from the Azure Key Vault.
Vault Name: The name of the Azure Key Vault to retrieve secrets from.
Vault Secrets to retrieve: Specify the names of the Secrets to be returned from Azure Key Vault, in the format: SecretName SecretVersion | OutputVariableName where:
- SecretName is the name of the Secret to retrieve.
- SecretVersion is the optional version of the Secret to retrieve. If this value isn't specified, the latest version will be retrieved.
- OutputVariableName is the optional Octopus output variable name to store the secret's value in. If this value isn't specified, an output name will be generated dynamically.
Note: Multiple fields can be retrieved by entering each one on a new line.
Print output variable names: Write out the Octopus output variable names to the task log. Default: False.
Az PowerShell Module version (optional): If you wish to use a specific version of the Az PowerShell module (rather than the default), enter the version number here. e.g. 5.9.0.

Note: The version specified must exist on the machine.

Az PowerShell Install Location (optional): If you wish to provide a custom path to the Az PowerShell module (rather than the default), enter the value here.

Note: The Module must exist at the specified location on the machine. This step template will not download the Module.

Using the step

The Azure Key Vault - Retrieve Secrets step is added to deployment and runbook processes in the same way as other steps.

After you've added the step to your process, fill out the parameters in the step:

After you've filled in the parameters, you can execute the step in a runbook or deployment process. On successful execution, any matching secrets will be stored as sensitive output variables. If you've configured your step to print the variable names, they'll appear in the task log:

In subsequent steps, output variables created from matching secrets can be used in your deployment or runbook.

Tip: Remember to replace Azure Key Vault - Retrieve Secrets with the name of your step for any output variable names.

Conclusion

The step template covered in this post demonstrates that it's easy to integrate with Azure Key Vault, and make use of secrets stored there with your Octopus deployments or runbooks.

Happy deployments!

This post was originally published at octopus.com.

Using HashiCorp Vault with Octopus Deploy

Mark Harrison — Mon, 24 May 2021 10:19:57 +0000

Storing sensitive values in Octopus Deploy solves many problems. If your organization has standardized on a secrets manager though, that might mean storing sensitive values twice, making secrets management more complicated.

Octopus has supported the concept of sensitive variables since Octopus 2.0, but customers often ask about support for secret managers. One in particular is HashiCorp Vault.

In this post, I walk through a number of new HashiCorp Vault step templates designed to retrieve secrets from Vault for use in your deployments or runbooks.

In this post

Introduction
Authentication
- LDAP login step
  - LDAP login parameters
  - Using the LDAP login step
- AppRole login step
  - AppRole login parameters
  - Using the AppRole login step
- AppRole best practises
- AppRole Get Wrapped SecretID step
  - AppRole Get Wrapped SecretID parameters
  - Using the AppRole Get Wrapped SecretID step
- AppRole Unwrap SecretID step
  - AppRole Unwrap SecretID parameters
  - Using the Unwrap SecretID step
- AppRole Unwrap SecretID and Login step
  - AppRole Unwrap SecretID parameters
  - Using the Unwrap SecretID step
Retrieving secrets
- Retrieve KV (v1) secrets step
  - Retrieve KV (v1) secrets parameters
  - Using Retrieve KV (v1) secrets step
- Retrieve KV (v2) secrets step
  - Retrieve KV (v2) secrets parameters
  - Using Retrieve KV (v2) secrets step
Conclusion
Learn more

Introduction

This post assumes some familiarity with custom step templates and the Octopus Community Library. To learn more about these, you can read Ryan Rousseau's two-part series on creating your own step template and publishing it to the library.

In addition, this post doesn't go into great detail about Vault server concepts or how to configure a Vault server.

The step templates covered in this post perform both Vault authentication and secret retrieval for both versions 1 and 2 of the Key-Value (kv) Secrets Engine.

All of the step templates make use of the Vault HTTP API so there are no additional dependencies required to use them, except being able to connect to your Vault server. They've all been tested using Vault version 1.7.1 and can run on both Windows and Linux (with Powershell Core installed).

Authentication

Before interacting with Vault, you must authenticate against an auth method. Vault offers a number of different authentication options. The following step templates have been created to support Vault authentication:

LDAP login
AppRole login
AppRole Get wrapped SecretID
AppRole Unwrap SecretID
AppRole Unwrap SecretID and Login

The AppRole method is the recommended way to authenticate with Vault for servers.

Upon authentication with Vault, a token is generated that can be used in further interactions with Vault.

LDAP login step

The HashiCorp Vault - Login with LDAP step template authenticates with a Vault Server using the LDAP authentication method. This allows Vault integration without having to duplicate username or password configuration.

You might choose to authenticate using LDAP if you already have an LDAP server available and use service accounts to control access to sensitive information.

Microsoft's Active Directory supports LDAP using Active Directory Lightweight Directory Services.

After authentication, the client_token from the Vault response will be made available as a sensitive output variable named LDAPAuthToken for use in other steps.

LDAP login parameters

The step template has the following parameters:

Vault Server URL: The URL of the Vault instance you are connecting to, including the port (The default is 8200).
API version: Choose the API version to use from a drop-down list. Currently, there is only one option: v1.
LDAP Auth Login path: The path that the LDAP method is mounted at. The default is /auth/ldap.
Username: The LDAP username.
Password: The LDAP password.

Using the LDAP login step

The LDAP login step is added to deployment and runbook processes in the same way as other steps.

After you've added the step to your process, fill out the parameters in the step:

You can then execute the step in a runbook or deployment process. On successful execution, the sensitive output variable name containing the token is displayed in the task log:

In subsequent steps, the output variable #{Octopus.Action[HashiCorp Vault - Login with LDAP].Output.LDAPAuthToken} can be used to authenticate and retrieve secrets.

Tip: Remember to replace HashiCorp Vault - Login with LDAP with the name of your step for any output variable names.

AppRole login step

The HashiCorp Vault - Login with AppRole step template authenticates with a Vault Server using the AppRole authentication method. This is perfect for use with Octopus. HashiCorp themselves recommend it for machines or apps:

This auth method is oriented to automated workflows (machines and services), and is less useful for human operators.

With an AppRole, a machine can log in with:

RoleID, think of this as the username in an authentication pair.
SecretID, think of this as the password in an authentication pair.

Don't store the SecretID:
Storing the RoleID in Octopus as a sensitive variable is a good way to ensure it remains encrypted until required.

However, the same is not recommended for the SecretID.

A SecretID, just like a password is designed to expire. Storing the SecretID could also provide the capability to retrieve all secrets as both the RoleID and SecretID would be available.

We recommend you use the more secure Get wrapped SecretID and Unwrap SecretID and Login step templates, as they use one of the best practices response wrapping.

If you use the AppRole login step template, we recommend you provide the SecretID at execution time using a sensitive prompted variable.

Once authenticated, the client_token from the Vault response will be made available as a sensitive output variable named AppRoleAuthToken for use in other steps.

AppRole login parameters

The step template has the following parameters:

Vault Server URL: The URL of the Vault instance you are connecting to, including the port (The default is 8200).
API version: Choose the API version to use from a drop-down list. Currently, there is only one option: v1.
App Role Path: The path where the approle auth method is mounted.
Role ID: The RoleID of the AppRole.
Secret ID: The SecretID of the AppRole.

Using the AppRole login step

The AppRole login step is added to deployment and runbook processes in the same way as other steps.

After you've added the step to your process, fill out the parameters in the step:

You can then execute the step in a runbook or deployment process. On successful execution, the sensitive output variable name containing the token is displayed in the task log:

In subsequent steps, the output variable #{Octopus.Action[HashiCorp Vault - Login with AppRole].Output.AppRoleAuthToken} can be used to authenticate and retrieve secrets.

Tip: Remember to replace HashiCorp Vault - Login with AppRole with the name of your step for any output variable names.

AppRole best practices

The AppRole authentication method is considered a trusted-broker method. This means that the onus of trust rests in the system acting as the authentication intermediary (the broker) between the client (typically an Octopus deployment target) and Vault.

An important best practice is to avoid storing an AppRole SecretID. Instead, use response wrapping to provide a wrapping token that will provide an access mechanism to retrieve a SecretID when required. This method of obtaining a SecretID is also known as a Pull mode as it requires the SecretID to be fetched or pulled from the AppRole.

The Vault documentation contains recommended patterns when using AppRole authentication.

Here's a summary of the recommendations:

Use a secured system to act as the broker for retrieving a wrapped SecretID.
Use response wrapping to obtain a SecretID.
Limit the number of uses and Time-to-live (TTL) value for a SecretID to prevent overuse.
Avoid anti-patterns such as having the broker retrieve secrets.

Secure the broker:
Since the trust rests on the broker, we strongly recommend using the Octopus Server's built-in worker, or a highly-secured external worker to act as the broker. It would be responsible for retrieving a wrapped SecretID and passing that value to the machine (the client) that authenticates with Vault.

To support these recommended practices, three additional AppRole step templates have been created:

AppRole Get Wrapped SecretID
AppRole Unwrap SecretID
AppRole Unwrap SecretID and Login

AppRole Get Wrapped SecretID step

The HashiCorp Vault - AppRole Get Wrapped Secret ID step template generates a response-wrapped SecretID for the AppRole authentication method.

The step template authenticates with a Vault Server using a token to retrieve a wrapped SecretID. The response contains a wrapping token and other metadata such as the creation path for the token.

This value can be used to validate no malfeasance has occurred. The wrapping token can then be used to retrieve the actual SecretID value.

The token used to authenticate to retrieve a wrapped SecretID should be of limited scope and should only be allowed to retrieve wrapped SecretIDs. Consider creating a long-lived Vault token as this presents only a minor risk.

After the response has been received from the Vault server, two sensitive output variables are created for use in other steps:

WrappedToken This is the wrapped token from the response, used to retrieve the actual SecretID value.
WrappedTokenCreationPath This is the creation path for the token. It allows you to validate no malfeasance has occurred.

AppRole Get Wrapped SecretID parameters ### AppRole Get Wrapped SecretID step

The step template uses the following parameters:

Vault Server URL: The URL of the Vault instance you are connecting to, including the port (The default is 8200).
API version: Choose the API version from a drop-down list. Currently, there is only one option: v1.
App Role Path: The path where the AppRole auth method is mounted.
Role Name: The role name of the AppRole.
Time-to-live (TTL): The TTL in seconds of the response-wrapping token itself. The default is: 120s
Auth Token: The token used to authenticate with Vault to generate a response-wrapped SecretID.

Using the AppRole Get Wrapped SecretID step

The Get Wrapped SecretID step is added to deployment and runbook processes in the same way as other steps.

After you've added the step to your process, fill out the parameters in the step:

You can then execute the step in a runbook or deployment process. On successful execution, the sensitive output variable names are displayed in the task log:

In subsequent steps, the output variables can be used to validate and retrieve the actual SecretID value:

#{Octopus.Action[HashiCorp Vault - AppRole Get Wrapped Secret ID].Output.WrappedToken}
#{Octopus.Action[HashiCorp Vault - AppRole Get Wrapped Secret ID].Output.WrappedTokenCreationPath}

Tip: Remember to replace HashiCorp Vault - AppRole Get Wrapped Secret ID with the name of your step for any output variable names.

AppRole Unwrap SecretID step

The HashiCorp Vault - AppRole Unwrap Secret ID step template retrieves and unwraps a SecretID for an AppRole using a wrapping token.

The secret_id from the Vault response will be made available as a sensitive output variable named UnwrappedSecretID for use in other steps.

AppRole Unwrap SecretID parameters

The step template uses the following parameters:

Vault Server URL: The URL of the Vault instance you are connecting to, including the port (The default is 8200).
API version: Choose the API version to use from a drop-down list. Currently, there is only one option: v1.
Wrapped Token: The wrapping token used to retrieve the actual Secret ID from Vault.
Token Creation Path: Optional The creation path for the wrapped token. If this value is provided, the step template will perform a wrapping lookup to validate no malfeasance has occurred.

Using the Unwrap SecretID step

The Unwrap SecretID step is added to deployment and runbook processes in the same way as other steps.

After you've added the step to your process, fill out the parameters in the step:

Note the use of sensitive output variables in the step parameters. In this example, the values are created using the Get Wrapped SecretID step template named HashiCorp Vault - AppRole Get Wrapped Secret ID.

You can then execute the step in a runbook or deployment process. On successful execution, the sensitive output variable names are displayed in the task log:

In subsequent steps, the output variable #{Octopus.Action[HashiCorp Vault - AppRole Unwrap Secret ID].Output.UnwrappedSecretID} can be used to authenticate with Vault and receive a token that can then be used to retrieve secrets.

Tip: Remember to replace HashiCorp Vault - AppRole Unwrap Secret ID with the name of your step for any output variable names.

AppRole Unwrap SecretID and Login step

The HashiCorp Vault - AppRole Unwrap Secret ID and Login step template is provided as a convenient way to combine two step templates used with Vault into one:

AppRole Unwrap SecretID: It retrieves and unwraps a SecretID for an AppRole using a wrapping token.
AppRole login: It authenticates with Vault using a supplied RoleID and the unwrapped SecretID.

It's designed as the second part of a two-step workflow with Vault:

Get a wrapped SecretID using the AppRole Get wrapped SecretID step template.
Provide the wrapped SecretID stored in a sensitive output variable from the first step to this step template to unwrap and authenticate.

Once authenticated, the client_token from the Vault response will be made available as a sensitive output variable named AppRoleAuthToken for use in other steps.

AppRole Unwrap SecretID and Login parameters

The step template uses the following parameters:

Vault Server URL: The URL of the Vault instance you are connecting to, including the port (The default is 8200).
API version: Choose the API version to use from a drop-down list. Currently, there is only one option: v1.
App Role Path: The path where the AppRole auth method is mounted.
Role ID: The RoleID of the AppRole.
Wrapped Token: The wrapping token used to retrieve the actual Secret ID from Vault.
Token Creation Path: Optional The creation path for the wrapped token. If this value is provided, the step template will perform a wrapping lookup to validate no malfeasance has occurred.

Using the Unwrap SecretID and Login step

The Unwrap SecretID and Login step is added to deployment and runbook processes in the same way as other steps.

After you've added the step to your process, fill out the parameters in the step:

Note the use of sensitive output variables in the step parameters. In this example, the values were created using the Get Wrapped SecretID step template named HashiCorp Vault - AppRole Get Wrapped Secret ID.

You can then execute the step in a runbook or deployment process. On successful execution, the sensitive output variable names are displayed in the task log:

In subsequent steps, the output variable #{Octopus.Action[HashiCorp Vault - AppRole Unwrap Secret ID and Login].Output.AppRoleAuthToken} can be used to authenticate, and retrieve secrets.

Tip: Remember to replace HashiCorp Vault - AppRole Unwrap Secret ID and Login with the name of your step for any output variable names.

Retrieving secrets

After you've authenticated with Vault, you receive an authentication token that can be used to retrieve secrets. Secrets in Vault are stored in a secrets engine, of which there are many different types.

The step templates created to support retrieving secrets focus on the Key-Value (kv) Secrets Engine as it's a generic Key-Value store used to store arbitrary secrets:

Retrieve KV (v1) secrets step
Retrieve KV (v2) secrets step

Retrieve KV (v1) secrets step

The HashiCorp Vault - Key Value (v1) retrieve secrets step template retrieves one or more secrets stored in a v1 Key-Value secrets engine.

Retrieving a single secret requires:

The path to the secret.
An authentication token with permission to access the secret.
Optionally, a list of field names to retrieve.

An advanced feature of the step template offers support for retrieving multiple secrets at once. This requires changing the Secrets retrieval method parameter to Multiple vault keys.

It's also possible to recursively retrieve secrets. This is useful when you want to retrieve all secrets for a given path.

Retrieve KV (v1) secrets parameters

The step template uses the following parameters:

Vault Server URL: The URL of the Vault instance you are connecting to, including the port (The default is 8200).
API version: Choose the API version to use from a drop-down list. Currently, there is only one option: v1.
Auth Token: The token used to authenticate to retrieve secrets.
Secrets Path: The full path to the secret(s) you want to retrieve. The value should include both the path where the secrets engine is mounted, as well as the path to the secret itself.
Secrets retrieval method: Choose between retrieving a single secret or multiple secrets. Retrieving a single secret is the equivalent of a vault kv get command using the Get method. Retrieving multiple secrets is the equivalent of the combination of both a vault kv list command using the List method and then subsequent vault kv get commands for each secret.
Recursive retrieval: If multiple secrets are being retrieved, should any sub-folders also be enumerated and retrieved? The default is: False.
Field names: Choose specific fields to be retrieved from identified secrets. This is useful when you only want to retrieve specific fields from one or more secrets. You can optionally include a name for the output variable.
Print output variable names: Write out the Octopus output variable names to the task log. The default is: False.

Using Retrieve KV (v1) secrets step

The Key Value (v1) retrieve secrets step is added to deployment and runbook processes in the same way as other steps.

After you've added the step to your process, fill out the parameters in the step:

Note the use of the sensitive output variable in the Auth Token parameter. In this example, the value was created using the Unwrap SecretID and Login step template named HashiCorp Vault - AppRole Unwrap Secret ID and Login.

In subsequent steps, output variables created from matching secrets can be used in your deployment or runbook.

Tip: Remember to replace HashiCorp Vault - Key Value (v1) retrieve secrets with the name of your step for any output variable names.

Retrieve KV (v2) secrets step

The HashiCorp Vault - Key Value (v2) retrieve secrets step template retrieves one or more secrets stored in a v2 Key-Value secrets engine.

One of the key advantages of the v2 Key-Value secrets engine is its support for versioned secrets. This is useful if you need to roll back secrets in the event of unintentional data loss.

Retrieving a single secret requires:

The path to the secret,
An authentication token with permission to access the secret.
Optionally, a list of field names to retrieve.

This step template offers advanced features:

Support for retrieving multiple secrets at once. This requires changing the Secrets retrieval method parameter to Multiple vault keys. It's also possible to recursively retrieve secrets. This is useful to retrieve all secrets for a given path.
Support for retrieving a specific version of a secret. This is only supported when retrieving a single secret.

For each secret retrieved, a sensitive output variable is created for use in subsequent steps. By default, only a count of the number of variables created will be shown in the task log. To see the names of the variables in the Task log, change the Print output variable names parameter to True.

Retrieve KV (v2) secrets parameters

The step template uses the following parameters:

Vault Server URL: The URL of the Vault instance you are connecting to, including the port (The default is 8200).
API version: Choose the API version to use from a drop-down list. Currently, there is only one option: v1.
Auth Token: The token used to authenticate to retrieve secrets.
Secrets Path: The full path to the secret(s) you want to retrieve. The value should include both the path where the secrets engine is mounted, as well as the path to the secret itself.
Secrets retrieval method: Choose between retrieving a single secret or multiple secrets. Retrieving a single secret is the equivalent of a vault kv get command using the Get method. Retrieving multiple secrets is the equivalent of the combination of both a vault kv list command using the List method and then subsequent vault kv get commands for each secret.
Recursive retrieval: If multiple secrets are being retrieved, should any sub-folders also be enumerated and retrieved? The default is: False.
Secret Version: Optional When retrieving a single secret, choose the version of the secret to retrieve. For example, if you want version 2 of all field values in a secret, enter the value 2.
Field names: Choose specific fields to be retrieved from identified secrets. This is useful when you only want to retrieve specific fields from one or more secret(s). You can optionally include a name for the output variable.
Print output variable names: Write out the Octopus output variable names to the task log. The default is: False.

Using Retrieve KV (v2) secrets step

The Key Value (v2) retrieve secrets step is added to deployment and runbook processes in the same way as other steps.

After you've added the step to your process, fill out the parameters in the step:

Note the use of the sensitive output variable in the Auth Token parameter. In this example, the value was created using the Unwrap SecretID and Login step template named HashiCorp Vault - AppRole Unwrap Secret ID and Login.

In subsequent steps, the output variables created from matching secrets can be used in your deployment or runbook.

Tip: Remember to replace HashiCorp Vault - Key Value (v2) retrieve secrets with the name of your step for any output variable names.

Conclusion

The templates covered in this post show how it's possible to extend the functionality of Octopus and retrieve secrets from Vault, or any other secrets manager, and use them in your deployments or runbooks.

Happy deployments!

Learn more

For further information, you can read:

The AppRole Pull Authentication tutorial showing how to retrieve SecretIDs securely.
HashiCorp Vault documentation for the K/V v1 Secrets Engine.
HashiCorp Vault documentation for the K/V v2 Secrets Engine.

This post was originally published at octopus.com.

Ask Octopus - How do I set the master key for my Octopus container?

Ryan Rousseau — Fri, 22 Jan 2021 17:27:05 +0000

You have Octopus running in a container. Now you need to upgrade or recreate the container. How do you reconnect to the existing database? Find out in this Ask Octopus!

Other resources:

https://help.octopus.com/t/how-do-i-set-the-master-key-in-for-octopus-running-in-a-container/26134

Do you have a question about how to use Octopus? Email advice@octopus.com and someone from our team will get you an answer. You might even see it in a future episode.

Join our community slack at https://octopus.com/slack

Ask Octopus - Can I host the Octopus Server in Docker?

Ryan Rousseau — Mon, 07 Dec 2020 19:12:05 +0000

One of our recent frequently asked questions has been about hosting an Octopus Server in Docker? Can you do it? What do you need to know? Find out in this Ask Octopus!

Other resources:

https://octopus.com/docs/installation/octopus-in-container

Do you have a question about how to use Octopus? Email advice@octopus.com and someone from our team will get you an answer. You might even see it in a future episode.

Join our community slack at https://octopus.com/slack

Ask Octopus - How can I set up Disaster Recovery?

Ryan Rousseau — Tue, 01 Dec 2020 14:38:03 +0000

Wondering what your Disaster Recovery options are for your Octopus Server? Find out in this Ask Octopus!

Other resources:

https://help.octopus.com/t/how-do-i-set-up-octopus-deploy-for-disaster-recovery/24081/2

Do you have a question about how to use Octopus? Email advice@octopus.com and someone from our team will get you an answer. You might even see it in a future episode.

Join our community slack at https://octopus.com/slack

Ask Octopus - Can I turn off health checks?

Ryan Rousseau — Mon, 23 Nov 2020 19:20:15 +0000

You don't need health checks to run very often or at all. Can you disable them completely? Find out in this Ask Octopus!

Other resources:

Do you have a question about how to use Octopus? Email advice@octopus.com and someone from our team will get you an answer. You might even see it in a future episode.

Join our community slack at https://octopus.com/slack

Ask Octopus - Can I have a test Octopus Server?

Ryan Rousseau — Fri, 13 Nov 2020 18:44:21 +0000

You need another Octopus Server instance to test upgrades. Do you need another license? Find out in this Ask Octopus!

Other resources:

Do you have a question about how to use Octopus? Email advice@octopus.com and someone from our team will get you an answer. You might even see it in a future episode.

Join our community slack at https://octopus.com/slack

Ask Octopus - Can I deploy database changes?

Ryan Rousseau — Fri, 06 Nov 2020 17:35:43 +0000

You're interested in automating your database deployments but not sure where to start? Check out the resources available in this Ask Octopus!

Other resources:

Do you have a question about how to use Octopus? Email advice@octopus.com and someone from our team will get you an answer. You might even see it in a future episode.

Join our community slack at https://octopus.com/slack

From Sysadmin to SRE

Josh Duffney — Fri, 06 Nov 2020 14:07:32 +0000

DevOps is everywhere! It’s used as hashtags, the name of products, and job titles. Digital Trends lists DevOps Engineer as the third-best tech job title for 2020. Indeed.com has over 4,000 job postings for the title DevOps engineer, and LinkedIn has over 3 times that number with more than 13,000 job postings.

Needless to say, DevOps is hot!

The job title Site Reliability Engineer(SRE) has emerged in recent years from Google. SRE isn’t yet as popular as its predecessor, but it is often used synonymously by recruiters.

In this post, I share my thoughts on how somebody new to the industry can aspire to these roles.

Looking at job postings will likely just confuse you. Everything from 10 years of C# experience to intimate knowledge of Active Directory domain trusts are listed in the job descriptions. With such a broad set of requirements, where do you even begin?

Start by understanding that DevOps and SRE are advanced job roles. You can start a career in web development by applying for entry-level positions and system administration at the help desk. DevOps and SRE don’t yet have that entry position. Typically a DevOps practitioner has already spent a few years working in technology. This is because DevOps and SRE roles require an understanding of development and infrastructure. It’s difficult to learn both disciplines at once, which is why most practitioners have an existing background in one or the other.

I started my career on the help desk. Moved up to a system administrator and eventually landed a gig as a senior systems engineer. When I first started hearing about DevOps through Twitter, blog posts, and conference talks, I thought it was reserved for startups. So I ignored it until my passion for automation brought me to the DevOps practice of Infrastructure as Code. It was episode 275 of the PowerScripting podcast with guest Steve Murawski where I first heard about DSC (Desired State Configuration). Infrastructure as Code was the single idea that pulled me into the world of DevOps. It’s the portal sysadmins of all operating system variants can use to transition into DevOps.

It isn’t about tools, but...

DevOps isn’t about tools! This is practically shouted at conferences by people seemingly wearing Braveheart face paint and drumming battle drums. They’re arguably correct in their assertion. But the tools you use shape the language you use and determine how you interact with other people to do your job. You literally live in the tools you use. Whether that’s email, slack, or like most developers, Git.

In a traditional company, you’ll find developers using one set of tooling and sysadmins using another. This only reinforces the silos.

If you ever hope to break down those silos you need to do more than change titles, form joint distribution groups, and tell people to work together. You need to give each team opportunities to interact. This happens naturally and organically if the tools they use to do their jobs overlap.

Sharing tools creates a shared language. This isn’t senior leadership advice, this applies to individual contributors. If you’re a sysadmin looking to move into a DevOps role this applies to you. DevOps isn’t about tools, but tools are a good place to start.

Learn to code from the command-line

Automation is the starting point. It is the starting point because it gets you writing code. If you look over the technologies listed in DevOps Engineer job postings, you’ll notice a trend. All of them mention some form of coding. This does not mean you need to attend a coding Bootcamp, learn Java or C#. Any scripting or programming language will suffice. What’s important is that you pick a language that’s practical for you to apply to your job. If you work on Linux, Bash and Python are a natural fit. If you work on Windows, PowerShell is the way to go.

Starting to learn a coding language is easy. There’s an endless number of blog posts, YouTube videos, books, and Pluralsight courses for you to consume.

What’s difficult is figuring out how to apply it at work. I suggest you start by automating things that suck. Find routine and mundane tasks and work to automate those. The scripting or programming language you choose to learn will become the hammer in your toolbox. As with any tool collection, a hammer is just the beginning. You’ll need to add other tools. Within the context of infrastructure automation, those tools go by the names of Ansible, Chef, Puppet, Terraform, Azure Resource Manager Templates, and Cloudformation. These tools are valuable because they abstract. Leaving you with less code to write and a better framework for managing your infrastructure. If Infrastructure as Code is the portal, automation is how you open it.

Start at the source

Long gone are the days of relying on shell history and file shares to store your code. You need something better. You need source control. Source control is a broad topic that has entire books dedicated to it. The good news is that you don’t need to understand it in depth. You just need enough information to become competent. Learning just a few commands is all you need to hit the ground running.

Git is by far the most popular source control system. And GitHub is by far the most popular hosted Git provider. Creating an account on GitHub and uploading your code there is a great place to start. GitHub allows you to create public and private repositories. If you choose to upload to a public repository, make sure your code is sanitized and can be open sourced.

Creating public repositories also serves as public artifacts of your work. You can use them to demonstrate your skills and knowledge, and you can also use them as a reference. Not everything you learn will stay in your head, but by using Git it will be logged in your commit history.

Learning Git will take some effort, but once you learn how to use it, you won’t go back. In fact, if you start using Git outside of work, you won’t want to work without it. Chances are someone within your organization is using Git or another source control system. Ask around and find out how you can gain access to the source control system, then create a repository for your team and help onboard them. Onboarding your team will be more difficult than getting your boss on board. There will be a learning curve for your team, but your manager will agree after you say the word audit.

All the advantages of Git are not immediately apparent. Focus on the value it adds upfront vs. in the future. That immediate value is auditable code through a commit history, increased collaboration, and easier management than a file share. Most importantly though, source control unlocks everything else.

Pull requests mean deployments

Tooling that was traditionally administered by release engineers has become table stakes for anyone writing code.

Yes, scripts are code too.

Release engineering is a sub-discipline of software engineering which focuses on the compilation, assembly, and delivery of source code. There is a lot you don’t need to know about release engineering, but two types of tools exist that you won’t want to live without; continuous integration and continuous delivery collectively known as CI/CD.

TeamCity, Octopus Deploy, Jenkins, Azure DevOps, and GitHub Actions are all tools that live in this space. Focusing on tools is going to confuse you because each of these tools contain features that allow you to build CI/CD systems. But, continuous integration and continuous delivery are actually software engineering practices.

To simplify, think of continuous integration as automating the build phase of software, and continuous delivery as automating the release and deployment of that software. Continuous integration, continuous delivery, and source control combine to make a release pipeline. A release pipeline is a conceptual process that takes your code from source to production for you. It is through source control and CI/CD that you can get out of the business of clicking buttons.

Scenario: You’ve automated the deployment, provisioning, and configuration of new infrastructure, and all the infrastructure code is stored in source control. You are at the point where you can scale the infrastructure up and down using this automation. However, it has become difficult to identify which version of the code was last deployed, deployments are done manually at the command-line, and typos frequently make it into the codebase.

Solution: Build a release pipeline for infrastructure code. Automate all the steps you take to deploy the infrastructure code by making it programmatic and non-interactive. Then you can start to build out the release pipeline. Starting with source control, the release pipeline will be triggered by commits and or pull requests. Next, in the build stage lint your code to reduce typos and control code quality. After your code passes the lint and other testing you deploy it in the release stage, which deploys the infrastructure code. Implementing a release pipeline frees you from the manual deployment of automation. It’s automating the automation in a sense. The goal is always to click fewer buttons.

Conclusion

Everything points back to automation, it’s what makes all of this possible. Without automation the portal would not open. Entering the portal, source control is your stronghold. With a stronghold in place, release pipelines carry you through the fog into the uncharted territory of the developer. Learning the technology and practices listed in this article will open many possible paths on your DevOps skill tree. Where you go next, will be up to you.

Resources:

Josh Duffney is a Site Reliability Engineer. He writes, presents, teaches, and tweets about Automation, DevOps, Cloud, and optimizing output while minimizing input.

Ask Octopus - How do I ignore messages written to stderr?

Ryan Rousseau — Fri, 30 Oct 2020 17:16:51 +0000

A tool that you're using in your deployments is writing informational logs to stderr, but you don't want the messages to show up as errors. What do you do? Find out in this Ask Octopus!

Do you have a question about how to use Octopus? Email advice@octopus.com and someone from our team will get you an answer. You might even see it in a future episode.

Join our community slack at https://octopus.com/slack