DEV Community: Igor Zhivilo

Scheduled backup of Vault secrets with Jenkins on Kubernetes

Igor Zhivilo — Tue, 14 Sep 2021 12:11:32 +0000

I am a DevOps engineer at Cloudify.co and I will share in this post my experience related to automation of Vault backup creation using Jenkins scheduled job and simple python script which I built to create dump of vault secrets.

Let's start.

What is HashiCorp's Vault?

Vault is a tool for securely accessing secrets. A secret is anything that you want to tightly control access to, such as API keys, passwords, certificates, and more. Vault provides a unified interface to any secret while providing tight access control and recording a detailed audit log.
https://www.vaultproject.io/

Prerequisites:

Vault Installed
Jenkins Installed

My Setup

EKS Kubernetes cluster
Vault runs on EKS cluster
Jenkins runs on EKS cluster

You can read in this tutorial how to run Jenkins on EKS cluster:
https://igorzhivilo.com/jenkins/ci-cd-future-k8s-jenkins

What you will learn from this post?

How to use python hvac library for authentication with Vault programmatically and backup vault secrets.
What is AppRole authentication mechanism in Vault and how to enable/create it.
How to create scheduled backup for Vault secrets with Jenkins pipeline on k8s.

AppRole authentication method in Vault

How can an application programmatically request a token so that it can read secrets from Vault?

Using the AppRole which is an authentication mechanism within Vault to allow machines or apps to acquire a token to interact with Vault and using the policies you can set access limitations for your app.

It uses RoleID and SecretID for login.

Create AppRole and policy for Jenkins

I explained how to do it in detail in my blog post: https://igorzhivilo.com/jenkins/how-to-read-vault-secrets-from-declarative-pipeline

After you applied everything I wrote in this post:

enabled approle in vault
v2 kv secrets engine enabled
applied all needed policies

eventually you will get role_id and secret_id which will be used programmatically with 'python hvac'.

Another way for 3rd step (apply all needed policies) is to create policy using Vault's UI:

$ kubectl port-forward -n vault svc/vault 8200
Forwarding from 127.0.0.1:8200 -> 8200
Forwarding from [::1]:8200 -> 8200

Go to policy tab -> Create ACL Policy

path "sys/auth/approle" {
  capabilities = [ "create", "read", "update", "delete", "sudo" ]
}
path "sys/auth/approle/*" {
  capabilities = [ "create", "read", "update", "delete" ]
}
path "auth/approle/*" {
  capabilities = [ "create", "read", "update", "delete", "list" ]
}
path "sys/policies/acl/*" {
  capabilities = [ "create", "read", "update", "delete", "list" ]
}
path "secret/data/jenkins/*" {
  capabilities = [ "create", "read", "update", "delete", "list" ]
}
path "secret/metadata/jenkins/*" {
  capabilities = [ "create", "read", "update", "delete", "list" ]
}

and then run via vault CLI:

$ vault write auth/approle/role/jenkins token_policies=jenkins \
 token_ttl=1h token_max_ttl=4h

# Get RoleID and SecretID
$ vault read auth/approle/role/jenkins/role-id
$ vault write -f auth/approle/role/jenkins/secret-id

Test that you created correctly role_id/secret_id

$ vault write auth/approle/login \
    role_id=ROLE_ID \
    secret_id=SECRET_ID

Testing authentication with vault using python hvac and appRole

Simple python script to test auth with vault:

import hvac

VAULT_URL = 'http://vault.vault.svc.cluster.local:8200'
client = hvac.Client(url=VAULT_URL)
client.auth.approle.login(
    role_id = self.role_id,
    secret_id = self.secret_id
)

assert client.is_authenticated()

first run 'pip install hvac'.

I am running this script from pod with python container inside of my Kubernetes cluster.

URL of vault in k8s cluster: 'http://vault.vault.svc.cluster.local:8200'

You will see authentication error if authentication is failed, if you do, make sure you applied all the needed policies, enabled applrole, and generated properly role_id / secret_id.

Validate role_id/secret_id is correct using vault CLI:

$ vault write auth/approle/login \
    role_id=YOU_ROLE_ID \
    secret_id=YOU_SECRET_ID

Get the list of secrets under 'jenkins' vault_prefix (CLI)

In my case, vault_prefix looks like: 'secret/data/jenkins' and all secrets stored under 'jenkins' prefix:

$ vault kv list secret/jenkins

Keys
----
aws
git
web-app1
web-app2
...

Each key in list has additional subset of keys, for example 'aws' has access_key_id/secret_access_keys

Getting the secrets list (python)

secrets_list_response = client.secrets.kv.v2.list_secrets(path = 'jenkins')

print('The following keys are available under "jenkins" prefix: {keys}'.
  format(keys=','.join(secrets_list_response['data']['keys'])))

If you have a permissions error on the secrets list, check you have access to metadata, that what you should see in UI for 'jenkins policy':

path "secret/metadata/jenkins/*" {
  capabilities = [ "create", "read", "update", "delete", "list" ]
}

If you don't, add using the UI or vault CLI:

$ tee jenkins-policy-metadata.hcl <<"EOF"
  path "secret/metadata/jenkins/*" {
    capabilities = [ "read" ]
  }
  EOF

$ vault policy write jenkins jenkins-policy-metadata.hcl

Get a specific secret (python)

secret_response = client.secrets.kv.v2.read_secret(path = 'jenkins/aws')

print(secret_response)

If you have a permission error, check you have access to data in UI of Vault:

path "secret/data/jenkins/*" {
  capabilities = [ "create", "read", "update", "delete", "list" ]
}

VaultHandler

I created VaultHandler which you can find here.

https://github.com/warolv/vault-backup

You can use it to:

Dump all your secrets as encrypted yaml/json files, or you can store it without encryption.
Get a list of your secrets.
Print all secrets nicely.
Populate Vault from yaml/json dumps to a specific 'vault_prefix'.

Also, I think to extend it to use different auth methods, besides appRole, create CLI, to run it in the command line and much more :-)

If the idea sounds interesting, add stars to the repo or clone it, I will know this way you like the idea.

Create Jenkins scheduled job for daily vault backup

I am using Vault Plugin in Jenkins https://plugins.jenkins.io/hashicorp-vault-plugin to add secrets as env variables during job execution. Read more about how to integrate this plugin into jenkins here: https://igorzhivilo.com/jenkins/how-to-read-vault-secrets-from-declarative-pipeline
During job execution POD will be created with 2 contaienrs: awscli to use aws s3 utility, and push created encrypted dump to private s3 bucket (vault-backups), python to run VaultHandler.*

def configuration = [vaultUrl: "${VAULT_URL}",  vaultCredentialId: "vault-role-app", engineVersion: 2]

def secrets = [
  [path: 'secret/jenkins/aws', engineVersion: 2, secretValues: [
    [envVar: 'AWS_ACCESS_KEY_ID', vaultKey: 'aws_access_key_id'],
    [envVar: 'AWS_SECRET_ACCESS_KEY', vaultKey: 'aws_secret_access_key']]],
  [path: 'secret/jenkins/vault-backup', engineVersion: 2, secretValues: [
    [envVar: 'VAULT_ADDR', vaultKey: 'vault_url'],
    [envVar: 'ROLE_ID', vaultKey: 'role_id'],
    [envVar: 'SECRET_ID', vaultKey: 'secret_id'],
    [envVar: 'VAULT_PREFIX', vaultKey: 'vault_prefix'],
    [envVar: 'DUMP_ENCRYPTION_PASSWORD', vaultKey: 'encryption_password']]],
]

def podTemplate = """
                apiVersion: v1
                kind: Pod
                spec:
                  containers:
                    - name: awscli
                      image: amazon/aws-cli
                      command:
                      - cat
                      tty: true
                    - name: python
                      image: python:3.6
                      command:
                      - cat
                      tty: true
                """.stripIndent().trim()

pipeline {
  agent {
    kubernetes {
      defaultContainer 'jnlp'
      yaml "${podTemplate}"
    }
  }

  environment {
    AWS_DEFAULT_REGION = "eu-west-1"
  }

  stages {
    stage('Backup Jenkins'){
      steps {
        container('python'){
          dir("${env.WORKSPACE}/pipelines-k8s/vault-backup/") {
            withVault([configuration: configuration, vaultSecrets: secrets]){
              sh """#!/bin/bash
                pip install -r requirements.txt
                python -u vault_handler.py
                tar -zcvf vault_secrets.json.enc.tar.gz vault_secrets.json.enc
              """
            }
          }
        }
        container('awscli'){
          dir("${env.WORKSPACE}/pipelines-k8s/vault-backup/") {
            withVault([configuration: configuration, vaultSecrets: secrets]){
              sh '''
                aws s3 cp vault_secrets.json.enc.tar.gz s3://vault-backups/$(date +%Y%m%d%H%M)/vault_secrets.json.enc.tar.gz
              '''
            }
          }
        }
      }
    }
  }
}

Now create a new pipeline in jenkins: newitem -> pipeline and make it periodic (daily).

In this post, I described how to automate Vault backup creation using Jenkins scheduled job and simple python script which I built to create dump of vault secrets.

Thank you for reading, I hope you enjoyed it, see you in the next post.

If you want to be notified when the next post of this tutorial is published, please follow me on Twitter @warolv.

Original story on my blog: https://igorzhivilo.com/vault/scheduled-backup-vault-secrets/

Scheduled backup of Vault secrets with CronJob of Kubernetes

Igor Zhivilo — Tue, 14 Sep 2021 12:01:21 +0000

I am a DevOps engineer at Cloudify.co and I will share in this post my experience related to automation of Vault backup creation using Kubernetes CronJob.

This post is a continuation of the previous post: https://igorzhivilo.com/vault/scheduled-backup-vault-secrets

The repository with all the code: https://github.com/warolv/vault-backup

What is HashiCorp's Vault?

https://www.vaultproject.io/

My Setup

EKS Kubernetes cluster
Vault runs on EKS cluster

What you will learn from this post?

How to create a scheduled backup for Vault secrets with CronJob of Kubernetes.
How to add Prometheus alerts for failed jobs.

You can find all the code presented in my repository: https://github.com/warolv/vault-backup

Let's start.

Building the docker container

First, need to build a docker container based on python3 and include the code of vault_handler.py

Need clone the repo first with Docker file: 'git clone https://github.com/warolv/vault-backup'

Docker file:

FROM python:3

COPY requirements.txt /

RUN pip install -r requirements.txt

COPY vault_handler.py /

CMD [ "python", "./vault_handler.py" ]

Building image

# login to dockerhub
$ docker login -u YOUR_USERNAME -p YOUR_PASSWORD

# Build Docker
$ docker build -t vault-backup .

Validate docker container working properly

$ docker run --name test-vault-backup --rm vault-backup
Specify one of the commands below
print
print-dump
dump
populate

It's working, we got a list of commands from vault-backup:-)

Pushing vault-backup docker container to docker hub

$ docker tag vault-backup <Your Docker ID>/vault-backup:latest
$ docker push <Your Docker ID>/vault-backup:latest

In my case, it's 'warolv/vault-backup:latest', you can find an already built image there.

CronJob to run vault-backup on a daily basis

https://github.com/warolv/vault-backup/blob/main/examples/cronjob/cronjob.yaml

apiVersion: batch/v1beta1
kind: CronJob
metadata:
  name: vault-backup
spec:
  schedule: "0 1 * * *"
  jobTemplate:
    spec:
      template:
        spec:
          restartPolicy: Never
          nodeSelector:
            instance-type: spot
          containers:
            - name: awscli
              image: amazon/aws-cli:latest
              command:
                - "aws"
                - "s3"
                - "cp"
                - "/data/vault_secrets.enc"
                - "s3://jenkins-backups/vault_secrets.enc"
              imagePullPolicy: Always
              envFrom:
                - secretRef:
                    name: aws-creds-secret
              volumeMounts:
              - name: backup-dir
                mountPath: /data
          initContainers:
            - name: vault-backup
              image: warolv/vault-backup
              command: 
                - "python3"
                - "vault_handler.py"
                - "dump"
                - "-dp"
                - "/data/vault_secrets.enc"
              imagePullPolicy: Always
              envFrom:
                - secretRef:
                    name: vault-backup-secret
              volumeMounts:
              - name: backup-dir
                mountPath: /data
          volumes:
          - name: backup-dir
            emptyDir: {}

Explanation

First 'vault_backup.py' script will run from InitContainer and secrets dump will be created (vault_secrets.enc) and saved to /data folder which is a shared folder for both containers.
The second will run 'awscli' container which will be used to push the secrets dump to a private S3 bucket (AWS CLI is used to copy the secrets dump to the privare S3 bucket). Of course, S3 private bucket must exist.
Credentials for AWS CLI (AWS_ACCESS_KEY_ID/AWS_SECRET_ACCESS_KEY) and for vault_backup script exported to the environment as k8s secrets.
In this example, I am copying the dump to 's3://jenkins-backups/vault_secrets.enc', in the production use case I suggest adding a timestamp to dump of secrets to be something like vault_secrets_${timestamp}.enc

Creating secrets for CronJob

# create k8s secret for AWS
$ kubectl create secret generic aws-creds-secret \
--from-literal=AWS_ACCESS_KEY_ID=YOUR_AWS_ACCESS_KEY_ID \
--from-literal=AWS_SECRET_ACCESS_KEY=YOUR_AWS_SECRET_ACCESS_KEY

# create k8s secret with all needed data for vault-backup
$ kubectl create secret generic vault-backup-secret \
--from-literal=VAULT_ADDR=http://vault.vault.svc.cluster.local:8200 \
--from-literal=ROLE_ID=YOUR_ROLE_ID \
--from-literal=SECRET_ID=YOUR_SECRET_ID \
--from-literal=ENCRYPTION_KEY=ENCRYPTION_KEY \
--from-literal=VAULT_PREFIX=jenkins

It's only an example, you need to put real values.

Deploy vault-backup cronjob

$ kubectl apply -f examples/cronjob/cronjob.yaml

How to trigger a Job from CronJob?

In case you want to test your job is working properly:

$ kubectl create job --from=cronjob/vault-backup vault-backup-001

Adding alerts to Prometheus

I am using kube-state-metrics with Prometheus and we have these metrics available: https://github.com/kubernetes/kube-state-metrics/blob/master/docs/cronjob-metrics.md

Let's add an alert for 'failed job' and for cronjob which 'takes too much time', of course, it's only an example to give you an idea.

groups:
- name: cronjob.rules
  rules:
  - alert: SlowCronJob
    expr: time()-kube_cronjob_next_schedule_time > 1800
    for: 30m
    labels:
      severity: warning
    annotations:
      description: CronJob {{$labels.namespaces}}/{{$labels.cronjob}} is taking more than 30m to complete
      summary: CronJob taking more than 30m
  - alert: FailedJob
    expr: kube_job_status_failed  > 0
    for: 30m
    labels:
      severity: warning
    annotations:
      description: Job {{$labels.namespaces}}/{{$labels.job}} failed
      summary: Job failure

In this post, I described how to automate Vault backup creation using Kubernetes CronJob and a simple python script that I built.

Thank you for reading, I hope you enjoyed it, see you in the next post.

Original story on my blog: https://igorzhivilo.com/vault/scheduled-backup-vault-cronjob/

If you want to be notified when the next post of this tutorial is published, please follow me on Twitter @warolv.

Instagram: @warolv