DEV Community

Cover image for Securing access to Scaleway Elements API Keys from Gitlab CI
Onepoint x Stack Labs profile image Chabane R.
Chabane R. for Onepoint x Stack Labs

Posted on • Edited on

Securing access to Scaleway Elements API Keys from Gitlab CI

#scaleway #security #devops #gitlab

How many api keys are stored per day as variables in the Gitlab CI configuration?

When a Scaleway Elements API Key is saved in Gitlab, we face all the security issues of storing credentials outside of the cloud infrastructure: Access, authorization, key rotation, age, destruction, location, etc.

There are 2 common reasons for developers to store GCP credentials in Gitlab CI:

  • They use shared runners.
  • They use specific/group runners deployed in a Scaleway Kubernetes Kapsule cluster but do not use (or do not know about) Gitlab additional configurations.

The alternative that Gitlab CI proposes for users is to mount a secret volume to the runner pods that are created for each build.

Prerequisites

Install the following tools:

And create 2 projects in your Scaleway organization:

  • devops
  • development

Working with Kapsule

The first step is to create the Kapsule devops cluster and configuring our environment [1].

Generate the API Key for the devops project. You can create one by following the documentation How to generate an API key.

scw init
scw k8s cluster create name=kapsule-devops
Enter fullscreen mode Exit fullscreen mode

Let's create a pool for the runner jobs:

scw k8s pool create cluster-id=$(scw k8s cluster list | grep kapsule-devops | awk '{ print $1 }') name=dev node-type=GP1_XS size=2
Enter fullscreen mode Exit fullscreen mode

Add a taint

kubectl taint nodes gitlab-runner-jobs-dev-reserved=true:NoSchedule --selector=k8s.scaleway.com/pool-name=dev
Enter fullscreen mode Exit fullscreen mode
  • Configure kubectl to communicate with the cluster: 
scw k8s kubeconfig install kapsule-devops
Enter fullscreen mode Exit fullscreen mode
  • Create the namespace for the dev runner: 
kubectl create namespace dev
Enter fullscreen mode Exit fullscreen mode
  • Create the Kubernetes service account to use for specific runner: 
kubectl create serviceaccount --namespace dev app-deployer
Enter fullscreen mode Exit fullscreen mode
  • Generate an API Key for the specific runner.

Note: For easier visibility and auditing, I recommend to centrally store API keys in a dedicated project and in an external tools like Vault.

  • To allow the specific runner to impersonate the API Key we need to store the credentials in a Kubernetes secret. 
kubectl create secret generic dev-api-key --from-file ~/.config/scw/config.yaml -n dev
Enter fullscreen mode Exit fullscreen mode

Binding Kubernetes Secrets with Scaleway API Keys

Assign the API Key to the Gitlab runner

The next step is to mount the secret as a data volume [4].

  • Start by installing Helm: 
curl -fsSL -o get_helm.sh https://raw.githubusercontent.com/helm/helm/master/scripts/get-helm-3
chmod 700 get_helm.sh
./get_helm.sh
Enter fullscreen mode Exit fullscreen mode
  • Add Gitlab Helm package: 
helm repo add gitlab https://charts.gitlab.io
Enter fullscreen mode Exit fullscreen mode
  • Configure the runner:

Create the file values.yaml:

imagePullPolicy: IfNotPresent
gitlabUrl: https://gitlab.com/
unregisterRunners: true
terminationGracePeriodSeconds: 3600
concurrent: 10
checkInterval: 30
rbac:
  create: true
metrics:
  enabled: true
runners:
  image: ubuntu:18.04
  config: |
   [[runners]]
     [runners.kubernetes]
        [[runners.kubernetes.volumes.secret]]
          name = "dev-api-key"
          mount_path = "/root/.config/scw"
          read_only = true
  locked: true
  pollTimeout: 360
  protected: true
  serviceAccountName: app-deployer
  privileged: false
  secret: dev-runner-tokens
  namespace: dev
  builds:
    cpuRequests: 100m
    memoryRequests: 128Mi
  services:
    cpuRequests: 100m
    memoryRequests: 128Mi
  helpers:
    cpuRequests: 100m
    memoryRequests: 128Mi
  tags: "k8s-dev-runner"
  nodeSelector: 
     k8s.scaleway.com/pool-name: dev
  nodeTolerations:
    - key: "gitlab-runner-jobs-dev-reserved"
      operator: "Equal"
      value: "true"
      effect: "NoSchedule"
Enter fullscreen mode Exit fullscreen mode

You can find the description of each attribute in the Gitlab runner charts repository [2]

  • Get the Gitlab registration token from Project -> Settings -> CI/CD -> Runners in the Setup a specific Runner manually section and create the following secret: 
kubectl create secret generic dev-runner-tokens --from-literal=runner-token='' --from-literal=runner-registration-token='<TOKEN>' -n dev
Enter fullscreen mode Exit fullscreen mode
  • Install the runner: 
helm install -n dev app-dev-runner -f values.yaml gitlab/gitlab-runner
Enter fullscreen mode Exit fullscreen mode

Specific Runner with Kapsule

Using the specific runner in Gitlab CI

Create the pipeline .gitlab-ci.yml:

stages:
  - dev

infra:
  stage: dev
  image:
    name: scaleway/cli:v2.3.1
  script: 
    - /scw k8s cluster create name=kapsule-dev
    - /scw k8s pool create cluster-id=$(/scw k8s cluster list | grep kapsule-dev | awk '{ print $1 }') name=apps node-type=DEV1_M size=2
  tags:
    - k8s-dev-runner
  only:
    - main
Enter fullscreen mode Exit fullscreen mode

The job will create a Kapsule cluster in the development project. We can follow the same steps for a production environment.

Deploy Kapsule Dev from Kapsule DevOps using Gitlab CI

If you want to change the API Key, you just need to delete and recreate the dev-api-key secret.

Access the Kapsule cluster

You can follow the same steps to connect to the Kapsule Cluster from your Gitlab job.

  • Create a secret: 
kubectl create secret generic dev-kapsule-config --from-file ~/.kube/config -n dev
Enter fullscreen mode Exit fullscreen mode
  • and mount it as a data volume: 
        [[runners.kubernetes.volumes.secret]]
          name = "dev-kapsule-config"
          mount_path = "/root/.kube"
          read_only = true
Enter fullscreen mode Exit fullscreen mode

In this example, we used a kubeconfig with admin access. You should use RBAC to restrict access to only specific Kubernetes resources that the runner needs.

Implementing Gitlab Flow

Environment branches with Gitlab Flow is a branching strategy and workflow. Suppose you have additional environments like a pre-production environment and a production environment. Deploy the main branch to your development environment. To deploy to pre-production, create a merge request [3] from the main branch to the pre-prod branch. Go live by merging the pre-prod branch into the production branch.

Gitlab flow

Add the following script ./utils/autoMergeRequest.sh:

#!/bin/bash

[[ $CI_PROJECT_URL =~ ^https?://[^/]+ ]] && CI_PROJECT_URL="${BASH_REMATCH[0]}/api/v4/projects/"

BODY="{
    \"id\": ${CI_PROJECT_ID},
    \"source_branch\": \"${CI_COMMIT_REF_NAME}\",
    \"target_branch\": \"${TARGET_BRANCH}\",
    \"remove_source_branch\": false,
    \"title\": \"Deployment to ${TARGET_BRANCH}\",
    \"assignee_id\":\"${GITLAB_USER_MAILINGLIST_ID}\"
}";

LISTMR=`curl --silent "${CI_PROJECT_URL}${CI_PROJECT_ID}/merge_requests?state=opened" --header "PRIVATE-TOKEN:${PRIVATE_TOKEN}"`;
COUNTBRANCHES=`echo ${LISTMR} | grep -o "\"source_branch\":\"${CI_COMMIT_REF_NAME}\"" | wc -l`;

if [ ${COUNTBRANCHES} -eq "0" ]; then
    curl -X POST "${CI_PROJECT_URL}${CI_PROJECT_ID}/merge_requests" \
        --header "PRIVATE-TOKEN:${PRIVATE_TOKEN}" \
        --header "Content-Type: application/json" \
        --data "${BODY}";

    echo "Opened a new merge request: Deployment to ${TARGET_BRANCH} and assigned to you";
    exit;
fi

echo "No new merge request opened";
Enter fullscreen mode Exit fullscreen mode

It's a common pattern to receive the merge request from a mailing list. Create a Gitlab user with this mailing list and add the variable CI/CD GITLAB_USER_MAILINGLIST_ID with the user ID. You can get the ID using the URL https://gitlab.com/api/v4/users?username=<USERNAME>.

Add the Variable CI/CD PRIVATE_TOKEN, you can create one from Settings > Project Access Tokens.

Example of a gitlab-ci.yml:

stages:
  - build
  - dev
  - preprod
  - prod

build:
  stage: build
  script: 
    - echo 'build'
    - TARGET_BRANCH=main ./utils/autoMergeRequest.sh
  only:
    - /^feature\/*/
    - /^hotfix\/*/

dev:
  stage: dev
  script: 
    - echo 'deploy dev'
    - TARGET_BRANCH=preprod ./utils/autoMergeRequest.sh
  tags:
    - k8s-dev-runner
  only:
    - main

preprod:
  stage: preprod
  script: 
    - echo 'deploy preprod'
    - TARGET_BRANCH=prod ./utils/autoMergeRequest.sh
  tags:
    - k8s-preprod-runner
  only:
    - preprod

prod:
  stage: prod
  script: 
    - echo 'deploy prod'
  tags:
    - k8s-prod-runner
  only:
    - prod
Enter fullscreen mode Exit fullscreen mode

The preprod and prod branches need to be marked as protected. Go to Settings > Repository > Protected branches.

Conclusion

In this post, we created a devops cluster, we mounted the config.yml file to the specific runner, and we ended up deploying our Scaleway Elements and Kubernetes resources in an environment project.

This mechanism guarantees end-to-end security for your API Keys resources in Scaleway Elements.

If you have any questions or feedback, please feel free to leave a comment.

Otherwise, I hope I've convinced you to remove your API keys from Gitlab CI variables.

By the way, do not hesitate to share with peers 😊

Thanks for reading!

Documentation

[1] https://www.scaleway.com/en/docs/compute/kubernetes/api-cli/creating-managing-kubernetes-lifecycle-cliv2/
[2] https://gitlab.com/gitlab-org/charts/gitlab-runner/-/blob/main/values.yaml
[3] https://about.gitlab.com/blog/2017/09/05/how-to-automatically-create-a-new-mr-on-gitlab-with-gitlab-ci/
[4] https://kubernetes.io/docs/concepts/configuration/secret/#using-secrets-as-files-from-a-pod

Top comments (0)