DEV Community: Nathan Bridgewater

Setting Up a Basic Prometheus Deployment with CDK8s

Nathan Bridgewater — Mon, 19 Feb 2024 09:34:30 +0000

I was genuinely surprised by the scarcity of CDK8s examples available. Consequently, I decided to pitch in and share what I've learned by presenting how I setup a basic Prometheus deployment using CDK8s.

What is CDK8s i hear you ask?

CDK8s is a development framework tailored for Kubernetes. It enables users to define Kubernetes manifests using a library of constructs in their preferred programming language. In essence, CDK8s is akin to AWS CDK but designed for Kubernetes.

Prerequisites

Before we dive into the tutorial, make sure you have the following prerequisites:

Basic understanding of Kubernetes and Prometheus
CDK8s CLI installed
kubectl installed
A Running Kubernetes cluster (Minikube works well)

1. Initialise your CDK8s application

Begin by setting up your CDK8s app using the init command. While I'll be using TypeScript, there are alternative options available.

cdk8s init typescript-app

This will create a folder structure which looks something like:

my-cdk8s-app/
├── imports/
├── dist/
├── cdk8s.yaml
├── package.json
├── tsconfig.json
└── main.ts

The key file here is main.ts, this contains our CDK8s chart. Charts are just a container for your CDK8s code, they are synthesised into a single K8s manifest.

2. Add imports for necessary constructs

Import the required constructs at the top of your main.ts file to abstract elements like configmaps, volumes, deployments, etc.

main.ts

import { 
  ClusterRole, 
  ApiResource, 
  NonApiResource, 
  ServiceAccount, 
  Deployment, 
  Volume,
  ConfigMap } 
from 'cdk8s-plus-25'

3. Create a namespace for Prometheus and configure the chart

Run the following command in your terminal to create a namespace for Prometheus:

kubectl create namespace monitoring

Modify your main.ts file to ensure all API objects use this namespace:

new MyChart(app, 'CDK8s-Prometheus', { namespace: 'monitoring' })

4. Configure RBAC

For Prometheus to access the Kubernetes API, create a service account, cluster role, and role binding:

    const serviceAccount = new ServiceAccount(this, 'service-account')
    const clusterRole = new ClusterRole(this, 'cluster-role')

    clusterRole.allowRead(ApiResource.NODES) 
    clusterRole.allowRead(ApiResource.SERVICES)
    clusterRole.allowRead(ApiResource.ENDPOINTS)
    clusterRole.allowRead(ApiResource.PODS)
    clusterRole.allowRead(ApiResource.INGRESSES)
    clusterRole.allowGet(NonApiResource.of('/metrics'))


    clusterRole.bindInNamespace('monitoring', serviceAccount)

5. Create a file for Prometheus configuration

Add a prometheus.yml file to the root directory of your project with the following content. This is a simple setup where Prometheus scrape its own metrics

prometheus.yml

global:
  scrape_interval: 5s
scrape_configs:
  - job_name: prometheus
    static_configs:
      - targets:
        - localhost:9090

6. Create volumes for storage and config

Define two volumes to be mounted in the pods later—one for Prometheus config and another for storing metrics data:

    const configMap = new ConfigMap(this, 'configmap')
    configMap.addFile(join(__dirname + '/prometheus.yml'))

    const configVolume = Volume.fromConfigMap(this, 'config-volume', configMap)
    const dataVolume = Volume.fromEmptyDir(this, 'data-volume', 'data')

7. Create our deployment

Specify the deployment details for Prometheus in your main.ts file:

const deployment = new Deployment(this, 'deployment', {

      containers: [ 
        {
          name: 'prometheus',
          image: 'prom/prometheus',
          args: ['--config.file=/etc/prometheus/prometheus.yml'],
          portNumber: 9090,
          securityContext: {ensureNonRoot: false}

        }
      ]
    })

    deployment.containers[0].mount('/etc/prometheus/', configVolume)
    deployment.containers[0].mount('/data/', dataVolume)

8. Deploy your application

With everything set up, deploy your Prometheus instance:

Run:

kubectl apply -f dist/

Test your Prometheus deployment by accessing the UI using port forwarding:

kubectl port-forward <POD NAME> 8080:9090 -n=monitoring

Access the UI via http://localhost:8080

Congratulations! You've successfully set up a basic Prometheus deployment using CDK8s. From here, you can explore further customisation and optimisation based on your specific needs.

Say Goodbye to Hardcoded AWS Creds in GitHub Actions with OIDC Magic ⭐

Nathan Bridgewater — Wed, 27 Dec 2023 12:12:08 +0000

I've been tinkering with GitHub actions over the last few months, often times with some kind of AWS interaction in my workflow. As it turns out, there are simple and less simple methods of authenticating your workflows with AWS. Let's explore how to use GitHub's OIDC provider to free yourself from hardcoded AWS credentials.

Hardcoded access keys

Lets be real, we've all hardcoded access keys before. For beginners and people who are just looking to get something running, hardcoded credentials are great, it's the most straightforward way to authenticate your workflow with AWS.

Here's a typical example:

- name: Configure AWS credentials
  uses: aws-actions/configure-aws-credentials@v4
  with:
    aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
    aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
    aws-region: us-east-1

However, this does not align with best practices, duplicating and storing long lived credentials for your AWS account is not the most secure way to authenticate your workflows.

Enter OpenID Connect (OIDC)

OIDC, or OpenID Connect, is an authentication protocol that lets a third party (in our case, AWS) verify a user's identity through a JSON Web Token (JWT). With OIDC, authentication is outsourced to the OIDC provider. Once authenticated, the user can obtain a JWT from the identity provider, using it to authenticate with the third party.

In our scenario, GitHub acts as an OIDC provider for workflows. This means our workflow can request a JWT from GitHub and then use it to assume an AWS role by obtaining temporary credentials from AWS.

How to use GitHub's OIDC provider in your workflows

Let's walk through a simple example that builds a Docker image and pushes it to ECR using GitHub's OIDC provider. If you want to follow along, I've prepared a NodeJS application and Docker file in this following repository:

https://github.com/Nathan-Bridgewater/GitHub-AWS-OIDC.git

1. Create an identity provider in AWS

Firstly, we add GitHub as an identity provider. This configures AWS to accept OIDC tokens from a workflow and return temporary access credentials.

aws create-open-id-connect-provider \
--url https://token.actions.githubusercontent.com  \
--client-id-list sts.amazonaws.com \
--thumbprint-list 1b511abead59c6ce207077c0bf0e0043b1382612 \

2. Create the trust policy

Craft a JSON file containing a trust policy for an AWS role, specifying which GitHub workflows can assume our role.

trust-policy.json

{
  "Version": "2012-10-17",
  "Statement": [
      {
          "Effect": "Allow",
          "Action": "sts:AssumeRoleWithWebIdentity",
          "Principal": {
              "Federated": "arn:aws:iam::<ACCOUNT NUM>:oidc-provider/token.actions.githubusercontent.com"
          },
          "Condition": {
              "StringEquals": {
                  "token.actions.githubusercontent.com:aud": [
                      "sts.amazonaws.com"
                  ]
              },
              "StringLike": {
                  "token.actions.githubusercontent.com:sub": [
                      "repo:Nathan-Bridgewater/GitHub-AWS-OIDC:*"
                  ]
              }
          }
      }
  ]
}

3. Create the role

We need to establish a role for our federated workflow to assume, and attach a policy allowing our workflow to push images to ECR.

aws iam create-role \
--role-name github-workflow-role \
--assume-role-policy-document file://trust-policy.json 

aws iam attach-role-policy \
--role-name github-workflow-role \
--policy-arn arn:aws:iam::aws:policy/AmazonEC2ContainerRegistryPowerUser

4. Create an ECR repository

Set up an ECR repository to push Docker images.

aws ecr create-repository \
--repository-name nodejs-images

5. Create the GitHub workflow

push-docker-actions.yaml

name: push-docker-ecr-actions
run-name: ${{ github.actor }} is running the push-docker-ecr workflow
on:
  push:
    branches:
      - 'main'

permissions:
  id-token: write
  contents: read  

jobs:
  build-image-push-to-ecr:
    name: Build docker image and push to ecr
    runs-on: ubuntu-latest
    steps:
      - name: Checkout code
        uses: actions/checkout@v4
      - name: Configure AWS credentials
        uses: aws-actions/configure-aws-credentials@v4
        with:
          role-to-assume:  arn:aws:iam::<ACCOUNT NUM>:role/github-workflow-role
          aws-region: us-east-1
      - name: Login to Amazon ECR
        id: login-ecr
        uses: aws-actions/amazon-ecr-login@v1
      - name: Build, tag and push image to ECR
        env:
          ECR_REGISTRY: ${{ steps.login-ecr.outputs.registry }}
          ECR_REPOSITORY: nodejs-images
          IMAGE_TAG: latest
        run: |
          docker build -t $ECR_REGISTRY/$ECR_REPOSITORY:$IMAGE_TAG .
          docker push $ECR_REGISTRY/$ECR_REPOSITORY:$IMAGE_TAG

6. Test it out

Once added, your workflows should now authenticate with AWS without hardcoded credentials. Freedom awaits!