DEV Community: Kamesh Sampath

[Boost]

Kamesh Sampath — Thu, 27 Feb 2025 18:27:32 +0000

Let's Build Together: A Local Playground for Apache Polaris

Kamesh Sampath ・ Feb 25

#apachepolaris #snowflake #kubernetes #localstack

Let's Build Together: A Local Playground for Apache Polaris

Kamesh Sampath — Tue, 25 Feb 2025 04:01:17 +0000

Why I Built a Developer-First Apache Polaris Starter Kit ?

As builders, we all know the pain of setting up a new development environment. Hours spent configuring dependencies, troubleshooting integration issues, and getting different services to play nicely together. When I started working with Apache Polaris, I faced these same challenges – and decided to do something about it.

The Challenge: Getting Started with Apache Polaris

Apache Polaris is a powerful open source Iceberg REST catalog implementation, originally contributed to the Apache Software Foundation by Snowflake. This donation to open source has made enterprise-grade data catalog capabilities accessible to the broader data community via simple REST APIs.

Setting up Polaris in a development environment can be challenging. You need:

A robust container orchestration platform
A working metastore (typically PostgreSQL)
S3-compatible storage
Various security configurations and credentials

Each of these components requires careful setup and configuration. For builders just getting started or wanting to experiment with Polaris, this overhead can be a significant barrier.

The Solution: A Complete Development Environment

This is why I created an open source starter kit that provides everything needed to get Polaris up and running in a local development environment. The project follows the true spirit of open source collaboration, building upon and integrating with other excellent open source tools in the ecosystem.

The kit automates the setup of:

A lightweight k3s Kubernetes cluster using k3d
LocalStack for AWS S3 emulation
PostgreSQL metastore with proper configurations
All necessary security credentials and configurations

A key aspect of this starter kit is its comprehensive automation using Ansible. The polaris-forge-setup directory houses Ansible playbooks that:

Automate the entire setup process
Verify if components are ready for use
Handle catalog setup and configuration
Provide cleanup capabilities for development iterations
Enable smooth transitions to higher environments

This automation-first approach serves two purposes:

Immediate Development: Developers can get started quickly with minimal manual intervention
Production Readiness: The Ansible scripts serve as a template for scaling to higher environments, making it easier to adapt the setup for staging or production use cases

By keeping everything open source and focusing on community-driven development, we ensure that builders can:

Learn from the implementation
Customize for their specific needs
Contribute improvements back to the community
Build upon a foundation of trusted open source tools

What is Snowflake OpenCatalog?

Snowflake OpenCatalog is an enterprise-grade implementation and managed service of upstream Polaris, making it incredibly easy to integrate with your existing data stack. By handling the operational complexities of running Polaris at scale, it allows teams to focus on their data applications:

Managed Infrastructure: Snowflake handles all operational aspects including:
- Polaris server management and scaling
- Security and access control
- High availability and reliability
- Regular updates and maintenance
Enterprise Integration: Seamless connectivity with:
- Snowflake's ecosystem of data services
- Popular query engines and tools
- Existing data governance frameworks
- Enterprise security systems
Production-Ready Features:
- Advanced access controls and auditing
- Cross-region and cross-cloud support
- Enterprise-grade SLAs
- Professional support

From Local Development to Enterprise Scale

This starter kit provides an ideal path for builders working with Apache Polaris and considering OpenCatalog for production deployment. By working with the upstream version in this development environment, you:

Gain hands-on experience with core concepts
Understand the underlying architecture
Can prototype and test implementations
Build expertise that transfers to OpenCatalog
Have a clear path to production scaling

When you're ready to move to production, the concepts and patterns you've learned here will help you make the most of OpenCatalog's enterprise capabilities while Snowflake handles the operational complexity.

Technical Design Decisions

Why Kubernetes with k3s and k3d?

While Docker Compose is often the go-to choice for local development environments, Apache Polaris's distributed nature benefits significantly from Kubernetes's capabilities. Here's why:

Advanced Networking: Kubernetes provides sophisticated networking between components:
- Automatic service discovery and DNS resolution
- Internal load balancing for scalable services
- Ingress management for external access
- Network policies for traffic control
Declarative Configuration: Using tools like Helm and Kustomize, we can:
- Maintain separate configurations for different environments
- Version control our infrastructure setup
- Apply consistent changes across deployments
- Manage complex dependencies between services
Reliable State Management:
- StatefulSets for databases and stateful services
- PersistentVolumes for durable storage
- Backup and restore capabilities
- Data replication when needed
Security and Configuration:
- Native secrets management
- Role-Based Access Control (RBAC)
- ConfigMaps for configuration management
- Service accounts for component authentication
Production Readiness:
- Same tools and patterns used in production
- Easy scaling of components
- Built-in monitoring and logging
- Consistent behavior across environments

I specifically chose k3s because it's lightweight and perfect for development environments. Using k3d allows us to run k3s in Docker containers, making it even more convenient for local development. It provides a full Kubernetes experience without the resource overhead of something like minikube.

LocalStack for S3 Integration

While we could have required developers to use actual AWS S3, LocalStack provides a perfect local alternative. It emulates AWS services locally, which means:

No cloud costs during development
No need for AWS credentials
Faster development cycles
Ability to work offline

PostgreSQL as the Metastore

PostgreSQL was a natural choice for the metastore. It's:

Well-documented and widely used
Easy to containerize
Highly reliable
Supported out of the box by Polaris

Kustomize for Deployment Management

Kustomize allows us to manage Kubernetes manifests in a clean, declarative way. It makes it easy to:

Maintain different configurations for different environments
Override settings without modifying base configurations
Keep configurations DRY and maintainable

Getting Started

Let me walk you through how to get up and running with this starter kit.

Ensure you have the prerequisites installed:

# Required tools and their version checks:

# Docker (Desktop or Engine)
docker --version
# Example output: Docker version 24.0.7

# Kubernetes CLI
kubectl version --client
# Example output: Client Version: v1.28.2

# k3d (>= 5.0.0)
k3d version
# Example output: k3d version v5.6.0

# Python (>= 3.11)
python --version
# Example output: Python 3.12.1

# uv (Python packaging tool)
uv --version
# Example output: uv 0.1.12

# Task
task --version
# Example output: Task version: v3.34.1

# LocalStack (>= 3.0.0)
localstack --version
# Example output: 3.0.0

Sign-up for Localstack

Initial Setup

Clone the repository and set up your environment:

git clone https://github.com/snowflake-labs/polaris-local-forge
cd polaris-local-forge

# Set up environment variables
export PROJECT_HOME="$PWD"
export KUBECONFIG="$PWD/.kube/config"
export K3D_CLUSTER_NAME=polaris-local-forge
export K3S_VERSION=v1.32.1-k3s1
export FEATURES_DIR="$PWD/k8s"

Python Environment Setup

# Install uv
pip install uv

# Set up Python environment
uv python pin 3.12
uv venv
source .venv/bin/activate  # On Unix-like systems
uv sync

Deploy the Environment

The setup process is automated through several scripts:

# Generate required sensitive files
$PROJECT_HOME/polaris-forge-setup/prepare.yml

# Create and set up the cluster
$PROJECT_HOME/bin/setup.sh

# Wait for deployments to be ready
$PROJECT_HOME/polaris-forge-setup/cluster_checks.yml --tags namespace,postgresql,localstack

Deploy Polaris

This is where things get interesting - deploying Polaris itself. You have two options for the container images:

Option 1: Use Pre-built Images

Apache Polaris doesn't currently publish official images, but you can use our pre-built images with PostgreSQL dependencies:

docker pull ghcr.io/snowflake-labs/polaris-local-forge/apache-polaris-server-pgsql
docker pull ghcr.io/snowflake-labs/polaris-local-forge/apache-polaris-admin-tool-pgsql

Option 2: Build Images Locally

Alternatively, you can build the images from source:

# Update IMAGE_REGISTRY in Taskfile.yml, then run:
task images

If you choose to build locally, remember to update the image references in:

k8s/polaris/deployment.yaml
k8s/polaris/bootstrap.yaml
k8s/polaris/purge.yaml

Deploy and Verify

Apply the Kubernetes manifests:

# Apply Polaris manifests
kubectl apply -k $PROJECT_HOME/k8s/polaris

# Verify deployments and jobs
$PROJECT_HOME/polaris-forge-setup/cluster_checks.yml --tags polaris

Setting Up Your First Catalog

Before creating your first catalog, configure your AWS environment variables:

export AWS_ENDPOINT_URL=http://localstack.localstack:14566
export AWS_ACCESS_KEY_ID=test
export AWS_SECRET_ACCESS_KEY=test
export AWS_REGION=us-east-1

# Run the catalog setup
$PROJECT_HOME/polaris-forge-setup/catalog_setup.yml

Pro Tip: You can customize the default catalog settings by modifying values in polaris-forge-setup/defaults/main.yml. This file contains configurable parameters for your catalog, principal roles, and permissions.

Play with the Catalog

Once your catalog is set up, you can explore its functionality using the provided Jupyter notebook. The notebook notebooks/verify_setup.ipynb walks you through:

Creating a namespace
Defining a table
Inserting sample data
Verifying data storage in LocalStack

This hands-on exploration helps you understand how Polaris integrates with:

The PostgreSQL metastore for catalog management
LocalStack's S3 emulation for data storage
The overall Apache Iceberg table format structure

You can visually verify your setup by checking the LocalStack console at https://app.localstack.cloud/inst/default/resources/s3/polardb, where you'll see:

Catalog storage structure
Metadata files
Actual data files

Video Walkthrough

For a detailed visual guide of setting up and using this development environment, check out my walkthrough video:

This video demonstrates the entire process from initial setup to running your first queries.

Troubleshooting Tips

If you run into issues, here are some helpful commands for debugging:

# Check Polaris server logs
kubectl logs -f -n polaris deployment/polaris

# Check PostgreSQL logs
kubectl logs -f -n polaris statefulset/postgresql

# Check LocalStack logs
kubectl logs -f -n localstack deployment/localstack

# Check events in the polaris namespace
kubectl get events -n polaris --sort-by='.lastTimestamp'

The Impact: Streamlined Development Experience

With this starter kit, what used to take days of setup and configuration now takes minutes. Builders can focus on creating and experimenting with Polaris rather than wrestling with infrastructure setup.

The kit is open source and available on GitHub. I welcome contributions and feedback from the community. Together, we can make the development experience even better for everyone working with Apache Polaris.

Building should be about creating, not configuring. This starter kit aims to remove the friction from getting started with Apache Polaris, allowing builders to focus on what matters most – creating great applications.

Dont forget to check another project where I used this starter kit https://github.com/kameshsampath/balloon-popper-demo.

Related Projects and Tools

Core Components

Apache Polaris - Data Catalog and Governance Platform
PyIceberg - Python library for Apache Iceberg
LocalStack - AWS Cloud Service Emulator
k3d - k3s in Docker
k3s - Lightweight Kubernetes Distribution
Ansible - Automation Platform

Development Tools

Docker - Container Platform
Kubernetes - Container Orchestration
Helm - Kubernetes Package Manager
kubectl - Kubernetes CLI
uv - Python Packaging Tool

Elements of Event Driven Architecture(EDA)

Kamesh Sampath — Wed, 08 Nov 2023 08:06:04 +0000

Well, we encounter lots of data in our everyday life e.g. weather reports, flight timings, food deliveries etc., All these data are continuous and flowing from variety of sources. Such continuously flowing data is called a Data Stream.

A raw data(stream) is useless for any application unless it has some identifiable element to it.

Let me explain it with few examples, Joe joined Acme crop as a Developer on 25 October 2023, Mini placed an order for two pizzas at 12:00 PM. In these data examples if we take out the verbs(actions) "joined" and "ordered" along with the time "25 Oct" and "12:00 PM" , the whole data become useless.

For an application to effectively use the data stream, the data in the stream should be an event, a data that has time and action associated with it. A stream of data as events brings a great value to applications via analytics, triggers, chaining etc.,

With Mini placed an order for two pizzas at 12:00 PM event, we could extract the following analytical information,

Is Mini placing order for two pizzas everyday?
Is the order placed at 12:00PM everyday?
Is the order is delivered by the same food chain?
What pizzas were ordered?
Are food chains delivering the orders on time ?

The process of using the events and building such analytical information is called Data Processing. Data Processing could be done by a human, an application, an IoT device etc.,

With any data streaming scenario there is always two essential primitive entities,

Event Producer is someone or something that produces an event in our example above Mini is event producer who places the order for pizza.

Event Consumer is again someone or something that consumes or uses the event produced by the Event Producer. Taking the same Mini's order example there could be Pizza house that takes, processes and delivers the order.

As we understood the Event Producers produces events that are being consumed by Event Consumers, software architectures started to evolve around building applications that act as an Event Producer/Consumer and leverage these streams of events. An architectural style of building applications around events is called an Event Driven Architecture(EDA).

When building applications using EDA, enforces few basic requirements on the Platform and a software Framework that will be used to build such applications.

The platform that will be used to build the such applications need to be:

Scalable - as events are continuously flowing there could be sudden spikes to number of events that might come in, the platform need to be scalable or elastic to handle such spikes
Durable - as events can be consumed immediately or bit late in time, the platform should support a mechanism of delivering events durably at time of need
Resilient - The platform should be capable of handling failures and recovering from them without data loss
Data Retention - Retaining of data until a configurable amount time
Responding to Events - The platform should also be able to respond to events at bare minimum acknowledge the event on receive
Ordering - As events are associated with time, ordering of the events helps consumers who need to process them in specific order e.g within a time range or date range etc.,

A sole platform alone might not be enough to build an effective EDA styled application. There is greater need for an integration into the platform via plugins, API etc., In other words, a framework that is extensible and pluggable, and works on common semantics.

The framework should support:

Data Sources - the sources from where the events are generated i.e. Event Producers
Data Sinks - the destinations where the processed event is drained into i.e. Event Consumers
API - An interface to connect and work with the platform, data sources and data sinks.

Some great of Data Streaming platforms,

Apache Kafka - Developed at LinkedIn and Opensources to Apache Software Foundation
Redpanda - is a simple, powerful, and cost-efficient streaming data platform that is compatible with Kafka® APIs while eliminating Kafka complexity.

Apache Kafka also supports processing of streaming data using its ecosystem of Streaming API and ksqlDB. But for an effective architecture, it is always nice to have the core data streaming and data processing to be decoupled(Separation of Concerns). Such decoupling helps in processing data from heterogeneous sources e.g. Apache Kafka, Database, File System CSV files etc.,

Apache Flink is one such framework and distributed processing engine for stateful computations over unbounded(Apache Kafka) and bounded data streams(Database).

Just to summarise we learnt,

What is a Data Stream and an Event
What is a Data Producer and Data Consumer
An architecture style that is used to build application around Events
An effective EDA platform
Some great platform and frameworks that could be used to build EDA applications.

Trigger CI using Terraform Cloud

Kamesh Sampath — Mon, 10 Apr 2023 01:23:33 +0000

Continuous Integration(CI) pipelines needs a target infrastructure to which the CI artifacts are deployed. The deployments are handled by CI or we can leverage Continuous Deployment pipelines. Modern day architecture uses automation tools like terraform, ansible to provision the target infrastructure, this type of provisioning is called IaC.

Usually CI/CD and IaC don't run in tandem. Many times we want to trigger the CI pipeline only when the target infrastructure is ready to bootstrap with software components that are required by CI/CD pipelines.

As part of this DIY blog let us tackle the aforementioned problem with an use case.

Use Case

As CI/CD user I would like to provision a Kubernetes Cluster on Google Cloud Platform(GKE) using Terraform. The successful provision of the cluster should notify a CI pipeline to start bootstrapping ArgoCD on to GKE.

What you need ?

A Terraform Cloud Account. Create a workspace on the terraform cloud to be used for this exercise.
Google Cloud Account used to create the Google Kubernetes Engine(GKE) cluster.
Though we can use any CI platform, for this demo we will use Harness CIas our CI platform. You can do a free tier signup from here.

Demo Sources

The demo uses the following git repositories a sources,

IaC vanilla-gke: the terraform source repository that will be used with terraform cloud to provision GKE.
Kubernetes manifests bootstrap-argocd: the repository that holds kubernetes manifests to bootstrap argo CD on to the GKE cluster
Harness CI Pipeline tfc-notification-demo

Fork and Clone the Sources

To make fork and clone easier we will use gh CLI. Download the add gh to your $PATH.

Let us create a directory where we want to place all our demo sources,

mkdir -p "$HOME/tfc-notification-demo"
cd "$HOME/tfc-notification-demo"
export DEMO_HOME="$PWD"

IaC

Clone and fork vanilla-gke repo,

gh repo clone harness-apps/vanilla-gke
cd vanilla-gke
gh repo fork
export TFC_GKE_REPO="$PWD"

Bootstrap Argo CD Sources

Clone and fork bootstrap-argocd repo,

cd ..
gh repo clone harness-apps/bootstrap-argocd
cd bootstrap-argocd
gh repo fork
export ARGOCD_BOOTSTRAP_REPO="$PWD"

Harness CI Pipeline

Clone and fork tfc-notification-demo repo,

cd ..
gh repo clone harness-apps/tfc-notification-demo
cd tfc-notification-demo
gh repo fork
export TFC_DEMO_REPO="$PWD"

For rest of the blog we will reference the repositories vanilla-gke and bootstrap-argocd and tfc-notification-demo as $TFC_GKE_REPO, $ARGOCD_BOOTSTRAP_REPO and $TFC_DEMO_REPO.

Harness CI

In the following sections we will define and create the resources required to define a CI pipeline using Harness platform.

Create Harness Project

Create new harness project named terraform_integration_demos using Harness Web Console,

Update its details as shown,

Follow the wizard leaving rest to defaults and on the last screen choose Continuous Integration,

Click Go to Module to go to project home page.

Define New Pipeline

Click Pipelines to define a new pipeline,

For this demo will be doing manual clone, hence disable the clone,

Click on Pipelines and delete the default Build pipeline,

Add `harnessImage` Docker Registry Connector

As part of pipelines we will be pulling image from DockerHub. harnesImage Docker Registry Connector helps pulling the public Docker Hub images as an anonymous user.

Let us configure an harnesImage connector as described in docker registry connectors. The pipelines we create as part of the later section will use this connector.

Configure GitHub

GitHub Credentials

Create a GitHub PAT for the account where you have have forked the repositories $TFC_GKE_REPO and $ARGOCD_BOOTSTRAP_REPO. We will refer to the token as $GITHUB_PAT.

From the Project Setup click Secrets,

Update the encrypted text secret details as shown,

Click Save to save the secret,

Connector

As we need to clone the sources from GitHub, we need to define a GitHub Connector, from the Project Setup click Connectors,

From connector list select GitHub,

Enter the name as GitHub,

Click Continue to enter the connector details,

Click Continue and update the GitHub Connector credentials,

When selecting the Personal Access Token make sure you select the GitHub PAT that we defined in previous section,

Click Continue and use select Connect through Harness Platform,

Click Save and Continue to run the connection test, if all went well the connection should successful,

Google Cloud Service Account Secret

We need Google Service Account(GSA) credentials(JSON Key) to query the GKE cluster details and create resources on it.

Set environment

export GCP_PROJECT="the Google Cloud Project where Kubernetes Cluster is created"
export GSA_KEY_FILE="path where to store the key file"

Create SA

gcloud iam service-accounts create gke-user \
  --description "GKE User" \
  --display-name "gke-user"

IAM Binding

Add permissions to the user to be able to provision kubernetes resources,

gcloud projects add-iam-policy-binding $GCP_PROJECT \
  --member="serviceAccount:$GSA_NAME@$GCP_PROJECT.iam.gserviceaccount.com" \
  --role="roles/container.admin"

Download And Save GSA Key

IMPORTANT: Only security admins can create the JSON keys. Ensure the Google Cloud user you are using has Security Admin role.

gcloud iam service-accounts keys create "${GSA_KEY_FILE}" \
    --iam-account="gke-user@${GCP_PROJECT}.iam.gserviceaccount.com"

GSA Secret

Get back to the Project Setup click Secrets,

Add the GSA secret details as shown,

IMPORTANT: When you browse and select make sure you select the $GSA_KEY_FILE as the file for the secret.

Click Save to save the secret,

Terraform Workspace

On your terraform cloud account create a new workspace called vanilla-gke. Update the workspace settings to use Version Control and make it point to $TFC_GKE_REPO.

Configure the workspace with following variables,

For more details on available variables, check Terraform Inputs.

IMPORTANT: The GOOGLE_CREDENTIALS is Google Service Account JSON Key with permissions to create GKE cluster. Please check the https://github.com/harness-apps/vanilla-gke#pre-requisites for the required roles and permissions. This key will be used by Terraform to create the GKE cluster. When you add the key to terraform variables, you need to make it as base64 encoded e.g. cat YOUR_GOOGLE_CREDENTIALS_KEY_FILE | tr -d \\n

Going forward we will refer to the Terraform Workspace as $TF_WORKSPACE

Lookup your terraform cloud organizations

And set it's value to the variable $TF_CLOUD_ORGANIZATION.

Create need Terraform API Token that can be used to pull the outputs of terraform run(cloud). From your terraform user settings Create an API token,

And save the API token to the variable $TF_TOKEN_app_terraform_io. We will use this variable in CI pipeline.

Harness CI Pipeline

Getting back to Harness web console, navigate to your project terraform_integration_demos, click Pipelines and Create a Pipeline --> Import From Git,

Update the pipeline details as shown,

IMPORTANT: Make sure the Name of the pipeline is bootstrap argocd pipeline to make the import succeed with defaults.

Click the bootstrap argocd pipeline from the list to open the Pipeline Studio and click on the stage Bootstrap Argo CD to bring up the pipeline steps,

You can click on each step to see the details.

The Pipeline uses the following secrets,

google_application_credentials - the GSA credentials to manipulate GKE
terraform_cloud_api_token - the value of $TF_TOKEN_app_terraform_io
terraform_workspace - the value $TF_WORKSPACE
terraform_cloud_organization - the value $TF_CLOUD_ORGANIZATION

We already added google_application_credentials secret as part of the earlier section. Following the similar pattern let us add the terraform_cloud_api_token, terraform_workspace and terraform_cloud_organization as text secrets.

HINT:
From the Project Setup click Secrets,

TIP: You can also skip adding terraform_workspace and terraform_cloud_organization, we can extract the values from the webhook payload using the expressions <+trigger.payload.workspace_name> and <+trigger.payload.organization_name> respectively.

Notification Trigger

For the Harness CI pipelines to listen to Terraform Cloud Events we need to define a Trigger, navigate back to pipelines and select the bootstrap argocd pipeline --> Triggers,

Click Add New Trigger to add a new webhook trigger(Type: Custom),

On the Configuration page enter the name of the trigger to be tfc notification,

Leave rest of the fields to defaults and click Continue, leave the Conditions to defaults and click Continue.

On the Pipeline Input update the Pipeline Reference Branch to be set to main

NOTE: The Pipeline Reference Branch does not have any implication with this demo as we do manual clone of resources.

Click Create Trigger to create and save the trigger.

Copy Webhook URL

Let us refer to this value as $TRIGGER_WEBHOOK_URL.

Terraform Notification

On your terraform cloud console navigate to the workspace Settings --> Notifications,

Click Create Notification and select Webhook as the Destination,

Update the notification details as shown,

Since we need to bootstrap argo CD only on create events we set the triggers to happen only on Completed,

Click Create Notification to finish the creation of notification.

NOTE: The creation would have fired a notification, if the cluster is not ready yet the pipeline would have failed.

Congratulations!!. With this setup any new or updates thats done to the $TFC_GKE_REPO will trigger a plan and apply on Terraform Cloud. A Completed plan will trigger the bootstrap argocd pipline to run and apply the manifests from $BOOTSTRAP_ARGOCD_REPO on the GKE cluster.

An example of successful pipeline run

Summary

By using the terraform notifications feature we were able to make the CI pipelines listen to IaC events and run the CI pipelines as needed.

Using Workload Identity Continuous Integration(CI) Pipelines

Kamesh Sampath — Thu, 23 Mar 2023 05:22:53 +0000

Overview

In the first part of this series we understood what a Workload Identity is and in the second part how it can help in doing keyless Google API invocations by deploying a demo application. In this blog we learn how to use Workload Identity with SaaS Continuous Integration(CI) providers.

Many SaaS Continuous Integration(CI) providers e.g Harness CI uses what is called a Delegate

Harness Delegate is a service you run in your local laptop or on Cloud to connect your artifacts, infrastructure, collaboration, verification and other providers, with Harness Manager.

Here are some advantages of using Delegates,

Total control of source code, as they can be within organisations cloud infrastructure
Cloud Cost Optimisations as CI pipelines can leverage organisations existing cloud infrastructure
Leverage the native cloud services from CI pipelines

In this tutorial we will deploy a Harness Delegate on to GKE and understand how enabling Workload Identity on GKE can simplify the CI Pipeline.

CI Pipeline UseCase

Builds go application, ideally you can build any application but go is taken as an example here.
Package application build artifact as a container image
Push the image to Google Artifact Registry(GAR).
Cache the build artifacts, dependencies(go modules) on to Google Cloud Storage(GCS) to make build process faster and quicker.

Tutorial

Before we get to tutorial make sure you have signed up for free tier Harness CI account.

Pre-requisites

A Google Cloud Account with a Service Account with roles:
- Kubernetes Engine Admin - to create GKE cluster
- Service Account roles used to create/update/delete Service Account
  - iam.serviceAccounts.actAs
  - iam.serviceAccounts.get
  - iam.serviceAccounts.create
  - iam.serviceAccounts.delete
  - iam.serviceAccounts.update
  - iam.serviceAccounts.get
  - iam.serviceAccounts.getIamPolicy
  - iam.serviceAccounts.setIamPolicy
  - (OR) simply you can add Service Account Admin and Service Account User roles
- Compute Network Admin - to create the VPC networks
Google Cloud SDK
terraform
kubectl
helm
Taskfile

Download Sources

git clone https://github.com/harness-apps/workload-identity-gke-demo.git && cd "$(basename "$_" .git)"
export DEMO_HOME="$PWD"

Environment Setup

Variables

When working with Google Cloud the following environment variables helps in setting the right Google Cloud context like Service Account Key file, project etc., You can use direnv or set the following variables on your shell,

export GOOGLE_APPLICATION_CREDENTIALS="the google cloud service account key json file to use"
export CLOUDSDK_ACTIVE_CONFIG_NAME="the google cloud cli profile to use"
export GOOGLE_CLOUD_PROJECT="the google cloud project to use"
export KUBECONFIG="$DEMO_HOME/.kube/config"

You can find more information about gcloud cli configurations at https://cloud.google.com/sdk/docs/configurations.

As you may need to override few terraform variables that you don't want to check in to VCS, add them to a file called .local.tfvars and set the following environment variable to be picked up by terraform runs,

export TFVARS_FILE=.local.tfvars

Check the Inputs section for all possible terraform variables that are configurable.

An example .local.tfvars looks like,

project_id                 = "my-awesome-gcp-project"
region                     = "asia-south1"
cluster_name               = "wi-demos"
kubernetes_version         = "1.24."
harness_account_id         = "REPLACE WITH YOUR HARNESS ACCOUNT ID"
harness_delegate_token     = "REPLACE WITH YOUR HARNESS DELEGATE TOKEN"
harness_delegate_name      = "wi-demos-delegate"
harness_delegate_namespace = "harness-delegate-ng"
harness_manager_endpoint   = "https://app.harness.io/gratis"

Create Environment

We will use terraform to create a GKE cluster with WorkloadIdentity enabled for its nodes,

task init

Create GKE cluster

The terraform apply will creates a GKE Cluster,

task create_cluster

Deploy Harness Delegate

The following section deploys a Harness Delegate on to our GKE cluster. To be able to successfully deploy a Harness Delegate we need update the following values in our .local.tfvars file,

harness_account_id
harness_delegate_token
harness_delegate_namespace
harness_manager_endpoint
Use Account Id from Account Overview as the value for harness_account_id,

Use the Harness Cluster Hosting Account from the account details to find the matching endpoint URL. e.g for prod-2 it is https://app.harness.io/gratis and set that as value for harness_manager_endpoint.

TIP:
You can find the endpoint corresponding to your Harness Cluster Hosting Account from https://developer.harness.io/tutorials/platform/install-delegate/

Copy the default token from Projects --> Project Setup --> Delegates(Tokens) and set it as value for harness_delegate_token.

harness_delegate_name: defaults to harness-delegate
harness_delegate_namespace: defaults to harness-delegate-ng

With us having updated the .local.tfvars, run the following command to deploy the Harness Delegate,

task deploy_harness_delegate

NOTE: It will take some time for delegate to come up and get connected.

Wait for the delegate to be connected before proceeding to next steps.

You can view status of the delegate from the Project --> Project Setup --> Delegates page,

You can also check the running Harness delegate pods by using kubectl,

kubectl get pods -n harness-delegate-ng

The output should be something like the pod name may vary based on your harness_delegate_name value.

NAME                                 READY   STATUS    RESTARTS   AGE
harness-delegate-6bfd78d5cb-5h8x9   1/1     Running   0          2m23s

Build Application

Having deployed the Harness delegate, let us build a CI pipeline that will build and push the same go app to GAR.

Import Template

The sources already has build stage template that can be used to create the CI pipeline.

Navigate to your Harness Account, Account Overview --> Organizations and select default.

From the Organization overview page select Templates,

Click New Template and choose Import From Git option,

Fill the wizard with values as shown,

NOTE: If you want to use your fork of harness-apps/workload-identity-gke-demo then update Repository with your fork.

Create Pipeline

Navigate to Builds --> Pipelines, click Create Pipeline.

Click Add Stage and click Use template, choose ko_gar_build_push template that we imported earlier and click Use template to complete import.

Enter details about the stage,

Click Setup Stage to create the stage and fill other details i.e Template Inputs,

We use default namespace to run builder pods. The build pod runs with a Kubernetes Service Account(KSA) harness-builder.

NOTE:
The harness-builder KSA is mapped to Google IAM Service Account(GSA) harness-delegate to inherit the GCP roles using Workload Identity in this case to push the images to Google Artifact Registry(GAR).

Click Run to run the pipeline to see the image build and pushed to GAR,

As successful run would have pushed the image into GAR in this example its asia-south1-docker.pkg.dev/pratyakshika/demos/lingua-greeter:latest

Cleanup

To clean up all the Google Cloud resources that were created as part of this demo,

task destroy

Summary

By using Workload Identity Delegate we have simplified and secured our CI pipelines which can now use any Google API services by configuring the GSA with right roles/permissions. The CI SaaS platform no longer need to store/update the Google API credentials.

Having deployed Workload Identity Delegate you can also do keyless signing of your container images using Google Application Credentials. For more info check cosign.

For more tutorials and documentation please visit https://developer.harness.io.

Applying Workload Identity With A Demo

Kamesh Sampath — Mon, 13 Mar 2023 04:20:19 +0000

Overview

As part of the first part of the series we understood what is Workload Identity. In this DIY blog we will apply Workload Identity to our GKE workloads by deploying a demo application called lingua-greeter to GKE, the lingua-greeter will call Translate API to translate the greeting text passed to it.

Glossary

Abbreviation	Expansion
API	Application Programming Interface
ACL	Access Control List
GCS	Google Cloud Storage
GKE	Google Kubernetes Engine
GSA	Google Service Account
IAM	Identity and Access Management
KSA	Kubernetes Service Account
RBAC	Role Based Access Control
SA	Service Account
VPC	Virtual Private Cloud

Pre-requisites

Google Cloud Account
With a Service Account with roles:
- Kubernetes Engine Admin - to create GKE cluster
- Service Account roles used to create/update/delete Service Account
- iam.serviceAccounts.actAs
- iam.serviceAccounts.get
- iam.serviceAccounts.create
- iam.serviceAccounts.delete
- iam.serviceAccounts.update
- iam.serviceAccounts.get
- iam.serviceAccounts.getIamPolicy
- iam.serviceAccounts.setIamPolicy Or simply you can add Service Account Admin and Service Account User roles
- Compute Network Admin - to create the VPC networks
Enable Translation API on your Google Cloud Account
Google Cloud SDK
terraform
kubectl
Taskfile

Optional

kustomize(Optional)
direnv(Optional)

Download Sources

git clone https://github.com/kameshsampath/workload-identiy-gke-demo && cd "$(basename "$_" .git)"
export DEMO_HOME="$PWD"

Environment Setup

Variables

export GOOGLE_APPLICATION_CREDENTIALS="the google cloud service account key json file to use"
export CLOUDSDK_ACTIVE_CONFIG_NAME="the google cloud cli profile to use"
export GOOGLE_CLOUD_PROJECT="the google cloud project to use"
export KUBECONFIG="$DEMO_HOME/.kube"

(e.g.)

export CLOUDSDK_ACTIVE_CONFIG_NAME=personal
export GOOGLE_APPLICATION_CREDENTIALS=~/.ssh/my-sa-key.json
export GOOGLE_CLOUD_PROJECT=my-awesome-project
export KUBECONFIG="$DEMO_HOME/.kube"

TIP If you are using direnv you can then create file .envrc.local and add the environment variables. They can then be loaded using direnv allow .

You can find more information about gcloud cli configurations at https://cloud.google.com/sdk/docs/configurations.

We will be using terraform to create the Google Cloud resources e.g. GKE Cluster with Workload Identity enabled, Google Service Accounts(GSA), IAM policies and bindings.

As you may need to override few terraform variables that you don't want to check in to VCS, add them to a file called .local.tfvars. Set the following environment variable to make terraform use the variable values form the file .local.tfvars

export TFVARS_FILE=.local.tfvars

Check the Inputs section for all possible terraform variables that are configurable.

Example

An example .local.tfvars that will use a Google Cloud project my-awesome-project, create a two node GKE cluster named wi-demo in region asia-south1 with Kubernetes version 1.24. from stable release channel. The machine type of each cluster node will be e2-standard-4. The demo will be deployed in Kubernetes namespace demo-apps, will use lingua-greeter as the Kubernetes Service Account.

app_ksa            = "lingua-greeter"
app_namespace      = "demo-apps"
cluster_name       = "wi-demo"
configure_app_workload_identity = false
gke_num_nodes      = 2
kubernetes_version = "1.24."
machine_type       = "e2-standard-4"
project_id         = "my-awesome-project"
region             = "asia-south1"
release_channel    = "stable"

NOTE: For rest of the section we assume that your tfvars file is called .local.tfvars

Application Overview

As part of the demo, let us deploy a Kubernetes application called lingua-greeter. The application exposes a REST API /:lang , that allows you to translate a text Hello World! into the language :lang using Google Translate client.

NOTE: The :lang is the BCP 47 language code.

Create Environment

We will use terraform to create a GKE cluster with WorkloadIdentity enabled for its nodes,

Initialize terraform and download its modules,

task init

Create GKE cluster

The terraform apply will creates a Kubernetes(GKE) Cluster,

task create_cluster

The terraform apply will create the following Google Cloud resources,

A Kubernetes cluster on GKE
A Google Cloud VPC that will be used with GKE
GKE is configured to use Workload Identity Pool. As refresher check How Workload Identity Works.

Deploy Application

To see Workload Identity in action we will deploy the application(workload) on to GKE in two parts,

Application is not enabled for Workload Identity
Application enabled for Workload Identity

Without Workload Identity Enabled

Create the namespace demo-apps to deploy the lingua-greeter application,

kubectl create ns demo-apps

Run the following command to deploy the application,

kubectl apply -n demo-apps -k $DEMO_HOME/app/config

Wait for application to be ready,

kubectl rollout status -n demo-apps deployment/lingua-greeter --timeout=60s

Get the application service LoadBalancer IP,

kubectl get svc -n demo-apps lingua-greeter

NOTE: If the EXTERNAL-IP is <pending> then wait for the IP to be assigned. It will take few minutes for the EXTERNAL-IP to be assigned.
You can use the following command to wait until External-IP is assigned,
while [ -z $(kubectl get svc -n demo-apps lingua-greeter -ojsonpath="{.status.loadBalancer.ingress[*].ip}") ]; do sleep .3; done;

Call Service

export SERVICE_IP=$(kubectl get svc -n demo-apps lingua-greeter -ojsonpath="{.status.loadBalancer.ingress[*].ip}")

Call the service to return the translation of Hello World! in Tamil(ta),

curl "http://$SERVICE_IP/ta"

The service should fail with a message,

{"message":"Internal Server Error"}

When you check the logs of the lingua-greeter pod,

kubectl logs -n demo-apps -lapp=lingua-greeter

You should see a message like,

 ____    __
  / __/___/ /  ___
 / _// __/ _ \/ _ \
/___/\__/_//_/\___/ v4.10.0
High performance, minimalist Go web framework
https://echo.labstack.com
____________________________________O/_______
                                    O\
⇨ http server started on [::]:8080
time="2023-03-10T07:36:35Z" level=error msg="googleapi: Error 401: Request had invalid authentication credentials. Expected OAuth 2 access token, login cookie or other valid authentication credential. See https://developers.google.com/identity/sign-in/web/devconsole-project.\nMore details:\nReason: authError, Message: Invalid Credentials\n"

As it describes you don't have authentication credentials to call the API. All Google Cloud API requires GOOGLE_APPLICATION_CREDENTIALS to allow client to authenticate itself before calling the API. If you check the deployment manifest we dont have one configured.

Configure Application To Use Workload Identity

Run the following command to configure the application to use Workload Identity,

task use_workload_identity

The terraform script rbac.tf does the following,

Create a GSA called translator
Add role roles/iam.workloadIdentityUser to translator with a single member serviceAccount:$GOOGLE_CLOUD_PROJECT.svc.id.goog[demo-apps/lingua-greeter]. This basically allows KSA to impersonate itself as GSA translator thereby allowing it to call Google Cloud services that are allowed for GSA translator, in this case to use Google Translate API.
Add IAM policy binding to translator for role roles/cloudtranslate.user which allows it to call the Google Translate API.
Finally an updated lingua-greeter KSA manifest $DEMO_HOME/k8s/sa.yaml, that is annotated with the GSA client_email that it is allowed to use, in this case it will be something like translator@$GOOGLE_CLOUD_PROJECT.iam.gserviceaccount.com.

apiVersion: v1
kind: ServiceAccount
metadata:
  name: lingua-greeter
  namespace: demo-apps
  annotations:
    iam.gke.io/gcp-service-account: "translator@$GOOGLE_CLOUD_PROJECT.iam.gserviceaccount.com"

Run the following command to update the Kubernetes SA lingua-greeter to use the Google IAM Service Account(GSA) translator using Workload Identity mechanics and call the Google Translate API,

kubectl apply -n demo-apps -f "$DEMO_HOME/k8s/sa.yaml"

Call the service again, the service should succeed with a response,

{"text":"Hello World!","translation":"வணக்கம் உலகம்!","translationLanguage":"ta"}

NOTE: Sometimes it may take few seconds for the pods to refresh the metadata, in such cases try to call the service after few seconds.

Cleanup

To clean up all the Google Cloud resources that were created as part of this demo,

task destroy

Summary

Deploy GKE cluster with Workload Identity enabled
Deploy lingua-greeter application to GKE
Create Google Service Account translator with permissions to call Google Translate API
Annotating Kubernetes Service Account lingua-greeter with Google Service Account translator allowing it to impersonate Google Service Account.

What is Workload Identity ?

Kamesh Sampath — Mon, 13 Mar 2023 04:09:15 +0000

Awesomeness of using Cloud and its ecosystem is that we don't need to reinvent the wheel, in other words no need to build/deploy services e.g. database, message queue,AI/ML etc. For any service need the APIs are just a click away.

Glossary

Abbreviation	Expansion
API	Application Programming Interface
ACL	Access Control List
GCS	Google Cloud Storage
GKE	Google Kubernetes Engine
GSA	Google Service Account
IAM	Identity and Access Management
KSA	Kubernetes Service Account
RBAC	Role Based Access Control
SA	Service Account
VPC	Virtual Private Cloud

Google Cloud provides a rich set of services(API) e.g. GKE, Translation, Pub/Sub etc., which can cater most of the distributed application needs with less or no effort.

GKE is one of the most commonly used Google Cloud Service that allows you to deploy Cloud Native services swiftly. In many occasions those applications deployed to GKE tend to leverage the services offered by Google Cloud e.g our application need to call Google Cloud Translation service, publish a message to Pub/Sub topic or upload/download files from GCS.

By virtue of GKE running on a Google Cloud infrastructure, the Kubernetes pods on GKE can't access any Google Cloud service at its wish and will.

Every API consumer in this case a GKE application needs to be authenticated and authorised to call a Google Cloud API. Google IAM provides mechanisms to generate credentials for consumers. For application consumers it is always recommended to use a SA, as Service Accounts are capable of being used as :

An principals - Access to Google API can be granted directly to a principal
A resource - Other principals e.g User, Group etc., can be added to a Service Account with specific role scopes.

Check out Service Account Permissions for more details.

Google Service Account Keys

Great! We understood that every consumer needs a Google IAM credentials to call Google Cloud Service API.

How do API consumers get this API Credential?

There few ways to do that, for this blog series we will use Google IAM Service Account(GSA). Each GSA can have one or more JSON Keys called the SA key. The SA key file has critical information like private_key_id, private_key, client_email, client_id etc., that will be used to identify the SA with with Google Cloud IAM and make necessary service calls.

IMPORTANT:

The JSON key needs to be stored securely

The Google IAM policies determines what authorisations are available for a associated GSA.

Using GSA JSON Key

All Google API client libraries by default looks for an environment variable GOOGLE_APPLICATION_CREDENTIALS, uses it perform required authentication and authorisation. So as developer once we have access to the SA JSON key we set the path of the file as the value of GOOGLE_APPLICATION_CREDENTIALS environment variable.

On GKE these credentials are usually stored as Kubernetes Secrets . The application pods can then mount on these secrets as files
and set the environment variable GOOGLE_APPLICATION_CREDENTIALS with the mounted file path.

Though this way of using static key file is quick and fast it lacks security and manageability:

The keys do not have an expiry and has to be manually rotated.
Key rotation and compromise has cascading effect i.e if the keys are compromised, it needs to regenerated and shared with all consumers.
On cloud its recommended to follow Principle of least privilege, which means a GSA be given very few permissions. So if your applications accesses multiple services then it might be required to generate multiple keys one for each service and it becomes harder to rotate all those static keys.

Is there a keyless way to call the APIs?

All that static key file does is to identify the caller using the details from the JSON key file with Google Cloud platform. Once its identity is proven a i.e. knowing the associated GSA; a short living access token is generated and shared with the consumer. The consumer can then use that token for making all authorised API calls.

So if the application pods i.e. workload can identify itself by some mechanism, then we might not need static key. Thats exactly what Workload Identity is used for.

Workload Identity allows a Kubernetes service account in your GKE cluster to act as an Google IAM Service account. Pods that use the configured KSA automatically authenticate as the IAM service account when accessing Google Cloud APIs.

When GKE is enabled with Workload Identity, the fixed workload identity pool helps Google Cloud IAM to understand the KSA and GSA associated with it i.e. KSA impersonates itself as GSA. With this impersonation the application running with that KSA can call the Google services as the GSA.

Check the official docs for more details on how Workload Identity works under the hoods.

So with Workload Identity we can work through the drawbacks of static key file usage and in addition:

Provide fine grained access control to each application with KSA and GSA combination.
The Workload Identity uses Short Lived Tokens than Long Living Static Keys thereby adding more security.
The application workloads no longer need to configure the GOOGLE_APPLICATION_CREDENTIALS and assume right keys are available for it.
For Ops the ACL/RBAC can be centrally managed from Google Cloud Web Console via the Google Cloud IAM

Summary

We need Google IAM Credentials to call any Google API
How to create Google Service Account Key(JSON) i.e. static key
How to use Google Service Account Key to call Google API
Use Google IAM Roles/Permissions is used to restrict what APIs a Google Service Account can call
How to Google Service Account keys are used by GKE application Pods
Finally What is a Workload Identity and how it helps in keyless API invocations

In next part of this series let us see how to apply Workload Identity with a demo.

Continuously Integrate Go Applications

Kamesh Sampath — Mon, 06 Mar 2023 16:05:59 +0000

At the end of this tutorial you will learn,

How to build Go application container image without using a Dockerfile
What are Secrets and how to add them to your Project
What are Connectors and how to add a Docker Registry Connector to your Project

Pre-requisites

Before you get started with the tutorial make sure you have the following accounts, credentials and tools,

A GitHub account, where you may need to fork the tutorial sources.
A Docker Registry account e.g DockerHub, Quay.io
A Harness CI free tier account.

The following tools are required to try building the sources locally for test and verification,

Drone CLI to build the application locally.
Docker Desktop

Overview

As part of this tutorial we will be building a simple Go REST API called fruits-api. The application uses a RDBMS(PostgreSQL or MySQL) or NOSQL(MongoDB) to store the fruits data.

Tutorial Source

The complete demo source is available here https://github.com/harness-apps/go-fruits-api, fork the repository on to your GitHub account. For rest of the tutorial we will refer to this repository as $TUTORIAL_GIT_REPO.

Building Application Locally

Languages and package formats have build specific tools. One of the core problems that a developer might face is to install the right version of those tools on their local machines. This approach has potential pit falls and leads to Works only on my machine scenarios.

Docker containers solved this problem and helped us to have clean environment that had right set of tools, encouraging the DevOps best practices right from the start. This approach also helps to identify the potential issues with the application at development stage.

Drone by Harness is an open source CI platform that can help building and testing on your local machines without the need of installing the tools as required by the programming languages.

But before we start to build the application, we need place to store the artifacts of the build i.e. container images. In container/Cloud Native world this is called a Container Registry e.g Docker Hub, Quay.io, Harbor etc.,

Configure Container Registry

Like any file you want to share with the world, storing them in an external spot makes them more accessible. A big benefit of using containers as a packaging format is the ecosystem of container registries out there. Your firm might have a registry provider such as Docker Hub, Quay.io, Harbor, Google Container Registry(GCR), Elastic Container Registry(ECR) etc.,

For this tutorial we will be using Docker Hub. If you do not have a registry available to you, you can create a Docker Hub account and then create a repository fruits-api, where we will push our fruits-api application container image.

With us having created the fruits-api repository, lets test our repository by building and pushing the image to the registry,

echo -n "$DOCKER_HUB_PASSWORD" |\
  docker login -u `$DOCKER_HUB_USERNAME` --password-stdin

Info

$DOCKER_HUB_USERNAME - Docker Hub username, the one you used while registering the for the Docker Hub account or the one you wish to use if you already have an account with Docker Hub.

$DOCKER_HUB_PASSWORD - Docker Hub user password

Let us clone the tutorial application from https://github.com/harness-apps/go-fruits-api,

#  clone go-fruits-api repository
git clone https://github.com/harness-apps/go-fruits-api.git \
  && cd "$(basename "$_" .git)"
# navigate to the clone repository folder
export TUTORIAL_HOME="$PWD"

Tip

GitHub Cli is very handy tool to work with the GitHub repositories from the command line.

Create your fork of the tutorial repository,

gh repo fork

Info

You can also create your fork from the tutorial repository https://github.com/harness-apps/go-fruits-api directly from GitHub.

To make things simple let use Drone by Harness to build and push the image from your laptops to the Docker Hub repository fruits-api,

Copy $TUTORIAL_HOME/.env.example to $TUTORIAL_HOME/.env,

cp $TUTORIAL_HOME/.env.example $TUTORIAL_HOME/.env

Edit the $TUTORIAL_HOME/.env and update it with following,

PLUGIN_REGISTRY=docker.io
PLUGIN_USERNAME=$DOCKER_HUB_USERNAME
PLUGIN_PASSWORD=$DOCKER_HUB_PASSWORD
PLUGIN_REPO=$DOCKER_HUB_USERNAME/fruits-api
PLUGIN_TAG=0.0.1

Info
Replace the $DOCKER_HUB_USERNAME, DOCKER_HUB_PASSWORD with your docker hub username and password values.

drone exec --env-file=.env

Info
It will take few mins for the build and push to complete as Drone will try to pull the container images if not exists.

If all went well your command line output(trimmed for brevity) should like,

...
[push:350] The push refers to repository [docker.io/$DOCKER_HUB_USERNAME/fruits-api:0.0.1]
[push:351] 639e874c7280: Preparing
[push:352] 96e320b34b54: Preparing
[push:353] c306578afebb: Preparing
[push:354] 96e320b34b54: Layer already exists
[push:355] c306578afebb: Pushed
[push:356] 639e874c7280: Pushed
...

You can check the pushed image at https://hub.docker.com/repository/docker/$DOCKER_HUB_USERNAME/fruits-api.

Tip
You can use tools like crane, that allows you check the image and its tags from cli
e.g.
 crane ls docker.io/$DOCKER_HUB_USERNAME/fruits-api

Simple enough locally to get your local build and packaging in. Our process to build and push the go application looks like,

These sequence of steps is referred to as a Pipeline in Continuous Integration(CI) world.

The drone pipeline build and push step uses ko-build which can build go container images without the need for Dockerfile. It also allows you to build the multi arch/platform images with much ease.

The drone exec that we did earlier is OK as long you are playing/learning a technology in other words laptop use cases, when you are working on a team to deliver some enterprise application then it becomes super critical that this process be centralized and automated. Harness Platform helps you do exactly that and much more.

The next sections this tutorial helps you get started on the building your CI Pipeline using Harness platform.

Your First Continuous Integration Pipeline

If you took a closer look at what your machine was doing during those local builds, the machine was bogged down for a few moments. For yourself, that is fine, but imagine having to support 10’s or 100’s or even 1000’s of engineers, this process can be taxing on systems. Luckily, modern Continuous Integration Platforms are designed to scale with distributed nodes. Harness Continuous Integration is designed to scale and simplify getting your local steps externalized; this is the Continuous Integration Pipeline. Let’s enable Harness Continuous Integration to mimic your local steps and create your first CI Pipeline. Once you are done, you will have a repeatable, consistent, and distributed build process.

There are a few Harness resources to create along the way, which this guide will walk through step-by-step.There are two paths to take. One path is to have Harness host all of the needed infrastructure for a distributed build. The second is to bring your own infrastructure for the distributed build.

Hosted Infrastructure:

Bring Your Own Infrastructure:

For this tutorial we will be using the Hosted Infrastructure as thats the only infrastructure available for Free Tier.

Starting off with Harness

Harness is a Platform which has lot of modules, but for this tutorial we will focus on the Continuous Integration(CI) module.

First, sign up for a Harness account to get started.

GitHub Personal Access Token(PAT)

Assuming you are leveraging GitHub, Harness will need access to the repository. It is recommended to use GitHub Personal Access Token(PAT) as a mode of providing Github credentials.

If you have not created a PAT before, on your GitHub account navigate to Settings -> Developer Settings -> Personal Access Tokens.

Important

Make sure to jot down the token as the token will only be displayed once. For rest of the tutorial we will refer to this token value as $GITHUB_PAT.

If you plan to bring in your PAT then make sure it has the scopes admin:repo_hook and user.

Create Project

Harness Platform organizes the resources like pipelines, secrets, connectors at various scopes such as Account, Organization and Project. For this tutorial we will create all our resources at Project scope.

On the new project page, click Create Project to create a new project named Fruits API.

Leave other options to defaults and click Save and Continue. On the modules select Continuous Integration,

Now you are ready to wire in the pieces to Harness Continuous Integration.

Create Your First Pipeline

In the Build Module Harness Continuous Integration, walking through the wizard is the fastest path to get your build running. Click Get Started. This will create a basic Pipeline for you.

Click Get Started, select GitHub as the repository to use, and enter your GitHub Access Token $GITHUB_PAT and finally click Test Connection to verify your credentials work,

Click Continue, click Select Repository to select the Git Hub Repository that you want to build [the sample is called go-fruits-api].

Info
Please ensure the repository you select here is your fork of https://github.com/harness-apps/go-fruits-api.

Can leverage one of the Starter Configs or create a Starter Pipeline. In this case if leveraging the example app which is Go based, leveraging the Go Starter Configuration works fine.

Click Create Pipeline to start adding the pipeline steps.

There are two ways to add your pipeline steps, visual or YAML. For rest of the tutorial we will use the visual editor.

The scaffolding would have added a single step called Build Go App. In the upcoming sections we will add the other steps like lint, test and push.

Before we get to adding other steps, we need some resources that the steps require namely secrets and connectors.

Create Docker Hub Password Secret

Navigate to Project Setup --> Secrets,

Click + New Secret and select Text,

Fill your Docker Hub password on the Add new Encrypted Text window,

Create Docker Hub Registry Connector

Next let we need to add Connector that allows us to connect and later push the image to our Docker Hub repository.

Navigate to Project Setup --> Connectors,

Click + New Connector and select Docker registry,

On the new connector wizard Overview screen, enter the name of the connector as docker hub,

Click Continue to configure the credentials,

Info

Update the Username with your $DOCKER_HUB_USERNAME

For the Password field click Create or Select a Secret to select the secret docker hub password.

Click Continue and use the Harness Platform as the connectivity mode option,

Click Save and Continue to perform the connectivity test,

Click Finish to complete the creation of Connector resource.

Now you are all set to add other steps to the Build Go pipeline.

Update Pipeline

Navigate to the Projects --> Pipelines,

Click Build Go pipeline,

Delete the existing Build Go App step by clicking the x that appears when you hover over the step.

Click Save to save the pipeline.

Click Add Step to add a new step called lint, from the Step Library choose step type as Run and configure the step with details:

Name:

lint

Description:

Lint the go application

Select the Shell to be Bash.

Command:

golangci-lint run

Click Apply Changes to save the step and click Save to save the pipeline.

As did earlier click Add Step to add a new step called test, from the Step Library choose step type as Run and configure the step with details:

Name:

test

Description:

Test the go application

Select the Shell to be Bash.

Command:

go test -timeout 30s -v ./...

While building the application locally we used SQLite as our database. The go application can also run with PostgreSQL or MySQL or Mongodb. For this tutorial we will be using MySQL.

For the test step to connect to the mysql service add the following environment variables to the step configuration.

FRUITS_DB_TYPE: mysql
MYSQL_HOST: "mysql"
MYSQL_PORT: 3306
MYSQL_ROOT_PASSWORD: superS3cret!
MYSQL_PASSWORD: pa55Word!
MYSQL_USER: demo
MYSQL_DATABASE: demodb

The environment variables could be added by clicking + Add under Environment Variables section of the step configuration,

Click Apply Changes to save the step.

Tip
You can awake step configuration screen by clicking the step on the visual editor.

Click Save to save the pipeline.

How can the test step connect to MySQL database ?

Harness Pipelines support a concept called as Service Dependency, it is a detached service that's accessible to all Steps in a Stage. Service dependencies support workflows such as

Integration testing: You can set up a service and then run tests against this service.
Running Docker-in-Docker: You can set up a dind service to process Docker commands in Run Steps.

In our tutorial we will use the Integration testing workflow to make the test step to connect to MySQL and run the integration test cases against it.

Add the MySQL Service Dependency

On the Pipeline editor click Add Service Dependency,

Configure the MySQL Dependency Service with details:

Name:

mysql

Description:

the mysql or mariadb server that will be used for testing.

Select the Container Registry to be docker hub.

Image:

mariadb

The service dependency need to be configured with the same environment variables that we added to test step.

MYSQL_PORT: 3306
MYSQL_ROOT_PASSWORD: superS3cret!
MYSQL_PASSWORD: pa55Word!
MYSQL_USER: demo
MYSQL_DATABASE: demodb

Click Apply Changes to save the step and then click Save to save the pipeline.

Lint and Test the Application

Let us verify if were able to lint and test our go application.

Click Run from the pipeline editor page,

Leaving everything to defaults namely Git Branch and Branch Name to be main, click Run Pipeline to start the pipeline run. If all ran well you should see a successful pipeline run as shown,

Tip
You can click on each step to view the logs of the respective step

Having tasted the success with our pipeline run, let us add the other step of building and pushing the go application to the container registry.

Build and Push Image to Container Registry

As did earlier navigate to the Projects --> Pipelines,

And click Build Go pipeline to open the pipeline editor,

Click Add Step to add a new step called build and push, from the Step Library choose step type as Run and configure the step with details,

Name:

build and push

Description:

Build go application

Choose Bash to be the Shell

Command:

echo -n "$DOCKER_HUB_PASSWORD" | ko auth login docker.io -u "$DOCKER_HUB_USERNAME" --password-stdin
ko build --bare --platform linux/amd64 --platform linux/arm64 .

We also need to configure few environment variables that are required by ko to build and push the image to fruits-api container repository.

Update the Environment Variables section with following values,

DOCKER_HUB_USERNAME: $DOCKER_HUB_USERNAME
DOCKER_HUB_PASSWORD: <+secrets.getValue("docker_hub_password")>
KO_DOCKER_REPO: docker.io/$DOCKER_HUB_USERNAME/fruits-api

Info

As marked ensure the DOCKER_HUB_PASSWORD is of type Expression

secrets.getValue is an expression that allows to get the value from the secret docker_hub_password, that was created earlier in the tutorial. Check the docs for more info

All $DOCKER_HUB_USERNAME references should your Docker Hub Username

Click Apply Changes to save the step and click Save to save the pipeline.

With those changes saved, you are ready to lint, test, build and push your go application to container registry(DockerHub).

Run CI Pipeline

As did earlier click Run from the pipeline editor window,

Leaving everything to defaults namely Git Branch and Branch Name to be main, click Run Pipeline to start the pipeline run.

Now you are ready to execute. Click "Run Pipeline".

Once a successful run, head back to Docker Hub, and tag latest is there!

This is just the start of your Continuous Integration journey. It might seem like multiple steps to get your local build in the platform, but it unlocks the world of possibilities.

Exercise

The https://github.com/harness-apps/go-fruits-api has another branch mongodb. Adapt your pipeline so that it build and test the code from mongodb branch.

Continuing on Your Continuous Integration Journey

You can now execute your builds whenever you want in a consistent fashion. Can modify the trigger to watch for SCM events so upon commit, for example, the Pipeline gets kicked off automatically. All of the objects you create are available for you to re-use. Lastly, you can even save your backing work / have it as part of your source code. Everything that you do in Harness is represented by YAML; feel free to store it as part of your project.

After you have built your artifact, the next step is to deploy your artifact. This is where Continuous Delivery steps in and make sure to check out some other CD Tutorials.

Simplify Your Dockerfile

Kamesh Sampath — Mon, 20 Feb 2023 09:19:23 +0000

With rustlang gaining lots of popularity, I thought to give it a try. As cloud native application developer, the first thing I thought was to build was simple REST API that greets the user by name.

After building the application locally, the next immediate was to containerise it. Though we have an official rustlang image, I want to build a customised image that will allow me to build cross platform container images namely linux/arm64 and linux/amd64. I don't want to dwell into those details as that demands its own blog post ;).

So my Dockerfile with all rust specific tools and dependencies looks like:

The Dockerfile has two stages bins and final. bins is used to do all binary builds that are required by final stage. The final stage builds the final image that can be used with to build multi-arch container images out of our rust applications.

Though it is not a complicated Dockerfile, if you notice the last RUN instruction of the final stage, it is complex and hard to debug typically when one of the commands fails to run. It is also hard read and understand the RUN instruction. Doing multiple RUN instructions to split the commands, is not recommended as it creates new layers and your your final image will be bloated in size.

NOTE: rustlang builder images are usually bigger ~ 700mb(compressed) by virute of dependencies that it needs e.g gcc, cross compilation linkers etc.,

I was then thinking of ways to simplify this Dockerfile though not from size point of view but atleast making it simple to read and understand.

The target was to to have one RUN instruction but to split the commands into individual steps without compromising on the size.

I then stumbled upon Taskfile -- Task is a task runner / build tool -- which is similar to [GNU Make](https://www.gnu.org/software/make/) but way simpler. Taskfile helped me to make the Dockerfile simpler.

I kind of moved the whole set RUN instruction into a Taskfile:

Though it is verbose, but helps in understanding commands we are running as part of the Docker build. With descriptions, comments, conditions it becomes more powerful and self documented in explaining what is being executed and when it will be executed.

Updating the Dockerfile file results in:

#syntax=docker/dockerfile:1.3-labs

FROM --platform=$TARGETPLATFORM rust:1.67-alpine3.17 AS bins

ARG TARGETPLATFORM

RUN --mount=type=cache,target=/usr/local/cargo/registry \
  apk add -U --no-cache alpine-sdk gcompat go-task \
  && cargo install cargo-zigbuild

## The core builder that can be used to build rust applications
FROM --platform=$TARGETPLATFORM alpine:3.17 as final

ARG TARGETPLATFORM
ARG rust_version=1.67.1
ARG rustup_version=1.25.2
ARG user_id=1001
ARG user=builder

ENV USER_ID=$user_id \
  USER=$user \
  RUST_VERSION=$rust_version \
  RUSTUP_VERSION=$rustup_version \
  RUSTUP_HOME=/usr/local/rustup \
  CARGO_HOME=/usr/local/cargo \
  PATH=/usr/local/cargo/bin:$PATH \
  RUST_VERSION=1.67.1

COPY --from=bins /usr/bin/go-task /usr/local/bin/task

COPY --from=bins /usr/local/cargo/bin/cargo-zigbuild /usr/local/cargo/bin/

COPY tasks/Taskfile.root.yaml ./Taskfile.yaml

RUN task

As we moved all our commands to Taskfile the RUN instruction now has to just run the task command, which will then run the default from the Taskfile.

To summarise we,

Wrote a multi stage Dockerfile to build multi arch rust app container
Moved all the instructions from RUN to Taskfile
Used the task command in RUN
Moving commands to Taskfile allows us to run/test the tasks separately before using them in Dockerfile. For more usage check the TaskFile documentation.

For a end to end example refer to rust-greeter that uses rust-zig-builder.

Build and sign application containers

Kamesh Sampath — Thu, 19 Jan 2023 07:49:42 +0000

Overview

Open Source software has been heart of every software development model. With its increased usage means the software the are built are susceptible to threats and vulnerabilities.

What is the Supply Chain problem ?

Supply Chain problem is how and where vulnerabilities are introduced into the software supply chain. It is usually introduced at Source or Dependencies level and gets seeped into the software artifact consumed by the end user a.k.a the Consumer.

The threats could be introduced at two levels which forms the basis of software integrity,

Source Integrity
- Compromised Source Repo
- Unauthorised Source Code change
Build Integrity
- Build from modified source
- Compromise build process
- Use compromised dependency
- Upload modified package
- Compromise package repo
- Use compromised package

SLSA

The Supply chain Levels Software Artifact(SLSA) puts a security framework in place that each software build can follow, ensuring the integrity of the built artifact.

There are four Levels of maturity in SLSA,

Level	Description	Example
1	Documentation of the build process	Continuous Integration(CI)
2	Tamper resistance of the build service	Hosted source/build
3	Extra resistance to specific threats	Security controls on host
4	Highest levels of confidence and trust	Two-party review + hermetic builds

As with any process, maturing with SLSA levels is a continuous improvement process. As part of the blog and a simple tutorial using Harness Platform, which will allow us to document our build process(SLSA Level 1).

IMPORTANT: All the levels requires us to have build Provenance, since it deserves its own blog post let us revisit it as part of another blog post. If you want to learn about provenance please do visit this great blog.

Tutorial

With containers being the heart of Cloud Native application development, it has become even more critical to ensure the integrity of the containers. One of the ways to do this to sign and verify the container images.sigstore is a open source project that empowers software developers to securely sign the container images.

As part of this tutorial we will,

Understand how to build container image, sign/verify the image using sigstore cosign utility
Integrate cosign as part of Continuous Integration(CI) using Harness CI.

Summary

Signing alone is not sufficient to ensure the overall security of any software, adopting SLSA and continuous improvement of the build process(SLSA levels) is very critical. By using Harness Platform we documented our build process and also implicitly started to move towards SLSA Level 2 by using a Host source (GitHub) and build(Harness CI).

Simplify Golang Multi Architecture Container Builds

Kamesh Sampath — Wed, 26 Oct 2022 01:47:10 +0000

With arm64 based laptops getting very popular it has become a need for the container developers to build a multi architecture images e.g. build amd64 on arm64 machine.

In my case I use Apple M1, as part of my day job I build,test and deploy container applications; by default my macbook produces linux/arm64 images. If I need to deploy the same image to any cloud services e.g. Google Cloud Run or Google Kubernetes Engine , then I need to have the linux/amd64 image as well.

For such use cases the we usually write multiple Dockerfile for each architecture, a manifest file etc., there nothing wrong with that approach but then it will soon become too hard to maintain.

So what we need is a tool chain that can,

Have same process to build image locally as well as for cloud usage. I found GoReleaser to be apt for this requirement.
Is declarative, typically mapping to standard steps that will be part of building a golang application i.e. test, build and push image to container registry. Drone help to setup a build pipeline in declarative way and has plugins for all major tools/platforms including one for GoReleaser.
Build multi architecture images. Buildx helps us to build multi architecture image using the standard docker build semantics. Adding to our tools there is a Drone plugin Drone Buildx Plugin that allow us to just plug it into our pipeline.

The end to end demo of this blog post is available on my GitHub Repo.

Clone it one to your laptop for quick reference,

git clone https://github.com/kameshsampath/go-hello-world.git \
&& cd "$(basename "$_" .git)"
export DEMO_HOME="$PWD"

For the rest of the post we will refer to this cloned folder as $DEMO_HOME.

For this demo we will be using a short lived container registry called ttl.sh for pushing and pulling the demo app image.
Download and Drone CLI to your path.

NOTE: You are welcome to use your own registry. Please check the Drone Buildx Plugin documentation on how to configure the extra parameters like username and password.

Let us setup some environment variables that we will be using as part of the demo

# a unique uid as image identifier, it needs to be in the lowercase
export IMAGE_NAME=$(uuidgen | tr '[:upper:]' '[:lower:]')
# short lived image for 10 mins
export IMAGE_TAG=10m

And for convenience we will save these values on to .env file which we will use to pass environment variables to Drone Pipelines and Docker runs.

envsubst < "$DEMO_HOME/.env.example" | tee "$DEMO_HOME/.env"

NOTE: The example above uses envsubst to update the file. If you don't have envsubst installed you can manually update the .env file.

Let us examine the Drone pipeline .drone.yml that we have,

It is very simple Drone Pipeline that has three steps ,

test that runs the golang tests
build that uses the GoReleaser to build the golang application
push that pushes the application to the container registry that we had configured earlier.

The push steps also specifies the platforms linux/arm64 and linux/amd64 that instructs the docker buildx build to perform multi architecture build that will produce a container image that is compatible with the respective platforms.

The push step uses the Dockerfile above to perform the build.

There are few important things to note,

The comment # syntax=docker/dockerfile:1.4 instructs the docker build to use buildx Dockerfile semantics from docker/dockerfile:1.4 . Though it does not have big significance in our case but we ensure the we use buildx to perform the builds. Buildx is default from Docker v18.x and above.
The ARG TARGETARCH allows the docker build to know the architecture thats is being build for e.g. amd64 or arm64 etc.,
The base image gcr.io/distroless/base allows to build a tiny image with just the binary of our application in it :).

That's pretty much need to build our multi architecture golang application, run the following drone command to start that pipeline that will build push the image to the container registry.

drone exec --trusted --env-file=.env

Once the build is done run the following command to start the built container locally,

docker-compose up

Doing a simple curl to http://localhost:8080 that should say "Hello World".

Lastly, a small caveat when using GoReleaser. GoReleaser, when cross compiling the code, creates distribution folders like linux_arm64 and linux_amd64_v1 under the dist folder, where the built binaries for the platforms (linux/arm64 and linux/amd64) are saved.

The folder names follows a pattern like <platform>_<arch>_<version> with version being optional. For amd64 GoReleaser creates a binary folder like linux_amd64_v1 that made it difficult to pick the binary by using the arg TARGETARCH from within the Dockerfile; as TARGETARCH just returns only the arch name without version. In our case just amd64.

As a workaround we need to write GoReleaser build post hook script as shown below that will rename the folder linux_amd64_v1 to linux_amd64 which then allows us to pick the right binary based on the architecture using the TARGETARCH,

COPY server_linux_${TARGETARCH}/server /bin/server

NOTE Special thanks to my son Rithul Kamesh who volunteered to write that little script for me 😊.

To summarize what we did,

Use Buildx to build multi architecture container images
Leveraged Drone Pipelines to define a declarative builds
Used GoReleaser to have standard configuration to build golang applications
Finally leveraged GoReleaser, Buildx via Drone plugins to perform reproducible multi architecture builds

I have also written Drone CI Docker Extension that allows you run this demo or any other Drone pipelines right from your Docker for Desktop. If you have minute please do try it out and let me know your valuable feedback/improvements.

Continuous Integration with Drone on Kubernetes

Kamesh Sampath — Thu, 21 Jul 2022 11:21:00 +0000

Overview

Over the past few years, lots of organizations have started to adopt Cloud Native architectures. Despite the adoption of Cloud Native architectures, many companies haven’t achieved optimal results. Wondering why? One of the reasons is our adherence to traditional ways of building and deploying Cloud Native applications.

Kubernetes has become the de facto Cloud Native deployment platform, solving one of the main Cloud Native problems: "deploying" applications quickly, efficiently and reliably. It offers radically easy scaling and fault tolerance. Despite this, not many Continuous Integration(CI) systems utilize the benefits of Kubernetes. None of the existing build systems offer the capabilities that are native to Kubernetes like in-cluster building, defining the build resources using CRDs, leveraging underlying security and access controls, etc. These missing features of Kubernetes made the Cloud Native architectures to be less effective and more complex.

Let me introduce an Open Source project Drone -- a cloud native self-service Continuous Integration platform -- . 10 years ago old Drone was the first CI tool to leverage containers to run pipeline steps independent of each other., Today, with over 100M+ Docker pulls, and the most GitHub stars of any Continuous Integration solution, Drone offers a mature, Kubernetes based CI system harnessing the scaling and fault tolerance characteristics of Cloud Native architectures. Drone help solve the next part of the puzzle by running Kubernetes native in-cluster builds.

In this blog, let see how we setup kind and Drone together on our laptops to build Kubernetes native pipelines which could then be moved to cloud platforms like Harness CI for a broader team based development.

This blog is a tutorial where I explain the steps required to use KinD and Drone to set up CI with Kubernetes on your local machine. At the end of these steps, you will have a completely functional Kubernetes + CI setup that can help you build and deploy Cloud Native applications on to Kubernetes on your laptop.

Required tools

To complete this setup successfully, we need the following tools on your laptop,

All linux distributions adds envsubst via gettext package. On macOS, it can be installed using Homebrew like brew install gettext.

Demo Sources

The accompanying code for this blog i.e. the demo sources is available on my GitHub repo. Let us clone the same on to our machine,

git clone https://github.com/kameshsampath/drone-on-k8s && \
  cd "$(basename "$_" .git)"
export PROJECT_HOME="${PWD}"

NOTE: Through out this blog we will use the name $PROJECT_HOME to refer to demo sources folder that we cloned above .

Alright, we are all set to get started!!

## Setup Kubernetes Cluster

As said earlier, we will use kind as our local Kubernetes cluster. But for this blog we will do the following customisations,

Set up a local container registry where we can push and pull container images that will be used in our Kubernetes. Check the kind docs for more details.
Do extra port mappings to make allow us to access the Drone Server and Gitea git repository

To make things easier, all the aforementioned customisations has been compiled into a utility script $PROJECT_HOME/bin/kind.sh. To start the KinD cluster with these customisations just do,

$PROJECT_HOME/bin/kind.sh

Version Control System

Without Version Control System(VCS) CI makes no sense. One of the primary goal of this blog is to show how to run local VCS so that you can build your applications without a need for external VCS like GitHub, Gitlab etc., For our setup we will use on Gitea -- A painless, self-hosted Git service--. Gitea is so easy to set up and does provide helm charts for Kubernetes based setup.

Helm Values

The contents of the helm values file that will be used to setup Gitea is shown below. The settings are self-explanatory for more details check the cheat sheet.

service:
  http:
    # the Kubernetes service gitea-http'  service type
    type: NodePort 
    # the gitea-http service port
    port: 3000
    # this port will be used in KinD extra port mappings to allow accessing the 
    # Gitea server from our laptops
    nodePort: 30950
gitea:
  # the admin credentials to access Gitea typically push/pull operations
  admin:
    # DON'T use username admin as its reserved and gitea will 
    # fail to start
    username: demo
    password: demo@123
    email: admin@example.com
  config:
    server:
      # for this demo we will use http protocol to access Git repos
      PROTOCOL: http
      # the port gitea will listen on
      HTTP_PORT: 3000
      # the Git domain - all the repositories will be using this domain
      DOMAIN: gitea-127.0.0.1.sslip.io
      # The clone base url e.g. if repo is demo/foo the clone url will be 
      # http://gitea-127.0.0.1.sslip.io:3000/demo/foo
      ROOT_URL: http://gitea-127.0.0.1.sslip.io:3000/
    webhook:
      # since we will deploy to local network we will allow all hosts
      ALLOWED_HOST_LIST: "*"
      # since we are in http mode disable TLS
      SKIP_TLS_VERIFY: true

Add the gitea helm repo,

helm repo add gitea-charts https://dl.gitea.io/charts/
helm repo update

Run the following command to deploy Gitea,

helm upgrade \
  --install gitea gitea-charts/gitea \
  --values $PROJECT_HOME/helm_vars/gitea/values.yaml \
  --wait

A successful deployment of gitea should show the following services in the default namespace when running the command,

kubectl get pods,svc -lapp.kubernetes.io/instance=gitea

NAME                                  READY   STATUS    RESTARTS   AGE
pod/gitea-0                           1/1     Running   0          4m32s
pod/gitea-memcached-b87476455-4kqvp   1/1     Running   0          4m32s
pod/gitea-postgresql-0                1/1     Running   0          4m32s

NAME                                TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)          AGE
service/gitea-http                  NodePort    10.96.55.25     <none>        3000:30950/TCP   4m32s
service/gitea-memcached             ClusterIP   10.96.176.235   <none>        11211/TCP        4m32s
service/gitea-postgresql            ClusterIP   10.96.59.23     <none>        5432/TCP         4m32s
service/gitea-postgresql-headless   ClusterIP   None            <none>        5432/TCP         4m32s
service/gitea-ssh                   ClusterIP   None            <none>        22/TCP           4m32s

Environment Variables

As a convenience let us set few environment variables which will be used by the commands in upcoming sections of the blog.

Gitea

# Gitea domain
export GITEA_DOMAIN="gitea-127.0.0.1.sslip.io"
# Gitea URL
export GITEA_URL="http://${GITEA_DOMAIN}:3000"

You can access Gitea in your browser like open ${GITEA_URL}. Default credentials demo/demo@123.

Drone

# the drone server host
export DRONE_SERVER_HOST="drone-127.0.0.1.sslip.io:8080"
# the drone server web console
export DRONE_SERVER_URL="http://${DRONE_SERVER_HOST}"

Drone Gitea oAuth Application

Drone will use Gitea for pulling/pushing the source code and to add webhooks to trigger builds on source change. To do these actions it requires an oAuth application to be configured on Gitea.

The demo sources has little utility called gitea-config that helps in creating the oAuth application in Gitea and clone and create the quickstart repository as drone-quickstart on Gitea. We will use drone-quickstart* repository to validate our setup.

$PROJECT_HOME/bin/gitea-config-darwin-arm64 \
  -g "${GITEA_URL}" -dh "${DRONE_SERVER_URL}"

NOTE: Use gitea-config binary corresponding to your os and architecture. In the command above we used macOS arm64 binary

The gitea-config utility creates a .env file under $PROJECT_HOME/k8s which has few Drone environment variables that will be used while deploying Drone server in upcoming steps,

DRONE_GITEA_CLIENT_ID: The Gitea oAuth Client ID
DRONE_GITEA_CLIENT_SECRET: The Gitea oAuth Client Secret
DRONE_RPC_SECRET: The unique secret to identify the server and runner, a simple generation like openssl rand -hex 16

Deploy Drone

For our demo the Drone server will be deployed on to a namespace called drone,

kubectl create ns drone

Add drone helm repo,

helm repo add drone https://charts.drone.io
helm repo update

The following content will be used as helm values file to deploy Drone server,

service:
  # the Drone Kubernetes service type
  type: NodePort
  port: 8080
  # this port will be used in KinD extra port mappings to allow accessing the 
  # drone server from our laptops
  nodePort: 30980

extraSecretNamesForEnvFrom:
   # all the other as $PROJECT_HOME/k8s/.env variables are loaded via this secret
   # https://docs.drone.io/server/reference/
  - drone-demos-secret

env:
  # the Drone server host typically what the drone runners will use to 
  # communicate with the server
  DRONE_SERVER_HOST: drone-127.0.0.1.sslip.io:8080
  # Since we run Gitea in http mode we will skip TLS verification
  DRONE_GITEA_SKIP_VERIFY: true
  # The url where Gitea could be reached, typically used while 
  # cloning the sources
  # https://docs.drone.io/server/provider/gitea/
  DRONE_GITEA_SERVER: http://gitea-127.0.0.1.sslip.io:3000/
  # For this local setup and demo we wil run Drone in http mode
  DRONE_SERVER_PROTO: http

Run the following helm command to deploy Drone server,

helm upgrade --install drone drone/drone \
  --values $PROJECT_HOME/helm_vars/drone/values.yaml \
  --namespace=drone \
  --post-renderer  k8s/kustomize \
  --wait

A successful Drone deployment should show the following resources in drone namespace,

kubectl get pods,svc,secrets -n drone

NAME                         READY   STATUS    RESTARTS   AGE
pod/drone-5bb66b9d97-hbpl5   1/1     Running   0          9s

NAME            TYPE       CLUSTER-IP      EXTERNAL-IP   PORT(S)          AGE
service/drone   NodePort   10.96.184.123   <none>        8080:30980/TCP   9s

NAME                                 TYPE                 DATA   AGE
secret/drone-demos-secret            Opaque               3      9s

Host Aliases

As you have noticed we use Magic DNS for Gitea and Drone. This will cause name resolution issues inside the Drone and Gitea pods, because the url gitea-127.0.0.1.sslip.io resolves to 127.0.0.1 on the Drone server pod. But for our setup to work we need gitea-127.0.0.1.sslip.io to be resolved to the gitea-http Kubernetes service on our cluster.

In order to achieve that we use the Kubernetes host aliases to add extra host entries(/etc/hosts) in Drone pods that will resolve gitea-127.0.0.1.sslip.io to the ClusterIP of the gitea-http service.

There are multiple techniques that allows us to add host entires to Kubernetes deployments. The first one we used in the earlier helm command to deploy Drone server is called helm post renderer. The post renderer allowed us to patch the Drone deployment from the helm chart with the hostAliases for gitea-127.0.0.1.sslip.io resolving to gitea-http's ClusterIP address.

As we did with Drone deployments to resolve the Gitea, we also need to make the Gitea pods resolve the Drone server when trying to send the webhook payload to trigger the build.

This time let us use kubectl patching technique to update the Gitea statefuleset deployments to resolve drone-127.0.0.1.sslip.io to drone service's ClusterIP .

The patch that will be applied to the Gitea statefulset is as shown below.

{
  "spec": {
    "template": {
      "spec": {
        "hostAliases": [
          {
            "ip": "${DRONE_SERVICE_IP}",
            "hostnames": ["drone-127.0.0.1.sslip.io"]
          }
        ]
      }
    }
  }
}

Run the following command to patch and update the gitea statefulset deployment,

export DRONE_SERVICE_IP="$(kubectl get svc -n drone drone -ojsonpath='{.spec.clusterIP}')"
kubectl patch statefulset gitea -n default --patch "$(envsubst<$PROJECT_HOME/k8s/patch.json)"

TIP: To replace the environment variables in the patch we use envsubst

Wait for the Gitea pods to be updated and restarted,

kubectl rollout status statefulset gitea --timeout 30s

You can check the updates to the gitea pod's /etc/hosts file by running the command,

kubectl exec -it gitea-0 -n default cat /etc/hosts

It should have a entry like,

# Entries added by HostAliases.
10.96.184.123   drone-127.0.0.1.sslip.io

Where 10.96.184.123 is the drone service ClusterIP on my setup, run the following command to verify it,

kubectl get svc -n drone drone

NAME    TYPE       CLUSTER-IP      EXTERNAL-IP   PORT(S)          AGE
drone   NodePort   10.96.184.123   <none>        8080:30980/TCP   6m14s

You can do similar checks with Drone pods and ensure that Drone pods /etc/hosts has entry for gitea-127.0.0.1.sslip.io mapping to gitea-http ClusterIP.

What we did so far,

Deployed a customized Kubernetes cluster using kind
Deployed Gitea on to our Kubernetes cluster
Deployed Drone Server on to our Kubernetes Cluster
Created an oAuth application on Gitea to authorize Drone server
Create a repository on Gitea that will be used to test our step

Deploy Drone Kubernetes Runner

To run the Drone pipelines on Kubernetes we need deploy the Drone Kubernetes Runner.

Deploy the drone-runner-kube with following values,

extraSecretNamesForEnvFrom:
   # all the other as env variables are loaded via this secret
  - drone-demos-secret
env:
  # the url to reach the Drone server
  # we point it to the local drone Kubernetes service drone on port 8080
  DRONE_RPC_HOST: "drone:8080"

Run the helm install to deploy the drone-runner-kube,

helm upgrade --install drone-runner-kube drone/drone-runner-kube \
  --namespace=drone \
  --values $PROJECT_HOME/helm_vars/drone-runner-kube/values.yaml  \
  --wait

Querying the Kubernetes resources on drone namespace should now return the drone-runner-kube pod and service,

kubectl get pods,svc -n drone -lapp.kubernetes.io/name=drone-runner-kube

NAME                                     READY   STATUS    RESTARTS   AGE
pod/drone-runner-kube-59f98956b4-mbr9c   1/1     Running   0          41s

NAME                        TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)    AGE
service/drone-runner-kube   ClusterIP   10.96.196.54   <none>        3000/TCP   41s

Open the Drone server web console in your browser using the URL ${DRONE_SERVER_URL} follow the on screen instructions to complete the registration and activation of our drone-quickstart repository.

Lets run our first pipeline

Let us clone the quickstart repository to the folder of our choice on our local machine,

git clone http://gitea-127.0.0.1.sslip.io:3000/demo/drone-quickstart

NOTE: The default git credentials to push is demo/demo@123

Open the drone-quickstart project with your favorite editor, try to make some changes for e.g add some dummy text README to trigger a build. Your build will fail as shown below,

Don't worry, that's what we are going to fix now. We need to do the same thing of adding hostAliases to our drone pipeline pods as well update the .drone.yml with ClusterIP of gitea-http and hostnames with entry for gitea-127.0.0.1.sslip.io so that our Drone pipeline pods are able to clone the sources from our Gitea repository.

The following snippet shows the updated .drone.yml with entries for host aliases,

NOTE: Your ClusterIP of gitea-http* may vary, to get the ClusterIP of the **gitea-http service run the command kubectl get svc gitea-http -ojsonpath='{.spec.clusterIP}'

---
kind: pipeline
type: kubernetes
name: default

steps:
- name: say hello
  image: busybox
  commands:
  - echo hello world
- name: good bye hello
  image: busybox
  commands:
  - echo good bye

# updates to match your local setup
host_aliases:
  # kubectl get svc gitea-http -ojsonpath='{.spec.clusterIP}'
  - ip: 10.96.240.234
    hostnames:
      - "gitea-127.0.0.1.sslip.io"

trigger:
  branch:
  - main

Commit and push the code to trigger a new Drone pipeline build, and you will see it being successful.

Cleanup

When you are done with experimenting, you can clean up the setup by running the following command

kind delete cluster --name=drone-demo

Summary

We now have a fully functional CI with Drone on Kubernetes. You no longer need to build your Cloud (Kubernetes) Native applications, but can Continuously Integrate with much ease and power.

Just summarize what we did in this blog,

Deployed a customized Kubernetes cluster using kind
Deployed Gitea on to our Kubernetes cluster
Deployed Drone Server on to our Kubernetes Cluster
Created an oAuth application on Gitea to authorize drone
Created a repository on Gitea that will be used to test our step
Deployed Drone Kubernetes runner to run pipelines on Kubernetes Cluster
Built our Quick start application using Drone pipelines on Kubernetes
Leveraged Kubernetes host aliases to add host entries to our deployments to resolve local URLs

DEV Community: Kamesh Sampath

[Boost]

Let's Build Together: A Local Playground for Apache Polaris

Kamesh Sampath ・ Feb 25

Let's Build Together: A Local Playground for Apache Polaris

Why I Built a Developer-First Apache Polaris Starter Kit ?

The Challenge: Getting Started with Apache Polaris

The Solution: A Complete Development Environment

What is Snowflake OpenCatalog?

From Local Development to Enterprise Scale

Technical Design Decisions

Why Kubernetes with k3s and k3d?

LocalStack for S3 Integration

PostgreSQL as the Metastore

Kustomize for Deployment Management

Getting Started

Initial Setup

Python Environment Setup

Deploy the Environment

Deploy Polaris

Option 1: Use Pre-built Images

Option 2: Build Images Locally

Deploy and Verify

Setting Up Your First Catalog

Play with the Catalog

Video Walkthrough

Troubleshooting Tips

The Impact: Streamlined Development Experience

Related Projects and Tools

Core Components

Development Tools

Elements of Event Driven Architecture(EDA)

Trigger CI using Terraform Cloud

Use Case

What you need ?

Demo Sources

Fork and Clone the Sources

IaC

Bootstrap Argo CD Sources

Harness CI Pipeline

Harness CI

Create Harness Project

Define New Pipeline

Add harnessImage Docker Registry Connector

Configure GitHub

GitHub Credentials

Connector

Google Cloud Service Account Secret

Set environment

Create SA

IAM Binding

Download And Save GSA Key

GSA Secret

Terraform Workspace

Harness CI Pipeline

Notification Trigger

Copy Webhook URL

Terraform Notification

Summary

Using Workload Identity Continuous Integration(CI) Pipelines

Overview

CI Pipeline UseCase

Tutorial

Pre-requisites

Download Sources

Environment Setup

Variables

Create Environment

Create GKE cluster

Deploy Harness Delegate

Build Application

Import Template

Create Pipeline

Cleanup

Summary

Applying Workload Identity With A Demo

Overview

Glossary

Pre-requisites

Optional

Add `harnessImage` Docker Registry Connector