Securing Apps Deployed via Jenkins X Using Ambassador Edge Stack

Overview

Microservice orchestrated by Kubernetes sometimes need not be public-facing.

There are many options to secure an endpoint, however, in this post we walk through configuring Datawire's Ambassador Edge Stack to protect an app which has been put through CI/CD and published via Jenkins X. We enable SSO to leverage existing identity management using Azure Active Directory all in AWS and EKS.

I also walk you through installing Jenkins X on AWS EKS first, then installing and configuring Ambassador. Of course, if you have a Jenkins X cluster already, you can skip to the Setup Ambassador Edge Stack section.

GitHub Repo: You can clone the repo specific branch.

Create EKS Cluster

For this scenario, we will use AWS EKS

Create an EKS cluster using eksctl using the following command

eksctl create cluster  \
--name drymartini  \
--full-ecr-access

Once the cluster is Active you are ready to install Jenkins X.

Configure Route 53 DNS

Part of our scenario is to configure Jenkins X with a custom domain. To do this, we must first configure a Route 53 Managed Zone then delegate your subdomain to Route 53. In my case, my domain is managed by Google Domains, and so I’ve created an NS record pointing to the NS servers given to me by Route 53 after I created my Hosted Zone.

1. Create a Managed Zone for subdomain
2. Delegate your subdomain to AWS Route 53 from the vendor hosting your domain.
3. Test it using dig jx.eks.sharepointoscar.com (in my case).

NOTE: Testing the Hosted Zone can be done from the AWS Console, and ultimately you want a response with NOERROR

Install Jenkins X using Boot

With the cluster created and DNS configured, we are ready to install Jenkins X using Jenkins X Boot

We will be using a stable distribution of the open-source Jenkins X project, otherwise known as CJXD CloudBees Jenkins X Distribution. The advantage of using this edition vs. the straight OSS is that it has been tested, and certified for EKS, it also allows for a controlled platform upgrade within your enterprise environment.

Install Jenkins X CLI

# download binary for CJXD (CloudBees Jenkins X Distro) based on your platform.
# https://docs.cloudbees.com/docs/cloudbees-jenkins-x-distribution 
curl -L https://storage.googleapis.com/artifacts.jenkinsxio.appspot.com/binaries/cjxd/latest/jx-darwin-amd64.tar.gz | tar xzv

# move to bin
sudo mv jx /usr/local/bin

# test
➜ jx version
NAME               VERSION
jx                 2.0.1245+cjxd.8
Kubernetes cluster v1.14.10-gke.27
kubectl            v1.15.0
helm client        2.14.3
git                2.21.0
Operating System   Mac OS X 10.14.6 build 18G3020

Execute JX Boot

Our next step is to install Jenkins X on top of the EKS cluster. We do this by using a declarative yaml approach.

During the curl of the jx binary, a couple of yaml files were downloaded to the same directory where we executed that command.

We will use the jx-requirements-eks.yml

Step 1 - Modify the jx-requirements-eks.yml file with basic settings

The basic settings we will modify in order to execute an initial jx boot installation will be as follows (please modify your file accordingly)

cluster:
 clusterName: drymartini
 environmentGitOwner: jenkins-oscar #github organization
 environmentGitPublic: true #my org is on free tier, so public repos
 provider: eks
 region: us-west-2

Next we modify the main ingress field as follows:

ingress:
 domain: jx.eks.sharepointoscar.com #Route 53 Managed Zone
 externalDNS: true
 ignoreLoadBalancer: true
 namespaceSubDomain: . # personally a dot makes for cleaner FQDN entries
 tls:
   email: [e-mail]
   enabled: true
   production: true

HashiCorp Vault Initial Configuration

In order for the jx boot command to run successfully the first time, we need to specify an IAM username in the specific jx-requirements-eks.yml file section as follows:

vault:
 aws:
   autoCreate: true
   iamUserName: [e-mail] #IAM Username
 disableURLDiscovery: true

For details on IAM permissions, please refer to the documentation.

Step 2 - Execute JX Boot CLI Command

Having modified the appropriate fields in the jx-requirements-eks.yml, we are ready to execute the jx boot CLI command as follows:

$ jx boot -r jx-requirements.eks.yml

The CLI will ask a few questions.

The cluster configuration repository is cloned immediately. This repository will be added to your GitHub Organization, and any future changes will require a Pull Request, the GitOps way!

IMPORTANT: Any future executions of jx boot, will require you to execute it from the root of the cluster configuration repository that was cloned during the first run. This effectively means the original jx-requirements-eks.yml file is no longer used in subsequent executions.

Setup Ambassador Edge Stack

To setup Ambassador, you can follow the simple instructions on their wonderful Docs site.

Install from MacOS

Download it with a curl command:

sudo curl -fL https://metriton.datawire.io/downloads/darwin/edgectl -o /usr/local/bin/edgectl && sudo chmod a+x /usr/local/bin/edgectl

The installer will provision a load balancer, configure TLS, and provide you with an edgestack.me subdomain. The edgestack.me subdomain allows the Ambassador Edge Stack to automatically provision TLS and HTTPS for a domain name, so you can get started right away.

Once you have it configured, you are ready to move to next step.

Create AD App Registration

Setup Azure App Registration as per the instructions on Docs site.

Configure Filter

Create a file oauth_filter.yaml and paste this content. Then deploy it via kubectl apply -f oauth_filter.yaml

apiVersion: getambassador.io/v2
kind: Filter
metadata:
  name: azure-skiapp-ad
  namespace: jx-staging
spec:
  OAuth2:
    authorizationURL: https://login.microsoftonline.com/<AD_TENANT_ID>/v2.0  ## URL where Ambassador Edge Stack can find OAuth2 descriptor
    clientID: <APP_CLIENT_ID> ## OAuth2 client from your IdP
    secret: <APP_SECRET> ## Secret used to access OAuth2 client
    protectedOrigins:
    - origin: https://skiapp.dev.sharepointoscar.com

Configure Filter Policy

Create a file oauth_filter_policy.yaml and paste this content. Then deploy it via kubectl apply -f oauth_filter_policy.yaml

apiVersion: getambassador.io/v2
kind: FilterPolicy
metadata:
  name: azure-skiapp-policy
  namespace: jx-staging
spec:
  rules:
      # Requires authentication on requests from hostname
    - host: "skiapp.dev.sharepointoscar.com"
      path: "*"
      filters:
        - name: azure-skiapp-ad

Additional Config via Ambassador Portal

To access the portal, execute the CLI as follows:

edgectl login --namespace=ambassador <yourinstancename>.edgestack.me

1. Create new Host entry. In my case, it is for skiapp.dev.sharepointoscar.com. I let Ambassador handle TLS as you can see.

1. Create a new Mapping, it should look like the following.

Import App Into Jenkins X

Import app from GitHub as follows. NOTE: Please make sure to fork the app before importing it. You are effectively importing it from your own GitHub Organization.

jx import --url <GITHUB_REPO_URL>

Modify App Ingress Config

When you import an application into Jenkins X, it automatically creates a Helm Chart! In this step, we need to modify the Helm Chart to tell Jenkins X, it should not use exposecontroller component to expose the app, which it did automatically.

Modify the Helm Chart

Under the helm/myapp chart, you will find a file called values.yaml which we need to modify as follows.

Original Service configuration.

service:
  name: skiapp
  type: ClusterIP
  externalPort: 80
  internalPort: 8080
  annotations:
    fabric8.io/expose: "true"
    fabric8.io/ingress.annotations: "kubernetes.io/ingress.class: nginx"

Modified Service

service:
  name: skiapp
  type: ClusterIP
  externalPort: 80
  internalPort: 8080
  annotations:
    getambassador.io/config: |
        ---
        apiVersion: ambassador/v1
        kind: Mapping
        name: skiapp-service_mapping
        host: skiapp.dev.sharepointoscar.com
        prefix: /
        service: skiapp:80

With this configuration in place, I can go to https://skiapp.dev.sharepointoscar.com and will get redirected to Microsoft login screen.

Watch the Webinar

https://www.youtube.com/watch?v=PUEno5NTSKA&feature=youtu.be

Summary

We now have an app which was put through CI/CD using Jenkins X - protected using Ambassador, and using SSO backed by Azure Active Directory. You can use Okta or any other IdP you environment uses, as long as it is using OAuth2 in this case.

Cheers,

@SharePointOscar