DEV Community: Buster Styren

Lock down your Kubernetes services with OAuth2 Proxy

Buster Styren — Wed, 12 Apr 2023 08:44:41 +0000

In this tutorial I will show how you can use Oauth2 Proxy to limit access to your Kubernetes services only to members of your GitHub organization, team or just to yourself.

If you're like me then you might have a long list of self-hosted third-party services running inside of your Kubernetes clusters, such as Grafana, Sentry.io, Elastic and Jaeger. Some of these services may have their own mechanism for authentication, some can be configured with an external OAuth provider, and some won't have any authentication mechanism at all.

How can we easily expose our services over the internet with a proper audit trail and without letting any unauthorized users in?

One surprisingly simple solution is to create a GitHub OAuth app and configure your ingresses to use a proxy that permits access based on which GitHub teams a user is part of.

That's a lot of words, but I'll show you that setting it up is just minutes of work and the result is protected internal services with just two ingress annotations:



nginx.ingress.kubernetes.io/auth-signin: https://oauth2.symbiosis.host/oauth2/start?rd=https://$host$uri
nginx.ingress.kubernetes.io/auth-url: https://oauth2.symbiosis.host/oauth2/auth

Requirements

A Kubernetes cluster
Helm
Kubectl
NGINX ingress controller
Cert-manager (to generate TLS certs for OAuth2 Proxy, optional)

What is OAuth2 Proxy?

OAuth2 Proxy is a reverse proxy that authenticates users for a whole range of different Oauth identity providers, such as Keycloak, Google, GitHub, OpenID Connect, and more.

Different providers have different configurations. What makes this powerful in the case of GitHub is the ability to allow or reject requests based on properties such as GitHub organization membership, team membership, email domain and others.

A running Oauth2 Proxy will expose endpoints for signing in, signing out, callbacks used by the OAuth provider, and more. These endpoints can in turn be used to configure NGINX Ingress to ensure that requests to a specific ingress are only performed by a user that is properly signed in and has the proper authority.

However, before we can install and configure OAuth2 Proxy we first have to configure the OAuth provider.

Creating an OAuth app on GitHub

Sign in to GitHub and browse to Settings → Developer settings → OAuth Apps → New OAuth App, fill in the name and page of your application.

The authorization callback URL is the URL that GitHub will redirect to when the authorization is finished. This URL will not be reachable as we haven't configured the OAuth2 Proxy ingress yet. However, it should have the following structure, with example.com replaced by your domain name:

https://example.com/oauth2/callback

Once the app is created generate a client secret and save the generated value. We will use this value soon to configure OAuth2 Proxy.

Installing OAuth2 Proxy with Helm

I've used Helm to install OAuth2 Proxy and found it to be pretty convenient. Check out the Helm chart for the full list of parameters.

Below is an example values.yaml file that permits access to any user that is part of the "example-org" GitHub organization, with TLS certificates generated by cert-manager.



config:
  clientID: example-client-id  # Replace with your GitHub OAuth app client ID
  configFile: |
    provider = "github"
    scope = "user:email read:org"
    github_org = "example-org"  # Replace with your GitHub org name, or github_team=team_name to limit access to a specific team
    email_domains = [ "*" ]  # Replace with [ "example.com" ] to limit access to users with specific email domain
    cookie_domains = [ "example.com" ]  # Replace with domain names that the proxy is allowed to redirect to after auth, prepend . for wildcards, i.e. ".example.com"
    whitelist_domains = [ "example.com" ]  # Same as above

ingress:
  enabled: true
  path: /oauth2
  pathType: Prefix
  className: nginx
  annotations:
    acme.cert-manager.io/http01-edit-in-place: "true"
    cert-manager.io/cluster-issuer: letsencrypt  # Replace with name of your cert-manager ClusterIssuer
    kubernetes.io/tls-acme: "true"
  hosts:
  - example.com  # Replace with host address for the oauth2-proxy ingress
  tls:
  - hosts: 
    - example.com  # Same as above
    secretName: oauth2-tls

Make sure to replace example-client-id, example-org and example.com with your GitHub OAuth app client ID, GitHub organization and domain name.

You may have to edit the cert-manager annotations based on your own configuration, for example by using the cert-manager.io/issuer annotation for namespaced certificate issuers.

We also need to configure a cookie secret that is used by OAuth2 Proxy to encrypt and decrypt user session cookies. Any base64 encoded string is valid, a secure base64 encoded string can be generated using openssl with the following command:



openssl rand -base64 32 | head -c 32 | base64

Finally, we can add the OAuth2 Proxy helm repository and install the chart with our values file, cookie and client secrets:



helm repo add oauth2-proxy https://oauth2-proxy.github.io/manifests
helm install oauth2-proxy oauth2-proxy/oauth2-proxy --file values.yaml --set config.cookieSecret=example-cookie-secret --set config.clientSecret=example-client-secret

oauth2-proxy without cert-manager

If you don't have cert-manager installed you can either create the TLS certificate manually or disable TLS altogether:



config:
  clientID: example-client-id  # Replace with your GitHub OAuth app client ID
  configFile: |
    github_org = "example-org"  # Replace with your GitHub org name, or github_team=team_name to limit access to a specific team
    scope = "user:email read:org"
    email_domains = [ "*" ]  # Replace with [ "example.com" ] to limit access to users with specific email domain
    provider = "github"
    cookie_secure = false
    cookie_domains = [ "example.com" ]  # Replace with domain names that the proxy is allowed to redirect to after auth
    whitelist_domains = [ "example.com" ]  # Same as above

ingress:
  enabled: true
  path: /oauth2
  pathType: Prefix
  className: nginx
  hosts:
  - example.com  # Replace with host address for the oauth2-proxy ingress

Note that you need to change the schema of the callback and auth URLs to http://, both in the GitHub OAuth configuration and in the annotations below.

Configuring our ingress to use OAuth2 Proxy

Now that OAuth2 Proxy is up and running, we can configure our ingress to protect our internal services with GitHub OAuth.

Add the following annotations to your ingress:



nginx.ingress.kubernetes.io/auth-signin: https://example.com/oauth2/start?rd=https://$host$uri
nginx.ingress.kubernetes.io/auth-url: https://example.com/oauth2/auth

Make sure to replace example.com with your own domain name.

The first annotation auth-signin will redirect unauthenticated requests to the OAuth2 Proxy login page. After the user logs in and authorizes the application, they will be redirected back to the original requested URL thanks to the rd query parameter that fowards the redirect URL to the proxy.

The second annotation auth-url specifies the authentication URL that is used to verify the user's session.

Use the commands below if you don't have a service to expose yet in order to launch a protected hello-world application. Make sure to replace example.com with your domain name.



kubectl create deployment web --image=gcr.io/google-samples/hello-app:1.0
kubectl expose deployment web --port=8080
kubectl create ingress web --class nginx '--rule=example.com/=web:8080' \
  --annotation 'nginx.ingress.kubernetes.io/auth-signin=https://example.com/oauth2/start?rd=https://$host$uri' \
  --annotation 'nginx.ingress.kubernetes.io/auth-url=https://example.com/oauth2/auth'

Accessing an endpoint protected by OAuth2 Proxy will redirect you to a GitHub OAuth sign-in page.

Make sure to grant access to the organization that you are authenticating against for OAuth2 Proxy to be able to find it and verify your authority.

If everything is set up correctly you should be redirected to your protected endpoint after signing in.

Limitations

Unfortunately, OAuth2 Proxy cannot be configured to allow specific GitHub teams or organizations to access a resource on the ingress level. If you wish to limit access to ingress1 only to team1 while also limiting access to ingress2 only to team2 you will have to configure multiple OAuth2 Proxy instances and GitHub OAuth applications.

Your services are still exposed on the internet. A security vulnerability in GitHub OAuth, the OAuth2 Proxy or the configuration might expose your service to the whole wide world. Protecting internal services behind a private network is always a good idea.

Putting it all together

To conclude, we've set up a GitHub OAuth app that we've configured with OAuth2 Proxy to allow access to layer 7 ingresses only to the members of our GitHub organization or team. We can use the aforementioned annotations to tag any ingress that we wish to only protect to authenticated users.

Oauth2-proxy isn't the silver bullet for securing internal services, rather it is an easy way to secure many different services or UIs without much work. It can also act as a first line of defense even for services that are only routable on a private network, or services that offer their own authentication mechanism, like Grafana.

You can check out OAuth2 Proxy's official documentation for more tips, tricks and config parameters.

How to save a fortune with self hosted GitHub runners

Buster Styren — Mon, 27 Feb 2023 19:05:42 +0000

GitHub has made it possible to run GitHub Actions using your own self-hosted runners. Thanks to the Actions Runner Controller it is surprisingly easy to run actions in your Kubernetes clusters.

In this post we will show how to install Actions Runner Controller into an existing Kubernetes cluster to run customized runners at a fraction of the cost.

Why?

At Symbiosis we run a lot of tests on each commit, so we've spent a considerable time to make sure they run quickly and can perform complex integration tests.

Using GitHub's own runners is therefore not ideal, as the average commit would cost us almost a dollar each and sadly we make a lot of small changes.

So we could either pay $40 for 5000 minutes of a 2 CPU github runner. Or we could pay $2 to rent a 2 CPU 8 GB Kubernetes node for 5000 minutes and run our actions on there instead.

And as we see below, can also customize our runners to add even more flexibility over the default GitHub runners by heavily modify the runtime.

Prerequisites

To follow this tutorial you need:

A Kubernetes cluster
NGINX ingress (or other ingress controller)
Certmanager (optional)

Installing Actions Runner Controller

The Actions Runner Controller (ARC) is the service responsible for monitoring your selected repositories and firing up new runners.

We will show you how to install it using kubectl but using helm is just as easy.

kubectl create -f https://github.com/actions-runner-controller/actions-runner-controller/releases/download/v0.25.2/actions-runner-controller.yaml

Connecting Actions Runner Controller to GitHub

So we have the controller running, that's great. Now we need to authenticate so that commits, PRs, comments or any other event can be picked up by the controller and trigger the start of a runner.

We have two options, either we can create a Personal Access Token (PAT) with admin access to our repos or we can create a github app and install it into the repos instead.

For simplicity we will authenticate using a PAT.

Personal Access Token (PAT)

Create a token under Settings > Developer settings > Personal access token. Make sure you have admin access to the repos your runners will run on.

Select the repo (Full control) permission, and if your runners will run in an organization you need to select the following permissions as well:

admin:org (Full control)
admin:public_key (read:public_key)
admin:repo_hook (read:repo_hook)
admin:org_hook (Full control)
notifications (Full control)
workflow (Full control)

Next, let's store the token we just created in a secret that our controller can use for authentication.

kubectl create secret generic controller-manager \
    --namespace=actions-runner-system \
    --from-literal=github_token=<YOUR PERSONAL ACCESS TOKEN>

Creating a workflow

Before we move on it is perhaps a good time to create an actual workflow that will eventually trigger our self-hosted runner.

name: Run test on PRs
on:
  pull_request: {}
jobs:
  test:
    name: "Run tests"
    runs-on: [self-hosted]
    steps:
    - name: Checkout repo
      uses: actions/checkout@master
    - name: Run tests
      run: yarn test

This workflow triggers on commits to pull requests and runs yarn test. Lets put it into .github/workflow/test-workflow.yaml and push the changes to our repository.

Notice the runs-on: [self-hosted] option that will instruct GitHub to select any of your own self-hosted runners. Don't worry, you can be more specific about which type of runner to use. More on that later.

Webhook & Ingress

Runners can trigger based on either push or pull-based mechanics. For example by polling or by configuring webhooks. Most triggers come with certain drawbacks, some spawn too many runners and some spawn too few which may put your actions on a slow moving queue.

However there is the workflowJob trigger that has none of these drawbacks, but requires us to create an Ingress and configure a GitHub webhook. So this step isn't strictly necessary but we can assure you it's worth the effort.

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: actions-runner-controller-github-webhook-server
  namespace: actions-runner-system
  annotations:
    kubernetes.io/ingress.class: nginx
    nginx.ingress.kubernetes.io/backend-protocol: "HTTP"
spec:
  tls:
  - hosts:
    - your.domain.com
    secretName: your-tls-secret-name
  rules:
  - http:
      paths:
      - path: /actions-runner-controller-github-webhook-server
        pathType: Prefix
        backend:
          service:
            name: actions-runner-controller-github-webhook-server
            port:
              number: 80

This Ingress is configured for NGINX ingress, so make sure to edit it depending on your ingress controller. It also assumes that cert-manager is configured to automatically provision a TLS certificate.

Next step is to define the webhook in GitHub. Go to Settings > Webhooks > Add webhook in your target repository.

First let's set the payload URL to point to the ingress, for example using the details above: https://your.domain.com/actions-runner-controller-github-webhook-server. Set content type to json and enable the Workflow Jobs permission.

Once it is done you can create the webhook and go to Recent Deliveries to verify that the ingress can be reached successfully.

Listening on events

We have our controller running, it's authenticated and we have a workflow. The only thing left is to create the actual runners.

Now, we could just create a Runner resource and be done with it, but just like a Pod it wouldn't have any replicas or any autoscaling.

Instead we create a RunnerDeployment and a HorizontalRunnerAutoscaler. And for any Kubernetes user you will notice plenty of similarities to regular Deployments and HPAs.

apiVersion: actions.summerwind.dev/v1alpha1
kind: RunnerDeployment
metadata:
  name: actions-runners
spec:
  template:
    spec:
      repository: myorg/myrepo
---
apiVersion: actions.summerwind.dev/v1alpha1
kind: HorizontalRunnerAutoscaler
metadata:
  name: actions-runners
spec:
  minReplicas: 0
  maxReplicas: 5
  scaleTargetRef:
    kind: RunnerDeployment
    name: actions-runners
  scaleUpTriggers:
  - githubEvent:
      workflowJob: {}
    duration: "30m"

Applying the above manifest will launch a deployment that will scale up to five concurrent runners. Remember to change the manifest to track the repository of choice (and make sure the access token has access to it).

Voilà! We're now able to create a pull request to verify that the runner is automatically triggered.

Using labels to identify runners

In a repo with many workflows, for example a monorepo, it may be necessary to run many different runners at once.

In order to more carefully select which runner to use for a specific workflow we can define custom labels:

apiVersion: actions.summerwind.dev/v1alpha1
kind: RunnerDeployment
metadata:
  name: actions-runners
spec:
  template:
    spec:
      repository: myorg/myrepo
      labels:
      - my-label

With this label we're able to select this runner by setting both self-hosted and my-label in our workflow:

name: Run test on PRs
on:
  pull_request: {}
jobs:
  test:
    name: "Run tests"
    runs-on: [self-hosted, my-label]
    steps:
    - name: Checkout repo
      uses: actions/checkout@master
    - name: Run tests
      run: yarn test

Customizing runners with custom volumes

Runners can be configured to pass through volumes from the host system, or to attach a PVC to runners.

At Symbiosis we use PVCs to expose KVM to our runners, in order to run integrations tests with vitualization enabled. We also use PVCs to attach large images that are used to set up a multi-tenant cloud environment for integration testing.

Custom volumes can also be used for layer caching, to improve the speed of building OCI images.

The below runner will provision a 10Gi PVC that will be shared. Notice that we're using RunnerSet instead of RunnerDeployment. This resource functions much like the StatefulSet in that it will allocate the runner on a node where the volume can be properly mounted.

apiVersion: actions.summerwind.dev/v1alpha1
kind: RunnerDeployment
metadata:
  name: actions-runners
spec:
  template:
    spec:
      repository: myorg/myrepo
      volumeMounts:
      - mountPath: /runner/work
        name: pvc
      volumes:
      - name: pvc
        ephemeral:
          volumeClaimTemplate:
            spec:
              accessModes: [ "ReadWriteOnce" ]
              resources:
                requests:
                  storage: 10Gi

This PVC can be used to store any data that we need between runs without having to store an unnecessarily large amount of data in the GitHub actions cache! At the time of writing this article 100GiB of storage using GitHub runners would costs $24/mo. With cloud providers like Linode, Symbiosis or Scaleway that cost would be closer to $8/mo.

To summarize

Running your own actions runners requires some upfront configuration but come with a list of benefits such as:

Reduced costs
Attaching custom volumes (such as hostpath or PVCs) to your runners
Customizing images or adding sidecars
Integrate workflow runs into the existing Kubernetes observability stack

Therefore, we highly recommend running your own runners to save cost, simplify management and increase flexibility by bringing the runners into the Kubernetes ecosystem.

Check out Symbiosis here.

Symbiosis: The cloud platform for Kubernetes

Buster Styren — Tue, 29 Nov 2022 13:15:03 +0000

As an engineer, you are faced with many decisions when building a Kubernetes environment.

How do I set up GitOps & CI/CD?
How do I create application delivery pipelines?
How do I keep costs down?
How do I quickly boot new prod-like environments?
How can I set up self-service for engineering teams?
How to avoid maintaining docker-compose files for local development or testing?

For years I've been configuring and maintaining tens of tools on-top of EKS to create an efficient and automated setup for day-2 operations and simplify deployment of new services.

But, maintaining a complicated k8s stack can take considerable effort as infrastructure evolves and changes over time. Moreover, the high costs and sluggishness of EKS or GKE make it harder to use Kubernetes outside of running prod and staging environments.

Symbiosis is a managed Kubernetes service that plugs some of these holes to make it simple for DevOps to manage day-2 operations and for developers to build, test and deploy.

In this guide I will take you through how to create a k8s cluster and add your projects to Symbiosis.

Getting started

Install the CLI with brew install symbiosis-cloud/tap/sym.

Then login with sym login to authenticate to your Symbiosis team. Make sure you have an account set up already.

Creating your first k8s cluster is as easy as issuing sym cluster create -- or use our terraform provider for IaC.

The kube context is automatically installed, so you can access the cluster with kubectl, k9s and other tools that read the kubeconfig file.

Well done! You have a working k8s cluster. If you're new to k8s you might read our guide on how to deploy a container to your cluster.

Let's look into what else Symbiosis can offer!

Projects

With Projects you can link your GitHub repositories to automatically create k8s environments for any occasion. For example in development, testing or as a staging environment in preparation for a release.

The core idea of Projects is to offer the simplicity of PaaS but without any opinionated or limiting abstractions.

Engineers should be able to easily build, test and release on prod-like environments without breaking a sweat.

Projects is currently in closed preview.

The `sym.yaml` file

The sym.yaml file is placed at the root of your repositories and define the structure of your project.

deploy:
  helm:
  - chart: "./charts"
    values:
      vaultToken: "{{.secret.VAULT_TOKEN}}"
      replicas: 2
test:
- image: "backend:latest"
  command: "go test"

Define and customize your Helm charts or Kustomize manifests to have them wired to your project.

With the file in place we can instantly boot a cluster that runs our project with sym run.

Tests run inside your cluster and can be used to easily define complex end-to-end or integration tests that span across many services.

Running your test suite in the cluster with sym test.

CI/CD

Pipelines are automatically created to build, test and ship your projects. In practice this means a preview cluster is created that will run your tests.

The preview environment can also be used to help with PR reviews or used to share progress with your team.

GitOps

In the projects settings you can configure a production cluster. Merging a PR will automatically apply any changes to your prod environment.

Note: Some users may need to use Symbiosis for dev and CI/CD but deploy to a cluster in AWS or Google Cloud. It's on our roadmap to add options to deploy to other providers.

Don't lose sleep over k8s bills

Kubernetes shouldn't have to cost a fortune. At Symbiosis we offer compute at less than half of AWS or DigitalOcean.

We're also committed to making sure our platform is efficient. Did you know that EKS nodes reserve 1.6GiB of memory? Meaning a smaller cluster with 4GB nodes will waste at least 41% of all memory.

Cheap bandwidth reduce the friction when using multiple clouds and services. We're determined to combat vendor lock-in effects by charging $5 per TB of traffic, compared to $92 with AWS.

Going further

To learn more about Symbiosis I recommend reading our docs.

Keep tabs on what we're doing over on twitter.

Happy coding! 🖤