Lock down your Kubernetes services with OAuth2 Proxy

In this tutorial I will show how you can use Oauth2 Proxy to limit access to your Kubernetes services only to members of your GitHub organization, team or just to yourself.

If you're like me then you might have a long list of self-hosted third-party services running inside of your Kubernetes clusters, such as Grafana, Sentry.io, Elastic and Jaeger. Some of these services may have their own mechanism for authentication, some can be configured with an external OAuth provider, and some won't have any authentication mechanism at all.

How can we easily expose our services over the internet with a proper audit trail and without letting any unauthorized users in?

One surprisingly simple solution is to create a GitHub OAuth app and configure your ingresses to use a proxy that permits access based on which GitHub teams a user is part of.

That's a lot of words, but I'll show you that setting it up is just minutes of work and the result is protected internal services with just two ingress annotations:

nginx.ingress.kubernetes.io/auth-signin: https://oauth2.symbiosis.host/oauth2/start?rd=https://$host$uri
nginx.ingress.kubernetes.io/auth-url: https://oauth2.symbiosis.host/oauth2/auth

Requirements

• A Kubernetes cluster
• Helm
• Kubectl
• NGINX ingress controller
• Cert-manager (to generate TLS certs for OAuth2 Proxy, optional)

What is OAuth2 Proxy?

OAuth2 Proxy is a reverse proxy that authenticates users for a whole range of different Oauth identity providers, such as Keycloak, Google, GitHub, OpenID Connect, and more.

Different providers have different configurations. What makes this powerful in the case of GitHub is the ability to allow or reject requests based on properties such as GitHub organization membership, team membership, email domain and others.

A running Oauth2 Proxy will expose endpoints for signing in, signing out, callbacks used by the OAuth provider, and more. These endpoints can in turn be used to configure NGINX Ingress to ensure that requests to a specific ingress are only performed by a user that is properly signed in and has the proper authority.

However, before we can install and configure OAuth2 Proxy we first have to configure the OAuth provider.

Creating an OAuth app on GitHub

Sign in to GitHub and browse to Settings → Developer settings → OAuth Apps → New OAuth App, fill in the name and page of your application.

The authorization callback URL is the URL that GitHub will redirect to when the authorization is finished. This URL will not be reachable as we haven't configured the OAuth2 Proxy ingress yet. However, it should have the following structure, with example.com replaced by your domain name:

https://example.com/oauth2/callback

Once the app is created generate a client secret and save the generated value. We will use this value soon to configure OAuth2 Proxy.

Installing OAuth2 Proxy with Helm

I've used Helm to install OAuth2 Proxy and found it to be pretty convenient. Check out the Helm chart for the full list of parameters.

Below is an example values.yaml file that permits access to any user that is part of the "example-org" GitHub organization, with TLS certificates generated by cert-manager.

config:
  clientID: example-client-id  # Replace with your GitHub OAuth app client ID
  configFile: |
    provider = "github"
    scope = "user:email read:org"
    github_org = "example-org"  # Replace with your GitHub org name, or github_team=team_name to limit access to a specific team
    email_domains = [ "*" ]  # Replace with [ "example.com" ] to limit access to users with specific email domain
    cookie_domains = [ "example.com" ]  # Replace with domain names that the proxy is allowed to redirect to after auth, prepend . for wildcards, i.e. ".example.com"
    whitelist_domains = [ "example.com" ]  # Same as above

ingress:
  enabled: true
  path: /oauth2
  pathType: Prefix
  className: nginx
  annotations:
    acme.cert-manager.io/http01-edit-in-place: "true"
    cert-manager.io/cluster-issuer: letsencrypt  # Replace with name of your cert-manager ClusterIssuer
    kubernetes.io/tls-acme: "true"
  hosts:
  - example.com  # Replace with host address for the oauth2-proxy ingress
  tls:
  - hosts: 
    - example.com  # Same as above
    secretName: oauth2-tls

Make sure to replace example-client-id, example-org and example.com with your GitHub OAuth app client ID, GitHub organization and domain name.

You may have to edit the cert-manager annotations based on your own configuration, for example by using the cert-manager.io/issuer annotation for namespaced certificate issuers.

We also need to configure a cookie secret that is used by OAuth2 Proxy to encrypt and decrypt user session cookies. Any base64 encoded string is valid, a secure base64 encoded string can be generated using openssl with the following command:

openssl rand -base64 32 | head -c 32 | base64

Finally, we can add the OAuth2 Proxy helm repository and install the chart with our values file, cookie and client secrets:

helm repo add oauth2-proxy https://oauth2-proxy.github.io/manifests
helm install oauth2-proxy oauth2-proxy/oauth2-proxy --file values.yaml --set config.cookieSecret=example-cookie-secret --set config.clientSecret=example-client-secret

oauth2-proxy without cert-manager

If you don't have cert-manager installed you can either create the TLS certificate manually or disable TLS altogether:

config:
  clientID: example-client-id  # Replace with your GitHub OAuth app client ID
  configFile: |
    github_org = "example-org"  # Replace with your GitHub org name, or github_team=team_name to limit access to a specific team
    scope = "user:email read:org"
    email_domains = [ "*" ]  # Replace with [ "example.com" ] to limit access to users with specific email domain
    provider = "github"
    cookie_secure = false
    cookie_domains = [ "example.com" ]  # Replace with domain names that the proxy is allowed to redirect to after auth
    whitelist_domains = [ "example.com" ]  # Same as above

ingress:
  enabled: true
  path: /oauth2
  pathType: Prefix
  className: nginx
  hosts:
  - example.com  # Replace with host address for the oauth2-proxy ingress

Note that you need to change the schema of the callback and auth URLs to http://, both in the GitHub OAuth configuration and in the annotations below.

Configuring our ingress to use OAuth2 Proxy

Now that OAuth2 Proxy is up and running, we can configure our ingress to protect our internal services with GitHub OAuth.

Add the following annotations to your ingress:

nginx.ingress.kubernetes.io/auth-signin: https://example.com/oauth2/start?rd=https://$host$uri
nginx.ingress.kubernetes.io/auth-url: https://example.com/oauth2/auth

Make sure to replace example.com with your own domain name.

The first annotation auth-signin will redirect unauthenticated requests to the OAuth2 Proxy login page. After the user logs in and authorizes the application, they will be redirected back to the original requested URL thanks to the rd query parameter that fowards the redirect URL to the proxy.

The second annotation auth-url specifies the authentication URL that is used to verify the user's session.

Use the commands below if you don't have a service to expose yet in order to launch a protected hello-world application. Make sure to replace example.com with your domain name.

kubectl create deployment web --image=gcr.io/google-samples/hello-app:1.0
kubectl expose deployment web --port=8080
kubectl create ingress web --class nginx '--rule=example.com/=web:8080' \
  --annotation 'nginx.ingress.kubernetes.io/auth-signin=https://example.com/oauth2/start?rd=https://$host$uri' \
  --annotation 'nginx.ingress.kubernetes.io/auth-url=https://example.com/oauth2/auth'

Accessing an endpoint protected by OAuth2 Proxy will redirect you to a GitHub OAuth sign-in page.

Make sure to grant access to the organization that you are authenticating against for OAuth2 Proxy to be able to find it and verify your authority.

If everything is set up correctly you should be redirected to your protected endpoint after signing in.

Limitations

Unfortunately, OAuth2 Proxy cannot be configured to allow specific GitHub teams or organizations to access a resource on the ingress level. If you wish to limit access to ingress1 only to team1 while also limiting access to ingress2 only to team2 you will have to configure multiple OAuth2 Proxy instances and GitHub OAuth applications.

Your services are still exposed on the internet. A security vulnerability in GitHub OAuth, the OAuth2 Proxy or the configuration might expose your service to the whole wide world. Protecting internal services behind a private network is always a good idea.

Putting it all together

To conclude, we've set up a GitHub OAuth app that we've configured with OAuth2 Proxy to allow access to layer 7 ingresses only to the members of our GitHub organization or team. We can use the aforementioned annotations to tag any ingress that we wish to only protect to authenticated users.

Oauth2-proxy isn't the silver bullet for securing internal services, rather it is an easy way to secure many different services or UIs without much work. It can also act as a first line of defense even for services that are only routable on a private network, or services that offer their own authentication mechanism, like Grafana.

You can check out OAuth2 Proxy's official documentation for more tips, tricks and config parameters.