DEV Community: Jax Gauthier

Getting Started with Cloudflare Workers

Jax Gauthier — Tue, 11 Feb 2020 17:00:00 +0000

I have recently had the need to redirect a few subdomains to another domain, due to old links still being used by public individuals.

I wanted a solution that would not require running another web server, so I started looking into a few different options.

Cloudflare Page Rules

While Page Rules do want I want, they were either not flexible enough (could only target a single subdomain), and would be relatively expensive for the number I would need.

Cloudflare Workers

Cloudflare Workers are pretty cool. They allow you to run JS code at the Cloudflare edge. They are basically functions-as-a-service, which are super fast and customizable.

I was able to set up a function in less than 5 minutes that allows me to redirect any subdomain to another domain.

The function I am using looks like this:

async function handleRequest(request) {
  return Response.redirect(someURLToRedirectTo, code)
}
addEventListener('fetch', async event => {
  event.respondWith(handleRequest(event.request))
})
/**
 * Example Input
 * @param {Request} url where to redirect the response
 * @param {number?=301|302} type permanent or temporary redirect
 */
const someURLToRedirectTo = 'https://homelab.blog'
const code = 302 // Temporary Redirect

Basically, it handles requests by sending a 302 Temporary Redirect to my current blog page.

Additionally, the function is super fast. It seems significantly faster than if I were to run a webserver to handle the redirect myself.

It is also using the routes function. You still have to define a DNS name for the URLs you want to redirect, and I currently just CNAME them to the workers.dev hostname of the worker.

This is just a start of what workers/functions are able to do, and I am excited to see where else I can use them to make my life easier.

Configuring Istio with OIDC authentication

Jax Gauthier — Tue, 04 Feb 2020 23:51:17 +0000

In this blog post, we will look at the first part of my ideal setup, which is to secure inbound communication via an authenticating reverse proxy (OAuth2_Proxy), and Keycloak.

This setup will use the follow technologies:

Istio (ingress gateway)
1. Certmanager (certificates) - not covered in this post
OAuth2_Proxy (controls the OIDC flow)
1. Redis (session storage)
Keycloak (OIDC Provider)

Istio

Istio is a service mesh that allows you to define and secure services in your Kubernetes cluster. In my lab, I use it as the ingress gateway for my cluster, and I am planning on using it to secure service-to-service communication using mutual-tls.

Istio will require a valid certificate for the gateway, you can either set this up via cert-manager, or by importing a certificate into your cluster manually.

Here are some example config files for setting up basic services in Istio:

Istio Gateway:

apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: gateway
  namespace: istio-system
spec:
  selector:
    istio: ingressgateway
  servers:
  - hosts:
    - '*'
    port:
      name: http
      number: 80
      protocol: HTTP
    tls:
      httpsRedirect: true
  - hosts:
    - '*'
    port:
      name: https
      number: 443
      protocol: HTTPS
    tls:
      credentialName: changeme
      httpsRedirect: false
      mode: SIMPLE
      privateKey: sds
      serverCertificate: sds

Destination Rule:

apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
  name: oauth-corp-justin-tech-com
  namespace: istio-system
spec:
  host: oauth.corp.justin-tech.com
  trafficPolicy:
    loadBalancer:
      simple: ROUND_ROBIN
    portLevelSettings:
    - port:
        number: 443
      tls:
        mode: SIMPLE
        sni: oauth.corp.justin-tech.com

Virtual Service:

oauthproxy-service:

apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: oauthproxy-service
  namespace: default
spec:
  gateways:
  - gateway.istio-system
  hosts:
  - oauth.corp.justin-tech.com
  - oauth.justin-tech.com
  http:
  - route:
    - destination:
        host: oauthproxy-service
        port:
          number: 4180
      weight: 100

the virtual service for your application, I am using Rancher-Demo:

apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: rancher-demo
  namespace: default
spec:
  gateways:
  - gateway.istio-system
  hosts:
  - rancher-demo.corp.justin-tech.com
  http:
  - route:
    - destination:
        host: rancher-demo
        port:
          number: 80
      weight: 100

The Envoy Filter, which is the part that starts the redirection for unauthenticated users. This is similar to the auth-url function of Nginx-Ingress. This acts on your Ingress-Gateway workload.

apiVersion: networking.istio.io/v1alpha3
kind: EnvoyFilter
metadata:
  name: authn-filter
spec:
  workloadLabels:
    istio: ingressgateway
  filters:
  - filterConfig:
      http_service:
        server_uri:
          uri: http://oauthproxy-service.default.svc.cluster.local
          cluster: outbound|4180||oauthproxy-service.default.svc.cluster.local
          timeout: 1.5s
        authorizationRequest:
          allowedHeaders:
            patterns:
            - exact: "cookie"
            - exact: "x-forwarded-access-token"
            - exact: "x-forwarded-user"
            - exact: "x-forwarded-email"
            - exact: "authorization"
            - exact: "x-forwarded-proto"
            - exact: "proxy-authorization"
            - exact: "user-agent"
            - exact: "x-forwarded-host"
            - exact: "from"
            - exact: "x-forwarded-for"
            - exact: "accept"
            - prefix: "x-forwarded"
            - prefix: "x-auth-request"
        authorizationResponse:
          allowedClientHeaders:
            patterns:
            - exact: "location"
            - exact: "proxy-authenticate"
            - exact: "set-cookie"
            - exact: "authorization"
            - exact: "www-authenticate"
            - prefix: "x-forwarded"
            - prefix: "x-auth-request"
          allowedUpstreamHeaders:
            patterns:
            - exact: "location"
            - exact: "proxy-authenticate"
            - exact: "set-cookie"
            - exact: "authorization"
            - exact: "www-authenticate"
            - prefix: "x-forwarded"
            - prefix: "x-auth-request"
    filterName: envoy.ext_authz
    filterType: HTTP
    listenerMatch:
      portNumber: 443
      listenerType: GATEWAY

The important parts are to set the server_uri. The allowed lists of headers is probably more than what is needed, but it works for me.

This final part is optional, if you omit this part, you would be able to use the standard OAuth2_Proxy setup which is to send the cookies to the client directly, instead of using Redis as a session store. This configuration uses Istio's JWT authentication validation to ensure that every request to your service is authenticated by your issuer. You could expand on this by requiring specific groups per service, and by doing client certificate validation (which you could also couple with Keycloak's client certificate validation), for the best authentication of incoming requests.

This is set per service with the target definition.

apiVersion: authentication.istio.io/v1alpha1
kind: Policy
metadata:
  name: ingress-auth-policy
spec:
  targets:
  - name: rancher-demo
  origins:
  - jwt:
      issuer: "https://auth.justin-tech.com/auth/realms/Justin-Tech"
      jwksUri: "https://auth.justin-tech.com/auth/realms/Justin-Tech/protocol/openid-connect/certs"
  principalBinding: USE_ORIGIN

Oauth2_Proxy

OAuth2_Proxy performs the OIDC flow for unauthenticated users. We will need to use the Redis session store, as having the ID Token sent to the browser results in cookies that are too long, and splitting the cookie into parts does not work with Istio (see https://pusher.github.io/oauth2_proxy/configuration#configuring-for-use-with-the-nginx-auth_request-directive for an example that spits the cookies with Nginx Ingress support).

Oauth2_Proxy Deployment example:

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    k8s-app: oauth2-proxy
  name: oauth2-proxy
  namespace: default
spec:
  replicas: 1
  selector:
    matchLabels:
      k8s-app: oauth2-proxy
  template:
    metadata:
      labels:
        k8s-app: oauth2-proxy
    spec:
      containers:
        - args:
            - --provider=oidc
            - --provider-display-name="Keycloak"
            - --email-domain=corp.justin-tech.com
            - --email-domain=justin-tech.com
            - --pass-access-token=true
            - --pass-authorization-header=true
            - --set-authorization-header=true
            - --oidc-issuer-url=https://auth.justin-tech.com/auth/realms/Justin-Tech
            - --login-url=https://auth.justin-tech.com/auth/realms/Justin-Tech/protocol/openid-connect/auth
            - --redeem-url=https://auth.justin-tech.com/auth/realms/Justin-Tech/protocol/openid-connect/token
            - --validate-url=https://auth.justin-tech.com/auth/realms/Justin-Tech/protocol/openid-connect/userinfo
            - --http-address=0.0.0.0:4180
            - --cookie-expire=4h0m0s
            - --whitelist-domain=".corp.justin-tech.com"
            - --cookie-domain=.justin-tech.com
            - --standard-logging=true
            - --auth-logging=true
            - --request-logging=true
            - --skip-provider-button=true
            - --upstream=static://
            - --redis-connection-url=redis://redis-master.default.svc.cluster.local:6379
            - --session-store-type=redis
          env:
            - name: OAUTH2_PROXY_CLIENT_ID
              value: rancher-demo
            - name: OAUTH2_PROXY_CLIENT_SECRET
              value:
            # docker run -ti --rm python:3-alpine python -c 'import secrets,base64; print(base64.b64encode(base64.b64encode(secrets.token_bytes(16))));'
            - name: OAUTH2_PROXY_COOKIE_SECRET
              value:
            - name: OAUTH2_PROXY_COOKIE_DOMAIN
              value: .justin-tech.com
          image: quay.io/pusher/oauth2_proxy:v4.1.0
          imagePullPolicy: Always
          name: oauth2-proxy
          ports:
            - containerPort: 4180
              protocol: TCP
---
apiVersion: v1
kind: Service
metadata:
  labels:
    k8s-app: oauth2-proxy
  name: oauthproxy-service
  namespace: default
spec:
  ports:
    - name: http
      port: 4180
      protocol: TCP
      targetPort: 4180
  selector:
    k8s-app: oauth2-proxy

The above config is what I am using to deploy OAuth2_Proxy, some of the configuration is probably unnecessary. Some important parts are the URLs for your OIDC Provider (Keycloak in my case), and the cookie domain, if you have a domain and subdomains that are being used.

Redis

I am using the stable/redis helm chart, with minimal configuration explained below. Redis is needed in order to pass JWT tokens from Keycloak to Istio, otherwise the cookies are too large and get split (which is not supported easily in Istio). The downside is that currently OAuth2_Proxy does not support a password on the Redis connection.

The standard values.yaml from redis is fine to use, though you can change a few options:

Enable persistence for the Redis master and slaves
Enable metrics (if you want)

Keycloak

i will not cover Keycloak too much in this post, as I have other posts available which cover setting up Keycloak. One thing to note is that you need to set the Redirect URL to either a wildcard, or by setting up one OAuth2_Proxy service per service, or have your services available at different URIs instead of subdomains. This could be a security issue as any redirect url could be set, please consider this before implementing.

Altogether, this setup allows you to use Istio as your ingress-gateway, which provides a lot of features if you are using the rest of their service mesh, including mutual TLS inside the cluster, and integration with Kiali, for example. It allows you to use a service for starting/handling the OIDC flow (if you are using third party applications, or just want an authenticating proxy in front of your applications), and allows you to use any OIDC provider to do it.

Getting Started with Kubernetes (at home) — Part 3

Jax Gauthier — Sat, 04 May 2019 16:00:00 +0000

In the first two parts of this series, we looked at setting up a production Kubernetes cluster in our labs. In part three of this series, we are going to deploy some services to our cluster such as Guacamole and Keycloak.

Step-by-step documentation and further service examples are here.

Guacamole

Guacamole is a very useful piece of software that allows you to remotely connect to your devices via RDP, SSH, or other protocols. I use it extensively to access my lab resources, even when I am at home.

You can use this Helm Chart to install Guacamole on your Kubernetes cluster. The steps are as follows:

Clone the Guacamole Helm Chart from here
Apply any changes to values.yaml such as the annotations and ingress settings.
Deploy the Helm Chart from the apache-guacamole-helm-chart directory
- helm install . -f values.yaml --name=guacamole --namespace=guacamole
An ingress is automatically created by the Helm Chart, and you can access it based on the hosts: section of values.yaml

Once you have deployed the Helm Chart, you should be able to access Guacamole at the ingress hostname specified in values.yaml.

Keycloak

Keycloak is an open-source single sign-on solution that is similar to Microsoft’s ADFS product. You can read more about Keycloak on my blog.

There is a stable Keycloak Helm Chart available in the default Helm repo, which we will be using to deploy Keycloak, you can find it here.

Apply any changes to values.yaml
Deploy the helm chart stable/keycloak with values
- helm install --name keycloak stable/keycloak --values values.yaml
Create an ingress
- kubectl apply -f ingress.yaml
Get the default password for the keycloak user.
- kubectl get secret --namespace default keycloak-http -o jsonpath="{.data.password}" | base64 --decode; echo

Though this article is on the shorter side, hopefully it exemplifies how easy it can be to run services in Kubernetes. We mainly looked at pre-made Helm Charts in this article, however deploying a service without a Chart can also be just as easy. I prefer using charts as I find it easier to manage than straight Kubernetes manifest files.

You can checkout my public Kubernetes repo at https://gitlab.com/just.insane/kubernetes/ for further information and more service examples.

Getting Started with Kubernetes (at home) — Part 2

Jax Gauthier — Sat, 27 Apr 2019 20:20:02 +0000

In the first part of the series, we looked at installing a bare bones Kubernetes cluster in some CentOS 7 VMs. In this part, we are going to look at setting up some back-end services, like a load balancer and ingress.

Some important things needed to properly run a Kubernetes cluster are a storage class or manual storage configuration via volumes, a load balancer though not strictly needed makes accessing services much easier, and an ingress.

Step-by-step documentation is here.

Setting up Helm

Helm is known as the “package manager for Kubernetes”, it is a tool that is used to template Kubernetes manifest files in a way that makes it easy to install new applications. I use Helm for all my service installation on Kubernetes due to it’s ease, and simplicity when getting started.

If you did not enable the optional Helm installation via Kubespray, we should do that first.

1. Download Helm

If you are on a Mac, you can install Helm via brew install kubernetes-helm. The Helm documentation has instructions for other methods of installation (unpacking a .zip and adding it to the $PATH).

2. Setup a Tiller user with RBAC

You can use this rbac-config.yaml file to setup the user, download it and run the following commands:

kubectl create -f rbac-config.yaml

3. Initialize Helm

Now we have to initialize Helm by deploying Tiller. The simplest method is to run helm init --service-account tiller. We have to specify the service account for tiller to use due to RBAC.

Persistent Storage

Since Docker, and by extension Kubernetes requires storage for any persistent data, it is important to provide Kubernetes some storage. If you were using a cloud provider, this would be handled for you, and you would use the providers default storage solution. In our case, we need to tell Kubernetes what storage to use.

We are going to use NFS storage for our cluster, which requires you to have an existing NFS server.

Once we have an NFS server, we are going to use the nfs-client-provisioner, which is available via Helm, to access that storage via Kubernetes storage class.

Simply run helm install stable/nfs-client-provisioner -name nfs-client -namespace kube-system -set nfs.server=10.0.40.5 -set nfs.path=/mnt/Shared/kubernetes

Let’s break that down a bit, the above command does the following:

Tell Helm to install the Helm Chart stable/nfs-client-provisioner
Give the release a name nfs-client
Install nfs-client to the kube-system namespace
Set the NFS Server IP to 10.0.40.5 you should change this as necessary
Set the NFS Path to /mnt/Shared/kubernetes you should change this to the share path on your NFS server

We should now check the status of nfs-client to ensure it is running correctly, via helm status nfs-client which should show ready.

Install MetalLB

MetalLB is a load balancer for Kubernetes that allows Kubernetes services to retrieve IP addresses when needed that are on the LAN or WAN. This greatly simplifies network access to services in a bare-metal (or VM) based Kubernetes cluster.

MetalLB has some requirements that we must comply with, namely, we have to have a Kubernetes cluster, a supported cluster network configuration, some IPv4 addresses, and optionally hardware that supports BGP. The first two are automatically handled by Kubespray in section 1, and for the IP addresses, you can provide a block that is outside your DHCP server’s scope on the same network as the Kubernetes nodes. I have not used the BGP option, as I do not have BGP hardware available.

You can install MetalLB via Helm with the following command helm install -name metallb -f values.yaml stable/metallb an example values.yaml file can be found here. Notice the configInline section that defines the IP addresses MetalLB will hand out.

Install Nginx-Ingress

Nginx-Ingress is an Ingress service for Kubernetes. Ingresses serve to manage external access to services in the cluster. It is important to understand how nginx-ingress works, so I recommend you read through the GitHub documentation.

Once you have done that, we can install Nginx-Ingress via Helm. You can review the values.yamlas an example, the main thing to change being the default-ssl-certificate that Nginx should use.

Once you have done that, you can install it via helm install -name nginx-ingress -f values.yaml stable/nginx-ingress -namespace nginx-ingress which does the following.

Tell Helm to install the Helm Chart stable/nginx-ingress
With the release name nginx-ingress
To the nginx-ingress namespace
With the values from values.yaml

Since the type is set to loadBalancer, MetalLB will automatically provide an external IP to the service from the provided IP addresses, which you can point your DNS or Port Forwards to in order to access your cluster’s services.

Install Cert-Manager

Cert-Manager is a complex service that allows us to automatically retrieve and renew SSL certificates from certificate providers. It works with Nginx-Ingress to ensure that all of our services are available via a secured connection over HTTPS.

There are a number of steps required to setup Cert-Manager correctly. You should start by reviewing the Helm Chart documentation here.

Prerequisites

Install the CustomResourceDefinition resources separately

kubectl apply -f [https://raw.githubusercontent.com/jetstack/cert-manager/release-0.7/deploy/manifests/00-crds.yaml](https://raw.githubusercontent.com/jetstack/cert-manager/release-0.7/deploy/manifests/00-crds.yaml)

Create the namespace for cert-manager

kubectl create namespace cert-manager

Label the cert-manager namespace to disable resource validation

kubectl label namespace cert-manager certmanager.k8s.io/disable-validation=true

Add the Jetstack Helm repository

helm repo add jetstack [https://charts.jetstack.io](https://charts.jetstack.io)

Update your local Helm chart repository cache

helm repo update

Installation via Helm

Make any changes to values.yaml notably the podDnsConfig/nameservers
Deploy the Helm Chart

helm install -name cert-manager -namespace cert-manager -f values.yaml jetstack/cert-manager

Verify the Installation

This is covered in the official documentation which you can find here. It has you create a test issuer and get a self-signed certificate to ensure everything is working properly.

Create the Production Issuer

Note : Creation of a test issuer is omitted for brevity, but please ensure you try creating a test issuer first so you do not hit the Let’s Encrypt rate limits.

If you are using Let’s Encrypt and Cloudflare, you can use this letsencrypt.yaml file. Remember to change the values as needed, especially anything marked with changeme.

Once you are ready, run kubectl apply -f letsencrypt.yaml to create the issuer.

Create Cloudflare API Key Secret

Now we need to add our Cloudflare API key into a Kubernetes secret. You can use cloudflare-secret.yaml as an example.

Once you have made your changes, create the secret with kubectl apply -f cloudflare-secret.yaml.

Create the Default Certificate

We can now create the default certificate that Nginx-Ingress will use for serving our sites. See certificate.yaml for an example.

Once you have made your changes to the file, with regard to the names and domains, you can create the certificate with kubectl apply -f certificate.yaml.

Create an Ingress

Now we can create an ingress for the Kubernetes dashboard, so instead of going to https://masternode:6443/proxy/ ect, we can go directly to https://kubernetes.domain.com

The first step is to create the dashboard ingress, which we can do with ingress.yaml once you have made your changes, you can create the ingress with kubectl apply -f ingress.yaml

You can now go to https://kubernetes.domain.com to access your Kubernetes dashboard, after you add the domain to DNS via the nginx-ingress external IP.

You now have a completely setup Kubernetes cluster running at home that you can test your services on, or use as a homelab.

Getting Started with Kubernetes (at home) — Part 1

Jax Gauthier — Sat, 27 Apr 2019 03:44:18 +0000

When you think about Kubernetes, you probably think AWS or GCP, a nice managed service where you can easily spin up resources and build applications on top of them. This is great, and honestly the best way to experience Kubernetes. However, if all you need is a lab to mess around in and experiment, or learn new things in, this can be very cost inefficient. That is why we are going to look at setting up Kubernetes ourselves.

In this post, we are going to look at the initial deployment of Kubernetes, from creating our nodes (in this case CentOS 7 VMs) to getting a cluster up and running.

Part 1: Installing our VMs

The first step is to create some VMs. I use a custom vCenter template in my lab, but if you do not have one of those, you can follow these simple steps.

You will need to complete these steps on at least 1 machine, however more is certainly better to get the full benefit of Kubernetes.

Your machine/VM should have at least 1 core and 3Gb of RAM.

Install CentOS 7 from the USB ISO image, a basic install is fine
Create a user for Ansible access. This user should be part of the sudo users group, and ideally have passwordless SSH authentication
Assign static IP Addresses to your hosts. Optionally set a hostname.
I hate to say it, but the official docs say to disable the firewall between the nodes, and I was unable to find documentation on which ports are needed.
Optionally, add your hosts to DNS.

Part 2: Kubernetes Installation

We are going to be using Kubespray for our cluster, as it makes creating and updating a Kubernetes cluster very simple and straightforward.

You can find the official docs here.

The quick guide is as follows:

Install the dependencies

sudo pip install -r requirements.txt
Copy the example Inventory

cp -rfp inventory/sample inventory/mycluster
Build the inventory, you can use the built in builder, or take a look here for an example. It is fine to have a single master, but the kube-master and etcd sections should be the same.
Deploy the cluster

ansible-playbook -i inventory/mycluster/hosts.yml --become --become-user=root cluster.yml

Then all you have to do is wait while Kubespray deploys your cluster automatically. On my 6 node cluster, it usually takes about 10–15 minutes for the cluster to be completely setup and running.

Note that in the Kubespray inventory there are a couple of options which are useful to enable. First, in the addons.yaml file, it is a good idea to enable Helm and the Kubernetes Dashboard automatic deployments. It may also be beneficial to enable kube_basic_auth in the k8s-cluster.yaml file, if you are having issues with the default token based authentication. If you decide to do this later, you can simply make the change and then re-run the deployment with the command in step 4 above.

Part 3: Testing the Cluster

You can test that your cluster is up and running with the following commands:

kubectl cluster-info which should return something like:

Kubernetes master is running at https://10.0.40.245:6444
kubectl get nodes which displays the state of all of your nodes.

You can find more information about how I have setup Kubernetes at my Gitlab repo, which has helpful code snippets, full configuration files, as well as expanded documentation.

What is Helm-Vault?

Jax Gauthier — Thu, 25 Apr 2019 16:01:00 +0000

Helm-Vault is a new application designed to protect secrets contained in Helm Chart’s values.yaml files.

The Problem:

The problem with using Helm with Kubernetes is that there is no good way to secure your private configuration items stored in the YAML configuration files.

There are multiple reasons you may want to do this, such as to audit who is accessing what secrets, or to prevent unauthorized modifications to the environment. Another reason may be that you want to have publicly available documentation without worrying about scrubbing the files.

Current Solutions:

Currently available solutions require you to significantly modify the Helm Chart, or encrypt the entire document with GPG or a hosted KMS solution.

This can be a pain to manage, who wants to use GPG all the time to work with files? What happens if the key is lost? What happens if the internet is down and you can’t access your KMS provider?

What makes Helm-Vault different:

With Helm-Vault, you get full access to the structure and content of the YAML documents, even when they are in an “encrypted” state, this provides you a lot more flexibility. When publishing charts, you can seamlessly provide your encrypted YAML files, and with the deliminator, provide easily notable locations where information needs to be changed.

If you run Hashicorp Vault internally, you will always have access to your secret data, and don’t have to worry about working with GPG keys, which makes life a lot easier for your developers.

Getting Started:

As long as you have Python 3.7, a working Hashicorp Vault environment, and a Vault token, you can get started with Helm-Vault in 2 easy steps:

pip3 install git+https://github.com/Just-Insane/helm-vault
helm plugin install https://github.com/Just-Insane/helm-vault

You can find information about using Helm-Vault in the README.

If you have any questions or concerns, feel free to open an issue on the project’s GitHub Issue Tracker.

The Importance of Security Testing

Jax Gauthier — Wed, 24 Apr 2019 23:08:28 +0000

While setting up a new Keycloak client in my lab over the weekend, I discovered something odd, a number of users in my Keycloak database who should not have been there.

Background:

In my homelab environment, I have been using Keycloak for securing my services, either directly for applications like Nextcloud, Confluence, and Jira, or indirectly, via a authenticated reverse proxy, for things that don’t such as single page applications like Hashicorp’s Consul or Jupyter. This has been a fine setup, and has allowed me to access my internal applications easily and securely from anywhere, with 2FA and seamless access when moving between services. Or so I thought.

Discovery:

After noticing some odd users in my Keycloak user’s list, I decided to do some digging. I knew I had initially allowed sign-ups to Keycloak to very locked down services, like read only access to parts of Confluence, and my internal issue tracker on Jira. These users were put into a default role in Keycloak called External-Users, which had very limited access to services on my network.

I decided to create a new user and see what all could be accessed, for the hell of it. I opened a new private window so that I could ensure I was not using cookies from my authenticated session, and went to work creating a new user. I was happily greeted with a Duo page (Duo is a 2FA provider that you can integrate with to protect many different services), until I realized there was an option to enroll myself as a user in Duo. That is when I realized there was an issue.

Breach:

After registering myself as a new user in Duo, I started to poke around, going first to Confluence and Jira, and seeing that I still had my limited permissions, was a bit relived. Then I went to Consul, and the page opened right to the main app, showing all my services humming away happily. Next I went to Jupyter, and immediately was greeted with the option of creating a new shell session.

Containment:

As soon as I was able to get access to Jupyter, I disabled signups on Keycloak, cleaned out the unknown users (some of which used legitimate emails), and set Duo to not allow users to enroll themselves. I also ensured there were no active sessions in Keycloak and started locking my network down to ensure there was nobody still inside the perimeter.

Root Cause:

The root cause of this issue was my reverse proxy. Knowing that most of my applications were secured behind it, especially the ones which do not handle their own authentication, I tore it apart. I didn’t have to look far to see the error in my ways.

Almost all of my proxied services had this block in the server section (private information omitted):

access_by_lua '
 local opts = {
 discovery = "$URL/.well-known/openid-configuration",
 redirect_uri_path = "/redirect_uri",
 redirect_uri_scheme = "https",
 client_id = "OpenResty",
 ssl_verify = "no",
 client_secret = "$ClientSecret",
 session_contents = {id_token=true}
 }
 local res, err = require("resty.openidc").authenticate(opts)

if err then
 ngx.status = 403
 ngx.say(err)
 ngx.exit(ngx.HTTP_FORBIDDEN)
 end
';

Let’s break that down a bit, shall we?

My reverse proxy is OpenResty, a modified Nginx server setup to run Lua. On CentOS7 you have to build Nginx from source to add an OIDC authentication plugin, with OpenResty, it is built in.

First, we define some options for the OIDC authentication:

The URL of the openid configuration information
The path to redirect to, this is a URI that does not exist in the backend application that should be redirected to after authentication with Keycloak so the proxy can check the authentication
The redirect scheme
The Keycloak Client ID
If SSL for Keycloak should be verified (this should be set to yes)
The client secret that you get from Keycloak
The session contents, in this case, we are just looking for the id_token
The authentication check, more on this below
What to do if there is an error, in this case, send a 403, print the error, and set HTTP_FORBIDDEN, then exit

Looking at the authentication check, do you see the problem? It is only checking that the person accessing the protected resource is authenticated to Keycloak. It does not check for their group, roles, or even to ensure they have a valid email address for the domain.

Since anyone could sign up to Keycloak, and anyone could enroll themselves in Duo, they would be authenticated, and would therefore pass this check automatically, giving anyone who tried access to these internal applications.

It Gets Worse:

Knowing that I had setup external Identity Providers in Keycloak, such as Facebook and Google (adding external Identity Providers in Keycloak lets people sign in with other services which handle their own authentication), I went back and tested these out.

I closed out my current private window, and opened a new one, going back to my Consul URL, and clicking on the login button for Google. When prompted, I entered a spare email address and password, and once I got through the Google verification, I was redirected back to Keycloak, and then to the main Consul page.

After checking in Keycloak, I didn’t see a new user created with this email. Since authentication was handled by Google, all Keycloak was caring about was if the user successfully authenticated with Google, and just passed the user back to the proxy as being authenticated in Keycloak.

Not only could anyone sign up for an account in Keycloak and add themselves to 2FA, they could completely bypass these steps without raising suspicion and not creating any user records.

Lessons Learned:

The lessons I learned through this were many, the first is to always double check everything, especially when it is security related. If I had checked these authentication flows when initially setting everything up, I would have caught these issues much sooner and this would not have been a problem.

Second, it is important to understand and read the documentation fully. This could have been easily mitigated or completely prevented by adding more checks to OpenResty, disabling sign up or third party authentication in Keycloak, or preventing user self-enrollment in Duo.

Third, check everything. If you enable a feature, ensure it does exactly what you are expecting it to do. If not, it is best to disable it, or get clarification as to why it is doing what it is.

Conclusion:

I messed up, spectacularly. Luckily, it was in a test environment and not production, and there was nothing overly sensitive on my network.

This is a mistake that I will not forget, and that I will bring with me in my career.

There is no inherent issue with the tools that I used, or even a combination of tools, it was purely a configuration issue on my part. I would still 100% recommend all of the tools mentioned in this post, and intend to keep on using them. I have gained more respect for them, and intend to put them to use to further secure my network in the future.

Future Thoughts:

As I rebuild my lab with a focus on Kubernetes and Zero Trust network design, the lessons I learned here will guide me in ensuring that everything is as secure as possible.

Keycloak SSO Part 2: Setting up Keycloak

Jax Gauthier — Wed, 24 Apr 2019 01:25:48 +0000

In part two of this series, we are going to look at setting up a standalone-ha deployment of Keycloak on two CentOS 7 servers.

There are three different deployment types for Keycloak, Standalone, Standalone-HA, and Domain Clustered. Standalone deployments are single servers, this is good for a dev or test environment, but not very useful for production use. Standalone-HA are one or more servers which can both be used to serve authentication requests. This method requires a shared database, and each server is configured manually. In a Domain deployment, there is a master server known as the domain controller, and one or more host controllers which serve authentication requests. This mode allows the host controllers to all have an updated configuration when it is changed on the domain controller, greatly reducing administration overhead with multiple servers.

Installation

Hardware requirements, as well as distribution directory structure and operation mode information can be found at https://www.keycloak.org/docs/latest/server_installation/index.html#installation

I use Ansible to deploy the folder structure for Keycloak and make sure all dependencies are setup correctly. This allows me to easily update and ensure that all my Keycloak servers are deployed correctly. My playbook calls https://github.com/andrewrothstein/ansible-keycloak with some configuration file changes.

Configuration

Reading and understanding the official documentation is essential to installing Keycloak in a secure manner, I highly recommend you follow the information there and use my configuration as a guide.

Operation Mode

The first thing to think about when deploying Keycloak is what operation mode you want to use. This will mostly come down to your environment, and the configuration of most modes is the same, just within different files. I am most experienced with Standalone-HA mode, so that is what we are going to work with in this series.

Configuration for this mode is done in the standalone-ha.xml configuration file found at $keycloak_home/standalone/configuration/standalone-ha.xml. This file needs to be edited on all servers in a standalone-ha cluster setup.

Relational Database Setup

The next thing we have to do is setup Keycloak to use a database, since we are going to be creating a deployment with multiple servers, we are going to need a shared database. Setting up a centrally accessible database is beyond the scope of this article. Just know that we are going to be using a PostgreSQL database that is hosted outside of both of the Keycloak servers.

Download a JDBC driver

The first step in setting up a database for Keycloak is to download a JDBC driver for the database. This allows Java to interact with the database. You can usually find these on the main site of your chosen database. For example, PostgreSQL's JDBC driver can be found here: https://jdbc.postgresql.org/download.html

Package the driver JAR and install

The official documentation is a good resource for how to package the driver for use with Keycloak, and there is no point in duplicating efforts. It can be found here: https://www.keycloak.org/docs/latest/server_installation/index.html#package-the-jdbc-driver

This boils down to adding a folder structure, copying the .jar file, and adding an .xml file like the following:

<?xml version="1.0" ?>
<module xmlns="urn:jboss:module:1.3" name="org.postgresql">

    <resources>
        <resource-root path="postgresql-9.4.1212.jar"/> <!-- update the filename to match your PostgreSQL JDBC driver file name -->
    </resources>

    <dependencies>
        <module name="javax.api"/>
        <module name="javax.transaction.api"/>
    </dependencies>
</module>

Making sure to update the path with the correct file name.

Declare and load the driver

This part, as well as modifying the datasource are a bit more advanced, so I will go over them in a bit more detail here, however the documentation is still very helpful.

We are going to look at the standalone-ha.xml file we were working on earlier, specifically the drivers XML block. In this block, we will be adding an additional driver. We can mostly copy the existing format of the h2 driver, and update the information for PostgreSQL. Below is an example of a driver in standalone-ha.xml

<drivers>
    <driver name="h2" module="com.h2database.h2">
        <xa-datasource-class>org.h2.jdbcx.JdbcDataSource</xa-datasource-class>
    </driver>
    <driver name="postgresql" module="org.postgresql">
        <xa-datasource-class>org.postgresql.xa.PGXADataSource</xa-datasource-class>
    </driver>
</drivers>

As we can see, the declaration of the driver is nearly identical to the pre-configured H2 database driver.

Modify the Keycloak Datasource

Below we will see an example of a working PostgreSQL datasource configuration.

<datasource jndi-name="java:jboss/datasources/KeycloakDS" pool-name="KeycloakDS" enabled="true" use-java-context="true">
    <connection-url>jdbc:postgresql://$URL:$PORT/$DATABASE</connection-url>
    <driver>postgresql</driver>
    <pool>
        <max-pool-size>20</max-pool-size>
    </pool>
    <security>
        <user-name>$USERNAME</user-name>
        <password>$PASSWORD</password>
    </security>
</datasource>

$URL = The URL or IP Address of the PostgreSQL server

$PORT = The port to connect to PostgreSQL database

$DATABASE = The name of the database that is configured for Keycloak

$USERNAME = The username that has access to the database specified above

$PASSWORD = The password of the user defined above

In the end, you should end up with a datasources section that looks like the following:

<subsystem xmlns="urn:jboss:domain:datasources:5.0">
    <datasources>
        <datasource jndi-name="java:jboss/datasources/KeycloakDS" pool-name="KeycloakDS" enabled="true" use-java-context="true">
            <connection-url>jdbc:postgresql://$URL:$PORT/$DATABASE</connection-url>
            <driver>postgresql</driver>
            <pool>
                <max-pool-size>20</max-pool-size>
            </pool>
            <security>
                <user-name>$USERNAME</user-name>
                <password>$PASSWORD</password>
            </security>
        </datasource>
        <drivers>
            <driver name="h2" module="com.h2database.h2">
                <xa-datasource-class>org.h2.jdbcx.JdbcDataSource</xa-datasource-class>
            </driver>
            <driver name="postgresql" module="org.postgresql">
                <xa-datasource-class>org.postgresql.xa.PGXADataSource</xa-datasource-class>
            </driver>
        </drivers>
    </datasources>
</subsystem>

Clustering

The above steps will get a basic setup going with a shared database, however to properly cluster Keycloak, there are a few more steps that need to be completed.

The relevent sections from the Keycloak documentation is below:

We have already completed steps 1 and 2 in setting up a cluster. There are some additional setup needed for the next two operations, parts of which are made serviceable here, but are covered in more detail at the above links.

Set up a load balancer

Identifying client IP addresses

It is very important that Keycloak is able to identify client IP addresses for various reasons, which are explained further in the docs. We will go over the changes that have to be made in standalone-ha.xml here.

You will need to configure the urn:jboss:domain:undertow:6.0 block to look like below:

<subsystem xmlns="urn:jboss:domain:undertow:6.0">
   <buffer-cache name="default"/>
   <server name="default-server">
      <ajp-listener name="ajp" socket-binding="ajp"/>
      <http-listener name="default" socket-binding="http" redirect-socket="https" proxy-address-forwarding="true"/>
      ...
   </server>
   ...
</subsystem>

Enable HTTPS with a Reverse Proxy

If you have a reverse proxy in front of Keycloak which handles your SSL connections and terminations, you need to make the following changes:

In urn:jboss:domain:undertow:6.0 block (configured above) change the redirect-socket from https to a socket binding which we will define.

<subsystem xmlns="urn:jboss:domain:undertow:6.0">
    ...
    <http-listener name="default" socket-binding="http"
        proxy-address-forwarding="true" redirect-socket="proxy-https"/>
    ...
</subsystem>

We will now need to add a new socket binding to the socket-binding-group element, like below:

<socket-binding-group name="standard-sockets" default-interface="public"
    port-offset="${jboss.socket.binding.port-offset:0}">
    ...
    <socket-binding name="proxy-https" port="443"/>
    ...
</socket-binding-group>

Testing the Cluster

Once the changes have been made on all of your Keycloak servers, we can manually start the Keycloak servers in any order. The command for doing so is

bin/standalone.sh --server-config=standalone-ha.xml

from the Keycloak home directory. The Keycloak servers will automatically configure themselves if they are connected to the same external database, and you can use your load balancer or reverse proxy to connect to either server to perform authentication operations.

Firewall

Ensure you have configured the firewall correctly, Keycloak listens on ports 8080 and 8443 by default. There may be additional ports that need to be opened based on your configuration.

Running at boot

Assuming your tests have passed and you can reach both of your Keycloak servers directly and through your load balancer, you are ready to setup a systemd unit file and have Keycloak start at boot.

Below is a copy of the systemd unit file I am using, which is placed at /etc/systemd/system/keycloak.service:

[Unit]
Description=Keycloak Identity Provider
After=syslog.target network.target
Before=httpd.service

[Service]
Environment=LAUNCH_JBOSS_IN_BACKGROUND=1 JAVA_HOME=/usr/local/java
User=keycloak
Group=keycloak
LimitNOFILE=102642
PIDFile=/var/run/keycloak/keycloak.pid
ExecStart=/usr/local/keycloak/bin/standalone.sh --server-config=standalone-ha.xml
#StandardOutput=null

[Install]
WantedBy=multi-user.target

Once you have completed this step, you can start and enable the service by running the below commands on all of your Keycloak servers:

systemctl enable keycloak

systemctl start keycloak

Keycloak SSO Part 1: What is single sign on?

Jax Gauthier — Thu, 06 Dec 2018 21:00:00 +0000

In this series, we will take a look at Keycloak, an open-source single sign on solution similar to Microsoft's ADFS product.

In this first part, we will take a brief look at what SSO and Keycloak are, and why they are used.

What is Single Sign On

Single sign-on (SSO) is a property of access control of multiple related, yet independent, software systems. Wikipedia

What this means is that we can use a single authentication service to allow users to login to other services, without providing a password to the service that is being logged into.

This is different than connecting each of these applications to LDAP, as this requires providing a username and password to every service that you want to login to.

Benefits of SSO

Central location for authentication allows for easier auditing and security.
Central location for configuration of authorization, ie, Jane is able to access email but not the CRM suite
Authentication to external services such as a hosted CRM suite are made possible without sending LDAP requests over the internet
Tokens are wrapped in SSL/TLS as part of the HTTPS connection, but can also be both signed and/or encrypted using keys known only between the identity provider and service for increased security

What is Keycloak

Keycloak is an open source program that allows you to setup a secure single sign on provider. It supports multiple protocols such as SAML 2.0 and OpenID Connect. It can also store user credentials locally or via an LDAP or Kerberos backend.

These features allows Keycloak to be highly configurable, but also fairly easy to install and setup.

More information about Keycloak can be found here: https://www.keycloak.org

Basic Workflow

The basic workflow when authenticating to a service that uses SAML/OIDC from the users perspective is as follows:

User goes to the web address of a SSO protected service (known as a service provider, or SP).
The service asks for a cookie that the users browser may have stored, which contains a token, if it finds a valid token in the browser, it logs the user in. If this token does not exist, or is invalid, the service directs the user to the configured Identity Provider (IDp).
Once the user reaches the IDp, they are presented with a login page.
The user logins in with their provided credentials. The IDp can compare these to locally stored credentials, or against an LDAP or Kerberos backend.
The user is given a token upon successful login which is stored as a cookie in the browser, and gets automatically redirected back to the service they were initially attempting to access. This token usually contains a username, as well as information regarding what the user has access to, if using a protocol like OIDC that supports authorization.
The service requests the token from the browser and if the token is valid, allows the user access to the service.

Token validation is done on a secure back channel between the service and Identity Provider, without involving the browser, to increase security.

Backing up FreeNAS to Backblaze B2

Jax Gauthier — Wed, 27 Sep 2017 23:09:00 +0000

If you use FreeNAS, it’s probably because you care about your data. Part of data security is ensuring the availability of your data. To that end, you need to ensure that said data is backed up. There are generally two reasonable ways to backup your data from FreeNAS. One, local backup (using ZFS replication), and two, cloud backup.

In this article, we will look at setting up cloud backups to Backblaze B2, an economical cloud backup solution similar to Amazon S3.

Step 1: Sign up

Sign up for a Backblaze account here. Once you have created an account, go to the “My Settings” tab, and under “Enabled Products”, check the box beside B2 Cloud Storage. This enables your account for using B2.

Step 2: Create a Bucket

Once you have enabled your account for B2, you need to create a bucket (where your files are stored). To do this, on the left side of the screen, select “Buckets” under B2 Cloud Storage. Then, select Create a Bucket. Also on this page, be sure to click on “Show Account ID and Application Key”, and mark down your Account ID and click “Create Application Key”. Also mark this down, as you will not be able to see it again, and will need it when we setup rclone in a later step.

Step 2a: Setup Caps and Alerts

While this step is optional, it is highly recommended so that you get notified about any charges against your account that you may not be expecting. I set mine to a cap of $1 a day for each section. This will give you 6Tb of storage, and a good number of API calls.

Step 2b: Have a look around your account

Have a look around your Backblaze account, there is a great get started guide available here.

Step 3: Setting up a FreeBSD Jail on FreeNAS

Login to your FreeNAS GUI, and go to the Jails section.
Click “Add Jail”.
Enter a Jail Name. I called mine “b2-backups”.
Click Ok, and your jail will be created. Note that this may take a little bit of time. You should be able to close the dialog box if needed, the jail will be created in the background.
Click on your Jail and click the “shell” button in the bottom left. This will open a shell session to the Jail.
Enter vi /etc./rc.conf and change sshd_enable="NO" to sshd_enable="YES". This will enable SSH to the jail.

! FreeBSD uses vim as a text editor, use i to insert text, del to delete the rest of the line, and the arrow keys to scroll. Save and exit by pressing the ESC key and then :we to save and quit.

! You will need to run passwd root and reboot in order to have SSH access, as well as PermitRootLogin yes in /etc/ssh/sshd_config.

!!!! At this point, you can switch over to SSH, if you prefer that to the shell in the FreeNAS GUI.

Install wget using pkg install wget, this will allow you to download the rclone binary.
Download the latest rclone binary: cd /tmp && wget https://downloads.rclone.org/rclone-v1.37-freebsd-amd64.zip.
run unzip rclone-v1.37-freebsd-amd64.zip to extract the binary. !!!! rclone version 1.37 is the latest stable release at the time of this writing
Copy the rclone executable to /usr/bin by running cd rclone-v1.37-freebsd-amd64 && cp ./rclone /usr/bin

Step 3a: Adding storage to the Jail

Create a new folder structure in the Jail, I put mine in /mnt/storage, where you will mount your FreeNAS datastores. It is a good idea to make a folder for each dataset you want to mount.
In the FreeNAS GUI, go to the Jails tab, and then the Storage sub-tab.
Click “Add Storage”
Select the Jail you want to add the storage to.
Select the source dataset.
Select the destination (this will be the folder structure in the jail that you created in Step 3a-1).
Optionally select read-only.
Leave “Create Directory” selected.
Click “Ok”.
Repeat steps 3a-4 to 3a-9 for each dataset you want to backup to B2.

Step 4: Configuring rclone

Run rclone config to initiate the configuration of rclone
Press n to create a new remote (a remote is what rclone uses to know where to copy/sync your files).
Enter a name, I choose b2.
Press 3.
Enter your account ID from your B2 account.
Enter your application Key from your B2 account.
Leave endpoint blank.
Press y to save the config.

Step 4a: Configuring encryption

Follow steps 1-3 from Step 4. ! Note, name this new remote different than the previous remote.
Press 5.
Enter the name of the remote you created in Step 4, number 3, followed by the name of your bucket. For example, b2:storage in my case.
Choose whether or not you want to encrypt the file names, selecting 1 does not encrypt file names. Selecting 2 encrypts the file names. I choose 2.
Choose y to type in your own password, choose g to generate a strong password randomly. If you choose g, you are given an option as to how strong of a password you want to generate.
Create a password for the salt. This is recommended if you have chosen to enter your own password in the previous section. Note that for security, these passwords should not be the same.
Select y to accept the configuration.

!! Note: The rclone config file is not encrypted by default, and Application Keys and your encryption passwords are stored in plaintext. It is recommended to set a password for the config file, and/or ensure the security of the rclone.conf file.

!! If you need to recover encrypted files from B2, you NEED both passwords (if you set two), otherwise your files will be completely unrecoverable.

Step 4b: Creating the bash script

In this section, we will look at creating the bash script we will use with cron in order to backup any changes to our local storage to B2.

Create a new file in /root, I called mine rclone-cron.sh.
Copy the following:

!/bin/sh if pidof -o %PPID -x "rclone-cron.sh"; then exit 1 fi echo starting storage sync rclone copy {/path/to/local/storage} {name of your crypt remote}: -v --log-file={/path/to/log/file} --min-age 15m --copy-links exit

Let’s break that down a bit and look at what the script actually does.

!/bin/sh - run the script with the sh terminal.

if pidof -o %PPID -x "rclone-cron.sh"; then - If the script is currently being run, then:

exit 1 - Do not run the script currently. This is good if your initial backup will take a while to run, as it won’t try to run rclone again.

fi - closes the if statement.

echo start storage sync - print to the terminal that the clone is starting.

rclone copy {/path/to/local/storage} {name of your crypt remote}: -v --log-file={/path/to/log/file} --min-age 15m --copy-links - runs rclone with the copy parameter (does not delete files deleted locally, alternatively change copy to sync to keep an exact copy on B2 (deletes files from B2 that are deleted locally)). Uses the -v flag for verbosity. --log-file={/path/to/log/file} Tells rclone where to create a log file. --min-age 15m Tells rclone not to sync files less than 15 minutes old, useful to ensure copied files are probably complete, instead of semi-completed. --copy-links Tells rclone to follow slinks.

exit - exits the script when the copy is finished.

Step 4c: Creating the cron entry

Run crontab -e to open the cron editor.
Enter 0 1 * * * /root/rclone-cron.sh - This will run the script we created in 4b once a day.

Step 5: Run the script!

chmod +x /root/rclone-cron.sh - makes the script executable
cd /root/ && ./rclone-cron.sh - runs the script. ! rclone does not run in the background. It is recommended to run the script in tmux or similar, or wait for the crontab to run, as the initial backup will probably take a long time if you have a lot of data like I do.

Step 5a: Check B2 console to see if it’s working.

Log into your back blaze account, and take a look at your bucket. You should see that files are being copied to B2.

This completes the guide on setting up rclone to backup to B2 on FreeNAS. Rclone can backup to many cloud providers, have a look at different providers if Backblaze is not your cup of tea.

Nginx: Reverse Proxy

Jax Gauthier — Mon, 25 Sep 2017 02:00:00 +0000

In this article we will look at what a reverse proxy is, as well as how to set one up on CentOS using Nginx.

What is a Reverse Proxy

A reverse proxy is a type of proxy server which retrieves resources on behalf of a client, from one or more servers. The collected information is then returend to the client as if it originated from the web server itself. It is the opposite of a forward proxy (one which allows it’s clients to contact any server), a reverse proxy allows it’s servers to be contacted by any client.

Why do I need a Reverse Proxy

You would need a reverse proxy for a number of potential use cases. The most common reason to use a reverse proxy is to increase security. For example, a reverse proxy can be a hardend computer in your DMZ, which then takes client traffic from the internet, and passes it to your internal web application servers. This would have two main security benefits:

Lower attack surface: Each system you expose to the internet increases the attack surface to get into your network. It only takes the compromise of one of those systems to wreak havoc on all of your internal systems.

Easier to patch/manage/secure: Having less resources exposed to the internet also means that you have less critial infrastructure to patch, manage, and secure. Of course, you should still definitely be doing patches on your internal systems, however, the best security is defence in depth, basically, having as many hard layers to go through before the attack is able to breach your inner network. Having a reverse proxy increases the number of layers before getting into your network.

Another reason you would want to setup a reverse proxy is if you have a number of resources inside your network, such as web and application servers/services, which you want to all be accessable outside of your network, but you only have one public IP address. Instead of having websites on different services, you can put all of those sites behind your reverse proxy on one IP and port (80/443), and then have clients connect to the proper backend based on the (sub)domain on which they are trying to connect.

Reverse Proxies also decrease your management costs, for example, instead of getting an SSL certificate and installing it on every web server you have, and then managing all of those certificates, you can get one certificate (either wildcard or SAN), and install it on your reverse proxy. This way, all traffic between the client and web server is encryted, but you don’t have to spend extra time and money dealing with installing certificates on each service behind the proxy. Note however, that doing this means that traffic between the reverse proxy and web server (inside your network) is unencrypted, unless you configure the internal site for HTTPS. This is usually not a big deal for a homelab environment or testing, but please be advised that it is less secure and should not be used in production environment. If you are in a production environment, ensure that traffic is encrypted between the reverse proxy and the internal application server.

What Reverse Proxy should I use

There are many types of reverse proxies, from dedicated enterprise hardware/software like F5 Big-IPs, all the way down to a regular web server, like Nginx or Apache. Each of these will have different Pros and Cons, which you should research yourself. However, for a small blog or homelab environment, a reverse proxy running Nginx will be fine.

Installing Nginx as a Reverse Proxy

Now that we have looked at what a Reverse Proxy is, and why we might want to use one, lets look at installing Nginx as a reverse proxy.

Step 1: Installing Nginx

In this step, we will look at installing Nginx on CentOS 7. Note that these steps should be similar for your prefered distribution, however, I make no promises.

First, update packages on your CentOS server:

$ sudo yum -y update

Install Nginx:

$ sudo yum -y install nginx

Start Nginx and enable (so that it starts at boot:

$ sudo systemctl start nginx
$ sudo systemctl enable nginx

Check that Nginx is running:

$ sudo systemctl status nginx

Once Nginx is running, we need to open the firewall so that our clients can access it.

$ sudo firewall-cmd --permanent --add-service=http
$ sudo firewall-cmd --permanent --add-service=https
$ sudo firewall-cmd --reload

Now that we have done that, we should verify that we are actually able to reach nginx. Go to http://<ip_of_your_host> in a web browser, and you should get an Nginx page. If you do not, ensure that Nginx is running, and that you have opened the firewall.

Step 2: Configuring Nginx

In this step, we will configure some sane defaults for our Nginx installation, including SSL and Proxy defaults.

First, let’s look at using Nginx’s snippets feature to configure the default SSL and Proxy settings in Nginx.

Proxy Defaults

In /etc/nginx/snippets/, create a file called proxy-defaults.conf, with the contents of:

proxy_redirect off;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $remote_addr;

Then add include snippets/proxy-defaults.conf to your server block.

SSL Defaults

First, let’s take a look at setting up the SSL certificates and dhparam files needed.

Getting an SSL certificate

We will look at setting up Let’s Encrypt, a free certificate authority.

Step 1 - Installing the Certbot Let’s Encrypt Client

Enable the EPEL repository:

$ sudo yum -y install epel-release

Install certbot-nginx:

$ sudo yum -y install certbot-nginx

For Cerbot to work, you need to specify your server name in /etc/nginx/nginx.conf

$ sudo vi /etc/nginx/nginx.conf

Find the existing server_name line:

server_name _;

Replace the _ underscore with your domain name:

server_name example.com www.example.com;

Verify that the syntax of the configuration file is correct:

$ sudo nginx -t

If this completes without errors, reload Nginx to load the new configuration:

$ sudo systemctl reload nginx

Step 2 - Obtaining a Certificate

There are a number of ways to obtain certificates from Let’s Encrypt through Certbot. We will use the Nginx plugin to take care of reconfiguring Nginx and reloading the config when certificates are updated.

$ sudo certbot --nginx --rsa-key-size 4096 -d example.com -d www.example.com

This will run certbot with the --nginx plugin, a key size of 4096 (more secure than the default 2048), and use -d to specify the domain names we would like the certificate to be valid for.

! You need to add every sub-domain you want to use Let’s Encrypt certificates, seperated with the -d flag.

If this is the first time you’ve run certbot on this server, you will be prompted to enter an email address and agree to the terms of service. It will then test to ensure that it can communicate with Let’s Encrypt’s servers.

Once that is complete, you will be asked how you’d like to configure HTTPS on your server. Choose the option you’d like to use, and then press Enter. You will then be told where your certificates are stored on the server.

Step 3 - Updating Diffie-Hellman Parameters

Currently, if you test your server using the SSL Labs test, you will notice that you only get a B grade. We can fix this be creating a new dhparam.pem file and adding it to our ssl-defaults.conf file (see the next section).

$ sudo openssl dhparam -out /etc/ssl/certs/dhparam.pem 4096

This will take a while. Make note of where the dhparam.pem file is located, as it will be needed in the next step.

Step 4 - Setting Up Auto Renewal

Let’s Encrypts certificates are only valid for ninety days. Therefore, we will need to setup a command to run regularly, which will check for expiring certificates and renew them automatically.

We will run the certbot renewal command daily, using cron. To do so, we need to edit a file called a crontab.

$ sudo crontab -e

Paste the following line, then save and close the file.

15 3 * * * /usr/bin/certbot renew --quiet

Let’s break that out and look at what the command is doing:

15 3 * * * - Run the following command at 3:15 am, every day. You may choose any time to run the command at.
/usr/bin/certbot - Calls the certbot utility.
renew - Tells Certbot to check all certificates installed on the system and update any that are set to expire in less than thirty days.
--quiet - Tells Certbot not to output information or wait for user input.

cron will now run this command daily. All installed cerificates will be automatically renewed and reloaded when there is thirty or less days before they expire.

Creating the ssl-defaults.conf snippet

In /etc/nginx/snippets/, create a file called ssl-defaults.conf, with the contents of:

# These next two lines depend on where certbot placed your SSL certificates
# edit them to match the location of the files
# ssl_certificate /path/to/ssl/certificate
# ssl_certificate_key /path/to/ssl/certificate_key

ssl_dhparam /etc/ssl/certs/dhparam.pem;
ssl_prefer_server_ciphers on;
ssl_protocols TLSv1 TLSv1.1 TLSv1.2;
ssl_ciphers 'ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-DSS-AES128-GCM-SHA256:kEDH+AESGCM:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA:ECDHE-ECDSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-DSS-AES128-SHA256:DHE-RSA-AES256-SHA256:DHE-DSS-AES256-SHA:DHE-RSA-AES256-SHA:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA256:AES256-SHA256:AES128-SHA:AES256-SHA:AES:CAMELLIA:!aNULL:!eNULL:!EXPORT:!DES:!RC4:!MD5:!PSK:!aECDH:!EDH-DSS-DES-CBC3-SHA:!EDH-RSA-DES-CBC3-SHA:!KRB5-DES-CBC3-SHA';
ssl_session_timeout 1d;
ssl_session_cache shared:SSL:50m;
ssl_stapling on;
ssl_stapling_verify on;
# The following setting can be dangerous, please research what it does before uncommenting.
# add_header Strict-Transport-Security max-age=15768000;

Then add include snippets/ssl-defaults.conf to your server block.

!! Note: These defaults may not fit your needs in every case. You may have to configure different settings in your server block.

! Also note that if you need to change these default settings, just remove include snippets/<item>-defaults.conf from the server block.

These default settings allow you to get an A+ from SSLLabs.

Default Server Block Template

Server blocks should be added to /etc/nginx/sites-available. The file should be named <subdomain>.example.com.conf.

server {
    listen 443 ssl;
    server_name www.example.com;
    ssl on;
        # Remember to comment these out if you need to change their defaults
        include snippets/ssl-defaults.conf;
        include snippets/proxy-defaults.conf;

    location / {
        proxy_pass http://<internal_ip>:<port>/;
        # Input any other settings you may need that are not already contained in the default snippets.
    }
}

After creating your server block, you need to link it to /etc/nginx/sites-enabled/ using the following command.

$ sudo ln -s /etc/nginx/sites-available/www.example.com.conf /etc/nginx/sites-enabled/

Test the Nginx config using the following command:

$ nginx -t

Once you have configured all of these settings, you need to reload nginx, so that it has the new config.

$ systemctl reload nignx

You should now be able to reach your internal site from the internet.

Ansible Part 2: Installing Telegraf

Jax Gauthier — Sat, 23 Sep 2017 04:00:00 +0000

In part two of this series, we will look at creating a playbook to deploy and configure Telegraf, the popular devops service for getting your system’s metrics into a time series database, for use with tools like Grafana.

As always, a link to the completed playbook can be found ~~here~~.

---
- hosts: all

  tasks:
  - name: Add telegraf repository
    yum_repository:
      name: influxdb
      description: InfluxDB Repository - RHEL $releasever
      baseurl: https://repos.influxdata.com/rhel/$releasever/$basearch/stable
      gpgcheck: yes
      gpgkey: https://repos.influxdata.com/influxdb.key

  - name: Install telegraf
    yum:
      name: telegraf
      state: latest

  - name: Copy config
    copy:
      src: ./telegraf.conf
      dest: /etc/telegraf/telegraf.conf
      owner: root
      group: root
      mode: 0644
      backup: yes
      force: no
      notify:
      - restart telegraf

  handlers:
  - name: restart telegraf
    service:
      name: telegraf
      state: restarted

This playbook is a bit more advanced then the last one we looked at, however, it gives you a good overview on the structure of a simple playbook, and how to define multiple tasks.

Let’s break it down below.

! Note that I won’t explain parts that are the same across this playbook and the previous, which can be found here.

First, we can see that there are three tasks, and a handler defined. Handlers are a way to tell Ansible to do <something>, but only when a task actually changes or updates something. In this playbook, the handler tells Ansible to restart telegraf if the final task in the playbook makes a change, otherwise telegraf won’t be restarted.

Second, let’s look at the first task. Called “Add telegraf repository”, in this task, we will find a new component of Ansible, called “yum_repository”, which will add a Yum repository to our system. Below that, we will see the vairables that are used for the repository. These are what gets added to the .repo file, and are based on the repository we are trying to add. In this file, you will see the information needed to install Telegraf.

Third, in the second task, you will see that we are again using the yum module to install a system service. For more information, please see the previous blog post in this series.

Fourth, we will find a task called “Copy config”, and as it sounds, it will copy a pre-made telegraf.conf file into the correct location for use by telegraf. Let’s take a further look at it.

This task is also using a new module called copy, and below that, we can see some of the available variables for the module. The first five lines are pretty basic, they define the source location for the file, the destination on the remote system, as well as the owner and permissons on the file. The next three lines are a bit more advanced.

backup tells Ansible to “Create a backup file including the timestamp information so you can get the original file back if you somehow clobbered it incorrectly”, this is very important if you are overwriting configuration files for programs, in the case that something breaks, you can always revert to the default configuration file if one exists.

force: no tells Ansible to only copy the file to the remote system if it does not already exist. “The default is yes, which will replace the remote file when contents are different than the source. If no, the file will only be transferred if the destination does not exist.” In my case, this file should not be changing once it’s on the remote system, setting force to no saves me a couple of seconds off the total run time of the playbook, and if I do ever change the conf file, I can easily change or remove the force variable from the playbook.

notify This is an interesting argument, but one that is very important to creating a strong playbook in Ansible. As I mentioned previously, the notify command can be used to call handlers. Handlers can do many things, but in this playbook, we only use them to programatically tell Ansible if we want to restart the telegraf service. If you want to learn more about Handlers, please see the Ansible documentation here.

Now we will look at the handlers section of the playbook. As you can see, the handler’s section looks very similar to a task, and that’s because it is. They are tasks that only run when you call them when something is changed. In our example, it doesn’t make sense to restart telegraf every single time the playbook is run, as this leads to downtime in our metrics, and that is generally not a good thing, and leads to a lot of alerts being sent out (you have alerts setup, right?).

This completes our post on using Ansible to setup Telegraf, as well as looking at a slightly more advanced playbook then we did previously.

P.S. If you want to see what you can do with Grafana and system metrics, try pressing the “Dashboard” button in the top left of the page, or going to ~~https://dashboard.justin-tech.com~~