'Doglabbing' ngrok: Many services, one standard AuthN path

Much like James from yesterday, we have another heavy Kubernetes homelab setup using ngrok as its gateway between the public internet and very personal self-hosted services.

Can you expect anything less from two engineers on ngrok's infrastructure team?

--

Stacks (Staff Software Engineer): Many services, one standard AuthN path

I’m a long-time Kubernetes user, and I wanted to run Kubernetes in my homelab because I like the ecosystem and I like trying new projects. I wanted a way to hit these projects when I’m not at home, and generally expose these services on the internet, with something simple that wouldn’t require a lot of feeding and wouldn’t force me to open up ports or deal with UPnP or anything like that.

I’m used to setting up K8s in a cloud environment, where I have easy access to a cloud load balancer, or even static IPs for on-prem, but when you’re at the mercy of your ISP, this is a lot trickier. See the MetalLB project for proof of how hard this can be based on your network.

ngrok's Traffic Policy and ability to put OAuth in front of my applications was a big plus. It’s an extra layer of protection to make sure I'm the only one accessing my homelab, and I don’t have to worry about security vulnerabilities in the apps themselves or doxxing my IP because people don’t see where I’m hosting everything.

And it just works wherever—if I unplugged my homelab and went to Cali for an offsite, and had internet access, it would be up and running again.

Ingress resources for Argo CD’s dashboard and gRCP server:

---
apiVersion: v1
kind: Service
metadata:
  name: argocd-server
  namespace: argocd
  annotations:
    k8s.ngrok.com/app-protocols: '{"https": "HTTPS", "grpc": "HTTPS"}'
spec:
  type: ClusterIP
  ports:
  - name: http
    port: 80
    protocol: TCP
    targetPort: 8080
  - name: https
    port: 443
    protocol: TCP
    # appProtocol: k8s.ngrok.com/http2 # OR "kubernetes.io/h2c"
    targetPort: 8080
  - name: grpc
    port: 444
    protocol: TCP
    appProtocol: k8s.ngrok.com/http2
    targetPort: 8080
  selector:
    app.kubernetes.io/name: argocd-server
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: argocd-ingress
  namespace: argocd
  annotations:
    external-dns.alpha.kubernetes.io/hostname: argocd.example.com
    external-dns.alpha.kubernetes.io/ttl: 1m
    k8s.ngrok.com/traffic-policy: google-oauth
    k8s.ngrok.com/mapping-strategy: endpoints-verbose
spec:
  ingressClassName: ngrok
  rules:
  - host: argocd.example.com
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: argocd-server
            port:
              name: https
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: argocd-grpc-ingress
  namespace: argocd
  annotations:
    external-dns.alpha.kubernetes.io/hostname: grpc.argocd.example.com
    external-dns.alpha.kubernetes.io/ttl: 1m
    k8s.ngrok.com/mapping-strategy: endpoints-verbose
    k8s.ngrok.com/traffic-policy: only-trusted-ips
spec:
  ingressClassName: ngrok
  rules:
  - host: grpc.argocd.example.com
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: argocd-server
            port:
              name: grpc

Referenced TrafficPolicy CRDs:

---
kind: NgrokTrafficPolicy
apiVersion: ngrok.k8s.ngrok.com/v1alpha1
metadata:
  name: google-oauth
spec:
  policy:
    on_http_request:
      - actions: # Phase 1 : Authenticate
        - type: oauth
          config:
            provider: google
      - expressions: # Phase 2 : Deny if not me
        - "!(actions.ngrok.oauth.identity.email in ['[e-mail]'])"
        actions:
        - type: custom-response
          config:
            status_code: 403
            content: "Forbidden"
---
kind: NgrokTrafficPolicy
apiVersion: ngrok.k8s.ngrok.com/v1alpha1
metadata:
  name: only-trusted-ips
spec:
  policy:
    on_tcp_connect:
    - actions:
      - type: restrict-ips
        config:
          enforce: true
          allow:
          - '203.0.113.0/32'        # My IPv4
          - '2001:db8:abcd:1234::1/128' # My IPv6

What's happening in these policies?

The google-oauth policy:

1. Forces every HTTP request to flow through a Google-provided OAuth flow.
2. Filters out Google emails that do not match [e-mail] and sends them a custom response.
3. Forwards all authenticated requests that do match the email onto the upstream service pod.

The only-trusted-ips policy:

1. Checks whether the IP of the request matches either a trusted IPv4 or IPv6 address, and if not, denies the request.
2. Forwards requests from matching IPs onto the upstream service pod.

--

Tomorrow, we're taking a swerve into self-hosting a Google Photos alternative with immich and protecting it not with OAuth, but OIDC. Can't wait to share it!

In the meantime, here are all the docs that Stacks used to build his gateway setup:

• ngrok Kubernetes Operator
• Using Ingresses
• oauth Traffic Policy action
• custom-response Traffic Policy action
• restrict-ips Traffic Policy action

Comments

[Roshan Sharma]

Nice setup, consolidating authentication with ngrok across multiple services is a smart move; it keeps things secure and consistent without adding too much overhead.
A pattern I could see myself using in a homelab.

[Joel Hans ngrok]

Yeah, that's exactly the idea! One config for auth so you don't have to worry about it in every service. Let me (Joel!) know if you ever have Qs about getting started.