Using FSM Ingress controller with osm-edge service mesh

Background

The Kubernetes Ingress API is designed with a separation of concerns, where the Ingress implementation provides an entry feature infrastructure managed by operations staff; it also allows application owners to control the routing of requests to the backend through rules.

The osm-edge supports multiple Ingress implementations to manage ingress traffic and provides the IngressBackend API to configure back-end services to receive access from trusted ingress points. This article focuses on the integration of osm-edge with FSM to manage ingress traffic.

Introduction to FSM

FSM is another open-source product from Flomesh for Kubernetes north-south traffic, gateway api contoller, and multi-cluster management. FSM uses Pipy, a programmable proxy at its core, and provides an Ingress controller, Gateway API controller, load balancer, cross-cluster service registration discovery, and more.

Integration with FSM

FSM is already integrated inside osm-edge and can be enabled during osm-edge installation; it can also be installed independently via helm for existing osm-edge mesh.

Prerequisites

• Kubernetes cluster, version 1.19.0 and higher
• Helm 3 CLI for standalone installation of FSM

Download the osm-edge CLI at

system=$(uname -s | tr [:upper:] [:lower:])
arch=$(dpkg --print-architecture)
release=v1.1.1
curl -L https://github.com/flomesh-io/osm-edge/releases/download/${release}/osm-edge-${release}-${system}-${arch}.tar.gz | tar -vxzf -
. /${system}-${arch}/osm version
cp . /${system}-${arch}/osm /usr/local/bin/

Integrated installation

export osm_namespace=osm-system 
export osm_mesh_name=osm 

osm install --set fsm.enabled=true \
    --mesh-name "$osm_mesh_name" \
    --osm-namespace "$osm_namespace" 

Standalone installation

If osm-edge is installed without FSM enabled, you can use a standalone installation to install it.

helm repo add fsm https://charts.flomesh.io

export fsm_namespace=osm-system 
helm install fsm fsm/fsm --namespace "$fsm_namespace" --create-namespace

Verify that all pods are up and running properly.

kubectl get pods -n osm-system
NAME READY STATUS RESTARTS AGE
repo-8756f76fb-2f78g 1/1 Running 0 5m53s
manager-866585bbd5-pbg7q 1/1 Running 0 5m53s
osm-bootstrap-7c6689ff57-47ksk 1/1 Running 0 5m53s
osm-controller-57888cfc7c-tnqxl 2/2 Running 0 5m52s
osm-injector-5f77898899-45f65 1/1 Running 0 5m53s
bootstrap-fd5894bcc-nr7hf 1/1 Running 0 5m53s
cluster-connector-local-68c7584c8b-qf7xm 1/1 Running 0 2m43s
ingress-pipy-6fb8c8b794-pgthl 1/1 Running 0 5m53s

Configuration

In order to authorize clients by restricting access to backend traffic, we will configure IngressBackend so that only ingress traffic from the ingress-pipy-controller endpoint can be routed to the backend service. In order to discover the ingress-pipy-controller endpoint, we need the osm-edge controller and the corresponding namespace to monitor it. However, to ensure that the FSM functions properly, it cannot be injected with sidecar.

kubectl label namespace "$osm_namespace" openservicemesh.io/monitored-by="$osm_mesh_name"

Save the external IP address and port of the entry gateway, which will be used later to test access to the backend application.

export ingress_host="$(kubectl -n "$osm_namespace" get service ingress-pipy-controller -o jsonpath='{.status.loadBalancer.ingress[0].ip}') "
export ingress_port="$(kubectl -n "$osm_namespace" get service ingress-pipy-controller -o jsonpath='{.spec.ports[? (@.name=="http")].port}')"
echo $ingress_host:$ingress_port

Deploying the sample service

The next step is to deploy the sample httpbin service.

## Create namespace kubectl create ns httpbin
kubectl create ns httpbin

# Add the namespace to the grid
osm namespace add httpbin

# Deploy the application
kubectl apply -f - <<EOF
apiVersion: v1
kind: ServiceAccount
metadata:
  name: httpbin
  namespace: httpbin
---
apiVersion: v1
kind: Service
metadata:
  name: httpbin
  namespace: httpbin
  labels:
    app: httpbin
    service: httpbin
spec:
  ports:
  - name: http
    port: 14001
  selector:
    app: httpbin
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: httpbin
  namespace: httpbin
spec:
  replicas: 1
  selector:
    matchLabels:
      app: httpbin
  template:
    metadata:
      labels:
        app: httpbin
    spec:
      serviceAccountName: httpbin
      containers:
      - image: kennethreitz/httpbin
        imagePullPolicy: IfNotPresent
        name: httpbin
        command: ["gunicorn", "-b", "0.0.0.0:14001", "httpbin:app", "-k", "gevent"]
        ports:
        - containerPort: 14001
EOF

Verify that the pod and service are created and the application is running successfully.

kubectl get pods,services -n httpbin
NAME READY STATUS RESTARTS AGE
pod/httpbin-54cc8cf5d-7vclc 2/2 Running 0 14s

NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/httpbin ClusterIP 10.43.105.166 <none> 14001/TCP 14s

Configure entry rules

Next we want to access the deployed httpbin service from outside the cluster and need to provide the ingress configuration rules.

kubectl apply -f - <<EOF
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: httpbin
  namespace: httpbin
  annotations:
    pipy.ingress.kubernetes.io/rewrite-target-from: /httpbin
    pipy.ingress.kubernetes.io/rewrite-target-to: /httpbin
spec:
  ingressClassName: pipy
  rules:
  - host: httpbin.org
    http:
      paths:
      - path: /httpbin
        pathType: Prefix
        backend:
          service:
            name: httpbin
            port:
              number: 14001
EOF

Accessing the httpbin service using the IP address and port recorded above will result in the following 502 Bad Gateway error response. This is because we have not set the ingress of the FSM as a trusted portal.

curl -sI http://"$ingress_host":"$ingress_port"/httpbin/get -H "Host: httpbin.org"
HTTP/1.1 502 Bad Gateway
content-length: 0
connection: keep-alive

Execute the following command to set the FSM ingress as a trusted entry.

kubectl apply -f - <<EOF
kind: IngressBackend
apiVersion: policy.openservicemesh.io/v1alpha1
metadata:
  name: httpbin
  namespace: httpbin
spec:
  backends:
  - name: httpbin
    port:
      number: 14001 # targetPort of httpbin service
      protocol: http
  sources:
  - kind: Service
    namespace: "$osm_namespace"
    name: ingress-pipy-controller
EOF

Try requesting the httpbin service again, and you will be able to access it successfully.

curl -sI http://"$ingress_host":"$ingress_port"/httpbin/get -H "Host: httpbin.org"
HTTP/1.1 200 OK
server: gunicorn/19.9.0
date: Thu, 18 Aug 2022 05:18:50 GMT
content-type: application/json
content-length: 241
access-control-allow-origin: *
access-control-allow-credentials: true
osm-stats-namespace: httpbin
osm-stats-kind: Deployment
osm-stats-name: httpbin
osm-stats-pod: httpbin-54cc8cf5d-7vclc
connection: keep-alive

Summary

FSM Ingress exposes the application access points within the Kubernetes cluster for easy management of portal traffic. osm-edge's IngressBackend API provides an additional line of defense against accidental data leakage by exposing services within the mesh to the public.