DEV Community: Flomesh

osm-edge: Using access control policies to access services with the service mesh

Ali Naqvi — Tue, 08 Nov 2022 06:46:19 +0000

Deploying a service mesh in a complex brownfield environment is a lengthy and gradual process requiring upfront planning, or there may exist use cases where you have a specific set of services that either aren't yet ready for migration or for some reason can not be migrated to service mesh.

This blog post will talk about the approaches which can be used to enable services outside of the service mesh to communicate with services within the osm-edge service mesh.

osm-edge forked from Open Service Mesh is a lightweight, extensible, cloud-native, SMI-compatible service mesh built purposely for Edge computing. osm-edge uses lightweight programmable proxy Pipy as a sidecar proxy.

osm-edge offers two ways to allow accessing services within the service mesh:

via Ingress
- FSM Ingress controller
- Nginx Ingress controller
Access Control
- Service
- IPRange

The first method to access the services in the service mesh is via Ingress controller, and treat the services outside the mesh as the services inside the cluster. The advantage of this approach is that the setup is simple and straightforward and the disadvantages are also apparent, as you cannot achieve fine-grained access control, and all services outside the mesh can access services within the mesh.

This article will focus on the second approach, which allows support for fine-grained access control on who can access services within the service mesh. This feature is newly added and available in release 1.2.0 osm-edge v1.2.0.

Access Control can be configured via two resource types: Service and IP range. In terms of data transmission, it supports plaintext transmission and mTLS-encrypted traffic.

Let's get started with a demo.

Demo environment preparation

Kubernetes cluster

Using the minimalist Kubernetes distribution k8e:

curl -sfL https://getk8e.com/install.sh | K8E_TOKEN=k8e-mesh INSTALL_K8E_EXEC="server --cluster-init --write-kubeconfig-mode 644 --write-kubeconfig ~/.kube/config" sh -

osm-edge CLI

system=$(uname -s | tr [:upper:] [:lower:])
arch=$(dpkg --print-architecture)
release=v1.2.0
curl -L https://github.com/flomesh-io/osm-edge/releases/download/${release}/osm-edge-${release}-${system}-${arch}.tar.gz | tar -vxzf -
./${system}-${arch}/osm version
cp ./${system}-${arch}/osm /usr/local/bin/

Install osm-edge

Execute the following commands to install the related components of osm-edge.

export osm_namespace=osm-system
export osm_mesh_name=osm

osm install \
     --mesh-name "$osm_mesh_name" \
     --osm-namespace "$osm_namespace" \
     --set=osm.image.pullPolicy=Always

Check to make sure all pods are up and running properly.

Deploy the sample application

#Mock target service
kubectl create namespace httpbin
osm namespace add httpbin
kubectl apply -n httpbin -f https://raw.githubusercontent.com/flomesh-io/osm-edge-docs/main/manifests/samples/httpbin/httpbin.yaml

#Mock external service
kubectl create namespace curl
kubectl apply -n curl -f https://raw.githubusercontent.com/flomesh-io/osm-edge-docs/main/manifests/samples/curl/curl.yaml

#Wait for the dependent POD to start normally
kubectl wait --for=condition=ready pod -n httpbin -l app=httpbin --timeout=180s
kubectl wait --for=condition=ready pod -n curl -l app=curl --timeout=180s

Demo

At this point, we send a request from service curl to target service httpbin by executing the following command.

kubectl exec "$(kubectl get pod -n curl -l app=curl -o jsonpath='{.items..metadata.name}')" -n curl -- curl -sI http://httpbin.httpbin:14001/get
command terminated with exit code 56

The access fails because by default the services outside the mesh cannot access the services inside the mesh and we need to apply an access control policy.

Before applying the policy, you need to enable the access control feature, which is disabled by default.

kubectl patch meshconfig osm-mesh-config -n "$osm_namespace" -p '{"spec":{"featureFlags":{"enableAccessControlPolicy":true}}}' --type= merge

Plaintext transfer

Data can be transferred in plaintext or with two-way TLS encryption. Plaintext transfer is relatively simple, so let's demonstrate the plaintext transfer scenario first.

Service-based access control

First, create a Service for service curl.

kubectl apply -n curl -f - <<EOF
apiVersion: v1
kind: Service
metadata:
  name: curl
  labels:
    app: curl
    service: curl
spec:
  ports:
    - name: http
      port: 80
  selector:
    app: curl
EOF

Next, create an access control policy with the source Service curl and the target service httpbin.

kubectl apply -f - <<EOF
kind: AccessControl
apiVersion: policy.openservicemesh.io/v1alpha1
metadata:
  name: httpbin
  namespace: httpbin
spec:
  backends:
  - name: httpbin
    port:
      number: 14001 # targetPort of httpbin service
      protocol: http
  sources:
  - kind: Service
    namespace: curl
    name: curl
EOF

Execute the command again to send the authentication request, and you can see that this time an HTTP 200 response is received.

kubectl exec "$(kubectl get pod -n curl -l app=curl -o jsonpath='{.items..metadata.name}')" -n curl -- curl -sI http://httpbin.httpbin:14001/get
HTTP/1.1 200 OK
server: gunicorn/19.9.0
date: Mon, 07 Nov 2022 08:47:55 GMT
content-type: application/json
content-length: 267
access-control-allow-origin: *
access-control-allow-credentials: true
osm-stats-namespace: httpbin
osm-stats-kind: Deployment
osm-stats-name: httpbin
osm-stats-pod: httpbin-69dc7d545c-qphrh
connection: keep-alive

Before continuing with the rest of the demonstration, execute the command kubectl delete accesscontrol httpbin -n httpbin to delete the policy you just created.

IP range-based access control, plaintext transfer

First, get the pod IP address of the service curl.

curl_pod_ip="$(kubectl get pod -n curl -l app=curl -o jsonpath='{.items[0].status.podIP}')"

Access control using IP ranges is simple, just set the access source type to IPRange and configure the IP address you just obtained.

kubectl apply -f - <<EOF
kind: AccessControl
apiVersion: policy.openservicemesh.io/v1alpha1
metadata:
  name: httpbin
  namespace: httpbin
spec:
  backends:
  - name: httpbin
    port:
      number: 14001 # targetPort of httpbin service
      protocol: http
  sources:
  - kind: IPRange
    name: ${curl_pod_ip}/32
EOF

Execute the command again to test if the control policy is in effect.

kubectl exec "$(kubectl get pod -n curl -l app=curl -o jsonpath='{.items..metadata.name}')" -n curl -- curl -sI http://httpbin.httpbin:14001/get
HTTP/1.1 200 OK
server: gunicorn/19.9.0
date: Mon, 07 Nov 2022 09:20:57 GMT
content-type: application/json
content-length: 267
access-control-allow-origin: *
access-control-allow-credentials: true
osm-stats-namespace: httpbin
osm-stats-kind: Deployment
osm-stats-name: httpbin
osm-stats-pod: httpbin-69dc7d545c-qphrh
connection: keep-alive

Remember to execute kubectl delete accesscontrol httpbin -n httpbin to clean up the policy.

The previous ones we used were plaintext transfers, next we look at encrypted transfers.

Encrypted transfers

The default access policy certificate feature is off, turn it on by executing the following command.

kubectl patch meshconfig osm-mesh-config -n "$osm_namespace" -p '{"spec":{"featureFlags":{"enableAccessCertPolicy":true}}}' --type=merge

Create AccessCert for the access source to assign a certificate for data encryption. The controller will store the certificate information in Secret curl-mtls-secret under the namespace curl, and here also assign SAN curl.curl.cluster.local for the access source.

kubectl apply -f - <<EOF
kind: AccessCert
apiVersion: policy.openservicemesh.io/v1alpha1
metadata:
  name: curl-mtls-cert
  namespace: httpbin
spec:
  subjectAltNames:
  - curl.curl.cluster.local
  secret:
    name: curl-mtls-secret
    namespace: curl
EOF

Redeploy curl and mount the system-assigned Secret to the pod.

kubectl apply -n curl -f - <<EOF
apiVersion: apps/v1
kind: Deployment
metadata:
  name: curl
spec:
  replicas: 1
  selector:
    matchLabels:
      app: curl
  template:
    metadata:
      labels:
        app: curl
    spec:
      serviceAccountName: curl
      nodeSelector:
        kubernetes.io/os: linux
      containers:
      - image: curlimages/curl
        imagePullPolicy: IfNotPresent
        name: curl
        command: ["sleep", "365d"]
        volumeMounts:
        - name: curl-mtls-secret
          mountPath: "/certs"
          readOnly: true
      volumes:
        - name: curl-mtls-secret
          secret:
            secretName: curl-mtls-secret
EOF

Service-based Access Control

The next step is to create an access control policy that uses encrypted transport. When configuring the target service, enable client certificate checking by specifying tls.skipClientCertValidation = false. For the access source, in addition to the Service type of access source, the SAN curl.curl.cluster.local should be specified via AuthenticatedPrincipal.

kubectl apply -f - <<EOF
kind: AccessControl
apiVersion: policy.openservicemesh.io/v1alpha1
metadata:
  name: httpbin
  namespace: httpbin
spec:
  backends:
  - name: httpbin
    port:
      number: 14001 # targetPort of httpbin service
      protocol: http
    tls:
      skipClientCertValidation: false
  sources:
  - kind: Service
    namespace: curl
    name: curl
  - kind: AuthenticatedPrincipal
    name: curl.curl.cluster.local
EOF

Test whether the access policy is effective. When sending the request, we have to specify the CA certificate, key and certificate to be used for the curl command of the access source, which will be responded to by HTTP 200 normally.

kubectl exec "$(kubectl get pod -n curl -l app=curl -o jsonpath='{.items..metadata.name}')" -n curl -- curl -ksI https://httpbin.httpbin:14001/get --cacert /certs/ca.crt --key /certs/tls.key --cert /certs/tls.crt
HTTP/2 200
server: gunicorn/19.9.0
date: Mon, 07 Nov 2022 10:44:05 GMT
content-type: application/json
content-length: 267
access-control-allow-origin: *
access-control-allow-credentials: true
osm-stats-namespace: httpbin
osm-stats-kind: Deployment
osm-stats-name: httpbin
osm-stats-pod: httpbin-69dc7d545c-qphrh

IP Range-Based Access Control

After service-based access control, IP range-based control is simple. You just need to specify the type of access source as IPRange and specify the IP address of the access source. As the application curl is redeployed, its IP address needs to be retrieved (perhaps you have discovered the drawbacks of IP range-based access control).

curl_pod_ip="$(kubectl get pod -n curl -l app=curl -o jsonpath='{.items[0].status.podIP}')"

kubectl apply -f - <<EOF
kind: AccessControl
apiVersion: policy.openservicemesh.io/v1alpha1
metadata:
  name: httpbin
  namespace: httpbin
spec:
  backends:
  - name: httpbin
    port:
      number: 14001 # targetPort of httpbin service
      protocol: http
    tls:
      skipClientCertValidation: false
  sources:
  - kind: IPRange
    name: ${curl_pod_ip}/32
  - kind: AuthenticatedPrincipal
    name: curl.curl.cluster.local
EOF

Send the request again for testing.

kubectl exec "$(kubectl get pod -n curl -l app=curl -o jsonpath='{.items..metadata.name}')" -n curl -- curl -ksI https://httpbin.httpbin:14001/get --cacert /certs/ca.crt --key /certs/tls.key --cert /certs/tls.crt
HTTP/2 200
server: gunicorn/19.9.0
date: Mon, 07 Nov 2022 10:58:55 GMT
content-type: application/json
content-length: 267
access-control-allow-origin: *
access-control-allow-credentials: true
osm-stats-namespace: httpbin
osm-stats-kind: Deployment
osm-stats-name: httpbin
osm-stats-pod: httpbin-69dc7d545c-qphrh

Bingo! The access is successful, indicating that our policy has taken effect

Summary

This blog post will talk about the approaches which can be used to enable services outside of the service mesh to communicate with services within the osm-edge service mesh. There may exist use cases where you have a specific set of services that aren't yet ready for migration to service mesh or for some reason can't be migrated yet.

The access control approaches presented in this article are suitable for solving such problems, and allow you to choose the appropriate type of access source and whether to transfer data in plaintext or encrypted, depending on your needs. The control is more granular than using a unified Ingress for access.

Pipy: Protecting Kubernetes Apps from SQL Injection & XSS Attacks

Ali Naqvi — Thu, 03 Nov 2022 13:12:06 +0000

Injection attacks sliding down to 3rd position in 2021 have been on the OWASP Top 10 list for many years and SQL Injection (SQLi) is a common injection technique used for attacking websites and web applications. Applications that do not cleanly separate user input from database commands are at risk of malicious input being executed as SQL commands. Successful injection attacks can lead to unauthorized access to sensitive data such as passwords, credit card details, and personal user information. Many recent high-profile data breaches have resulted from SQL injection attacks, resulting in reputational damage and regulatory fines.

The common and normal workaround which you might have seen with other solutions is using a list of regular expressions to filter out traffic, which works for some cases but falls short with some complex inputs or escaped inputs. We are not trying to bash Regular Expressions, they are good and have their own use cases, but they aren't a good fit for such use cases.

This blog post will demonstrate how to use Pipy an open-source programmable proxy to harden the security by adding an extra layer of shield to your applications which might otherwise be vulnerable to such attacks.

In a previous blog post announcing the latest release of Pipy 0.70.0, it was introduced that Pipy added the support of extensions called Native Module Interface (NMI) and we will be using Pipy NMI to develop a module to integrate with mature and stable open-source library libinject to scan incoming traffic against SQLi and XSS attacks before it reaches the application.

For the demo application, we will be using an open-source classical LAMP stack Vulnerable Mama Shop (VMS) which is intentionally developed to have SQLi flaws. We will have fun together by first hacking the basic application to demonstrate the SQLi attacks, then we will harden the application security by adding Pipy as a sidecar to block certain SQLi attacks.

Pre-requisites

This blog post assumes you have access to:

Running Kubernetes cluster (dev box)
kubectl

The demo source code is available on GitHub and can be downloaded from pipy-sqli-demo repository.

Deploy Kubernetes cluster and Vulnerable Mama Shop App

To run the demo locally, we recommend k3d a lightweight wrapper to run k3s (Rancher Lab’s minimal Kubernetes distribution) in docker.

$ k3d cluster create my-cluster -p 8080:30060@server:0

In the above command, we are creating a cluster with a single server and mapping port 30060 of the k3d container to the local 8080 port, usage of this port will become clear in the later steps of this tutorial.

Deploy Vulnerable Mama Shop App

We are going to deploy a simple online store application that comes installed with

Apache Webserver
MariaDB
PHP app that runs on Apache and connects to the MariaDB database

Use the text editor of your choice and create a YAML file called 1-app.yaml with the following contents:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: vms
spec:
  selector:
    matchLabels:
      app: vms
  template:
    metadata:
      labels:
        app: vms
    spec:
      containers:
        - name: vms
          image: naqvis/mamashop
          ports:
            - containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
  name: vms
spec:
  ports:
    - port: 80
      targetPort: 80
      nodePort: 30060
  selector:
    app: vms
  type: NodePort

Deploy the app

$ kubectl apply -f 1-app.yaml

Confirm that pod is up and running, as indicated by the value Running in the STATUS column. It can take 30-40 seconds for them to fully deploy, so it’s useful to run the command again to confirm all pods are running before continuing to the next step.

$ kubectl get pods
NAME                   READY   STATUS    RESTARTS   AGE
vms-6658dd4478-hn246   1/1     Running   0          2m12s

Query the Servcice

$ kubectl get svc vms
NAME   TYPE       CLUSTER-IP      EXTERNAL-IP   PORT(S)        AGE
vms    NodePort   10.43.244.253   <none>        80:30060/TCP   5m43s

Still remember at the beginning we did port mapping for the K3d container: 8080=>30060? Keen users may also find that we have created a NodePort service for our Vulnerable Mama Shop web application, and the node port is 30060.

Open the app in your browser by visiting the URL http://localhost:8080

Hack the App

Mama Shop is rather very basic, it comes with only 3 pages. The main page (shown above) is where a user can query items for sale through a drop-down box, a customer login page, and an about page. Play around with these 3 pages to see what each does and how each page works normally.

Try submitting a category and see how items are listed. The following shows a listing of items from the Drinks category.

As you may have noticed, selecting a category and hitting submit doesn't change the URL, so the application is doing a POST request and we will need some tool to intercept or view the traffic the web page is generating.

Normally you will be using a proxy tool like Blurp suite or OWASP ZAP to intercept and modify the request to the application. But to keep things simple, we will be using Firefox browser Developer Console to view and modify the requests.

Go back to your browser (Firefox in this demo) and turn on Web Developer Tools either via Tools | Browser Tools | Web Developer Tools menu option or via short-cut key Option+Cmd+I on Mac or similar on your OS. Move to the Network tab, and hit submit button on a web page to view the network traffic.

We can see all of the requests along with all details, a web page makes to the webserver to display the page. Hit Resend button at the top right of Request details window displayed at the bottom right.

Network action window shows that to browse items in a category, a URL encoded HTTP POST with a single parameter catid is sent to welcome.php. To test for SQL injection, it is common to modify user input and send a single quotation mark like '.

Modify Form Parameter

We will find out if the input is properly escaped with a simple experiment: changing the catid to a SQL query string, to see if it will cause a syntax error which may be shown on the webpage.

catid=' or 1 = 1 ; --

Make above change and hit send button at the bottom right of network action window.

Whoa, we are upto something, welcome.php function for displaying category items have a SQL injection vulnerability. The error message also tells us the database is MariaDB. And error is telling us that there is a syntax issue near or 1 = 1. Using our imaginations, we can assume the database query is something like

SELECT item_name, item_category, item_description from items_table where item_category = ' or 1 = 1 ; -- ;

If we were to change the catid ending to " or 1 = 1 -- ;, that should fix the quoting issue. It should select all rows from the database, which is useful in a hack.

The structure of the assumed SQL query is probably correct, although it is not necessarily the exact query that the developer has used. This is sufficient for devasting attacks such as dumping customer information.

Extract User Data

The customer login page tells us that the application contains customer data and this is likely stored in some sort of customer or user table.

Based on our information gathered so far we can make use of the MariaDB INFORMATION_SCHEMA and the SQL Union operator to retrieve database and tables details.

Change catid parameter to below to retrieve database name, which will be further used to retrieve table details.

catid=1000 union select database(), "A" , "B" from dual

The following shows the response from Vulnerable Mama Shop containing the database name, appdb.

Now we have database name, we can obtain the tables in the database

catid=1000 union select table_name,version, table_comment from information_schema.TABLES where table_schema = 'appdb'

So we get a list of all tables, We can safely assume users table in the database contains usernames and passwords.

We need to retrieve list of columns users table have and we will need to ensure that we use same number of columns as queried from products table for UNION to work.

catid=1000 union select table_name, COLUMN_NAME, DATA_TYPE from information_schema.COLUMNS where table_name = 'users'

Now that we know there are a total of six columns in the users table to contains details like firstname, lastname, nric, password, email are available from users table.

We have gathered enough information to dump out a users listing. The following input can be used to dump out a listing of users with their firstname, password and email address.

catid=1000 union select firstname, nric, email from users LIMIT 7, 100

The values for the LIMIT and offset can be worked out by observing how many item entries there are in a normal request. The offset value should remove these items. The LIMIT can be set a sufficiently large value so that customer records can be dumped out.

Play around and use these gathered information to see if you can login with these credentials.

Using Pipy Sidecar to block certain SQLi attacks

SQL injection occurs when unvalidated user input is mixed with SQL instructions, so developers should pay more attention to escape user input (such as use of parameterized queries), but you – the Kubernetes engineer – can also help avoid SQL injection by preventing this attack from reaching the app. That way, even if the app is vulnerable, attacks can still be stopped.

There are multiple options for protecting your apps, but for this blog post we will focus on injecting a sidecar container to proxy all of the traffic and deny any request that is detected as SQLi attacks.

For demonstration purposes, we are following the approach of manually injecting sidecar, but in reality, manually deploying proxies as sidecars isn’t the best solution. The complexity of your apps and architecture might require more fine–grain control. If your organization requires Zero Trust and has a need for end–to–end encryption, consider a service mesh like osm-edge a light weight, ultra fast, low resources, highly extensible, Service Mesh Interface (SMI) compatible, built for edge and cloud computing service mesh.

Deploying Pipy Sidecar

Create a YAML file called 2-app-sidecar.yaml with the contents below, and check out these noteworthy components:

A sidecar container running Pipy is started on port 8000.
The Pipy process forwards all traffic to the app.
Requests URI or POST body containing SQLi is declined
The service for the app routes all traffic to the Pipy sidecar container first.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: vms
spec:
  selector:
    matchLabels:
      app: vms
  template:
    metadata:
      labels:
        app: vms
    spec:
      containers:
        - name: vms
          image: naqvis/mamashop
          ports:
            - containerPort: 80
        - name: pipy # <-- sidecar
          image: naqvis/pipy-nmi
          env:
            - name: PIPY_CONFIG_FILE
              value: /etc/pipy/nmi/nmi.js
          ports:
            - containerPort: 8000
          volumeMounts:
            - mountPath: /etc/pipy/nmi
              name: pipy-pjs
      volumes:
        - name: pipy-pjs
          configMap:
            name: sidecar
---
apiVersion: v1
kind: Service
metadata:
  name: vms
spec:
  ports:
    - port: 80
      targetPort: 8000 # <-- the traffic is routed to the proxy
      nodePort: 30060
  selector:
    app: vms
  type: NodePort
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: sidecar
data:
  nmi.js: |-
    pipy({
      _rejected: undefined,
    })
      .import({
        __is_sqli: 'lib-inject',
        __is_xss: 'lib-inject',
        __sqli_fingerprint: 'lib-inject',
      })
      .listen(8000)
      .demuxHTTP().to(
        $=>$
            .use('/etc/pipy/modules/inject-nmi.so')
            .handleMessage(() => _rejected = (__is_sqli || __is_xss))
            .branch(
                () => _rejected === true, (
                $=>$
                .handleMessageStart(_ =>
                  console.log(`SQL Injection found with Fingerprint: ${__sqli_fingerprint}`))
                .replaceMessage(new Message({ status: 403 }, 'Forbidden'))
                ),
                () => _rejected === false, (
                  $=>$.muxHTTP().to(
                      $=>$.connect('localhost:80')
                  )
                )
            )
      )

Deploy the side

$ kubectl apply -f 2-app-sidecar.yaml

Wait for pods to be up and ready

$ kubectl get pods
NAME                  READY   STATUS    RESTARTS   AGE
vms-945f6f85c-8v7sb   2/2     Running   0          8m48s

Test the Sidecar

Test whether the sidecar is filtering traffic by returning to the app and trying the SQL injection again. Pipy sidecar blocks the request before it reaches the app!

Sidecar is blocking the traffic by returning 403 Forbidden. Run few more requests with any of the previous SQLi attempts we made and we will be getting 403 back in response.
We can validate that by looking at the logs of Pipy side car.

Conclusion

The purpose of learning about offensive techniques is to enable developers, defenders and security professionals to protect the confidentiality, the integrity and availability of critical information assets and services.

Kubernetes is not secure by default. This blog post made use of Vulnerable Mama Shop, a simple application based on LAMP stack to demonstrate SQL injection attacks and how one can use techniques like SideCar to protect against them.

But as the complexity of your apps and architecture grows, you might require more fine–grain control over your services. And if your organization requires Zero Trust and has a need for end–to–end encryption like mTLS, you should consider a service mesh like osm-edge a light weight, ultra fast, low resources, highly extensible, Service Mesh Interface (SMI) compatible, built for edge and cloud computing service mesh. When you have communication between services (east–west traffic), a service mesh allows you to control traffic at that level.

Pipy 0.70.0 is released!

Ali Naqvi — Fri, 21 Oct 2022 15:26:06 +0000

Pipy 0.70.0 is now available. This release consists of 170+ commits and comes with improvements to several areas including support for writing Pipy modules using C via Pipy Native Module Interface (NMI), PJS language extensions like looping construct, Map, Set, etc, protocols like Thrift, PROXY, TPROXY, Output UDP with Ephemeral ports, Encoder/Decoder for DNS protocol, and more. This version of Pipy comes with a fresh and new design of modules/plugin system, enhanced Pipy unique Admin Console, IDE is now more robust and supports more features like Intellisense, parameters highlighting, hints, etc.
This release continues on the great work of previous releases and optimization is done for PipyJS and code readability, better documentation, and some bug fixes.

This release was indeed a community effort and could not have been made possible without all of the hard work from everyone involved in active discussions and the Pipy project on GitHub.The Pipy community provides code submissions covering new functionality and bug fixes, documentation improvements, quality assurance testing, continuous integration environments, bug reports, and more. Everyone has done their part to make this release possible! If you’d like to join this amazing community, you can find it on GitHub, Slack, and the Pipy discussion groups.

Below we list the most remarkable changes in the Core API and PipyJS. For a detailed change log, refer to Pipy Release Page

Core

NMI (Native Module Interface) for dynamically loaded modules written in C
New module/plugin system by using chain() filters
Support loop logic in pipelines by using the new replay() filter
Builtin GUI frontend greatly reduced in size by using Brotli compression

Protocols

Support Thrift protocol
Support the Proxy protocol proposed by HAProxy
Support outbound UDP with ephemeral local ports
Support TPROXY on UDP

API

Implementation of the standard ECMAScript builtin objects Map and Set
Encode and decode DNS messages
Logging API now supports HTTPS receivers and Syslog
Expose subject alternative names of a certificate
Mux filters to support limit of maximum messages per session
Netmask API now supports IPv6

Documentation & Samples

Renovation of the documentation system by using TypeScript
All new tutorials with complete API reference
New samples for HTTP proxy common use cases

We would like to thank each and every contributor who was involved in this release.

Flomesh Ingress Controller with Kubernetes Multi-tenancy

Ali Naqvi — Mon, 03 Oct 2022 13:10:34 +0000

Background

It's very rare for organizations to have or provide a dedicated Kubernetes cluster to each tenant, as it's generally more cost-effective to share a cluster. Sharing clusters reduces expenses and streamlines management. Sharing clusters, however, also comes with difficulties like managing noisy neighbors and ensuring security.

Clusters can be shared in many ways. In some cases, different applications may run in the same cluster. In other cases, multiple instances of the same application may run in the same cluster, one for each end user. All these types of sharing are frequently described using the umbrella term multi-tenancy.

While Kubernetes does not have first-class concepts of end users or tenants, it provides several features to help manage different tenancy requirements. These are discussed below.

A common form of multi-tenancy is to share a cluster between multiple teams within an organization, each of whom may operate one or more workloads. In this scenario, members of the teams often have direct access to Kubernetes resources via tools such as kubectl, or indirect access through GitOps controllers or other types of release automation tools. There is often some level of trust between members of different teams, but Kubernetes policies such as RBAC, quotas, and network policies are essential to safely and fairly share clusters.
The other major form of multi-tenancy frequently involves a Software-as-a-Service (SaaS) vendor running multiple instances of a workload for customers. This business model is so strongly associated with this deployment style that many people call it "SaaS tenancy." In this scenario, the customers do not have access to the cluster; Kubernetes is invisible from their perspective and is only used by the vendor to manage the workloads. Cost optimization is frequently a critical concern, and Kubernetes policies are used to ensure that the workloads are strongly isolated from each other.

Tenant Isolation

There are several ways to design and build multi-tenant solutions with Kubernetes at its control plane, data plane, or at both levels. Each of these methods comes with its own set of tradeoffs that impact the isolation level, implementation effort, operational complexity, and cost of service.

Kubernetes Control plane isolation ensures that different tenants cannot access or affect each other's Kubernetes API resources, and Flomesh Service Mesh (FSM) Ingress controller provides isolated Ingress controllers for Kubernetes Namespaces.

In Kubernetes, a Namespace provides a mechanism for isolating groups of API resources within a single cluster. This isolation has two key dimensions:

Object names within a namespace can overlap with names in other namespaces, similar to files in folders. This allows tenants to name their resources without having to consider what other tenants are doing.
Many Kubernetes security policies are scoped to namespaces. For example, RBAC Roles and Network Policies are namespace-scoped resources. Using RBAC, Users and Service Accounts can be restricted to a namespace.

In a multi-tenant environment, a Namespace helps segment a tenant's workload into a logical and distinct management unit. A common practice is to isolate every workload in its namespace, even if multiple workloads are operated by the same tenant. This ensures that each workload has its own identity and can be configured with an appropriate security policy.

In this article, we will learn how to use the Flomesh Service Mesh Ingress controller to create physical isolation of Ingress controllers when hosting multiple tenants in your Kubernetes cluster.

Flomesh Service Mesh (FSM)

FSM is an open-source product from Flomesh for Kubernetes north-south traffic, gateway API controller, and multi-cluster management. FSM uses Pipy, a programmable proxy at its core, and provides an Ingress controller, Gateway API controller, load balancer, cross-cluster service registration discovery, and more.

FSM Ingress Controller supports a multi-tenancy model via its concept of NamespacedIngress CRD where it deploys a physically isolated Ingress Controller for requested Namespace.

For example, the YAML below defines an Ingress Controller that monitors on port 100 and also creates a LoadBalancer type service for it that listens on port 100.

apiVersion: flomesh.io/v1alpha1
kind: NamespacedIngress
metadata:
  name: namespaced-ingress-100
  namespace: test-100
spec:
  serviceType: LoadBalancer
  ports:
  - name: http
    port: 100
    protocol: TCP

Install FSM

FSM provides a standard Helm chart, which can be installed via the Helm CLI.

$ helm repo add fsm https://flomesh-io.github.io/fsm
$ helm repo update

$ helm install fsm fsm/fsm --namespace flomesh --create-namespace --set fsm.ingress.namespaced=true

Verify that all pods are up and running properly.

$ kubectl get po -n flomesh
NAME                                          READY   STATUS    RESTARTS   AGE
fsm-manager-6857f96858-sjksm                  1/1     Running   0          55s
fsm-repo-59bbbfdc5f-w7vg6                     1/1     Running   0          55s
fsm-bootstrap-8576c5ff4f-7qr7k                1/1     Running   0          55s
fsm-cluster-connector-local-8f8fb87f6-h7z9j   1/1     Running   0          32s

Create Sample Application

In this demo, we will be deploying httpbin service under a namespace httpbin.

# Create Namespace
kubectl create ns httpbin

# Deploy sample
kubectl apply -f https://raw.githubusercontent.com/flomesh-io/osm-edge-docs/main/manifests/samples/httpbin/httpbin.yaml -n httpbin

Creating a standalone Ingress Controller

The next step is to create a separate Ingress Controller for the namespace httpbin.

$ kubectl apply -f - <<EOF
apiVersion: flomesh.io/v1alpha1
kind: NamespacedIngress
metadata:
  name: namespaced-ingress-httpbin
  namespace: httpbin
spec:
  serviceType: LoadBalancer
  http:
    port:
      name: http
      port: 81
      nodePort: 30081
  resources:
    limits:
      cpu: 500m
      memory: 200Mi
    requests:
      cpu: 100m
      memory: 20Mi
EOF

After executing the above command, you will see an Ingress Controller running successfully under the namespace httpbin.

kubectl get po -n httpbin -l app=fsm-ingress-pipy
NAME                                        READY   STATUS    RESTARTS   AGE
fsm-ingress-pipy-httpbin-5594ffcfcc-zl5gl   1/1     Running   0          58s

At this point, there should be a corresponding Service under this namespace.

$ kubectl get svc -n httpbin -l app.kubernetes.io/component=controller
NAME                       TYPE           CLUSTER-IP     EXTERNAL-IP    PORT(S)        AGE
fsm-ingress-pipy-httpbin   LoadBalancer   10.43.62.120   192.168.1.11   81:30081/TCP   2m49s

Once you have the Ingress Controller, it's time to create the Ingress resource.

kubectl apply -f - <<EOF
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: httpbin
  namespace: httpbin
spec:
  ingressClassName: pipy
  rules:
  - host: httpbin.org
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: httpbin
            port:
              number: 14001
EOF

Now we have created Ingress resource, let's do a quick curl to see if things are working as expected.

For my local setup demo, LoadBalancer IP is 192.168.1.11, your IP might be different. So ensure you are performing a curl against your setup ExternalIP.

curl -sI http://192.168.1.11:81/get -H "Host: httpbin.org"
HTTP/1.1 200 OK
server: gunicorn/19.9.0
date: Mon, 03 Oct 2022 12:02:04 GMT
content-type: application/json
content-length: 239
access-control-allow-origin: *
access-control-allow-credentials: true
connection: keep-alive

Conclusion

In this blog post, you learned about Kubernetes multi-tenancy, features provided by Kubernetes, tenancy isolation levels, and how to use Flomesh Service Mesh (FSM) Ingress controller to set up isolated Ingress controller for namespaces.

Flomesh Service Mesh(FSM) from Flomesh is Kubernetes North-South traffic manager, provides Ingress controllers, Gateway API, Load Balancer, and cross-cluster service registration and service discovery. FSM uses Pipy - a programmable network proxy, as its data plane and is suitable for cloud, edge, and IoT.

Using FSM Ingress controller with osm-edge service mesh

Ali Naqvi — Thu, 18 Aug 2022 15:18:11 +0000

Background

The Kubernetes Ingress API is designed with a separation of concerns, where the Ingress implementation provides an entry feature infrastructure managed by operations staff; it also allows application owners to control the routing of requests to the backend through rules.

The osm-edge supports multiple Ingress implementations to manage ingress traffic and provides the IngressBackend API to configure back-end services to receive access from trusted ingress points. This article focuses on the integration of osm-edge with FSM to manage ingress traffic.

Introduction to FSM

FSM is another open-source product from Flomesh for Kubernetes north-south traffic, gateway api contoller, and multi-cluster management. FSM uses Pipy, a programmable proxy at its core, and provides an Ingress controller, Gateway API controller, load balancer, cross-cluster service registration discovery, and more.

Integration with FSM

FSM is already integrated inside osm-edge and can be enabled during osm-edge installation; it can also be installed independently via helm for existing osm-edge mesh.

Prerequisites

Kubernetes cluster, version 1.19.0 and higher
Helm 3 CLI for standalone installation of FSM

Download the osm-edge CLI at

system=$(uname -s | tr [:upper:] [:lower:])
arch=$(dpkg --print-architecture)
release=v1.1.1
curl -L https://github.com/flomesh-io/osm-edge/releases/download/${release}/osm-edge-${release}-${system}-${arch}.tar.gz | tar -vxzf -
. /${system}-${arch}/osm version
cp . /${system}-${arch}/osm /usr/local/bin/

Integrated installation

export osm_namespace=osm-system 
export osm_mesh_name=osm 

osm install --set fsm.enabled=true \
    --mesh-name "$osm_mesh_name" \
    --osm-namespace "$osm_namespace"

Standalone installation

If osm-edge is installed without FSM enabled, you can use a standalone installation to install it.

helm repo add fsm https://charts.flomesh.io

export fsm_namespace=osm-system 
helm install fsm fsm/fsm --namespace "$fsm_namespace" --create-namespace

Verify that all pods are up and running properly.

kubectl get pods -n osm-system
NAME READY STATUS RESTARTS AGE
repo-8756f76fb-2f78g 1/1 Running 0 5m53s
manager-866585bbd5-pbg7q 1/1 Running 0 5m53s
osm-bootstrap-7c6689ff57-47ksk 1/1 Running 0 5m53s
osm-controller-57888cfc7c-tnqxl 2/2 Running 0 5m52s
osm-injector-5f77898899-45f65 1/1 Running 0 5m53s
bootstrap-fd5894bcc-nr7hf 1/1 Running 0 5m53s
cluster-connector-local-68c7584c8b-qf7xm 1/1 Running 0 2m43s
ingress-pipy-6fb8c8b794-pgthl 1/1 Running 0 5m53s

Configuration

In order to authorize clients by restricting access to backend traffic, we will configure IngressBackend so that only ingress traffic from the ingress-pipy-controller endpoint can be routed to the backend service. In order to discover the ingress-pipy-controller endpoint, we need the osm-edge controller and the corresponding namespace to monitor it. However, to ensure that the FSM functions properly, it cannot be injected with sidecar.

kubectl label namespace "$osm_namespace" openservicemesh.io/monitored-by="$osm_mesh_name"

Save the external IP address and port of the entry gateway, which will be used later to test access to the backend application.

export ingress_host="$(kubectl -n "$osm_namespace" get service ingress-pipy-controller -o jsonpath='{.status.loadBalancer.ingress[0].ip}') "
export ingress_port="$(kubectl -n "$osm_namespace" get service ingress-pipy-controller -o jsonpath='{.spec.ports[? (@.name=="http")].port}')"
echo $ingress_host:$ingress_port

Deploying the sample service

The next step is to deploy the sample httpbin service.

## Create namespace kubectl create ns httpbin
kubectl create ns httpbin

# Add the namespace to the grid
osm namespace add httpbin

# Deploy the application
kubectl apply -f - <<EOF
apiVersion: v1
kind: ServiceAccount
metadata:
  name: httpbin
  namespace: httpbin
---
apiVersion: v1
kind: Service
metadata:
  name: httpbin
  namespace: httpbin
  labels:
    app: httpbin
    service: httpbin
spec:
  ports:
  - name: http
    port: 14001
  selector:
    app: httpbin
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: httpbin
  namespace: httpbin
spec:
  replicas: 1
  selector:
    matchLabels:
      app: httpbin
  template:
    metadata:
      labels:
        app: httpbin
    spec:
      serviceAccountName: httpbin
      containers:
      - image: kennethreitz/httpbin
        imagePullPolicy: IfNotPresent
        name: httpbin
        command: ["gunicorn", "-b", "0.0.0.0:14001", "httpbin:app", "-k", "gevent"]
        ports:
        - containerPort: 14001
EOF

Verify that the pod and service are created and the application is running successfully.

kubectl get pods,services -n httpbin
NAME READY STATUS RESTARTS AGE
pod/httpbin-54cc8cf5d-7vclc 2/2 Running 0 14s

NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/httpbin ClusterIP 10.43.105.166 <none> 14001/TCP 14s

Configure entry rules

Next we want to access the deployed httpbin service from outside the cluster and need to provide the ingress configuration rules.

kubectl apply -f - <<EOF
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: httpbin
  namespace: httpbin
  annotations:
    pipy.ingress.kubernetes.io/rewrite-target-from: /httpbin
    pipy.ingress.kubernetes.io/rewrite-target-to: /httpbin
spec:
  ingressClassName: pipy
  rules:
  - host: httpbin.org
    http:
      paths:
      - path: /httpbin
        pathType: Prefix
        backend:
          service:
            name: httpbin
            port:
              number: 14001
EOF

Accessing the httpbin service using the IP address and port recorded above will result in the following 502 Bad Gateway error response. This is because we have not set the ingress of the FSM as a trusted portal.

curl -sI http://"$ingress_host":"$ingress_port"/httpbin/get -H "Host: httpbin.org"
HTTP/1.1 502 Bad Gateway
content-length: 0
connection: keep-alive

Execute the following command to set the FSM ingress as a trusted entry.

kubectl apply -f - <<EOF
kind: IngressBackend
apiVersion: policy.openservicemesh.io/v1alpha1
metadata:
  name: httpbin
  namespace: httpbin
spec:
  backends:
  - name: httpbin
    port:
      number: 14001 # targetPort of httpbin service
      protocol: http
  sources:
  - kind: Service
    namespace: "$osm_namespace"
    name: ingress-pipy-controller
EOF

Try requesting the httpbin service again, and you will be able to access it successfully.

curl -sI http://"$ingress_host":"$ingress_port"/httpbin/get -H "Host: httpbin.org"
HTTP/1.1 200 OK
server: gunicorn/19.9.0
date: Thu, 18 Aug 2022 05:18:50 GMT
content-type: application/json
content-length: 241
access-control-allow-origin: *
access-control-allow-credentials: true
osm-stats-namespace: httpbin
osm-stats-kind: Deployment
osm-stats-name: httpbin
osm-stats-pod: httpbin-54cc8cf5d-7vclc
connection: keep-alive

Summary

FSM Ingress exposes the application access points within the Kubernetes cluster for easy management of portal traffic. osm-edge's IngressBackend API provides an additional line of defense against accidental data leakage by exposing services within the mesh to the public.

Benchmarking osm and osm-edge data planes

Ali Naqvi — Sun, 07 Aug 2022 04:51:14 +0000

osm-edge is built on top of Open Service Mesh (OSM) v1.1.0 codebase and is a lightweight service mesh for resource-sensitive cloud environments and edge computing scenarios. It uses osm as the control plane and Pipy as the data plane and features high performance, low resources, simplicity, ease of use, scalability, and compatibility (x86/arm64 support).

osm-edge is a simple, complete, and standalone service mesh and ships out-of-the-box with all the necessary components to deploy a complete service mesh. As a lightweight and SMI-compatible Service Mesh, osm-edge is designed to be intuitive and scalable.

For detailed introduction and documentation, refer to osm-edge documentation

In this experiment, benchmarks were conducted for osm-edge(pipy) (v1.1.0) and osm(envoy) (v1.1.0). The main focus is on service TPS, latency distribution when using two different meshes, and monitoring the resource overhead of the data plane.

osm-edge uses Pipy as the data plane; osm uses Envoy as the data plane.

Testing Environment

The benchmark was performed in a Kubernetes cluster running on Cloud. Cluster comprised of 2 worker nodes and one load generator node. Software and hardware configurations as below:

Kubernetes: k3s v1.20.14+k3s2
OS: Ubuntu 20.04
Nodes: 16c32g * 2
Load Generator: 8c16g

What features were tested?

Both meshes had permissive traffic mode enabled
Both meshes had mutual TLS enabled and were encrypting traffic and validating identity between all application pods.
Both meshes were reporting metrics, including L7 metrics, though these metrics were not consumed in this experiment.
Both meshes logged various messages at the INFO level by default. We did not configure logging.

What application was used for testing?

The test application uses the common SpringCloud microservices architecture. The application code is taken from flomesh-bookinfo-demo, which is a SpringCloud implementation of bookinfo application using SpringCloud. In the tests, not all services were used, but rather the API gateway and the Bookinfo Ratings service were selected.

External access to the service is provided through Ingress; the load generator uses the common Apache Jmeter 5.5.

For the tests, 2 paths were chosen: one is the direct access to the Bookinfo Ratings service via Ingress (hereafter ratings), and the other passes through the API Gateway (hereafter gateway-ratings) in the middle. The reason for choosing these two paths is to cover both single-sidecar and multiple-sidecar scenarios.

Procedure

We start the test with a non-mesh (i.e., no sidecar injection, hereafter referred to as non-mesh), and then test with osm-edge(pipy) and osm(envoy) mesh, respectively.

To simulate the resource-constrained scenario of the edge, the CPU used by the sidecar is limited when using the mesh, and the 1 core and 2 core scenarios are tested respectively.

Thus, there are 5 rounds of testing, with 2 different tests in each round.

Performance

Jmeter uses 200 threads during the test and runs the test for 5 minutes. Each round of testing is preceded by a 2-minute warm-up.

For sake of verbosity, only the results of the gateway-ratings paths are shown below with different sidecar resource constraints.

Note: The sidecar in the table refers to the API Gateway sidecar.

mesh	sidecar max CPU	TPS	90th	95th	99th	sidecar CPU usage	sidecar memory usage
non-mesh	NA	3283	87	89	97	NA	NA
osm-edge(pipy)	2	3395	77	79	84	130%	52
osm(envoy)	2	2189	102	104	109	200%	108
osm-edge(pipy)	1	2839	76	77	79	100%	34
osm(envoy)	1	1097	201	203	285	100%	105

Sidecar with 2 CPU core limitations

In the sidecar 2-core scenario, using the osm-edge(pipy) mesh gives a small TPS improvement over not using the mesh, and also improves latency. Improvement of TPS and latency is due the fact that Java NIO uses reactor pattern, while Pipy is built on C++ ASIO which uses the proactor pattern and the connection from Pipy to Java are long duplex connections. The sidecar for both API Gateway and Bookinfo Ratings is still not running out of 2 cores (only 65%), when the Bookinfo Ratings service itself is at its performance limit.

The TPS of the osm(envoy) mesh was down nearly 30%, and the API Gateway sidecar CPU was running full, which was the bottleneck.

In terms of memory, the memory usage of osm-edge(pipy) and osm(envoy) sidecar is 50 MiB and 105 MiB respectively, meaning osm-edge(pipy) was utilizing 52% less memory than osm(envoy).

TPS

Latency distribution

API gateway sidecar CPU usage

API gateway sidecar memory footprint

Bookinfo Ratings sidecar CPU usage

Bookinfo Ratings sidecar memory usage

Sidecar with 1 CPU core limitation

The difference is particularly noticeable in tests that limit the sidecar to 1 core CPU. At this point, the API Gateway sidecar becomes the performance bottleneck, with both the osm-edge and osm sidecar running out of CPU.

In terms of TPS, osm-edge(Pipy) drops by 12% and osm(Envoy)'s TPS drops by a staggering 65%.

TPS

Latency distribution

API gateway sidecar CPU usage

API gateway sidecar memory footprint

Bookinfo Ratings sidecar CPU usage

Bookinfo Ratings sidecar memory usage

Summary

The focus of this experiment was to benchmark the performance and resource consumption of both meshes in a resource-constrained environment. Experiment results revealed that osm-edge maintained high performance and made efficient use of allocated resources. This experiment proves that complete service mesh functionalities can be used and utilized for resource-constrained edge scenarios.

osm-edge is well-suited and ready for both cloud and resource-sensitive environments.

Announcing osm-edge 1.1: ARM support and more

Ali Naqvi — Thu, 28 Jul 2022 13:19:04 +0000

We're very happy to announce the release of osm-edge 1.1. This release is the culmination of months of effort from the flomesh.io team and we are very excited to unveil it today. osm-edge is built on top of Open Service Mesh v1.1.0 codebase and is built purposely for edge computing and uses lightweight, high-performant, cloud native, and programmable proxy Pipy as its data plane and sidecar proxy.

Why osm-edge?

In practice, we have encountered users from a variety of industries with similar requirements for service mesh. These industry users and scenarios include:

Energy and power companies. They want to build simple server rooms at each substation or gas station to deploy computing and storage capacity for the processing of data generated by devices within the coverage area of that location. They want to push traditional data center applications to these simple rooms and take full advantage of data center application management and operation capabilities
Telematics service providers. They want to build their simple computing environments in non-data center environments for data collection and service delivery to cars and vehicle owners. These environments may be near highway locations, parking lots, or high-traffic areas
Retailers. They want to build a minimal computing environment in each store, and in addition to supporting traditional capabilities such as inventory, sales, and payment collection, they also hope to introduce new capabilities of data collection, processing, and transmission
Medical institutions. They want to provide network capabilities at every hospital, or a simple point of care, so that in addition to providing digital service capability for patients, they can also complete data collection and data linkage with higher management departments at the same time
Teaching institutions, hospitals, and campuses alike. These campuses are characterized by a relatively regular and dense flow of people. They want to deploy computing resources near more crowd gathering points for delivering digital services, as well as collecting and processing data in real-time

These are typical edge computing scenarios, and they have similar needs:

Users want to bring the traditional data center computing model, especially microservices, and the related application delivery, management operation, and maintenance capabilities to the edge side
In terms of the working environment, users have to deal with factors such as power supply, limited computing power, and unstable network. Therefore, computing platforms are required to be more robust and can be deployed quickly or recover a computing environment completely in extreme situations
The number of locations (we call POP=Point of Presence) that usually need to be deployed is large and constantly evolving and expanding. The cost of a POP point, the price of maintenance, and the price of expansion are all important cost considerations.
Common, or low-end, PC servers are more often used in these scenarios to replace cloud-standard servers; low-power technology-based computing power such as ARM is further replacing low-end PC servers. On these hardware platforms, which are not comparable to cloud-standard servers, users still want to have enough computing power to handle the growth in functionality and data volume. The conflicting demands of computing moving closer to where the data is generated, the growth in data volume and functional requirements at the edge, and the limited computing resources at the edge require edge-side computing platforms to have better computing power efficiency ratios, i.e., running more applications and supporting larger data volumes with as little power and as few servers as possible
The fragility and a large number of POP points require better application support for multi-cluster, cross-POP failover. For example, if a POP point fails, the neighboring POP points can quickly share or even temporarily take over the computing tasks.

Compared with the cloud data center computing scenario, the three core and main differences and difficulties of edge computing are:

Edge computing requires support for heterogeneous hardware architectures. We see non-x86 computing power being widely used at the edge, often with the advantage of low power consumption and low cost
Edge computing POP points are fragile. This fragility is reflected in the fact that they may not have an extremely reliable power supply, or the power supply is not as powerful as a data center; they may operate in a worse environment than the constant temperature and ventilation of a data center; their networks may be narrowband and unstable
Edge computing is naturally distributed computing. In almost all edge computing scenarios, there are multiple POP points, and the number of POP points is continuously increasing. the POP points can disaster-proof each other and migrate to adjacent POP points in case of failure, which is a fundamental capability of edge computing

The evolution of Kubernetes to the edge side solves the difficulties of edge computing to a certain extent, especially against fragility; while the development of service mesh to the edge side focuses on network issues in edge computing, against network fragility, as well as providing basic network support for distributed, such as fault migration. In practice, container platforms, as today's de facto quasi-standard means of application delivery, are rapidly evolving to the edge side, with a large number of releases targeting edge features, such as k3s; but service mesh, as an important network extension for container platforms, are not quickly keeping up with this trend. It is currently difficult for users to find service mesh for edge computing scenarios, so we started the osm-edge an open source project with several important considerations and goals, namely

Support and compatibility with the SMI specification, so that it can meet users' needs for standardization of service mesh management
Full support for the ARM ecosystem, which is the "first-class citizen" or even the preferred computing platform for edge computing, and the Service Mesh should be fully adapted to meet this trend. osm-edge follows the ARM First strategy, which means that all features are developed, tested, and delivered on the ARM platform first
High performance and low resources. The service mesh as infrastructure should use fewer resources (CPU/MEM) while delivering higher performance (TPS/Latency) at the edge.

Features

Light-weight, high-performant, cloud-native, extensible
Out-of-the-box supports x86, ARM architectures
Easily and transparently configure traffic shifting for deployments
Secure service-to-service communication by enabling mutual TLS
Define and execute fine-grained access control policies for services
Observability and insights into application metrics for debugging and monitoring services
Integrate with external certificate management services/solutions with a pluggable interface
Onboard applications onto the mesh by enabling automatic sidecar injection of Pipy proxy

ARM Support

osm-edge 1.1 brings the oft-requested support for ARM (both for development & production use), Whether you’re focused on cost reduction with ARM-based compute such as AWS Graviton or simply want to run service mesh on your Raspberry Pi cluster, now you can!

Multi-cluster Kubernetes the Kubernetes way

osm-edge 1.1 comes bundled with Flomesh Service Mesh (FSM) a Kubernetes North-South traffic manager, provides Ingress controllers, Gateway API, Load Balancer, and cross-cluster service registration and service discovery.

With the help of FSM, osm-edge can now connect Kubernetes services across cluster boundaries in a way that’s secure, fully transparent to the application, and independent of network topology. Automated fail-over ability to automatically redirect all traffic from a failing or inaccessible service to one or more replicas of that service—including replicas on other clusters.

Multiple Sidecar Proxy support

To break the tight coupling on Pipy and open doors for 3rd parties to develop or make use of their data plane or sidecar proxies, we have refactored the OSM v1.1.0 codebase to make it generic and provide extension points. We strongly believe in and support open source and our proposal for this refactoring have been submitted to upstream for their review, discussion, and/or utilization.

And lot more

osm-edge 1.1 also has a tremendous list of other improvements, performance enhancements, and bug fixes, for a detailed change log, please refer to the Release page on Github. Below are some of the notable changes included in this release:

Refactoring of Proxy Control Plane component to stay generic and allow interaction with new sidecar proxy implementation
Added new driver.Driver interface that needs to be implemented by 3rd party vendors wishing to provide sidecar proxy for control plane
Added new driver.HealthProbes, driver.HealthProbe, driver.InjectorContext, driver.controllerContext struct to use with new sidecar proxy driver implementation
Refactored Envoy based proxy sidecar integration to a separate Driver, so that it works as an implementation of driver.Driver interface
Added pipy implementation as a new Proxy Control Plane component proxy Driver
Refactors Helm chart values.yaml and added items osm.sidecarClass, osm.sidecarImage, osm.sidecarWindowsImage, osm.sidecarDrivers List to allow configuring the proxy sidecar driver.
Pipy driver is the default driver of osm-edge distribution, but can be configured via cli flag --set=osm.sidecarClass=XXX where XXX refers to sidecar Driver.
osm-edge control plane images are now multi-architecture, built for Linux/amd64 and Linux/arm64
osm-edge test suite used images are now multi-architecture, built for Linux/amd64 and Linux/arm64
Optimization of scripts
Added Makefile targets for easier installation/setup
Updated scripts to setup a development environment on amd64 and arm64 architectures.

What's next and where to learn more?

For more documentation, a quick-start guide, and step-by-step tutorials please visit osm-edge-docs

osm-edge is a fork of open service mesh and we will strive to keep this fork in sync with its upstream and propose back major changes and/or feature proposals to upstream for broader benefits of the community. Both OSM and osm-edge are hosted on Github. If you have any feature request, question, or comment, we’d love to have you join the rapidly-growing community via Github Issues, Pull Requests, or osm slack channel!

MQTT policy enforcement with Pipy

Ali Naqvi — Fri, 17 Jun 2022 16:23:25 +0000

If you play in the IoT space, you may have heard of the MQTT. MQTT is an OASIS standard messaging protocol for the Internet of Things (IoT) that is becoming the defacto standard and is already in use in a wide variety of industries like automotive, manufacturing, telecommunications, oil, and gas.

The Internet of Things (IoT) can be loosely defined as a system of sensors and other devices interacting with industry systems all to enhance business operations. Industries like manufacturing, automotive, telecommunication, and oil & gas to name just a few, deploy massive numbers of sensors. These sensors in turn send critical telemetry data to analytics engines, where the data is analyzed for trends and/or anomalies, enabling organizations to better understand and improve their operations.

Although IoT contributes many novel applications to our daily life, one of its main concerns refers to IoT device security. Recent cyber-security incidents exposed the security pitfalls and vulnerabilities in IoT devices.

The purpose of this tutorial is to demonstrate how to enforce custom policies on devices communicating via MQTT by using Pipy as a MITM that sits between MQTT Providers & Consumers and MQTT servers (previously known as brokers).

Pipy is an open-source, lightweight, high-performance, programmable network proxy for cloud, edge, and IoT. Pipy versatile nature allows it to be used in a variety of use-cases ranging from (but not limited to) edge routers, load balancers & proxy solutions, API gateways, static HTTP servers, service mesh sidecars, admission controllers, policy engines, and other applications.

Pipy comes with built-in support for MQTT protocol versions 5.0 and 3.1.1 (aka version 4).

Having Pipy in the middle allows us to perform authentication, and authorization, applying rules like content validation, content replacement, allowing/denying consumers subscribing to particular topics, invoking business rules validations either inside Pipy or invoking already existing business domain services, etc. This list of policies and rules can be adapted or modified per your business case.

For demonstration purposes and simplicity's sake, we will be defining some simple rules in JSON format and have them applied at Pipy proxy, working with a single MQTT server, which won't be dealing with TLS support. But for more complex rules and/or policies you can expand the rules, and policies and update the Pipy script or integrate them with your existing systems. Pipy comes with built-in support for load balancing, routing, TLS termination/offloading, and much more. For more details and step-by-step tutorials, please refer to Pipy's Website.

Demo code is accompanied by a Docker Compose file which configures:

Eclipse Mosquitto as MQTT server
Pipy proxy
Expose MQTT port 1883 and Pipy Admin GUI port 6060

Sample policy and rules are defined in policy.json where we define 4 test accounts and some rules on which clients are allowed to subscribe/publish to which topics. Pipy proxy script will read this configuration file and enforce them before routing traffic to the upstream MQTT server. Refer to Publisher workflow and Subscriber workflow.

Source code is made available and can be downloaded from Github repository pipy-mqtt-policy-engine

Below diagrams details the architecture and flows between the various components:

Architecture Overview

Publisher workflow

Subscriber workflow

Running the Demo Project

First, download the tutorial code and initialize the components by calling Docker Compose:

$ git clone https://github.com/flomesh-io/pipy-demos.git
$ cd pipy-mqtt-policy-engine
$ sudo docker-compose up -d

This brings up a complete environment, including the Pipy, Eclipse Mosquitto MQTT server. Feel free to adjust the docker-compose.yaml or pipy/config/policy.json according to your specific needs.

For viewing Pipy proxy scripts, statistics, and other demos you can open Pipy GUI admin console by pointing your browser to http://localhost:6060.

Testing

You can use the MQTT client of your choice which supports MQTT protocol version either 5.0 or 3.1.1 (aka 4). If you don't have any choice then you try an all-round MQTT client MQTT Explorer.

Conclusion

This tutorial showed how you can leverage Pipy to enforce custom policies and rules without requiring any change in IoT devices and MQTT server.

Pipy from Flomesh is an open-source, extremely fast, and lightweight network traffic processor which can be used in a variety of use cases ranging from edge routers, load balancing & proxying (forward/reverse), API gateways, Static HTTP Servers, Service mesh sidecars, , admission controller, policy engine and other applications. Pipy is in active development and maintained by full-time committers and contributors, though still an early version, it has been battle-tested and in production use by several commercial clients.

What & Why of programmable proxy

Ali Naqvi — Tue, 12 Apr 2022 15:42:57 +0000

Question which gets asked too often is "When you say programmable proxy, what is a programmable proxy, and why do you need a programmable proxy?". This article tries to answer this question from different perspective. We will start with brief definition of what is a proxy? then move forward our discussion of evolution, proxy went through its development process, explaining the need for those stages and what benefits each stage provide and why such stage improvements were required. In end we will discuss several aspects of "programmability" and provide a summary of "why we need programmable proxy".

A proxy server is usually deployed in the middle of two isolated networks where it has access to both sides of the network and transfers data from one side to the other to achieve network connectivity. Proxy is a kind of serial network device, which has existed since the birth of computer networks. Proxy acts not only as a network connector, but it also opens a path to new functionality and usage scenarios like:

Routing. The proxy forwards data to different destinations according to the characteristics of the data defined.
Load balancing. During forwarding, data is distributed to different destinations to improve throughput and avoid destination single point of failure. Load balancing is becoming an area of proxy segmentation functionality
Fail over. In the forwarding process, when the destination fails, the proxy can forward the data to an alternate destination, providing uninterrupted service to the requester
Access control. Proxy can decide what traffic is allowed through and what needs to be blocked. Web Application Firewall (WAF) is a typical proxy application example
Identity management. Access control is often based on identity information, so proxies often have identity management capabilities as well
Network acceleration. proxy accelerates network access by caching data
Metrics collection. Proxy captures data statistics and summarizes it to NPM software for network optimization and network planning
Information security. In addition to access control, proxies can also be used for security auditing, TLS/SSL offloading, and data encryption to meet security requirements

In addition to proxying, routers provide the functionality of bridging two networks. Routers work at layer 3 of the network while Proxies work at layers above 3, like layer 4 and layer 7.

Proxy servers have gone through following evolutions during its development:

Configurations file era. Proxy software (mostly open source software), which makes up the majority of network infrastructure software, provides different functions in different domains, such as proxy for different protocols, proxy for load balancing, proxy for cache acceleration. This generation of proxy is all configuration based. User sets parameters, configures rules in the configuration file, and then starts the service process to execute the rules
Configuration DSL. Static configuration files are difficult to express complex logic, so many proxies introduced thin scripting capabilities on top of configuration, commonly referred to as "configuration languages" or Domain specific languages (DSL for short), such as Haprxoy's ACL or Varnish's VCL
Scripting language era. As the logic becomes more complex, it becomes more difficult to express that via configuration languages; At the same time, when the number of configuration languages reaches a certain extent, the management of configuration languages itself becomes difficult. Shell scripts, for example, one can write simple logic, but when the shell code reaches a certain level, it is often a step toward more structured scripting languages like Perl or Python. Proxy supports scripting language, which has both the convenience of scripting language and the structural advantage of programming language. Examples of this are OpenResty (Nginx + Lua) and Nginx Plus(Nginx + NJS). These examples also include proxy servers implemented by a number of application programming languages, such as Node.js based StrongLoop and Spring Gateway, which often have scripting capabilities of their own
Cluster age. Script language solves the difficulties in modularization and structuring of complex logic in proxy server. A further requirement at this point is to integrate proxies with other administrative control tools, hence the need for a REST or alike interface. The external control plane can dynamically update the logic in the script through the REST or alike interface. At the same time, the use of proxies has moved from single instances to cluster of proxies, so proxy software like Envoy and OpenResty based Kong often support clustering capabilities themselves, implementing clustering capabilities in some centralized or shared way, while providing REST interfaces. For proxies of type #4 mentioned above, cluster management is generally possible through configuration management; Configuration management tools can also expose REST interfaces. For example, the Ansible + Nginx solution implements similar capabilities to #5. In contrast, scheme #4 requires more components to form the scheme, while scheme #5 is more convergent
Cloud era. In this era, proxies are deployed in a distributed manner. The most common scenario is to deploy one proxy for each application process e.g Sidecar Proxy pattern. Adoption of distributed proxies, opened the doors for using different rules and policies for different upstream services, that is, multi-tenant capability. The various upstream services not only have logically independent rules and policies; Physical isolation is also provided, allowing granular management at the process and interface level. If we consider the control plane and data plane of the service mesh as a whole, then the service mesh is representative of this domain, typical examples are Istio+Envoy, Linkerd + Linkerd proxy. Pipy is a product of this age and falls under this category.

In the above evolution stages, each stage is an improvement over the previous one:

#2 adds basic scripting capabilities over #1. This basic scripting capability is augmented by dynamic capabilities on top of configuration files. For example, get the characteristics of the request at run time (such as HTTP headers), and then make dynamic logical decisions based on those characteristics to perform specific actions
#3 adds full scripting capabilities over #2, allowing scripting logic to be structured and modular. In the #2 stage, when logic becomes complex, the volume of scripts increases dramatically, and structured scripting capabilities become a necessity
#4 has more REST interfaces and clustering capabilities than #3. To horizontally expand the proxy capacity, you need to cluster multiple proxy instances. Instances in the cluster share configurations and scripts, and users can manage configurations and scripts through the REST interfaces
#5 has more distributed capabilities than #4, mainly because different instances within the same cluster run different scripts and configurations. In #4 mode, there are usually different configurations and policies for different upstream services, such as different authentication modes and access control mechanisms. As upstream services grow, the configurations of these different upstream services are logically separated but physically run in the same proxy process. This logically separated configuration and policy scripts running in the same physical process has some disadvantages: the more logic running in one process brings the more complexity; different upstream services share resources such as CPU and memory, affecting each other. If a script of an upstream service has a security vulnerability, the configurations of other upstream services may be leaked, resulting in security risks. #5 stage is an improvement over the #4 in that the proxy processes for each upstream service are independent and isolated from each other. They are managed by the same cluster manager, but the configuration and scripts at work are separated and isolated. This isolation is a strong requirement in a multi-tenant environment -- different upstream services belong to different tenants, and tenants should not affect each other or know each other's configurations. #5 can be thought of as the extreme mode of "#4 multi-clustering "-- at its most extreme, each process has its own configuration

Lets take another look at the evolution of proxies -- The Evolution of Requirements.

The first generation of proxies mainly implemented proxy functionality and provided basic configurable capabilities. At that time, network equipment, especially Serial network equipment, required proxy to be highly reliable; The real-time transmission of massive data requires high throughput, low latency and low resources. Like all software, proxy also required to support modularity and extensibility. Proxy in this stage was mainly developed in C language, and the corresponding development of extension modules were also developed in C language. Extension modules get loaded dynamically at the start of the process. To summarize, proxy requirements at this stage are: connectivity (network capabilities), ease of use (configurable via configuration files), reliability (requirements for cross-linking devices), high performance, scalability
The second generation of proxies got further improvements in extensibility and flexibility, such as some dynamic data acquisition and making logical judgment. The introduction of scripts further enhanced the ease of use. Support for combinatorial logic and dynamic data retrieval provided flexibility while improving scalability
The main improvements of the 3rd generation proxies over the 2nd generation proxies are manageability, developer friendliness and programmability. Scripting capabilities are widely used, mainly because it is difficult to develop and maintain extensions using C language, while scripting languages are easier to learn and provide fast turnarounds in developments when compared to compiled languages counterparts. Developers' development efficiency and the difficulty of massive script maintenance require this generation of proxies to use a more structured scripting language, and to maintain performance, resource utilization and other core capabilities of the previous generation. The use of structured and modular scripting language, opened the era of programmable proxies and required proxies to provide two level of possibilities, one is to use C language for development of core modules, and scripts to be used to program dynamic logic; In other words programmable includes the meaning of giving its user the power of development of core modules and dynamic logic.
The fourth generation of proxies starts with cluster support capabilities, which are improvements to manageability. With the support of REST interfaces, proxies become part of network infrastructure implementation, and a landing place for infra as code. REST interface capabilities improved proxies ability to be managed and considered a part of the ease of administration. External interfaces are also an important feature of programmable, and REST, as the most common form of interface, is also widely used in the proxy server world. At this point, programmable consists of three layers: programmable core modules as described in #3, programmable dynamic logic , and programmable external interfaces. The emergence of proxy server cluster reflects the change of scalability from function expansion to resource expansion. The emergence of REST interfaces provides the technical foundation for further self-service and managed services
The evolution of the fifth generation of proxy is driven by the popularity and rapid development of cloud computing. The elasticity, self-service, tenant, isolation, and metering of the cloud require that the proxy service software have the best capability of the cloud native. If the fourth generation of agents is for system administrators, the fifth generation of agents is for cloud services. While fully maintaining the characteristics of previous generations of proxy software, Cloud Ready is further incorporated. With the expansion of cloud computing to the edge, the fifth generation of proxies are also toward the development of heterogeneous hardware, heterogeneous software, low energy consumption, so this generation of proxies began to show the ability to optimize the integration of cloud and edge. The fifth generation of proxies in the programmable further evolution, from the core module, dynamic logic, external interface, increased cloud ability; Supports distributed, multi-tenant, and metering. Metering is a derivative requirement of multi-tenancy, which requires isolation on the one hand, and resources that can be measured at the smallest possible granularity on the other

Let's summarize the above discussion into a tabular format, with the first column identifying a specific requirement that the proxy meets; Header row represents proxies at different stages, with the typical examples or known software in parentheses. In each cell, we use * to indicate whether such capabilities are available and to what extent (1-5 *, 5 * for full support, and 1 * for basic support).

SN	Requirement	Configuration (squid, httpd, nginx)	Configuration Language (varnish, haproxy)	Scripting support (nginx+lua, nginx+js)	Clustering (kong, envoy)	Cloud (istio+envoy, linkerd, pipy)	Remarks
1.	Connectivity	* * * * *	* * * * *	* * * * *	* * * * *	* * * * *	Connectivity in the cloud era began using kernel technologies such as iptables and ebpf; Previously, there was only user Space process mode
2.	reliability	* * * * *	* * * * *	* * * * *	* * * * *	* * * * *	Reliability has always been the most important fundamental capability of proxy server
3.	high performance	* * *	* * * *	* * * * *	* * * * *	* * * * *	Performance includes throughput, latency, error rate, and deviation from the mean. Latency metrics P99, P999 and other. Early proxy software has a long tail effect, so the indicators above P99 are not as good as later software. Proxy with high-performance scripts often perform better than their predecessors when returning the same content. Agents using proactive technology are more stable (with less deviation from the mean) while providing the same performance
4.	flexibility	*	* *	* * *	* * * *	* * * * *	Compared with the fourth generation, the fifth-generation proxies significantly enhances the multi-protocol support capability, so we give this generation a five-star evaluation. Moreover, the processing model of the fifth generation can adapt to various protocols and has universality, which is better than that of the fourth generation
5.	scalability	*	* *	* * *	* * *	* * * *	Similar to Flexibility, the 5th generation proxies supports multiple protocols in addition to core function extension development, 7-layer logic extension development, so we give it one star more than the 4th generation
6.	hardware compatibility	* * * *	* * * *	* * * *	* * * *	* * * *	Proxies developed in C or C++ generally have better hardware compatibility and a more active community to migrate applications to new hardware architectures. Proxies developed using Rust, Go, and Lua have been relatively slow to migrate for hardware compatibility
7.	system compatibility	* * *	* * *	* * * *	* * * *	* * * * *	system mainly includes two aspects, one is the operating system, the other is the cloud platform. In terms of operating system compatibility, each generation of proxies is similar; However, in terms of cloud platform compatibility, both the fourth and fifth generation proxies have more fully considered and realized cloud compatibility. In contrast, the significant difference between the fifth generation and the fourth generation is the ability to support multi-tenant
8.	ease of management	* *	* *	* *	* * *	* * * *	ease of management is a function of the system operations & administrator roles. The first and second generations mainly use configuration files, based on which the use of configuration management tools to achieve automatic and batch management. In the third generation, in addition to managing configuration files, we need to further manage source files. But in essence, there is no significant difference from the first and second generation of ease of management. The fourth generation provides REST interfaces, which greatly improves the ease of management. In addition to REST, the fifth generation usually provides a control plane for the cloud to manage proxies; It also provides multiple external interfaces to meet other management requirements, such as monitoring, auditing, and statistics
9.	ease of use	*	*	*	* *	* * *	The primary users of the first three generations of proxies are ops & sys administrators. In the fourth generation, administrators began to provide some functions to users, and the as-a-service mode began to appear. The fifth generation takes into account more user scenarios and provides more rent-oriented capabilities
10.	ease of development	*	* *	* * *	* * * *	* * * * *	The development around the proxies includes two aspects, one is inside the proxy to achieve the functionality, another is outside the proxy to achieve the management ability of the proxy. The first three generations provide an interface for internal development; The latter two generations provide both internal and external interfaces. The significant improvement of the fifth generation compared with the fourth generation is the cloud interface
11.	core interface is programmable	*	*	*	*	*	Each generation of proxies provides the ability to extend core interfaces, but these interfaces are too low-level and difficult to master
12.	functionality extension is programmable	*	* *	* * *	* * * *	* * * * *	Providing the ability to extend functions more efficiently is part of the process that gets better with each generation of proxies. It is the core metric of programmable proxy
13.	protocol extensions are programmable				* *	* * *	The first three generations are mainly for single agreements, or fixed agreements. Starting with the fourth generation, users began to seek support for multiple protocols and custom protocols. In the fifth generation, protocol extension is considered as a core capability in the design
14.	modular scripting			* * *	* * * *	* * * *	Third generation proxies are beginning to pay more attention to script structuring; The fourth and fifth generations attempt to prepare for more structured programming, such as Envoy's attempt to provide multilingual support through WASM; Pipy introduces high-performance JS scripts for better structuring
15.	configuration management is programmable				* *	* * *	The first three generations of proxy configuration are mainly for ops management personnel, and external configuration management tools are based on this premise. The fourth generation supports REST management interfaces. The fifth generation further provides a standard cloud interface for configuration management
16.	resource extensions are programmable	*	*	*	* *	* * * *	In the first three generations, proxy capacity expansion is mainly to increase the number of threads or processes. The fourth generation provides scale-out capability for processes, known as clustering capability. On the basis of the fourth generation, the fifth generation on the one hand provides horizontal expansion capabilities, on the other hand provides capabilities under smaller resources to support more fine-grained metering and billing; That is, it not only supports incremental scaling, but also provides the ability to reduce scaling, and all of these capabilities provide programming interfaces
17.	tenant extension programmable					* * *	The cloud is something that has emerged at the same time as the fourth generation of proxy, and tenant, as a core feature of the cloud, is not well supported in the fourth generation. The fifth generation is designed on the premise of cloud, considering and providing tenants with the possibility of programming their own extensions

rows #11 to #17 in the table above are the specific aspects of Programmable Proxy. These aspects also constitute the answer to the question of Why Programmable Proxy:

The internal functions of the proxy need to be expanded, including the extension of the underlying core capabilities, the extension of supporting more protocols, and Layer7 processing capabilities (forwarding, routing, judgment, access control, etc.); These Layer7 processing power require a more convenient way of programming, that is, scripting and structured programming ability
Proxies need to provide interfaces externally to integrate into larger management systems (such as cloud platforms), including configuration management, resource management, and so on
Proxies need to provide extensible capabilities for different roles, including operations, administrators, resource providers, and tenants, all of which require the capability of Programmable to some extent
Also, like any programmable component, Programmable Proxy requires accompanying documentation, development manuals, code management, dependency management, build and deployment tools, and preferably a visual development and debugging environment. Only when these requirements are fully met can users better manage network traffic and the services carried on the traffic

Pipy + Redis + Sentinel = High available Redis

Ali Naqvi — Mon, 04 Apr 2022 13:04:02 +0000

Pipy is an open source programmable proxy for cloud, edge, and IoT. Pipy versatile nature allow it be used in multiple use cases. The main purpose of this article is to demonstrate the implementation of Pipy as TCP load balancer in front of a Redis master and slave (agent) replication to ensure that client and application connections are always entertained.

Our goals

We want to implement and achieve automatic failover capability in cases when designated Redis master node stop responding because of process termination, hardware failure, network hiccups or some uknown reasons, and Sentinel automatically fails over the master to one of the remaining agent nodes.
We want clients and application connecting to Redis are not impacted by failover mechanism.
We want whole failover mechanism to be fully transparent without requiring any change in client and applications configuration/settings.
We want all write (Redis SET or other user configured commands) requests are handled by master node without client and application knowing the topology in place.
We want all our requests are distributed equally to available nodes for making better use of allocated resources and high throuput.
....

Load balancing, proxying TCP connections, ensuring SET (or other user configured) redis commands are always routed to master node, health probing, logging to console, recording metrics etc is achieved by Pipy.

High availability for Redis deployment including master and agent implementation is achieved by implementing Redis Sentinel, which continuously performs monitoring, notification, and automatic failover if a master becomes unresponsive.

Quick introduction to technologies used

Pipy is an open source programmable proxy for cloud, edge, and IoT.
Redis is an open source, in-memory data store, benchmarked as the world’s fastest. Redis powers cutting edge applications and is known to enhance use cases such as real-time analytics, fast high-volume transactions, in-app social functionality, application job management, queuing and caching.
Redis Sentinel provides high availability for Redis when not using Redis Cluster.

Demo high-level deployment architecture

One Pipy proxy server to act as a TCP load balancer and keeping track of which Redis servers are available and ready to serve requests.
Three Redis servers provide the master-agent replication.
Three Sentinel instances for a robust deployment.

Setup

For demonstration purposes, demo code comes with Docker Compose script to spin up containers.

Full source of demo is available and can be downloaded from https://github.com/flomesh-io/pipy-demos/tree/main/pipy-redis-sentinel

Prerequisites

Clone Pipy Demos Repo or download demo code from pipy-redis-sentinel.
Spin up Pipy proxy, Redis, Sentinel containers
```
$ docker-compose up -d
```

Testing

Make sure all containers are up and running:

$ docker ps

CONTAINER ID   IMAGE                       COMMAND                  CREATED         STATUS         PORTS                                                 NAMES
0db7eba4f91d   sentinel                    "sentinel-entrypoint…"   6 seconds ago   Up 6 seconds   6379/tcp, 26379/tcp                                   redis_sentinel_3
e4a7b6b99074   sentinel                    "sentinel-entrypoint…"   6 seconds ago   Up 6 seconds   6379/tcp, 26379/tcp                                   redis_sentinel_2
4ec87846b1e3   naqvis/pipy-pjs:0.22.0-31   "/docker-entrypoint.…"   6 seconds ago   Up 5 seconds   6000/tcp, 0.0.0.0:6379->6379/tcp, :::6379->6379/tcp   pipy-proxy
8e81ddc5eb07   sentinel                    "sentinel-entrypoint…"   6 seconds ago   Up 4 seconds   6379/tcp, 26379/tcp                                   redis_sentinel_1
f1a533de6d41   redis:alpine                "docker-entrypoint.s…"   8 seconds ago   Up 6 seconds   6379/tcp                                              redis-slave2
a522c208b236   redis:alpine                "docker-entrypoint.s…"   8 seconds ago   Up 7 seconds   6379/tcp                                              redis-slave1
8065dec93c3d   redis:alpine                "docker-entrypoint.s…"   8 seconds ago   Up 7 seconds   6379/tcp                                              redis-master

Pipy is listening on TCP port 6379 on the docker host which is the standard Redis port and load balance across the 3 Redis containers (one Master & two Slave). Three Redis Sentinel contaners for a robust deployment and providing high availability for Redis.

Check that Pipy service is routing traffic to all Redis nodes by executing below commands. You don't need to provide docker host IP and port information, as Pipy is exposing Redis default port of 6379.

$ redis-cli info replication | grep role
role:master
$ redis-cli info replication | grep role
role:slave
$ redis-cli info replication | grep role
role:slave

Pipy comes with load balancing algorithms and demo script is configured to use RoundRobinLoadBalancer and you can see from above output that Pipy is sending requests to all configured Redis nodes in roundrobin fashion.

Now try some more Redis commands

$ redis-cli set hello world
OK

$ redis-cli get hello
"world"

$ redis-cli set foo bar
OK

We have configured proxy to use roundrobin algorithms, but we have seen that all SET requests are executed successfully. If we have simply followed the roundrobin algorithm to forward requests equally to each nodes, then our SET requests reaching out to slave nodes would have failed with error like:

(error) READONLY You can't write against a read only slave.

Failover Testing

Let's pause redis-master container to test automatic failover

$ docker pause redis-master

Sentinel will automatically detect that the master is missing, and it will choose a slave to promote that as master. Pipy during health-checks will detect that master node is down, it will mark that as unhealthy and will no longer send requests to that node until node become accessible again. Pipy will detect new master node and will forward all SET commands to newly promoted master node.

Run again some Redis commands to see if can access Redis without any problems.

$ redis-cli set abc 1234
OK

$ redis-cli get abc
"1234"

Bring back pause container, and Sentinel will mark that as slave node

$ docker unpause redis-master

That's it, You should now have everything You need to setup a fault tolerant and high available Redis when not using Redis Cluster.

Performance test

We performed various testes with pipy, haproxy, twemproxy, and single redis instance to find out the drop in through by applying proxy before redis.
following are the test result performed on our VM with 4C8G with OS 20.04.4 LTS (Focal Fossa) ARM64 running on our Mac M1 Max.

It is clear that adding any proxy before Redis is going to cause performance penalty, but that's the cost which one had to pay to achieve the benefits proxy brings.

Conclusion

You can achieve Redis high availability and maximum protection against any failure and disaster for Redis replication by combining the infrastructure services with the software configuration. With propery infrastructure setup and design, we can achieve high availability for Redis replication between master and slave by using different availability zones. Redis Sentinel monitors and performs the failover of Redis replication from master to slave.

Pipy is an open-source, extremely fast, and lightweight network traffic processor which can be used in a variety of use cases ranging from edge routers, load balancing & proxying (forward/reverse), API gateways, Static HTTP Servers, Service mesh sidecars, and many other applications. Pipy is in active development and maintained by full-time committers and contributors, though still an early version, it has been battle-tested and in production use by several commercial clients.

Pipy proxy helps improve SpringBoot REST Service QPS & Latency

Ali Naqvi — Wed, 30 Mar 2022 12:25:06 +0000

Spring and SpringBoot development platforms are one of the most popular development platform among Java developers.

We performed a simple benchmark of SpringBoot starter application running with default configurations with and without a sidecar Pipy proxy and measured QPS and Latency indicators.

Our results shows that deploying Pipy proxy as sidecar improved QPS and reduced latency by almost 10% in our testing environment.

Test Results

#	OS	Tool	Mode	QPS	Latency P50	P90	P99	P99.9
1	ubuntu	ab	springboot	46102
2	ubuntu	ab	pipy-->springboot	50303
3	ubuntu	wrk	springboot	71840	1.30ms	2.04ms	4.27ms
4	ubuntu	wrk	pipy-->springboot	73055	1.35ms	2.09ms	2.49ms
5	ubuntu	fortio	springboot	62249	0.0015217	0.00293947	0.00428543	0.00777612
6	ubuntu	fortio	pipy-->springboot	69233	0.0014350	0.00249741	0.00296226	0.00362949

From the test results, we observed: Accessing Springboot service via Pipy proxy results in higher QPS and lower latency as compared to directly accessing the SpringBoot service.

For detailed test reports along with benchmark procedure, software used, load testing tools, and scripts used refer to Pipy wiki link below:

BENCHMARK：Pipy as proxy for SpringBoot REST service

Announcing Pipy 0.30.0

Ali Naqvi — Fri, 25 Mar 2022 05:45:26 +0000

Pipy 0.30 is now available. It adds improvements to a number of areas including better documentation, more core controls, new filters, enhanced Cache and Metrics API, and some bug fixes. The Pipy Runtime API has expanded its coverage of SSL engines, asynchronus file read/write operations.

This release was truly a community effort and could not have been made possible without all of the hard work from everyone involved in active discussions and the Pipy project on GitHub.The Pipy community provides code submissions covering new functionality and bug fixes, documentation improvements, quality assurance testing, continuous integration environments, bug reports, and much more. Everyone has done their part to make this release possible! If you’d like to join this amazing community, you can find it on GitHub, Slack, and the Pipy discussion groups.

In the following sections, you will find a full list of changes included in this version.

Core

Add flow control mechanism for throttling any types of input
Add asynchronous file operations
Support OpenSSL engines

Filters

New filter tee
New filter throttleConcurrency
New filter demux (legacy demux/mux filters are renamed to demuxQueue/muxQueue)
Filter decodeMQTT to support protocolLevel option
Throttle filters support weak references as the account keys

API

Support TTL option for algo.Cache
New method stats.Metric.valueOf() for querying current metric value
Data.from() rejects input types other than strings

Major bug fixes

Fixed a bug where TLS requests could be stuck in the buffer
Fixed a dead loop when trying to read folders in codebases
Fixed some crashes when null is given to some cypto APIs
Fixed a crash when retrieving stats.Histogram sub-metrics
Fixed a bug where an HTTP 304 response would lead to forever waiting on the connection
Fixed a bug in Data::shift_while() that caused Dubbo decoding failures
The size of Data wasn't changed accordingly in some cases when packing

Build

Support static linking to libc
Support linking to external zlib

Documentation

Tremendous growth in documentation coverage has been done since the last release in English, Chinese and Japanese.

We would like to thank each and every contributor who was involved in this release.