DEV Community: Ali Naqvi

Pipy proxy: Creating HTTP Tunnel with 10 lines of code

Ali Naqvi — Tue, 27 Dec 2022 12:21:54 +0000

Pipy is an open-source, lightweight, high-performance, modular, programmable, cloud-native network stream processor that is ideal for a variety of use cases ranging from (but not limited to) edge routers, load balancers & proxy solutions, API gateways, static HTTP servers, service mesh sidecars, and other applications. Previously we have covered several use cases, and in this blog post will continue our learning of Pipy by implementing an HTTP Tunnel functionality by only using the PipyJS scripting language.

HTTP tunneling is a technique used to transmit data over the internet through a network that does not support the desired data transfer protocol. It involves encapsulating the data in a way that allows it to be transmitted over the internet using the HTTP protocol, which is widely supported by web servers and clients.

There are several reasons why HTTP tunneling might be used. For example, it can be used to bypass firewalls or other network security measures that block certain types of data transfer. It can also be used to access resources on a private network from a remote location or to connect to a network that uses a proprietary protocol not supported by the client.

To use HTTP tunneling, a client sends a request to a server using the HTTP protocol, with the desired data included in the request body. The server then sends a response back to the client using the HTTP protocol, with the data included in the response body. This allows the data to be transmitted over the internet using the HTTP protocol, bypassing any restrictions or limitations on the network.

Solution

Suppose we have two devices communicating with each other using a private protocol based on TCP, and the firewall only allows HTTP communication. The server is listening on an internal IP and port 8081. Here we use Pipy to simulate a TCP server that responds with "Hi, TCP!" (mimicking the private protocol) when it receives a request.

pipy()

  .listen('127.0.0.1:8081')
  .replaceData(
    () => new Data('Hi, TCP!\n')
  )

Next, we need a proxy on the DMZ side to provide HTTP tunneling. The client establishes an HTTP connection with the proxy (using the HTTP CONNECT method). When the connection is successful, the proxy establishes a connection with the destination server as requested by the client, connecting the client and server. The proxy then forwards the TCP stream sent by the client to the server and sends the server's response back to the client.

PipyJS script

The scripting part is simple. We have a proxy listening on port 80 that checks if the request header is an HTTP CONNECT request. If it is, the proxy retrieves the address and port of the destination server from the path in the request header and establishes a connection with the destination server. If it is not an HTTP CONNECT request, it is treated as a regular HTTP request and we return a 404 Not Found! here without implementing it.

pipy({
  _isTunnel: false,
  _target: null,
})

  //http tunnel
  .listen(80)
  .demuxHTTP().to($ => $
    .handleMessageStart(
      msg => msg.head.method === 'CONNECT' && (_isTunnel = true)
    )
    .branch(
      () => _isTunnel, (
      $ => $.acceptHTTPTunnel(
        msg => (
          _target = msg.head.path,
          new Message({ status: 200 })
        )
      ).to(
        $ => $.connect(() => _target)
      )
    ), (
      //common HTTP request
      $ => $.replaceMessage(new Message({ status: 404 }, 'Not Found!\n'))
    )
    )
  )

Here we have a special filter called acceptHTTPTunnel Filter that implements HTTP tunneling on the server side (there is another filter called connectHTTPTunnel Filter on the client side, which we will talk about later). It redirects the TCP stream after the HTTP CONNECT to a child pipeline (to filter in the above code).

Compared to a regular HTTP proxy, there is only the additional acceptHTTPTunnel filter.

Testing

Our proxy and server are running on host 192.168.1.110, and the client curl is running on host 192.168.1.11.

curl -p -x http://192.168.1.110:80 telnet://127.0.0.1:8081

curl uses the -x (--proxy) option to specify the proxy server and the -p (--proxytunnel) option to use the proxy HTTP tunnel.

What could be done next?

This time, our client curl happens to support using HTTP tunnels. What if the client doesn't support it? That's where connectHTTPTunnel, which we mentioned earlier, comes in. This filter provides HTTP tunneling functionality on the client side: first, it sends an HTTP CONNECT request to establish the tunnel, then it sends the TCP stream through the tunnel, as shown in the following figure.

Interested readers can practice their PipyJS skills by trying to implement this and let us know by posting comments.

Summary

We used about 10 lines of code to add tunneling capabilities to an HTTP proxy. This article briefly introduced the implementation of tunneling, but the function of tunneling is not just to solve connectivity problems. High-level HTTP tunnels can also provide features that lower-level protocols cannot, such as various security authentications to enhance security.

The egress gateway in the service mesh osm-edge provides HTTP tunneling functionality, using an HTTP tunnel to provide governance functions at the high-level protocol between the sidecar and the egress gateway. We will explain this in more detail when we introduce the egress gateway.

Writing a DNS proxy with Pipy

Ali Naqvi — Mon, 19 Dec 2022 13:22:11 +0000

Pipy is a programmable proxy that has previously been used for TCP/HTTP proxy, MQTT proxy, Dubbo proxy, Redis proxy, and Thrift proxy. Recently we were asked if Pipy can be used to program a DNS proxy? and our answer was super simple yes, and via this post we will demonstrate how Pipy can help you build DNS proxy with simple scripting knowledge.

By reading this article, you will learn:

Basic introduction to DNS and the process of DNS handling
Implementation of a DNS proxy using coding
Adding intelligent route resolution functionality to the proxy

DNS Introduction

The Domain Name System (DNS) is a hierarchical and distributed naming system for computers, services, and other resources in the Internet or other Internet Protocol (IP) networks. It associates various information with domain names assigned to each of the associated entities. Most prominently, it translates readily memorized domain names to the numerical IP addresses needed for locating and identifying computer services and devices with the underlying network protocols.[1] The Domain Name System has been an essential component of the functionality of the Internet since 1985.

-- excerpt from Wikipedia

Simplified version of the DNS processing flow:

A DNS client (such as a browser, application or device) sends a query request for the domain name example.com.
The DNS resolver receives the request, queries the local cache, and returns the local record if it is available and has not expired.
If the local cache is not hit, the DNS resolver will start at the DNS root server and work its way down, starting with the Top Level Domain (TLD) DNS server (in this case .com) and working its way down to the server that can resolve example.com.
The server that can resolve example.com becomes the Authoritative DNS name server, which the resolver accesses and receives the IP address and other related information, and then returns it to the client. Resolution is completed.

During past or recently you might have encounter the need to change DNS records to update the domain name's true direction, such as switching running environments, traffic interception, and DNS is also often used as one of the means of service discovery. Usually, the DNS server is either maintained by the service provider or the company's internal network team, which makes it inconvenient to modify DNS resolution records. Moreover, due to the cache design of DNS, each record has a TTL setting, which will not be updated again before the cache expires. Both a long and a short TTL are not appropriate.

The introduction of DNS proxies can solve this problem while enabling more functionality.

Next, we will demonstrate how to use Pipy to implement a DNS proxy (more accurately, a combination of a proxy and a server) that returns DNS query requests from custom records. At the same time, we will also add a feature to return different DNS records based on the client IP address to achieve intelligent route resolution. The scripts used in the demonstration can be downloaded from here.

Solution

As shown in the above figure, the DNS proxy provides similar functionality to the original resolver. However, when the cache expires or fails to hit, it will query custom resolution records. If there is a custom record, it will return the custom record; if not, it will query the DNS server as per the original process.

Implementation

Before starting, let's use the wireshark network packet capture to look at the format of DNS messages. The format of DNS query and response messages is the same and includes the following four parts:

Header: Contains the ID, flag, number of query entries, number of response entries, number of authority resource entries, and number of additional resource entries.
Flag part: This part identifies the message type, whether the name server is authoritative, whether the query is recursive, whether the request has been truncated, and the status.
Request part: Contains the domain name being/needing to be resolved and the record type (A, AAAA, MX, TXT, etc.). Each label in the domain name is prefixed with its length.
Response part: Contains resource records for the queried domain name.

In previous announcement Pipy 0.70.0 is released!, it was highlighted that Pipy 0.70.0 comes bundled with DNS encoder/decoder. And we will be using Pipy to decode the DNS packet into above mentioned parts.

PipyJS Code

The script logic is very simple. For ease of reading, it is divided into several modules according to function, and implements the parsing of several common types of records, including A, AAAA, CNAME, MX, TXT, and NS.

├── cache.js #Cache
├── main.js #main entry script
├── records.js #Logic for customizing records
├── records.json #Custom record content
├── smart-line.js #Logic for smart line parsing
└── smart-line.json #Configuration of smart-line parsing

Here is a part of the core code of main.js with annotations:

First, use DNS.decode() to decode the data stream
Then find the queried domain name and type from the result
Query the cache
If the cache fails to hit, query custom records
Intelligent route resolution
Return the response
If 3 and 4 are both unsuccessful, it will request the upstream DNS server and then cache and return the response.

.listen(5300, { protocol: 'udp' })
.replaceMessage(
  msg => (
    (query, res, record) => (
      query = DNS.decode(msg.body), //1
      query?.question?.[0]?.name && query?.question?.[0]?.type && ( //2
        record = getDNS(query.question[0].type + '#' + query.question[0].name) //3
        || local.query(query.question[0].name, query.question[0].type) //4
      ),
      record ? (
        record = line.filter(__inbound.remoteAddress, record), //5
        res = {},
        res.qr = res.rd = res.ra = res.aa = 1,
        res.id = query.id,
        res.question = [{
          'name': query.question[0].name,
          'type': query.question[0].type
        }],
        record.status === 'deny' ? (
          res.rcode = local.code.REFUSED
        ) : (
          res.answer = record.rr
        ),
        new Message(DNS.encode(res)) //6
      ) : (
        _forward = true,
        msg
      )
    )
  )()
)
.branch(
  () => _forward, $ => $
    .connect(() => `${config.upstreamDNSServer}:53`, { protocol: 'udp' }) //7
    .handleMessage(
      msg => (
        (res = DNS.decode(msg.body)) => (
          res?.question?.[0]?.name && res?.question?.[0]?.type &&
          !res?.rcode && (
            setDNS(res.question[0].type + '#' + res.question[0].name,
              {
                rr: res.answer,
                status: res.rcode == local.code.REFUSED ? 'deny' : null
              }
            )
          )
        )
      )()
    ),
  $ => $
)

Custom Record

Below is the content of the custom record, which is similar to the format of a DNS response. In order to support intelligent route resolution, some records have added label information: "labels": ["line1"].

[
  {
    "name": "example.com",
    "type": "A",
    "ttl": 60,
    "rdata": "192.168.139.10",
    "labels": ["line1"]
  },
  {
    "name": "example.com",
    "type": "A",
    "ttl": 60,
    "rdata": "192.168.139.11",
    "labels": ["line2"]
  },
  ...
  {
    "name": "example.com",
    "type": "MX",
    "ttl": 600,
    "rdata": {
      "preference": 10,
      "exchange": "mail2.example.com"
    }
  },
  {
    "name": "example.com",
    "type": "TXT",
    "ttl": 600,
    "rdata": "hi.pipy!"
  },
  {
    "name": "example.com",
    "type": "NS",
    "ttl": 600,
    "rdata": "ns1.example.com"
  },
  ...
]

Intelligent Route Resolution

The logic of intelligent route resolution is relatively simple. Set different route labels for different IP ranges, and return only records with the corresponding label in the response if the record has a label.

{
  "line1": [
    "192.168.1.110/32"
  ],
  "line2": [
    "127.0.0.1/32"
  ]
}

Testing

Start the proxy:

$ pipy main.js

As shown in the above configuration, 127.0.0.1 is the address of the local loopback network card, and 192.168.1.110 is the address of the local Ethernet network card, and the proxy listens on port 5300.

First, access the proxy using localhost, so that the client IP address obtained by the proxy is 127.0.0.1. When querying the records for example.com, it directly returns the record 192.168.139.11 corresponding to the route line2 of the local address.

$ dig @localhost -p 5300 a example.com

; <<>> DiG 9.10.6 <<>> @localhost -p 5300 a example.com
; (2 servers found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 25868
;; flags: qr aa rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 0

;; QUESTION SECTION:
;example.com.           IN  A

;; ANSWER SECTION:
example.com.        60  IN  A   192.168.139.11

;; Query time: 0 msec
;; SERVER: 127.0.0.1#5300(127.0.0.1)
;; WHEN: Tue Dec 13 21:09:38 CST 2022
;; MSG SIZE  rcvd: 56

Then access the proxy using 192.168.1.110, this time the client's address is 192.168.1.110, and the record returned is 192.168.139.10 of route line1.

$ dig @192.168.1.110 -p 5300 a example.com

; <<>> DiG 9.10.6 <<>> @192.168.1.110 -p 5300 a example.com
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 54165
;; flags: qr aa rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 0

;; QUESTION SECTION:
;example.com.           IN  A

;; ANSWER SECTION:
example.com.        60  IN  A   192.168.139.10

;; Query time: 0 msec
;; SERVER: 192.168.1.110#5300(192.168.1.110)
;; WHEN: Tue Dec 13 21:12:37 CST 2022
;; MSG SIZE  rcvd: 56

If I access from another machine, because no route is set, it will return two records.

$ dig @192.168.1.110 -p 5300 a example.com

; <<>> DiG 9.16.1-Ubuntu <<>> @192.168.1.110 -p 5300 a example.com
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 64873
;; flags: qr aa rd ra; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 0

;; QUESTION SECTION:
;example.com.           IN  A

;; ANSWER SECTION:
example.com.        60  IN  A   192.168.139.10
example.com.        60  IN  A   192.168.139.11

;; Query time: 0 msec
;; SERVER: 192.168.1.110#5300(192.168.1.110)
;; WHEN: Tue Dec 13 13:15:24 UTC 2022
;; MSG SIZE  rcvd: 83

Because only the route of the A record is set, other types of records are not affected.

$ $dig @localhost -p 5300 mx example.com

; <<>> DiG 9.10.6 <<>> @localhost -p 5300 mx example.com
; (2 servers found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 33492
;; flags: qr aa rd ra; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 0

;; QUESTION SECTION:
;example.com.           IN  MX

;; ANSWER SECTION:
example.com.        600 IN  MX  10 mail1.example.com.
example.com.        600 IN  MX  10 mail2.example.com.

;; Query time: 0 msec
;; SERVER: 127.0.0.1#5300(127.0.0.1)
;; WHEN: Tue Dec 13 21:18:27 CST 2022
;; MSG SIZE  rcvd: 117

Going Further

Those who have some understanding of Pipy may know the Repo mode, and those who are interested can refer to previously published articles on Pipy Blog and Pipy Reference Documentation. Using the Repo mode, all proxies (or DNS servers) on the host real-timely obtain updates of custom records from the Repo and refresh the cache.

Due to space constraints, it will not be discussed in depth here. Those interested can give it a try to implement and learn Pipy.

Conclusion

With this, Pipy has added another type of proxy. DNS is everywhere, and it is precisely because of this that problems can be solved at the DNS level.

If you have been following us and might have gone through previous Multi-cluster related blog posts, and in Kubernetes: Multi-cluster communication with Flomesh Service Mesh (Demo) we demonstrated cross cluster service communication, and there we easily scheduled request traffic to other clusters for processing. In this demo we demonstrated how to access service located on another Kubernetes cluster, but to caller it was transparently forwarded to respective cluster, how was that done?

Here, a small trick was used. When setting iptables rules to intercept traffic in the initialization container of the mesh, DNS traffic was also intercepted by the DNS proxy implemented by the sidecar (listening on 127.0.0.153:5300), and business traffic was intercepted through custom DNS records.

Kubernetes: Cross-cluster traffic scheduling - Access control

Ali Naqvi — Sun, 11 Dec 2022 13:14:40 +0000

Flomesh open source service mesh osm-edge is based on an implementation of the SMI (Service Mesh Interface) standard. SMI defines specifications for traffic identification, access control, telemetry, and management. In the previous article Kubernetes: Multi-cluster communication with Flomesh Service Mesh we covered the background, motivations, and goals of Kubernetes multi-cluster and part 2 Kubernetes: Multi-cluster communication with Flomesh Service Mesh (Demo) we demonstrated a detailed demo of how to use FSM in a multi-cluster environment and how to schedule policies for traffics.

This article will expand on the knowledge we covered before and demonstrate how to configure and enable cross-cluster access control based on SMI. With osm-edge support for multi-clusters, users can define and enforce fine-grained access policies for services running across multiple Kubernetes clusters. This allows users to easily and securely manage access to services and resources, ensuring that only authorized users and applications have access to the appropriate services and resources.

Before we start, let's review the SMI Access Control Specification. There are two forms of traffic policies in osm-edge: Permissive Mode and Traffic Policy Mode. The former allows services in the mesh to access each other, while the latter requires the provision of the appropriate traffic policy to be accessible.

SMI Access Control Policy

In traffic policy mode, SMI defines ServiceAccount-based access control through the Kubernetes Custom Resource Definition(CRD) TrafficTarget, which defines traffic sources (sources), destinations (destinations), and rules (rules). What is expressed is that applications that use the ServiceAccount specified in sources can access applications that have the ServiceAccount specified in destinations, and the accessible traffic is specified by rules.

For example, the following example represents a load running with ServiceAccount promethues sending a GET request to the /metrics endpoint of a load running with ServiceAccount service-a. The HTTPRouteGroup defines the identity of the traffic: i.e. the GET request to access the endpoint /metrics.

kind: HTTPRouteGroup
metadata:
  name: the-routes
spec:
  matches:
  - name: metrics
    pathRegex: "/metrics"
    methods:
    - GET
---
kind: TrafficTarget
metadata:
  name: path-specific
  namespace: default
spec:
  destination:
    kind: ServiceAccount
    name: service-a
    namespace: default
  rules:
  - kind: HTTPRouteGroup
    name: the-routes
    matches:
    - metrics
  sources:
  - kind: ServiceAccount
    name: prometheus
    namespace: default

So how does access control perform in a multi-cluster?

FSM's `ServiceExport`

FSM's ServiceExport is used to export services to other clusters, which is the process of service registration. The field spec.serviceAccountName of ServiceExport can be used to specify the ServiceAccount used for the service load.

apiVersion: flomesh.io/v1alpha1
kind: ServiceExport
metadata:
  namespace: httpbin
  name: httpbin
spec:
  serviceAccountName: "*"
  rules:
    - portNumber: 8080
      path: "/cluster-1/httpbin-mesh"
      pathType: Prefix

Next, we start the demonstration based on the last environment. Those who haven't read the previous article can refer to the previous article to set up the demo environment.

Deploy the application

Deploy the sample application

Deploy the httpbin application under the httpbin namespace (managed by the mesh, which injects sidecar) in clusters cluster-1 and cluster-3. Here we specify ServiceAccount as httpbin.

export NAMESPACE=httpbin
for CLUSTER_NAME in cluster-1 cluster-3
do
  kubectx k3d-${CLUSTER_NAME}
  kubectl create namespace ${NAMESPACE}
  osm namespace add ${NAMESPACE}
  kubectl apply -n ${NAMESPACE} -f - <<EOF
apiVersion: v1
kind: ServiceAccount
metadata:
  name: httpbin
---  
apiVersion: apps/v1
kind: Deployment
metadata:
  name: httpbin
  labels:
    app: pipy
spec:
  replicas: 1
  selector:
    matchLabels:
      app: pipy
  template:
    metadata:
      labels:
        app: pipy
    spec:
      serviceAccountName: httpbin
      containers:
        - name: pipy
          image: flomesh/pipy:latest
          ports:
            - containerPort: 8080
          command:
            - pipy
            - -e
            - |
              pipy()
              .listen(8080)
              .serveHTTP(new Message('Hi, I am from ${CLUSTER_NAME} and controlled by mesh!\n'))
---
apiVersion: v1
kind: Service
metadata:
  name: httpbin
spec:
  ports:
    - port: 8080
      targetPort: 8080
      protocol: TCP
  selector:
    app: pipy
---
apiVersion: v1
kind: Service
metadata:
  name: httpbin-${CLUSTER_NAME}
spec:
  ports:
    - port: 8080
      targetPort: 8080
      protocol: TCP
  selector:
    app: pipy
EOF

  sleep 3
  kubectl wait --for=condition=ready pod -n ${NAMESPACE} --all --timeout=60s
done

Deploy the httpbin application under the cluster-2 namespace httpbin, but do not specify a ServiceAccount and use the default ServiceAccount default.

export NAMESPACE=httpbin
export CLUSTER_NAME=cluster-2

kubectx k3d-${CLUSTER_NAME}
kubectl create namespace ${NAMESPACE}
osm namespace add ${NAMESPACE}
kubectl apply -n ${NAMESPACE} -f - <<EOF
apiVersion: apps/v1
kind: Deployment
metadata:
  name: httpbin
  labels:
    app: pipy
spec:
  replicas: 1
  selector:
    matchLabels:
      app: pipy
  template:
    metadata:
      labels:
        app: pipy
    spec:
      containers:
        - name: pipy
          image: flomesh/pipy:latest
          ports:
            - containerPort: 8080
          command:
            - pipy
            - -e
            - |
              pipy()
              .listen(8080)
              .serveHTTP(new Message('Hi, I am from ${CLUSTER_NAME}! and controlled by mesh!\n'))
---
apiVersion: v1
kind: Service
metadata:
  name: httpbin
spec:
  ports:
    - port: 8080
      targetPort: 8080
      protocol: TCP
  selector:
    app: pipy
---
apiVersion: v1
kind: Service
metadata:
  name: httpbin-${CLUSTER_NAME}
spec:
  ports:
    - port: 8080
      targetPort: 8080
      protocol: TCP
  selector:
    app: pipy
EOF

sleep 3
kubectl wait --for=condition=ready pod -n ${NAMESPACE} --all --timeout=60s

Deploy the curl application under the namespace curl in cluster cluster-2, which is managed by the mesh, and the injected sidecar will be fully traffic dispatched across the cluster. Specify here to use ServiceAccout curl.

export NAMESPACE=curl
kubectx k3d-cluster-2
kubectl create namespace ${NAMESPACE}
osm namespace add ${NAMESPACE}
kubectl apply -n ${NAMESPACE} -f - <<EOF
apiVersion: v1
kind: ServiceAccount
metadata:
  name: curl
---
apiVersion: v1
kind: Service
metadata:
  name: curl
  labels:
    app: curl
    service: curl
spec:
  ports:
    - name: http
      port: 80
  selector:
    app: curl
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: curl
spec:
  replicas: 1
  selector:
    matchLabels:
      app: curl
  template:
    metadata:
      labels:
        app: curl
    spec:
      serviceAccountName: curl
      containers:
      - image: curlimages/curl
        imagePullPolicy: IfNotPresent
        name: curl
        command: ["sleep", "365d"]
EOF

sleep 3
kubectl wait --for=condition=ready pod -n ${NAMESPACE} --all --timeout=60s

Export service

export NAMESPACE_MESH=httpbin
for CLUSTER_NAME in cluster-1 cluster-3
do
  kubectx k3d-${CLUSTER_NAME}
  kubectl apply -f - <<EOF
apiVersion: flomesh.io/v1alpha1
kind: ServiceExport
metadata:
  namespace: ${NAMESPACE_MESH}
  name: httpbin
spec:
  serviceAccountName: "httpbin"
  rules:
    - portNumber: 8080
      path: "/${CLUSTER_NAME}/httpbin-mesh"
      pathType: Prefix
---
apiVersion: flomesh.io/v1alpha1
kind: ServiceExport
metadata:
  namespace: ${NAMESPACE_MESH}
  name: httpbin-${CLUSTER_NAME}
spec:
  serviceAccountName: "httpbin"
  rules:
    - portNumber: 8080
      path: "/${CLUSTER_NAME}/httpbin-mesh-${CLUSTER_NAME}"
      pathType: Prefix
EOF
sleep 1
done

Test

We switch back to the cluster cluster-2.

kubectx k3d-cluster-2

The default route type is Locality and we need to create an ActiveActive policy to allow requests to be processed using service instances from other clusters.

kubectl apply -n httpbin -f  - <<EOF
apiVersion: flomesh.io/v1alpha1
kind: GlobalTrafficPolicy
metadata:
  name: httpbin
spec:
  lbType: ActiveActive
  targets:
    - clusterKey: default/default/default/cluster-1
    - clusterKey: default/default/default/cluster-3
EOF

In the curl application of the cluster-2 cluster, we send a request to httpbin.httpbin.

curl_client="$(kubectl get pod -n curl -l app=curl -o jsonpath='{.items[0].metadata.name}')"

kubectl exec "${curl_client}" -n curl -c curl -- curl -s http://httpbin.httpbin:8080/

A few more requests will see the following response.

Hi, I am from cluster-1 and controlled by mesh!
Hi, I am from cluster-2 and controlled by mesh!
Hi, I am from cluster-3 and controlled by mesh!
Hi, I am from cluster-1 and controlled by mesh!
Hi, I am from cluster-2 and controlled by mesh!
Hi, I am from cluster-3 and controlled by mesh!

Demo

Adjusting the traffic policy mode

Let's adjust the traffic policy mode of cluster cluster-2 so that the traffic policy can be applied.

kubectx k3d-cluster-2
export osm_namespace=osm-system
kubectl patch meshconfig osm-mesh-config -n "$osm_namespace" -p '{"spec":{"traffic":{"enablePermissiveTrafficPolicyMode":false}}}' --type=merge

In this case, if you try to send the request again, you will find that the request fails. This is because in traffic policy mode, inter-application access is prohibited if no policy is configured.

kubectl exec "${curl_client}" -n curl -c curl -- curl -s http://httpbin.httpbin:8080/
command terminated with exit code 52

Application Access Control Policy

The access control policy of SMI is based on ServiceAccount as mentioned at the beginning of the article, that's why our deployed httpbin service uses different ServiceAccount in cluster cluster-1, cluster-3, and cluster cluster-2.

cluster-1：httpbin
cluster-2：default
cluster-3：httpbin

Next, we will set different access control policies TrafficTarget for in-cluster and out-of-cluster services, differentiated by the ServiceAccount of the target load in TrafficTarget.

Execute the following command to create a traffic policy curl-to-httpbin that allows curl to access loads under the namespace httpbin that uses ServiceAccount default.

kubectl apply -n httpbin -f - <<EOF
apiVersion: specs.smi-spec.io/v1alpha4
kind: HTTPRouteGroup
metadata:
  name: httpbin-route
spec:
  matches:
  - name: all
    pathRegex: "/"
    methods:
    - GET
---
kind: TrafficTarget
apiVersion: access.smi-spec.io/v1alpha3
metadata:
  name: curl-to-httpbin
spec:
  destination:
    kind: ServiceAccount
    name: default
    namespace: httpbin
  rules:
  - kind: HTTPRouteGroup
    name: httpbin-route
    matches:
    - all
  sources:
  - kind: ServiceAccount
    name: curl
    namespace: curl
EOF

Multiple request attempts are sent and the service of cluster cluster-2 responds, while clusters cluster-1 and cluster-3 will not participate in the service.

Hi, I am from cluster-2 and controlled by mesh!
Hi, I am from cluster-2 and controlled by mesh!
Hi, I am from cluster-2 and controlled by mesh!

Execute the following command to check ServiceImports and you can see that cluster-1 and cluster-3 export services using ServiceAccount httpbin.

kubectl get serviceimports httpbin -n httpbin -o jsonpath='{.spec}' | jq

{
  "ports": [
    {
      "endpoints": [
        {
          "clusterKey": "default/default/default/cluster-1",
          "target": {
            "host": "192.168.1.110",
            "ip": "192.168.1.110",
            "path": "/cluster-1/httpbin-mesh",
            "port": 81
          }
        },
        {
          "clusterKey": "default/default/default/cluster-3",
          "target": {
            "host": "192.168.1.110",
            "ip": "192.168.1.110",
            "path": "/cluster-3/httpbin-mesh",
            "port": 83
          }
        }
      ],
      "port": 8080,
      "protocol": "TCP"
    }
  ],
  "serviceAccountName": "httpbin",
  "type": "ClusterSetIP"
}

So, we create another TrafficTarget curl-to-ext-httpbin that allows curl to access the load using ServiceAccount httpbin.

kubectl apply -n httpbin -f - <<EOF
kind: TrafficTarget
apiVersion: access.smi-spec.io/v1alpha3
metadata:
  name: curl-to-ext-httpbin
spec:
  destination:
    kind: ServiceAccount
    name: httpbin
    namespace: httpbin
  rules:
  - kind: HTTPRouteGroup
    name: httpbin-route
    matches:
    - all
  sources:
  - kind: ServiceAccount
    name: curl
    namespace: curl
EOF

After applying the policy, test it again and all requests are successful.

Hi, I am from cluster-2 and controlled by mesh!
Hi, I am from cluster-1 and controlled by mesh!
Hi, I am from cluster-3 and controlled by mesh!

Summary

Kubernetes multi-cluster service access control is an important feature for users who want to easily and securely manage access to services and resources across multiple Kubernetes clusters. By leveraging this feature, users can easily and securely scale their applications and services across multiple clusters, without having to worry about managing access control policies manually.

The goal of the MCS (Multi-cluster Service) API implementation is to be able to use services from other clusters as if they were services running inside the same cluster. While there is still a long way to go in the Kubernetes implementation itself, the SMI standard implementation of Service Mesh can be treated as a Service.

As with access control in this article, SMI traffic splitting can also support multi-cluster services. In the following article, we will introduce traffic splitting for multicluster services.

Kubernetes: Multi-cluster communication with Flomesh Service Mesh (Part 2)

Ali Naqvi — Fri, 02 Dec 2022 12:43:31 +0000

In Part 1 we covered the motives, goals, and architecture of Flomesh Service Mesh and in this blog post we are going to demonstrate how to use FSM and lightweight SMI-compatible Service Mesh osm-edge to achieve multi-cluster service discovery & communication.

Kubernetes: Multi-cluster communication with Flomesh Service Mesh

Ali Naqvi ・ Nov 28 '22

#kubernetes #multicluster #flomesh

Demo Architecture

For demonstration purposes we will be creating 4 Kubernetes clusters and high-level architecture will look something like the below:

As a convention and for this demo we will be creating a separate stand-alone cluster to serve as a control plane cluster, but that isn't strictly required as a separate cluster and it could be one of any existing cluster.

Demo Pre-requisites

kubectx: for switching between multiple kubeconfig contexts (clusters)
k3d: for creating multiple k3s clusters locally using containers
helm: for deploying FSM
docker: required to run k3d

Demo clusters & environment setup

In this demo, we will be using k3d a lightweight wrapper to run k3s (Rancher Lab’s minimal Kubernetes distribution) in docker, to create 4 separate clusters named control-plane, cluster-1, cluster-2, and cluster-3 respectively.

We will be using the HOST machine IP address and separate ports during the installation, for us to easily access the individual clusters. My demo host machine IP address is 192.168.1.110 (it might be different for your machine).

cluster	cluster ip	api-server port	LB external-port	description
control-plane	HOST_IP(192.168.1.110)	6444	N/A	control-plane cluster
cluster-1	HOST_IP(192.168.1.110)	6445	81	application-cluster
cluster-2	HOST_IP(192.168.1.110)	6446	82	Application Cluster
cluster-3	HOST_IP(192.168.1.110)	6447	83	Application Cluster

Network

Creates a docker bridge type network named multi-clusters, which run all containers.

docker network create multi-clusters

Find your machine host IP address, mine is 192.168.1.110, and export that as an environment variable to be used later.

export HOST_IP=192.168.1.110

Cluster creation

We are going to use k3d to create 4 clusters.

API_PORT=6444 #6444 6445 6446 6447
PORT=80 #81 82 83
for CLUSTER_NAME in control-plane cluster-1 cluster-2 cluster-3
do
  k3d cluster create ${CLUSTER_NAME} \
    --image docker.io/rancher/k3s:v1.23.8-k3s2 \
    --api-port "${HOST_IP}:${API_PORT}" \
    --port "${API_PORT}:6443@server:0" \
    --port "${PORT}:80@server:0" \
    --servers-memory 4g \
    --k3s-arg "--disable=traefik@server:0" \
    --network multi-clusters \
    --timeout 120s \
    --wait
    ((API_PORT=API_PORT+1))
    ((PORT=PORT+1))
done

Install FSM

Install FSM to newly created 4 clusters.

helm repo update
export FSM_NAMESPACE=flomesh
export FSM_VERSION=0.2.0-alpha.9
for CLUSTER_NAME in control-plane cluster-1 cluster-2 cluster-3
do 
  kubectx k3d-${CLUSTER_NAME}
  sleep 1
  helm install --namespace ${FSM_NAMESPACE} --create-namespace --version=${FSM_VERSION} --set fsm.logLevel=5 fsm fsm/fsm
  sleep 1
  kubectl wait --for=condition=ready pod --all -n $FSM_NAMESPACE
done

We have our clusters ready, now we need to federate them together, but before we do that, let's first understand the mechanics on how FSM is configured.

In Part 1 of this series, we stated that FSM provides a set of Kubernetes custom resources (CRD) for cluster connectors, and makes use of KEP-1645 ServiceExport and ServiceImport API for exporting and importing services. So let's take a quick look at them

`Cluster` CRD

When registering a cluster, we provide the following information.

The address (e.g. gatewayHost: cluster-A.host) and port (e.g. gatewayPort: 80) of the cluster
kubeconfig to access the cluster, containing the api-server address and information such as the certificate and secret key

apiVersion: flomesh.io/v1alpha1
kind: Cluster
metadata:
  name: cluster-A
spec:
  gatewayHost: cluster-A.host
  gatewayPort: 80
  kubeconfig: |+
    ---
    apiVersion: v1
    clusters:
    - cluster:
        certificate-authority-data:
        server: https://cluster-A.host:6443
      name: cluster-A
    contexts:
    - context:
        cluster: cluster-A
        user: admin@cluster-A
      name: cluster-A
    current-context: cluster-A
    kind: Config
    preferences: {}
    users:
    - name: admin@cluster-A
      user:
        client-certificate-data:
        client-key-data:

`ServiceExport` and `ServiceImport` CRD

For cross-cluster service registration, FSM provides the ServiceExport and ServiceImport CRDs from KEP-1645: Multi-Cluster Services API for ServiceExports.flomesh.io and ServiceImports.flomesh.io. The former is used to register services with the control plane and declare that the application can provide services across clusters, while the latter is used to reference services from other clusters.

For clusters cluster-A and cluster-B that join the cluster federation, a Service named foo exists under the namespace bar of cluster cluster-A and a ServiceExport foo of the same name is created under the same namespace. A ServiceImport resource with the same name is automatically created under the namespace bar of cluster cluster-B (if it does not exist, it is automatically created).

// in cluster-A
apiVersion: v1
kind: Service
metadata:
  name: foo
  namespace: bar
spec:
  ports:
  - port: 80
  selector:
    app: foo
---
apiVersion: flomesh.io/v1alpha1
kind: ServiceExport
metadata:
  name: foo
  namespace: bar
---
// in cluster-B
apiVersion: flomesh.io/v1alpha1
kind: ServiceImport
metadata:
  name: foo
  namespace: bar

The YAML snippet above shows how to register the foo service to the control plane of a multi-cluster. In the following, we will walk through a slightly more complex scenario of cross-cluster service registration and traffic scheduling.

Okay that was a quick introduction to the CRDs, so let's continue with our demo.

Federate clusters

We will enroll clusters cluster-1, cluster-2, and cluster-3 into the management of control-plane cluster.

export HOST_IP=192.168.1.110
kubectx k3d-control-plane
sleep 1
PORT=81
for CLUSTER_NAME in cluster-1 cluster-2 cluster-3
do
  cat <<EOF
apiVersion: flomesh.io/v1alpha1
kind: Cluster
metadata:
  name: ${CLUSTER_NAME}
spec:
  gatewayHost: ${HOST_IP}
  gatewayPort: ${PORT}
  kubeconfig: |+
`k3d kubeconfig get ${CLUSTER_NAME} | sed 's|^|    |g' | sed "s|0.0.0.0|$HOST_IP|g"`
EOF
((PORT=PORT+1))
done

Install osm-edge Service Mesh

Download osm CLI

Install the service mesh osm-edge to the clusters cluster-1, cluster-2, and cluster-3. The control plane does not handle application traffic and does not need to be installed.

export OSM_NAMESPACE=osm-system
export OSM_MESH_NAME=osm
for CLUSTER_NAME in cluster-1 cluster-2 cluster-3
do
  kubectx k3d-${CLUSTER_NAME}
  DNS_SVC_IP="$(kubectl get svc -n kube-system -l k8s-app=kube-dns -o jsonpath='{.items[0].spec.clusterIP}')"
osm install \
    --mesh-name "$OSM_MESH_NAME" \
    --osm-namespace "$OSM_NAMESPACE" \
    --set=osm.certificateProvider.kind=tresor \
    --set=osm.image.pullPolicy=Always \
    --set=osm.sidecarLogLevel=error \
    --set=osm.controllerLogLevel=warn \
    --timeout=900s \
    --set=osm.localDNSProxy.enable=true \
    --set=osm.localDNSProxy.primaryUpstreamDNSServerIPAddr="${DNS_SVC_IP}"
done

Deploy Demo application

Deploying mesh-managed applications

Deploy the httpbin application under the httpbin namespace of clusters cluster-1 and cluster-3 (which are managed by the mesh and will inject sidecar). Here the httpbin application is implemented by Pipy and will return the current cluster name.

export NAMESPACE=httpbin
for CLUSTER_NAME in cluster-1 cluster-3
do
  kubectx k3d-${CLUSTER_NAME}
  kubectl create namespace ${NAMESPACE}
  osm namespace add ${NAMESPACE}
  kubectl apply -n ${NAMESPACE} -f - <<EOF
apiVersion: apps/v1
kind: Deployment
metadata:
  name: httpbin
  labels:
    app: pipy
spec:
  replicas: 1
  selector:
    matchLabels:
      app: pipy
  template:
    metadata:
      labels:
        app: pipy
    spec:
      containers:
        - name: pipy
          image: flomesh/pipy:latest
          ports:
            - containerPort: 8080
          command:
            - pipy
            - -e
            - |
              pipy()
              .listen(8080)
              .serveHTTP(new Message('Hi, I am from ${CLUSTER_NAME} and controlled by mesh!\n'))
---
apiVersion: v1
kind: Service
metadata:
  name: httpbin
spec:
  ports:
    - port: 8080
      targetPort: 8080
      protocol: TCP
  selector:
    app: pipy
---
apiVersion: v1
kind: Service
metadata:
  name: httpbin-${CLUSTER_NAME}
spec:
  ports:
    - port: 8080
      targetPort: 8080
      protocol: TCP
  selector:
    app: pipy
EOF

  sleep 3
  kubectl wait --for=condition=ready pod -n ${NAMESPACE} --all --timeout=60s
done

Deploy the curl application under the namespace curl in cluster cluster-2, which is managed by the mesh.

export NAMESPACE=curl
kubectx k3d-cluster-2
kubectl create namespace ${NAMESPACE}
osm namespace add ${NAMESPACE}
kubectl apply -n ${NAMESPACE} -f - <<EOF
apiVersion: v1
kind: ServiceAccount
metadata:
  name: curl
---
apiVersion: v1
kind: Service
metadata:
  name: curl
  labels:
    app: curl
    service: curl
spec:
  ports:
    - name: http
      port: 80
  selector:
    app: curl
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: curl
spec:
  replicas: 1
  selector:
    matchLabels:
      app: curl
  template:
    metadata:
      labels:
        app: curl
    spec:
      serviceAccountName: curl
      containers:
      - image: curlimages/curl
        imagePullPolicy: IfNotPresent
        name: curl
        command: ["sleep", "365d"]
EOF

sleep 3
kubectl wait --for=condition=ready pod -n ${NAMESPACE} --all --timeout=60s

Export Service

Let's export services in cluster-1 and cluster-3

export NAMESPACE_MESH=httpbin
for CLUSTER_NAME in cluster-1 cluster-3
do
  kubectx k3d-${CLUSTER_NAME}
  kubectl apply -f - <<EOF
apiVersion: flomesh.io/v1alpha1
kind: ServiceExport
metadata:
  namespace: ${NAMESPACE_MESH}
  name: httpbin
spec:
  serviceAccountName: "*"
  rules:
    - portNumber: 8080
      path: "/${CLUSTER_NAME}/httpbin-mesh"
      pathType: Prefix
---
apiVersion: flomesh.io/v1alpha1
kind: ServiceExport
metadata:
  namespace: ${NAMESPACE_MESH}
  name: httpbin-${CLUSTER_NAME}
spec:
  serviceAccountName: "*"
  rules:
    - portNumber: 8080
      path: "/${CLUSTER_NAME}/httpbin-mesh-${CLUSTER_NAME}"
      pathType: Prefix
EOF
sleep 1
done

After exporting the services, FSM will automatically create Ingress rules for them, and with the rules, you can access these services through Ingress.


for CLUSTER_NAME_INDEX in 1 3
do
  CLUSTER_NAME=cluster-${CLUSTER_NAME_INDEX}
  ((PORT=80+CLUSTER_NAME_INDEX))
  kubectx k3d-${CLUSTER_NAME}
  echo "Getting service exported in cluster ${CLUSTER_NAME}"
  echo '-----------------------------------'
  kubectl get serviceexports.flomesh.io -A
  echo '-----------------------------------'
  curl -s "http://${HOST_IP}:${PORT}/${CLUSTER_NAME}/httpbin-mesh"
  curl -s "http://${HOST_IP}:${PORT}/${CLUSTER_NAME}/httpbin-mesh-${CLUSTER_NAME}"
  echo '-----------------------------------'
done

To view one of the ServiceExports resources.

kubectl get serviceexports httpbin -n httpbin -o jsonpath='{.spec}' | jq

{
  "loadBalancer": "RoundRobinLoadBalancer",
  "rules": [
    {
      "path": "/cluster-3/httpbin-mesh",
      "pathType": "Prefix",
      "portNumber": 8080
    }
  ],
  "serviceAccountName": "*"
}

The exported services can be imported into other managed clusters. For example, if we look at the cluster cluster-2, we can have multiple services imported.

kubectx k3d-cluster-2
kubectl get serviceimports -A

NAMESPACE   NAME                AGE
httpbin     httpbin-cluster-1   13m
httpbin     httpbin-cluster-3   13m
httpbin     httpbin             13m

Testing

Staying in the cluster-2 cluster (kubectx k3d-cluster-2), we test if we can access these imported services from the curl application in the mesh.

Get the pod of the curl application, from which we will later launch requests to simulate service access.

curl_client="$(kubectl get pod -n curl -l app=curl -o jsonpath='{.items[0].metadata.name}')"

At this point you will find that it is not accessible.

kubectl exec "${curl_client}" -n curl -c curl -- curl -s http://httpbin.httpbin:8080/

command terminated with exit code 7

Note that this is normal, by default no other cluster instance will be used to respond to requests, which means no calls to other clusters will be made by default. So how to access it, then we need to be clear about the global traffic policy GlobalTrafficPolicy.

Global Traffic Policy

Note that all global traffic policies are set on the user's side, so this demo is about setting global traffic policies on the cluster cluster-2 side. So before you start, switch to cluster cluster-2: kubectx k3d-cluster-2.

The global traffic policy is set via CRD GlobalTrafficPolicy.

type GlobalTrafficPolicy struct {  
   metav1.TypeMeta   `json:",inline"`  
   metav1.ObjectMeta `json:"metadata,omitempty"`  

   Spec   GlobalTrafficPolicySpec   `json:"spec,omitempty"`  
   Status GlobalTrafficPolicyStatus `json:"status,omitempty"`  
}
type GlobalTrafficPolicySpec struct {  
   LbType LoadBalancerType `json:"lbType"`  
   LoadBalanceTarget []TrafficTarget `json:"targets"`  
}

Global load balancing types .spec.lbType There are three types.

Locality: uses only the services of this cluster, and is also the default type. This is why accessing the httpbin application fails when we don't provide any global policy, because there is no such service in cluster cluster-2.
FailOver: proxies to other clusters only when access to this cluster fails, which is often referred to as failover, similar to primary backup.
ActiveActive: Proxy to other clusters under normal conditions, similar to multi-live.

The FailOver and ActiveActive policies are used with the targets field to specify the id of the standby cluster, which is the cluster that can be routed to in case of failure or load balancing. ** For example, if you look at the import service httpbin/httpbin in cluster cluster-2, you can see that it has two endpoints for the outer cluster, note that endpoints here is a different concept than the native endpoints.v1 and will contain more information. In addition, there is the cluster id clusterKey.

kubectl get serviceimports httpbin -n httpbin -o jsonpath='{.spec}' | jq

{
  "ports": [
    {
      "endpoints": [
        {
          "clusterKey": "default/default/default/cluster-1",
          "target": {
            "host": "192.168.1.110",
            "ip": "192.168.1.110",
            "path": "/cluster-1/httpbin-mesh",
            "port": 81
          }
        },
        {
          "clusterKey": "default/default/default/cluster-3",
          "target": {
            "host": "192.168.1.110",
            "ip": "192.168.1.110",
            "path": "/cluster-3/httpbin-mesh",
            "port": 83
          }
        }
      ],
      "port": 8080,
      "protocol": "TCP"
    }
  ],
  "serviceAccountName": "*",
  "type": "ClusterSetIP"
}

Routing Type - `Locality`

The default routing type is Locality, and as tested above, traffic cannot be dispatched to other clusters.

kubectl exec "${curl_client}" -n curl -c curl -- curl -s http://httpbin.httpbin:8080/
command terminated with exit code 7

Routing Type - `FailOver`

Since setting a global traffic policy for causes access failure, we start by enabling FailOver mode. Note that the global policy traffic, to be consistent with the target service name and namespace. For example, if we want to access http://httpbin.httpbin:8080/, we need to create GlobalTrafficPolicy resource named httpbin under the namespace httpbin.

kubectl apply -n httpbin -f  - <<EOF
apiVersion: flomesh.io/v1alpha1
kind: GlobalTrafficPolicy
metadata:
  name: httpbin
spec:
  lbType: FailOver
  targets:
    - clusterKey: default/default/default/cluster-1
    - clusterKey: default/default/default/cluster-3
EOF

After setting the policy, let's try it again by requesting.

kubectl exec "${curl_client}" -n curl -c curl -- curl -s http://httpbin.httpbin:8080/
Hi, I am from cluster-1!

The request is successful and the request is proxied to the service in cluster cluster-1. Another request is made, and it is proxied to cluster cluster-3, as expected for load balancing.

kubectl exec "${curl_client}" -n curl -c curl -- curl -s http://httpbin.httpbin:8080/
Hi, I am from cluster-3!

What will happen if we deploy the application httpbin in the namespace httpbin of the cluster cluster-2?

export NAMESPACE=httpbin
export CLUSTER_NAME=cluster-2

kubectx k3d-${CLUSTER_NAME}
kubectl create namespace ${NAMESPACE}
osm namespace add ${NAMESPACE}
kubectl apply -n ${NAMESPACE} -f - <<EOF
apiVersion: apps/v1
kind: Deployment
metadata:
  name: httpbin
  labels:
    app: pipy
spec:
  replicas: 1
  selector:
    matchLabels:
      app: pipy
  template:
    metadata:
      labels:
        app: pipy
    spec:
      containers:
        - name: pipy
          image: flomesh/pipy:latest
          ports:
            - containerPort: 8080
          command:
            - pipy
            - -e
            - |
              pipy()
              .listen(8080)
              .serveHTTP(new Message('Hi, I am from ${CLUSTER_NAME}!\n'))
---
apiVersion: v1
kind: Service
metadata:
  name: httpbin
spec:
  ports:
    - port: 8080
      targetPort: 8080
      protocol: TCP
  selector:
    app: pipy
---
apiVersion: v1
kind: Service
metadata:
  name: httpbin-${CLUSTER_NAME}
spec:
  ports:
    - port: 8080
      targetPort: 8080
      protocol: TCP
  selector:
    app: pipy
EOF

sleep 3
kubectl wait --for=condition=ready pod -n ${NAMESPACE} --all --timeout=60s

After the application is running normally, this time we send the request to test again. From the results, it looks like the request is processed in the current cluster.

kubectl exec "${curl_client}" -n curl -c curl -- curl -s http://httpbin.httpbin:8080/ 
Hi, I am from cluster-2!

Even if the request is repeated multiple times, it will always return Hi, I am from cluster-2!, which indicates that the services of same cluster are used in preference to the services imported from other clusters.

In some cases, we also want other clusters to participate in the service as well, because the resources of other clusters are wasted if only the services of this cluster are used. This is where the ActiveActive routing type comes into play.

Routing Type - `ActiveActive`

Moving on from the status above, let's test the ActiveActive type by modifying the policy created earlier and updating it to ActiveActive:

kubectl apply -n httpbin -f  - <<EOF
apiVersion: flomesh.io/v1alpha1
kind: GlobalTrafficPolicy
metadata:
  name: httpbin
spec:
  lbType: ActiveActive
  targets:
    - clusterKey: default/default/default/cluster-1
    - clusterKey: default/default/default/cluster-3
EOF

Multiple requests will show that httpbin from all three clusters will participate in the service. This indicates that the load is being proxied to multiple clusters in a balanced manner.

kubectl exec "${curl_client}" -n curl -c curl -- curl -s http://httpbin.httpbin:8080/
Hi, I am from cluster-1 and controlled by mesh!
kubectl exec "${curl_client}" -n curl -c curl -- curl -s http://httpbin.httpbin:8080/
Hi, I am from cluster-2!
kubectl exec "${curl_client}" -n curl -c curl -- curl -s http://httpbin.httpbin:8080/
Hi, I am from cluster-3 and controlled by mesh!

Summary

This concludes our series of introductions to Flomesh Service Mesh, an open-source solution from Flomesh to help you achieve Kubernetes multi-cluster services discovery and communication.

In part 1 of this series, we briefly touched on the use cases for multi-cluster requirements and talked about the motivation and goals of FSM and its architecture. In this part of the series we demonstrated how to implement cross-cluster traffic scheduling and load balancing of services, and try three different global traffic policies: local cluster scheduling only, failover, and global load balancing.

If you are interested in learning more about SMI-compatible Service Mesh, FSM, and Programmable Proxy, visit our website Flomesh.io for more detailed documentation, tutorials, and use cases.

Kubernetes: Multi-cluster communication with Flomesh Service Mesh

Ali Naqvi — Mon, 28 Nov 2022 13:22:22 +0000

Kubernetes has been quite successful in popularizing the idea of container clusters. Deployments have reached a point where many users are running multiple clusters and struggling to keep them running smoothly. Organizations need to run multiple Kubernetes clusters might fall into one of the below reasons (not an exhaustive list):

Location
- Latency (run the application as close to customers as possible)
- Jurisdiction (e.g. required to keep user data in-country)
- Data gravity (e.g. data exists in one provider)
Isolation
- Environment (e.g. development, testing, staging, prod, etc)
- Performance isolation (teams don't want to feel each other)
- Security isolation (sensitive data or untrusted code)
- Organizational isolation (teams have different management domains)
- Cost isolation (teams want to get different bills)
Reliability
- Blast radius (an infra or app problem in one cluster doesn't kill the whole system)
- Infrastructure diversity (an underlying zone, region, or provider outages does not bring down the whole system)
- Scale (the app is too big to fit in a single cluster)
- Upgrade scope (upgrade infra for some parts of your app but not all of it; avoid the need for in-place cluster upgrades)

There is currently no standard way to connect or even think about Kubernetes services beyond the single cluster boundary, and Kubernetes Multicluster SIG has put together a proposal KEP-1645 to extend Kubernetes Service concepts across multiple clusters.

Flomesh team has been spending time tackling the challenge of multicluster communication, integrating north-south traffic management capabilities into osm-edge SMI compatible service mesh, and contributing back to the Open Source community.

In this part of the series, we will be looking into motivation, goals, architecture of FSM multi-cluster support, its components, and how it integrates with osm-edge full-fledged, lightweight, SMI-compatible service mesh suitable for cloud-native and edge.

Flomesh Service Mesh

FSM is an open-source product from Flomesh for Kubernetes north-south traffic, gateway API controller, and multi-cluster management. FSM uses Pipy, a programmable proxy at its core, and provides an Ingress controller, Gateway API controller, load balancer, cross-cluster service registration discovery, and more.

Motivation

During our consultancy and support to the community, commercial clients, and enterprises we have seen multiple requests and desires (a few of which are cited above) on why they want to split their deployments across multiple clusters while maintaining mutual dependencies between workloads operating in those clusters. Currently, the cluster is a hard boundary, and service is opaque to a distant K8s consumer who may otherwise use metadata (e.g. endpoint topology) to better direct traffic. Users may want to use services distributed across clusters to support failover or temporarily during migration, however, this needs non-trivial customized solutions today.

Flomesh team aims to help the community by providing solutions to these problems.

Goals

Define a minimal API to support service discovery and consumption across clusters.
- Consume a service in another cluster.
- Consume a service deployed in multiple clusters as a single service.
When a service is consumed from another cluster its behavior should be predictable and consistent with how it would be consumed within its cluster.
Allow gradual rollout of changes in a multi-cluster environment.
Provide a stand-alone implementation that can be used without any coupling to any product and/or solution.
Transparent integration with osm-edge service mesh, for users who want to have multi-cluster support with service mesh functionality.
Fully open source and welcomes the community to participate and contribute.

Architecture

Control plane

osm-edge integration (managed cluster)

FSM provides a set of Kubernetes custom resources (CRD) for cluster connector, and make use of KEP-1645 ServiceExport and ServiceImport API for exporting and importing services.

FSM multi-cluster support is already in the alpha stage and is available on FSM Github, you are welcome to give it a try.

In the upcoming part of this series, we will be exploring FSM multi-cluster functionality via hands-on tutorial and demo, so stay tuned.

Summary

Kubernetes is no longer constrained to a single boundary, and use cases of having multiple Kubernetes clusters are becoming quite common. There wasn't a typical or standard way of connecting numerous clusters and providing a transparent means to access services across boundaries. Kubernetes multi-cluster special interest group is working on proposals to provide a common set of APIs that can be implemented by vendors and provide a consistent way for k8s users to access services within or across boundaries.

Flomesh Service Mesh(FSM) from Flomesh is a Kubernetes North-South traffic manager, that provides Ingress controllers, Gateway API, Load Balancer, and cross-cluster service registration and service discovery. FSM uses Pipy - a programmable network proxy, as its data plane and is suitable for cloud, edge, and IoT.

osm-edge: Using access control policies to access services with the service mesh

Ali Naqvi — Tue, 08 Nov 2022 06:46:19 +0000

Deploying a service mesh in a complex brownfield environment is a lengthy and gradual process requiring upfront planning, or there may exist use cases where you have a specific set of services that either aren't yet ready for migration or for some reason can not be migrated to service mesh.

This blog post will talk about the approaches which can be used to enable services outside of the service mesh to communicate with services within the osm-edge service mesh.

osm-edge forked from Open Service Mesh is a lightweight, extensible, cloud-native, SMI-compatible service mesh built purposely for Edge computing. osm-edge uses lightweight programmable proxy Pipy as a sidecar proxy.

osm-edge offers two ways to allow accessing services within the service mesh:

via Ingress
- FSM Ingress controller
- Nginx Ingress controller
Access Control
- Service
- IPRange

The first method to access the services in the service mesh is via Ingress controller, and treat the services outside the mesh as the services inside the cluster. The advantage of this approach is that the setup is simple and straightforward and the disadvantages are also apparent, as you cannot achieve fine-grained access control, and all services outside the mesh can access services within the mesh.

This article will focus on the second approach, which allows support for fine-grained access control on who can access services within the service mesh. This feature is newly added and available in release 1.2.0 osm-edge v1.2.0.

Access Control can be configured via two resource types: Service and IP range. In terms of data transmission, it supports plaintext transmission and mTLS-encrypted traffic.

Let's get started with a demo.

Demo environment preparation

Kubernetes cluster

Using the minimalist Kubernetes distribution k8e:

curl -sfL https://getk8e.com/install.sh | K8E_TOKEN=k8e-mesh INSTALL_K8E_EXEC="server --cluster-init --write-kubeconfig-mode 644 --write-kubeconfig ~/.kube/config" sh -

osm-edge CLI

system=$(uname -s | tr [:upper:] [:lower:])
arch=$(dpkg --print-architecture)
release=v1.2.0
curl -L https://github.com/flomesh-io/osm-edge/releases/download/${release}/osm-edge-${release}-${system}-${arch}.tar.gz | tar -vxzf -
./${system}-${arch}/osm version
cp ./${system}-${arch}/osm /usr/local/bin/

Install osm-edge

Execute the following commands to install the related components of osm-edge.

export osm_namespace=osm-system
export osm_mesh_name=osm

osm install \
     --mesh-name "$osm_mesh_name" \
     --osm-namespace "$osm_namespace" \
     --set=osm.image.pullPolicy=Always

Check to make sure all pods are up and running properly.

Deploy the sample application

#Mock target service
kubectl create namespace httpbin
osm namespace add httpbin
kubectl apply -n httpbin -f https://raw.githubusercontent.com/flomesh-io/osm-edge-docs/main/manifests/samples/httpbin/httpbin.yaml

#Mock external service
kubectl create namespace curl
kubectl apply -n curl -f https://raw.githubusercontent.com/flomesh-io/osm-edge-docs/main/manifests/samples/curl/curl.yaml

#Wait for the dependent POD to start normally
kubectl wait --for=condition=ready pod -n httpbin -l app=httpbin --timeout=180s
kubectl wait --for=condition=ready pod -n curl -l app=curl --timeout=180s

Demo

At this point, we send a request from service curl to target service httpbin by executing the following command.

kubectl exec "$(kubectl get pod -n curl -l app=curl -o jsonpath='{.items..metadata.name}')" -n curl -- curl -sI http://httpbin.httpbin:14001/get
command terminated with exit code 56

The access fails because by default the services outside the mesh cannot access the services inside the mesh and we need to apply an access control policy.

Before applying the policy, you need to enable the access control feature, which is disabled by default.

kubectl patch meshconfig osm-mesh-config -n "$osm_namespace" -p '{"spec":{"featureFlags":{"enableAccessControlPolicy":true}}}' --type= merge

Plaintext transfer

Data can be transferred in plaintext or with two-way TLS encryption. Plaintext transfer is relatively simple, so let's demonstrate the plaintext transfer scenario first.

Service-based access control

First, create a Service for service curl.

kubectl apply -n curl -f - <<EOF
apiVersion: v1
kind: Service
metadata:
  name: curl
  labels:
    app: curl
    service: curl
spec:
  ports:
    - name: http
      port: 80
  selector:
    app: curl
EOF

Next, create an access control policy with the source Service curl and the target service httpbin.

kubectl apply -f - <<EOF
kind: AccessControl
apiVersion: policy.openservicemesh.io/v1alpha1
metadata:
  name: httpbin
  namespace: httpbin
spec:
  backends:
  - name: httpbin
    port:
      number: 14001 # targetPort of httpbin service
      protocol: http
  sources:
  - kind: Service
    namespace: curl
    name: curl
EOF

Execute the command again to send the authentication request, and you can see that this time an HTTP 200 response is received.

kubectl exec "$(kubectl get pod -n curl -l app=curl -o jsonpath='{.items..metadata.name}')" -n curl -- curl -sI http://httpbin.httpbin:14001/get
HTTP/1.1 200 OK
server: gunicorn/19.9.0
date: Mon, 07 Nov 2022 08:47:55 GMT
content-type: application/json
content-length: 267
access-control-allow-origin: *
access-control-allow-credentials: true
osm-stats-namespace: httpbin
osm-stats-kind: Deployment
osm-stats-name: httpbin
osm-stats-pod: httpbin-69dc7d545c-qphrh
connection: keep-alive

Before continuing with the rest of the demonstration, execute the command kubectl delete accesscontrol httpbin -n httpbin to delete the policy you just created.

IP range-based access control, plaintext transfer

First, get the pod IP address of the service curl.

curl_pod_ip="$(kubectl get pod -n curl -l app=curl -o jsonpath='{.items[0].status.podIP}')"

Access control using IP ranges is simple, just set the access source type to IPRange and configure the IP address you just obtained.

kubectl apply -f - <<EOF
kind: AccessControl
apiVersion: policy.openservicemesh.io/v1alpha1
metadata:
  name: httpbin
  namespace: httpbin
spec:
  backends:
  - name: httpbin
    port:
      number: 14001 # targetPort of httpbin service
      protocol: http
  sources:
  - kind: IPRange
    name: ${curl_pod_ip}/32
EOF

Execute the command again to test if the control policy is in effect.

kubectl exec "$(kubectl get pod -n curl -l app=curl -o jsonpath='{.items..metadata.name}')" -n curl -- curl -sI http://httpbin.httpbin:14001/get
HTTP/1.1 200 OK
server: gunicorn/19.9.0
date: Mon, 07 Nov 2022 09:20:57 GMT
content-type: application/json
content-length: 267
access-control-allow-origin: *
access-control-allow-credentials: true
osm-stats-namespace: httpbin
osm-stats-kind: Deployment
osm-stats-name: httpbin
osm-stats-pod: httpbin-69dc7d545c-qphrh
connection: keep-alive

Remember to execute kubectl delete accesscontrol httpbin -n httpbin to clean up the policy.

The previous ones we used were plaintext transfers, next we look at encrypted transfers.

Encrypted transfers

The default access policy certificate feature is off, turn it on by executing the following command.

kubectl patch meshconfig osm-mesh-config -n "$osm_namespace" -p '{"spec":{"featureFlags":{"enableAccessCertPolicy":true}}}' --type=merge

Create AccessCert for the access source to assign a certificate for data encryption. The controller will store the certificate information in Secret curl-mtls-secret under the namespace curl, and here also assign SAN curl.curl.cluster.local for the access source.

kubectl apply -f - <<EOF
kind: AccessCert
apiVersion: policy.openservicemesh.io/v1alpha1
metadata:
  name: curl-mtls-cert
  namespace: httpbin
spec:
  subjectAltNames:
  - curl.curl.cluster.local
  secret:
    name: curl-mtls-secret
    namespace: curl
EOF

Redeploy curl and mount the system-assigned Secret to the pod.

kubectl apply -n curl -f - <<EOF
apiVersion: apps/v1
kind: Deployment
metadata:
  name: curl
spec:
  replicas: 1
  selector:
    matchLabels:
      app: curl
  template:
    metadata:
      labels:
        app: curl
    spec:
      serviceAccountName: curl
      nodeSelector:
        kubernetes.io/os: linux
      containers:
      - image: curlimages/curl
        imagePullPolicy: IfNotPresent
        name: curl
        command: ["sleep", "365d"]
        volumeMounts:
        - name: curl-mtls-secret
          mountPath: "/certs"
          readOnly: true
      volumes:
        - name: curl-mtls-secret
          secret:
            secretName: curl-mtls-secret
EOF

Service-based Access Control

The next step is to create an access control policy that uses encrypted transport. When configuring the target service, enable client certificate checking by specifying tls.skipClientCertValidation = false. For the access source, in addition to the Service type of access source, the SAN curl.curl.cluster.local should be specified via AuthenticatedPrincipal.

kubectl apply -f - <<EOF
kind: AccessControl
apiVersion: policy.openservicemesh.io/v1alpha1
metadata:
  name: httpbin
  namespace: httpbin
spec:
  backends:
  - name: httpbin
    port:
      number: 14001 # targetPort of httpbin service
      protocol: http
    tls:
      skipClientCertValidation: false
  sources:
  - kind: Service
    namespace: curl
    name: curl
  - kind: AuthenticatedPrincipal
    name: curl.curl.cluster.local
EOF

Test whether the access policy is effective. When sending the request, we have to specify the CA certificate, key and certificate to be used for the curl command of the access source, which will be responded to by HTTP 200 normally.

kubectl exec "$(kubectl get pod -n curl -l app=curl -o jsonpath='{.items..metadata.name}')" -n curl -- curl -ksI https://httpbin.httpbin:14001/get --cacert /certs/ca.crt --key /certs/tls.key --cert /certs/tls.crt
HTTP/2 200
server: gunicorn/19.9.0
date: Mon, 07 Nov 2022 10:44:05 GMT
content-type: application/json
content-length: 267
access-control-allow-origin: *
access-control-allow-credentials: true
osm-stats-namespace: httpbin
osm-stats-kind: Deployment
osm-stats-name: httpbin
osm-stats-pod: httpbin-69dc7d545c-qphrh

IP Range-Based Access Control

After service-based access control, IP range-based control is simple. You just need to specify the type of access source as IPRange and specify the IP address of the access source. As the application curl is redeployed, its IP address needs to be retrieved (perhaps you have discovered the drawbacks of IP range-based access control).

curl_pod_ip="$(kubectl get pod -n curl -l app=curl -o jsonpath='{.items[0].status.podIP}')"

kubectl apply -f - <<EOF
kind: AccessControl
apiVersion: policy.openservicemesh.io/v1alpha1
metadata:
  name: httpbin
  namespace: httpbin
spec:
  backends:
  - name: httpbin
    port:
      number: 14001 # targetPort of httpbin service
      protocol: http
    tls:
      skipClientCertValidation: false
  sources:
  - kind: IPRange
    name: ${curl_pod_ip}/32
  - kind: AuthenticatedPrincipal
    name: curl.curl.cluster.local
EOF

Send the request again for testing.

kubectl exec "$(kubectl get pod -n curl -l app=curl -o jsonpath='{.items..metadata.name}')" -n curl -- curl -ksI https://httpbin.httpbin:14001/get --cacert /certs/ca.crt --key /certs/tls.key --cert /certs/tls.crt
HTTP/2 200
server: gunicorn/19.9.0
date: Mon, 07 Nov 2022 10:58:55 GMT
content-type: application/json
content-length: 267
access-control-allow-origin: *
access-control-allow-credentials: true
osm-stats-namespace: httpbin
osm-stats-kind: Deployment
osm-stats-name: httpbin
osm-stats-pod: httpbin-69dc7d545c-qphrh

Bingo! The access is successful, indicating that our policy has taken effect

Summary

This blog post will talk about the approaches which can be used to enable services outside of the service mesh to communicate with services within the osm-edge service mesh. There may exist use cases where you have a specific set of services that aren't yet ready for migration to service mesh or for some reason can't be migrated yet.

The access control approaches presented in this article are suitable for solving such problems, and allow you to choose the appropriate type of access source and whether to transfer data in plaintext or encrypted, depending on your needs. The control is more granular than using a unified Ingress for access.

Pipy: Protecting Kubernetes Apps from SQL Injection & XSS Attacks

Ali Naqvi — Thu, 03 Nov 2022 13:12:06 +0000

Injection attacks sliding down to 3rd position in 2021 have been on the OWASP Top 10 list for many years and SQL Injection (SQLi) is a common injection technique used for attacking websites and web applications. Applications that do not cleanly separate user input from database commands are at risk of malicious input being executed as SQL commands. Successful injection attacks can lead to unauthorized access to sensitive data such as passwords, credit card details, and personal user information. Many recent high-profile data breaches have resulted from SQL injection attacks, resulting in reputational damage and regulatory fines.

The common and normal workaround which you might have seen with other solutions is using a list of regular expressions to filter out traffic, which works for some cases but falls short with some complex inputs or escaped inputs. We are not trying to bash Regular Expressions, they are good and have their own use cases, but they aren't a good fit for such use cases.

This blog post will demonstrate how to use Pipy an open-source programmable proxy to harden the security by adding an extra layer of shield to your applications which might otherwise be vulnerable to such attacks.

In a previous blog post announcing the latest release of Pipy 0.70.0, it was introduced that Pipy added the support of extensions called Native Module Interface (NMI) and we will be using Pipy NMI to develop a module to integrate with mature and stable open-source library libinject to scan incoming traffic against SQLi and XSS attacks before it reaches the application.

For the demo application, we will be using an open-source classical LAMP stack Vulnerable Mama Shop (VMS) which is intentionally developed to have SQLi flaws. We will have fun together by first hacking the basic application to demonstrate the SQLi attacks, then we will harden the application security by adding Pipy as a sidecar to block certain SQLi attacks.

Pre-requisites

This blog post assumes you have access to:

Running Kubernetes cluster (dev box)
kubectl

The demo source code is available on GitHub and can be downloaded from pipy-sqli-demo repository.

Deploy Kubernetes cluster and Vulnerable Mama Shop App

To run the demo locally, we recommend k3d a lightweight wrapper to run k3s (Rancher Lab’s minimal Kubernetes distribution) in docker.

$ k3d cluster create my-cluster -p 8080:30060@server:0

In the above command, we are creating a cluster with a single server and mapping port 30060 of the k3d container to the local 8080 port, usage of this port will become clear in the later steps of this tutorial.

Deploy Vulnerable Mama Shop App

We are going to deploy a simple online store application that comes installed with

Apache Webserver
MariaDB
PHP app that runs on Apache and connects to the MariaDB database

Use the text editor of your choice and create a YAML file called 1-app.yaml with the following contents:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: vms
spec:
  selector:
    matchLabels:
      app: vms
  template:
    metadata:
      labels:
        app: vms
    spec:
      containers:
        - name: vms
          image: naqvis/mamashop
          ports:
            - containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
  name: vms
spec:
  ports:
    - port: 80
      targetPort: 80
      nodePort: 30060
  selector:
    app: vms
  type: NodePort

Deploy the app

$ kubectl apply -f 1-app.yaml

Confirm that pod is up and running, as indicated by the value Running in the STATUS column. It can take 30-40 seconds for them to fully deploy, so it’s useful to run the command again to confirm all pods are running before continuing to the next step.

$ kubectl get pods
NAME                   READY   STATUS    RESTARTS   AGE
vms-6658dd4478-hn246   1/1     Running   0          2m12s

Query the Servcice

$ kubectl get svc vms
NAME   TYPE       CLUSTER-IP      EXTERNAL-IP   PORT(S)        AGE
vms    NodePort   10.43.244.253   <none>        80:30060/TCP   5m43s

Still remember at the beginning we did port mapping for the K3d container: 8080=>30060? Keen users may also find that we have created a NodePort service for our Vulnerable Mama Shop web application, and the node port is 30060.

Open the app in your browser by visiting the URL http://localhost:8080

Hack the App

Mama Shop is rather very basic, it comes with only 3 pages. The main page (shown above) is where a user can query items for sale through a drop-down box, a customer login page, and an about page. Play around with these 3 pages to see what each does and how each page works normally.

Try submitting a category and see how items are listed. The following shows a listing of items from the Drinks category.

As you may have noticed, selecting a category and hitting submit doesn't change the URL, so the application is doing a POST request and we will need some tool to intercept or view the traffic the web page is generating.

Normally you will be using a proxy tool like Blurp suite or OWASP ZAP to intercept and modify the request to the application. But to keep things simple, we will be using Firefox browser Developer Console to view and modify the requests.

Go back to your browser (Firefox in this demo) and turn on Web Developer Tools either via Tools | Browser Tools | Web Developer Tools menu option or via short-cut key Option+Cmd+I on Mac or similar on your OS. Move to the Network tab, and hit submit button on a web page to view the network traffic.

We can see all of the requests along with all details, a web page makes to the webserver to display the page. Hit Resend button at the top right of Request details window displayed at the bottom right.

Network action window shows that to browse items in a category, a URL encoded HTTP POST with a single parameter catid is sent to welcome.php. To test for SQL injection, it is common to modify user input and send a single quotation mark like '.

Modify Form Parameter

We will find out if the input is properly escaped with a simple experiment: changing the catid to a SQL query string, to see if it will cause a syntax error which may be shown on the webpage.

catid=' or 1 = 1 ; --

Make above change and hit send button at the bottom right of network action window.

Whoa, we are upto something, welcome.php function for displaying category items have a SQL injection vulnerability. The error message also tells us the database is MariaDB. And error is telling us that there is a syntax issue near or 1 = 1. Using our imaginations, we can assume the database query is something like

SELECT item_name, item_category, item_description from items_table where item_category = ' or 1 = 1 ; -- ;

If we were to change the catid ending to " or 1 = 1 -- ;, that should fix the quoting issue. It should select all rows from the database, which is useful in a hack.

The structure of the assumed SQL query is probably correct, although it is not necessarily the exact query that the developer has used. This is sufficient for devasting attacks such as dumping customer information.

Extract User Data

The customer login page tells us that the application contains customer data and this is likely stored in some sort of customer or user table.

Based on our information gathered so far we can make use of the MariaDB INFORMATION_SCHEMA and the SQL Union operator to retrieve database and tables details.

Change catid parameter to below to retrieve database name, which will be further used to retrieve table details.

catid=1000 union select database(), "A" , "B" from dual

The following shows the response from Vulnerable Mama Shop containing the database name, appdb.

Now we have database name, we can obtain the tables in the database

catid=1000 union select table_name,version, table_comment from information_schema.TABLES where table_schema = 'appdb'

So we get a list of all tables, We can safely assume users table in the database contains usernames and passwords.

We need to retrieve list of columns users table have and we will need to ensure that we use same number of columns as queried from products table for UNION to work.

catid=1000 union select table_name, COLUMN_NAME, DATA_TYPE from information_schema.COLUMNS where table_name = 'users'

Now that we know there are a total of six columns in the users table to contains details like firstname, lastname, nric, password, email are available from users table.

We have gathered enough information to dump out a users listing. The following input can be used to dump out a listing of users with their firstname, password and email address.

catid=1000 union select firstname, nric, email from users LIMIT 7, 100

The values for the LIMIT and offset can be worked out by observing how many item entries there are in a normal request. The offset value should remove these items. The LIMIT can be set a sufficiently large value so that customer records can be dumped out.

Play around and use these gathered information to see if you can login with these credentials.

Using Pipy Sidecar to block certain SQLi attacks

SQL injection occurs when unvalidated user input is mixed with SQL instructions, so developers should pay more attention to escape user input (such as use of parameterized queries), but you – the Kubernetes engineer – can also help avoid SQL injection by preventing this attack from reaching the app. That way, even if the app is vulnerable, attacks can still be stopped.

There are multiple options for protecting your apps, but for this blog post we will focus on injecting a sidecar container to proxy all of the traffic and deny any request that is detected as SQLi attacks.

For demonstration purposes, we are following the approach of manually injecting sidecar, but in reality, manually deploying proxies as sidecars isn’t the best solution. The complexity of your apps and architecture might require more fine–grain control. If your organization requires Zero Trust and has a need for end–to–end encryption, consider a service mesh like osm-edge a light weight, ultra fast, low resources, highly extensible, Service Mesh Interface (SMI) compatible, built for edge and cloud computing service mesh.

Deploying Pipy Sidecar

Create a YAML file called 2-app-sidecar.yaml with the contents below, and check out these noteworthy components:

A sidecar container running Pipy is started on port 8000.
The Pipy process forwards all traffic to the app.
Requests URI or POST body containing SQLi is declined
The service for the app routes all traffic to the Pipy sidecar container first.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: vms
spec:
  selector:
    matchLabels:
      app: vms
  template:
    metadata:
      labels:
        app: vms
    spec:
      containers:
        - name: vms
          image: naqvis/mamashop
          ports:
            - containerPort: 80
        - name: pipy # <-- sidecar
          image: naqvis/pipy-nmi
          env:
            - name: PIPY_CONFIG_FILE
              value: /etc/pipy/nmi/nmi.js
          ports:
            - containerPort: 8000
          volumeMounts:
            - mountPath: /etc/pipy/nmi
              name: pipy-pjs
      volumes:
        - name: pipy-pjs
          configMap:
            name: sidecar
---
apiVersion: v1
kind: Service
metadata:
  name: vms
spec:
  ports:
    - port: 80
      targetPort: 8000 # <-- the traffic is routed to the proxy
      nodePort: 30060
  selector:
    app: vms
  type: NodePort
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: sidecar
data:
  nmi.js: |-
    pipy({
      _rejected: undefined,
    })
      .import({
        __is_sqli: 'lib-inject',
        __is_xss: 'lib-inject',
        __sqli_fingerprint: 'lib-inject',
      })
      .listen(8000)
      .demuxHTTP().to(
        $=>$
            .use('/etc/pipy/modules/inject-nmi.so')
            .handleMessage(() => _rejected = (__is_sqli || __is_xss))
            .branch(
                () => _rejected === true, (
                $=>$
                .handleMessageStart(_ =>
                  console.log(`SQL Injection found with Fingerprint: ${__sqli_fingerprint}`))
                .replaceMessage(new Message({ status: 403 }, 'Forbidden'))
                ),
                () => _rejected === false, (
                  $=>$.muxHTTP().to(
                      $=>$.connect('localhost:80')
                  )
                )
            )
      )

Deploy the side

$ kubectl apply -f 2-app-sidecar.yaml

Wait for pods to be up and ready

$ kubectl get pods
NAME                  READY   STATUS    RESTARTS   AGE
vms-945f6f85c-8v7sb   2/2     Running   0          8m48s

Test the Sidecar

Test whether the sidecar is filtering traffic by returning to the app and trying the SQL injection again. Pipy sidecar blocks the request before it reaches the app!

Sidecar is blocking the traffic by returning 403 Forbidden. Run few more requests with any of the previous SQLi attempts we made and we will be getting 403 back in response.
We can validate that by looking at the logs of Pipy side car.

Conclusion

The purpose of learning about offensive techniques is to enable developers, defenders and security professionals to protect the confidentiality, the integrity and availability of critical information assets and services.

Kubernetes is not secure by default. This blog post made use of Vulnerable Mama Shop, a simple application based on LAMP stack to demonstrate SQL injection attacks and how one can use techniques like SideCar to protect against them.

But as the complexity of your apps and architecture grows, you might require more fine–grain control over your services. And if your organization requires Zero Trust and has a need for end–to–end encryption like mTLS, you should consider a service mesh like osm-edge a light weight, ultra fast, low resources, highly extensible, Service Mesh Interface (SMI) compatible, built for edge and cloud computing service mesh. When you have communication between services (east–west traffic), a service mesh allows you to control traffic at that level.

Pipy 0.70.0 is released!

Ali Naqvi — Fri, 21 Oct 2022 15:26:06 +0000

Pipy 0.70.0 is now available. This release consists of 170+ commits and comes with improvements to several areas including support for writing Pipy modules using C via Pipy Native Module Interface (NMI), PJS language extensions like looping construct, Map, Set, etc, protocols like Thrift, PROXY, TPROXY, Output UDP with Ephemeral ports, Encoder/Decoder for DNS protocol, and more. This version of Pipy comes with a fresh and new design of modules/plugin system, enhanced Pipy unique Admin Console, IDE is now more robust and supports more features like Intellisense, parameters highlighting, hints, etc.
This release continues on the great work of previous releases and optimization is done for PipyJS and code readability, better documentation, and some bug fixes.

This release was indeed a community effort and could not have been made possible without all of the hard work from everyone involved in active discussions and the Pipy project on GitHub.The Pipy community provides code submissions covering new functionality and bug fixes, documentation improvements, quality assurance testing, continuous integration environments, bug reports, and more. Everyone has done their part to make this release possible! If you’d like to join this amazing community, you can find it on GitHub, Slack, and the Pipy discussion groups.

Below we list the most remarkable changes in the Core API and PipyJS. For a detailed change log, refer to Pipy Release Page

Core

NMI (Native Module Interface) for dynamically loaded modules written in C
New module/plugin system by using chain() filters
Support loop logic in pipelines by using the new replay() filter
Builtin GUI frontend greatly reduced in size by using Brotli compression

Protocols

Support Thrift protocol
Support the Proxy protocol proposed by HAProxy
Support outbound UDP with ephemeral local ports
Support TPROXY on UDP

API

Implementation of the standard ECMAScript builtin objects Map and Set
Encode and decode DNS messages
Logging API now supports HTTPS receivers and Syslog
Expose subject alternative names of a certificate
Mux filters to support limit of maximum messages per session
Netmask API now supports IPv6

Documentation & Samples

Renovation of the documentation system by using TypeScript
All new tutorials with complete API reference
New samples for HTTP proxy common use cases

We would like to thank each and every contributor who was involved in this release.

Flomesh Ingress Controller with Kubernetes Multi-tenancy

Ali Naqvi — Mon, 03 Oct 2022 13:10:34 +0000

Background

It's very rare for organizations to have or provide a dedicated Kubernetes cluster to each tenant, as it's generally more cost-effective to share a cluster. Sharing clusters reduces expenses and streamlines management. Sharing clusters, however, also comes with difficulties like managing noisy neighbors and ensuring security.

Clusters can be shared in many ways. In some cases, different applications may run in the same cluster. In other cases, multiple instances of the same application may run in the same cluster, one for each end user. All these types of sharing are frequently described using the umbrella term multi-tenancy.

While Kubernetes does not have first-class concepts of end users or tenants, it provides several features to help manage different tenancy requirements. These are discussed below.

A common form of multi-tenancy is to share a cluster between multiple teams within an organization, each of whom may operate one or more workloads. In this scenario, members of the teams often have direct access to Kubernetes resources via tools such as kubectl, or indirect access through GitOps controllers or other types of release automation tools. There is often some level of trust between members of different teams, but Kubernetes policies such as RBAC, quotas, and network policies are essential to safely and fairly share clusters.
The other major form of multi-tenancy frequently involves a Software-as-a-Service (SaaS) vendor running multiple instances of a workload for customers. This business model is so strongly associated with this deployment style that many people call it "SaaS tenancy." In this scenario, the customers do not have access to the cluster; Kubernetes is invisible from their perspective and is only used by the vendor to manage the workloads. Cost optimization is frequently a critical concern, and Kubernetes policies are used to ensure that the workloads are strongly isolated from each other.

Tenant Isolation

There are several ways to design and build multi-tenant solutions with Kubernetes at its control plane, data plane, or at both levels. Each of these methods comes with its own set of tradeoffs that impact the isolation level, implementation effort, operational complexity, and cost of service.

Kubernetes Control plane isolation ensures that different tenants cannot access or affect each other's Kubernetes API resources, and Flomesh Service Mesh (FSM) Ingress controller provides isolated Ingress controllers for Kubernetes Namespaces.

In Kubernetes, a Namespace provides a mechanism for isolating groups of API resources within a single cluster. This isolation has two key dimensions:

Object names within a namespace can overlap with names in other namespaces, similar to files in folders. This allows tenants to name their resources without having to consider what other tenants are doing.
Many Kubernetes security policies are scoped to namespaces. For example, RBAC Roles and Network Policies are namespace-scoped resources. Using RBAC, Users and Service Accounts can be restricted to a namespace.

In a multi-tenant environment, a Namespace helps segment a tenant's workload into a logical and distinct management unit. A common practice is to isolate every workload in its namespace, even if multiple workloads are operated by the same tenant. This ensures that each workload has its own identity and can be configured with an appropriate security policy.

In this article, we will learn how to use the Flomesh Service Mesh Ingress controller to create physical isolation of Ingress controllers when hosting multiple tenants in your Kubernetes cluster.

Flomesh Service Mesh (FSM)

FSM Ingress Controller supports a multi-tenancy model via its concept of NamespacedIngress CRD where it deploys a physically isolated Ingress Controller for requested Namespace.

For example, the YAML below defines an Ingress Controller that monitors on port 100 and also creates a LoadBalancer type service for it that listens on port 100.

apiVersion: flomesh.io/v1alpha1
kind: NamespacedIngress
metadata:
  name: namespaced-ingress-100
  namespace: test-100
spec:
  serviceType: LoadBalancer
  ports:
  - name: http
    port: 100
    protocol: TCP

Install FSM

FSM provides a standard Helm chart, which can be installed via the Helm CLI.

$ helm repo add fsm https://flomesh-io.github.io/fsm
$ helm repo update

$ helm install fsm fsm/fsm --namespace flomesh --create-namespace --set fsm.ingress.namespaced=true

Verify that all pods are up and running properly.

$ kubectl get po -n flomesh
NAME                                          READY   STATUS    RESTARTS   AGE
fsm-manager-6857f96858-sjksm                  1/1     Running   0          55s
fsm-repo-59bbbfdc5f-w7vg6                     1/1     Running   0          55s
fsm-bootstrap-8576c5ff4f-7qr7k                1/1     Running   0          55s
fsm-cluster-connector-local-8f8fb87f6-h7z9j   1/1     Running   0          32s

Create Sample Application

In this demo, we will be deploying httpbin service under a namespace httpbin.

# Create Namespace
kubectl create ns httpbin

# Deploy sample
kubectl apply -f https://raw.githubusercontent.com/flomesh-io/osm-edge-docs/main/manifests/samples/httpbin/httpbin.yaml -n httpbin

Creating a standalone Ingress Controller

The next step is to create a separate Ingress Controller for the namespace httpbin.

$ kubectl apply -f - <<EOF
apiVersion: flomesh.io/v1alpha1
kind: NamespacedIngress
metadata:
  name: namespaced-ingress-httpbin
  namespace: httpbin
spec:
  serviceType: LoadBalancer
  http:
    port:
      name: http
      port: 81
      nodePort: 30081
  resources:
    limits:
      cpu: 500m
      memory: 200Mi
    requests:
      cpu: 100m
      memory: 20Mi
EOF

After executing the above command, you will see an Ingress Controller running successfully under the namespace httpbin.

kubectl get po -n httpbin -l app=fsm-ingress-pipy
NAME                                        READY   STATUS    RESTARTS   AGE
fsm-ingress-pipy-httpbin-5594ffcfcc-zl5gl   1/1     Running   0          58s

At this point, there should be a corresponding Service under this namespace.

$ kubectl get svc -n httpbin -l app.kubernetes.io/component=controller
NAME                       TYPE           CLUSTER-IP     EXTERNAL-IP    PORT(S)        AGE
fsm-ingress-pipy-httpbin   LoadBalancer   10.43.62.120   192.168.1.11   81:30081/TCP   2m49s

Once you have the Ingress Controller, it's time to create the Ingress resource.

kubectl apply -f - <<EOF
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: httpbin
  namespace: httpbin
spec:
  ingressClassName: pipy
  rules:
  - host: httpbin.org
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: httpbin
            port:
              number: 14001
EOF

Now we have created Ingress resource, let's do a quick curl to see if things are working as expected.

For my local setup demo, LoadBalancer IP is 192.168.1.11, your IP might be different. So ensure you are performing a curl against your setup ExternalIP.

curl -sI http://192.168.1.11:81/get -H "Host: httpbin.org"
HTTP/1.1 200 OK
server: gunicorn/19.9.0
date: Mon, 03 Oct 2022 12:02:04 GMT
content-type: application/json
content-length: 239
access-control-allow-origin: *
access-control-allow-credentials: true
connection: keep-alive

Conclusion

In this blog post, you learned about Kubernetes multi-tenancy, features provided by Kubernetes, tenancy isolation levels, and how to use Flomesh Service Mesh (FSM) Ingress controller to set up isolated Ingress controller for namespaces.

Flomesh Service Mesh(FSM) from Flomesh is Kubernetes North-South traffic manager, provides Ingress controllers, Gateway API, Load Balancer, and cross-cluster service registration and service discovery. FSM uses Pipy - a programmable network proxy, as its data plane and is suitable for cloud, edge, and IoT.

Using FSM Ingress controller with osm-edge service mesh

Ali Naqvi — Thu, 18 Aug 2022 15:18:11 +0000

Background

The Kubernetes Ingress API is designed with a separation of concerns, where the Ingress implementation provides an entry feature infrastructure managed by operations staff; it also allows application owners to control the routing of requests to the backend through rules.

The osm-edge supports multiple Ingress implementations to manage ingress traffic and provides the IngressBackend API to configure back-end services to receive access from trusted ingress points. This article focuses on the integration of osm-edge with FSM to manage ingress traffic.

Introduction to FSM

FSM is another open-source product from Flomesh for Kubernetes north-south traffic, gateway api contoller, and multi-cluster management. FSM uses Pipy, a programmable proxy at its core, and provides an Ingress controller, Gateway API controller, load balancer, cross-cluster service registration discovery, and more.

Integration with FSM

FSM is already integrated inside osm-edge and can be enabled during osm-edge installation; it can also be installed independently via helm for existing osm-edge mesh.

Prerequisites

Kubernetes cluster, version 1.19.0 and higher
Helm 3 CLI for standalone installation of FSM

Download the osm-edge CLI at

system=$(uname -s | tr [:upper:] [:lower:])
arch=$(dpkg --print-architecture)
release=v1.1.1
curl -L https://github.com/flomesh-io/osm-edge/releases/download/${release}/osm-edge-${release}-${system}-${arch}.tar.gz | tar -vxzf -
. /${system}-${arch}/osm version
cp . /${system}-${arch}/osm /usr/local/bin/

Integrated installation

export osm_namespace=osm-system 
export osm_mesh_name=osm 

osm install --set fsm.enabled=true \
    --mesh-name "$osm_mesh_name" \
    --osm-namespace "$osm_namespace"

Standalone installation

If osm-edge is installed without FSM enabled, you can use a standalone installation to install it.

helm repo add fsm https://charts.flomesh.io

export fsm_namespace=osm-system 
helm install fsm fsm/fsm --namespace "$fsm_namespace" --create-namespace

Verify that all pods are up and running properly.

kubectl get pods -n osm-system
NAME READY STATUS RESTARTS AGE
repo-8756f76fb-2f78g 1/1 Running 0 5m53s
manager-866585bbd5-pbg7q 1/1 Running 0 5m53s
osm-bootstrap-7c6689ff57-47ksk 1/1 Running 0 5m53s
osm-controller-57888cfc7c-tnqxl 2/2 Running 0 5m52s
osm-injector-5f77898899-45f65 1/1 Running 0 5m53s
bootstrap-fd5894bcc-nr7hf 1/1 Running 0 5m53s
cluster-connector-local-68c7584c8b-qf7xm 1/1 Running 0 2m43s
ingress-pipy-6fb8c8b794-pgthl 1/1 Running 0 5m53s

Configuration

In order to authorize clients by restricting access to backend traffic, we will configure IngressBackend so that only ingress traffic from the ingress-pipy-controller endpoint can be routed to the backend service. In order to discover the ingress-pipy-controller endpoint, we need the osm-edge controller and the corresponding namespace to monitor it. However, to ensure that the FSM functions properly, it cannot be injected with sidecar.

kubectl label namespace "$osm_namespace" openservicemesh.io/monitored-by="$osm_mesh_name"

Save the external IP address and port of the entry gateway, which will be used later to test access to the backend application.

export ingress_host="$(kubectl -n "$osm_namespace" get service ingress-pipy-controller -o jsonpath='{.status.loadBalancer.ingress[0].ip}') "
export ingress_port="$(kubectl -n "$osm_namespace" get service ingress-pipy-controller -o jsonpath='{.spec.ports[? (@.name=="http")].port}')"
echo $ingress_host:$ingress_port

Deploying the sample service

The next step is to deploy the sample httpbin service.

## Create namespace kubectl create ns httpbin
kubectl create ns httpbin

# Add the namespace to the grid
osm namespace add httpbin

# Deploy the application
kubectl apply -f - <<EOF
apiVersion: v1
kind: ServiceAccount
metadata:
  name: httpbin
  namespace: httpbin
---
apiVersion: v1
kind: Service
metadata:
  name: httpbin
  namespace: httpbin
  labels:
    app: httpbin
    service: httpbin
spec:
  ports:
  - name: http
    port: 14001
  selector:
    app: httpbin
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: httpbin
  namespace: httpbin
spec:
  replicas: 1
  selector:
    matchLabels:
      app: httpbin
  template:
    metadata:
      labels:
        app: httpbin
    spec:
      serviceAccountName: httpbin
      containers:
      - image: kennethreitz/httpbin
        imagePullPolicy: IfNotPresent
        name: httpbin
        command: ["gunicorn", "-b", "0.0.0.0:14001", "httpbin:app", "-k", "gevent"]
        ports:
        - containerPort: 14001
EOF

Verify that the pod and service are created and the application is running successfully.

kubectl get pods,services -n httpbin
NAME READY STATUS RESTARTS AGE
pod/httpbin-54cc8cf5d-7vclc 2/2 Running 0 14s

NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/httpbin ClusterIP 10.43.105.166 <none> 14001/TCP 14s

Configure entry rules

Next we want to access the deployed httpbin service from outside the cluster and need to provide the ingress configuration rules.

kubectl apply -f - <<EOF
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: httpbin
  namespace: httpbin
  annotations:
    pipy.ingress.kubernetes.io/rewrite-target-from: /httpbin
    pipy.ingress.kubernetes.io/rewrite-target-to: /httpbin
spec:
  ingressClassName: pipy
  rules:
  - host: httpbin.org
    http:
      paths:
      - path: /httpbin
        pathType: Prefix
        backend:
          service:
            name: httpbin
            port:
              number: 14001
EOF

Accessing the httpbin service using the IP address and port recorded above will result in the following 502 Bad Gateway error response. This is because we have not set the ingress of the FSM as a trusted portal.

curl -sI http://"$ingress_host":"$ingress_port"/httpbin/get -H "Host: httpbin.org"
HTTP/1.1 502 Bad Gateway
content-length: 0
connection: keep-alive

Execute the following command to set the FSM ingress as a trusted entry.

kubectl apply -f - <<EOF
kind: IngressBackend
apiVersion: policy.openservicemesh.io/v1alpha1
metadata:
  name: httpbin
  namespace: httpbin
spec:
  backends:
  - name: httpbin
    port:
      number: 14001 # targetPort of httpbin service
      protocol: http
  sources:
  - kind: Service
    namespace: "$osm_namespace"
    name: ingress-pipy-controller
EOF

Try requesting the httpbin service again, and you will be able to access it successfully.

curl -sI http://"$ingress_host":"$ingress_port"/httpbin/get -H "Host: httpbin.org"
HTTP/1.1 200 OK
server: gunicorn/19.9.0
date: Thu, 18 Aug 2022 05:18:50 GMT
content-type: application/json
content-length: 241
access-control-allow-origin: *
access-control-allow-credentials: true
osm-stats-namespace: httpbin
osm-stats-kind: Deployment
osm-stats-name: httpbin
osm-stats-pod: httpbin-54cc8cf5d-7vclc
connection: keep-alive

Summary

FSM Ingress exposes the application access points within the Kubernetes cluster for easy management of portal traffic. osm-edge's IngressBackend API provides an additional line of defense against accidental data leakage by exposing services within the mesh to the public.

Benchmarking osm and osm-edge data planes

Ali Naqvi — Sun, 07 Aug 2022 04:51:14 +0000

osm-edge is built on top of Open Service Mesh (OSM) v1.1.0 codebase and is a lightweight service mesh for resource-sensitive cloud environments and edge computing scenarios. It uses osm as the control plane and Pipy as the data plane and features high performance, low resources, simplicity, ease of use, scalability, and compatibility (x86/arm64 support).

osm-edge is a simple, complete, and standalone service mesh and ships out-of-the-box with all the necessary components to deploy a complete service mesh. As a lightweight and SMI-compatible Service Mesh, osm-edge is designed to be intuitive and scalable.

For detailed introduction and documentation, refer to osm-edge documentation

In this experiment, benchmarks were conducted for osm-edge(pipy) (v1.1.0) and osm(envoy) (v1.1.0). The main focus is on service TPS, latency distribution when using two different meshes, and monitoring the resource overhead of the data plane.

osm-edge uses Pipy as the data plane; osm uses Envoy as the data plane.

Testing Environment

The benchmark was performed in a Kubernetes cluster running on Cloud. Cluster comprised of 2 worker nodes and one load generator node. Software and hardware configurations as below:

Kubernetes: k3s v1.20.14+k3s2
OS: Ubuntu 20.04
Nodes: 16c32g * 2
Load Generator: 8c16g

What features were tested?

Both meshes had permissive traffic mode enabled
Both meshes had mutual TLS enabled and were encrypting traffic and validating identity between all application pods.
Both meshes were reporting metrics, including L7 metrics, though these metrics were not consumed in this experiment.
Both meshes logged various messages at the INFO level by default. We did not configure logging.

What application was used for testing?

The test application uses the common SpringCloud microservices architecture. The application code is taken from flomesh-bookinfo-demo, which is a SpringCloud implementation of bookinfo application using SpringCloud. In the tests, not all services were used, but rather the API gateway and the Bookinfo Ratings service were selected.

External access to the service is provided through Ingress; the load generator uses the common Apache Jmeter 5.5.

For the tests, 2 paths were chosen: one is the direct access to the Bookinfo Ratings service via Ingress (hereafter ratings), and the other passes through the API Gateway (hereafter gateway-ratings) in the middle. The reason for choosing these two paths is to cover both single-sidecar and multiple-sidecar scenarios.

Procedure

We start the test with a non-mesh (i.e., no sidecar injection, hereafter referred to as non-mesh), and then test with osm-edge(pipy) and osm(envoy) mesh, respectively.

To simulate the resource-constrained scenario of the edge, the CPU used by the sidecar is limited when using the mesh, and the 1 core and 2 core scenarios are tested respectively.

Thus, there are 5 rounds of testing, with 2 different tests in each round.

Performance

Jmeter uses 200 threads during the test and runs the test for 5 minutes. Each round of testing is preceded by a 2-minute warm-up.

For sake of verbosity, only the results of the gateway-ratings paths are shown below with different sidecar resource constraints.

Note: The sidecar in the table refers to the API Gateway sidecar.

mesh	sidecar max CPU	TPS	90th	95th	99th	sidecar CPU usage	sidecar memory usage
non-mesh	NA	3283	87	89	97	NA	NA
osm-edge(pipy)	2	3395	77	79	84	130%	52
osm(envoy)	2	2189	102	104	109	200%	108
osm-edge(pipy)	1	2839	76	77	79	100%	34
osm(envoy)	1	1097	201	203	285	100%	105

Sidecar with 2 CPU core limitations

In the sidecar 2-core scenario, using the osm-edge(pipy) mesh gives a small TPS improvement over not using the mesh, and also improves latency. Improvement of TPS and latency is due the fact that Java NIO uses reactor pattern, while Pipy is built on C++ ASIO which uses the proactor pattern and the connection from Pipy to Java are long duplex connections. The sidecar for both API Gateway and Bookinfo Ratings is still not running out of 2 cores (only 65%), when the Bookinfo Ratings service itself is at its performance limit.

The TPS of the osm(envoy) mesh was down nearly 30%, and the API Gateway sidecar CPU was running full, which was the bottleneck.

In terms of memory, the memory usage of osm-edge(pipy) and osm(envoy) sidecar is 50 MiB and 105 MiB respectively, meaning osm-edge(pipy) was utilizing 52% less memory than osm(envoy).

TPS

Latency distribution

API gateway sidecar CPU usage

API gateway sidecar memory footprint

Bookinfo Ratings sidecar CPU usage

Bookinfo Ratings sidecar memory usage

Sidecar with 1 CPU core limitation

The difference is particularly noticeable in tests that limit the sidecar to 1 core CPU. At this point, the API Gateway sidecar becomes the performance bottleneck, with both the osm-edge and osm sidecar running out of CPU.

In terms of TPS, osm-edge(Pipy) drops by 12% and osm(Envoy)'s TPS drops by a staggering 65%.

TPS

Latency distribution

API gateway sidecar CPU usage

API gateway sidecar memory footprint

Bookinfo Ratings sidecar CPU usage

Bookinfo Ratings sidecar memory usage

Summary

The focus of this experiment was to benchmark the performance and resource consumption of both meshes in a resource-constrained environment. Experiment results revealed that osm-edge maintained high performance and made efficient use of allocated resources. This experiment proves that complete service mesh functionalities can be used and utilized for resource-constrained edge scenarios.

osm-edge is well-suited and ready for both cloud and resource-sensitive environments.

Announcing osm-edge 1.1: ARM support and more

Ali Naqvi — Thu, 28 Jul 2022 13:19:04 +0000

We're very happy to announce the release of osm-edge 1.1. This release is the culmination of months of effort from the flomesh.io team and we are very excited to unveil it today. osm-edge is built on top of Open Service Mesh v1.1.0 codebase and is built purposely for edge computing and uses lightweight, high-performant, cloud native, and programmable proxy Pipy as its data plane and sidecar proxy.

Why osm-edge?

In practice, we have encountered users from a variety of industries with similar requirements for service mesh. These industry users and scenarios include:

Energy and power companies. They want to build simple server rooms at each substation or gas station to deploy computing and storage capacity for the processing of data generated by devices within the coverage area of that location. They want to push traditional data center applications to these simple rooms and take full advantage of data center application management and operation capabilities
Telematics service providers. They want to build their simple computing environments in non-data center environments for data collection and service delivery to cars and vehicle owners. These environments may be near highway locations, parking lots, or high-traffic areas
Retailers. They want to build a minimal computing environment in each store, and in addition to supporting traditional capabilities such as inventory, sales, and payment collection, they also hope to introduce new capabilities of data collection, processing, and transmission
Medical institutions. They want to provide network capabilities at every hospital, or a simple point of care, so that in addition to providing digital service capability for patients, they can also complete data collection and data linkage with higher management departments at the same time
Teaching institutions, hospitals, and campuses alike. These campuses are characterized by a relatively regular and dense flow of people. They want to deploy computing resources near more crowd gathering points for delivering digital services, as well as collecting and processing data in real-time

These are typical edge computing scenarios, and they have similar needs:

Users want to bring the traditional data center computing model, especially microservices, and the related application delivery, management operation, and maintenance capabilities to the edge side
In terms of the working environment, users have to deal with factors such as power supply, limited computing power, and unstable network. Therefore, computing platforms are required to be more robust and can be deployed quickly or recover a computing environment completely in extreme situations
The number of locations (we call POP=Point of Presence) that usually need to be deployed is large and constantly evolving and expanding. The cost of a POP point, the price of maintenance, and the price of expansion are all important cost considerations.
Common, or low-end, PC servers are more often used in these scenarios to replace cloud-standard servers; low-power technology-based computing power such as ARM is further replacing low-end PC servers. On these hardware platforms, which are not comparable to cloud-standard servers, users still want to have enough computing power to handle the growth in functionality and data volume. The conflicting demands of computing moving closer to where the data is generated, the growth in data volume and functional requirements at the edge, and the limited computing resources at the edge require edge-side computing platforms to have better computing power efficiency ratios, i.e., running more applications and supporting larger data volumes with as little power and as few servers as possible
The fragility and a large number of POP points require better application support for multi-cluster, cross-POP failover. For example, if a POP point fails, the neighboring POP points can quickly share or even temporarily take over the computing tasks.

Compared with the cloud data center computing scenario, the three core and main differences and difficulties of edge computing are:

Edge computing requires support for heterogeneous hardware architectures. We see non-x86 computing power being widely used at the edge, often with the advantage of low power consumption and low cost
Edge computing POP points are fragile. This fragility is reflected in the fact that they may not have an extremely reliable power supply, or the power supply is not as powerful as a data center; they may operate in a worse environment than the constant temperature and ventilation of a data center; their networks may be narrowband and unstable
Edge computing is naturally distributed computing. In almost all edge computing scenarios, there are multiple POP points, and the number of POP points is continuously increasing. the POP points can disaster-proof each other and migrate to adjacent POP points in case of failure, which is a fundamental capability of edge computing

The evolution of Kubernetes to the edge side solves the difficulties of edge computing to a certain extent, especially against fragility; while the development of service mesh to the edge side focuses on network issues in edge computing, against network fragility, as well as providing basic network support for distributed, such as fault migration. In practice, container platforms, as today's de facto quasi-standard means of application delivery, are rapidly evolving to the edge side, with a large number of releases targeting edge features, such as k3s; but service mesh, as an important network extension for container platforms, are not quickly keeping up with this trend. It is currently difficult for users to find service mesh for edge computing scenarios, so we started the osm-edge an open source project with several important considerations and goals, namely

Support and compatibility with the SMI specification, so that it can meet users' needs for standardization of service mesh management
Full support for the ARM ecosystem, which is the "first-class citizen" or even the preferred computing platform for edge computing, and the Service Mesh should be fully adapted to meet this trend. osm-edge follows the ARM First strategy, which means that all features are developed, tested, and delivered on the ARM platform first
High performance and low resources. The service mesh as infrastructure should use fewer resources (CPU/MEM) while delivering higher performance (TPS/Latency) at the edge.

Features

Light-weight, high-performant, cloud-native, extensible
Out-of-the-box supports x86, ARM architectures
Easily and transparently configure traffic shifting for deployments
Secure service-to-service communication by enabling mutual TLS
Define and execute fine-grained access control policies for services
Observability and insights into application metrics for debugging and monitoring services
Integrate with external certificate management services/solutions with a pluggable interface
Onboard applications onto the mesh by enabling automatic sidecar injection of Pipy proxy

ARM Support

osm-edge 1.1 brings the oft-requested support for ARM (both for development & production use), Whether you’re focused on cost reduction with ARM-based compute such as AWS Graviton or simply want to run service mesh on your Raspberry Pi cluster, now you can!

Multi-cluster Kubernetes the Kubernetes way

osm-edge 1.1 comes bundled with Flomesh Service Mesh (FSM) a Kubernetes North-South traffic manager, provides Ingress controllers, Gateway API, Load Balancer, and cross-cluster service registration and service discovery.

With the help of FSM, osm-edge can now connect Kubernetes services across cluster boundaries in a way that’s secure, fully transparent to the application, and independent of network topology. Automated fail-over ability to automatically redirect all traffic from a failing or inaccessible service to one or more replicas of that service—including replicas on other clusters.

Multiple Sidecar Proxy support

To break the tight coupling on Pipy and open doors for 3rd parties to develop or make use of their data plane or sidecar proxies, we have refactored the OSM v1.1.0 codebase to make it generic and provide extension points. We strongly believe in and support open source and our proposal for this refactoring have been submitted to upstream for their review, discussion, and/or utilization.

And lot more

osm-edge 1.1 also has a tremendous list of other improvements, performance enhancements, and bug fixes, for a detailed change log, please refer to the Release page on Github. Below are some of the notable changes included in this release:

Refactoring of Proxy Control Plane component to stay generic and allow interaction with new sidecar proxy implementation
Added new driver.Driver interface that needs to be implemented by 3rd party vendors wishing to provide sidecar proxy for control plane
Added new driver.HealthProbes, driver.HealthProbe, driver.InjectorContext, driver.controllerContext struct to use with new sidecar proxy driver implementation
Refactored Envoy based proxy sidecar integration to a separate Driver, so that it works as an implementation of driver.Driver interface
Added pipy implementation as a new Proxy Control Plane component proxy Driver
Refactors Helm chart values.yaml and added items osm.sidecarClass, osm.sidecarImage, osm.sidecarWindowsImage, osm.sidecarDrivers List to allow configuring the proxy sidecar driver.
Pipy driver is the default driver of osm-edge distribution, but can be configured via cli flag --set=osm.sidecarClass=XXX where XXX refers to sidecar Driver.
osm-edge control plane images are now multi-architecture, built for Linux/amd64 and Linux/arm64
osm-edge test suite used images are now multi-architecture, built for Linux/amd64 and Linux/arm64
Optimization of scripts
Added Makefile targets for easier installation/setup
Updated scripts to setup a development environment on amd64 and arm64 architectures.

What's next and where to learn more?

For more documentation, a quick-start guide, and step-by-step tutorials please visit osm-edge-docs

osm-edge is a fork of open service mesh and we will strive to keep this fork in sync with its upstream and propose back major changes and/or feature proposals to upstream for broader benefits of the community. Both OSM and osm-edge are hosted on Github. If you have any feature request, question, or comment, we’d love to have you join the rapidly-growing community via Github Issues, Pull Requests, or osm slack channel!

DEV Community: Ali Naqvi

Pipy proxy: Creating HTTP Tunnel with 10 lines of code

Solution

PipyJS script

Testing

What could be done next?

Summary

Writing a DNS proxy with Pipy

DNS Introduction

Solution

Implementation

PipyJS Code

Custom Record

Intelligent Route Resolution

Testing

Going Further

Conclusion

Kubernetes: Cross-cluster traffic scheduling - Access control

SMI Access Control Policy

FSM's ServiceExport

Deploy the application

Deploy the sample application

Export service

Test

Demo

Adjusting the traffic policy mode

Application Access Control Policy

Summary

Kubernetes: Multi-cluster communication with Flomesh Service Mesh (Part 2)

Kubernetes: Multi-cluster communication with Flomesh Service Mesh

Ali Naqvi ・ Nov 28 '22

Demo Architecture

Demo Pre-requisites

Demo clusters & environment setup

Network

Cluster creation

Install FSM

Cluster CRD

ServiceExport and ServiceImport CRD

Federate clusters

Install osm-edge Service Mesh

Deploy Demo application

Deploying mesh-managed applications

Export Service

Testing

Global Traffic Policy

Routing Type - Locality

Routing Type - FailOver

Routing Type - ActiveActive

Summary

Kubernetes: Multi-cluster communication with Flomesh Service Mesh

Flomesh Service Mesh

Motivation

Goals

Architecture

Summary

osm-edge: Using access control policies to access services with the service mesh

Demo environment preparation

Kubernetes cluster

osm-edge CLI

Install osm-edge

Deploy the sample application

Demo

Plaintext transfer

Service-based access control

IP range-based access control, plaintext transfer

Encrypted transfers

Service-based Access Control

IP Range-Based Access Control

Summary

Pipy: Protecting Kubernetes Apps from SQL Injection & XSS Attacks

Pre-requisites

Deploy Kubernetes cluster and Vulnerable Mama Shop App

Deploy Vulnerable Mama Shop App

Hack the App

Modify Form Parameter

Extract User Data

Using Pipy Sidecar to block certain SQLi attacks

Deploying Pipy Sidecar

Test the Sidecar

FSM's `ServiceExport`

`Cluster` CRD

`ServiceExport` and `ServiceImport` CRD

Routing Type - `Locality`

Routing Type - `FailOver`

Routing Type - `ActiveActive`