Kubernetes: Multi-cluster communication with Flomesh Service Mesh

Kubernetes has been quite successful in popularizing the idea of ​​container clusters. Deployments have reached a point where many users are running multiple clusters and struggling to keep them running smoothly. Organizations need to run multiple Kubernetes clusters might fall into one of the below reasons (not an exhaustive list):

• Location
  • Latency (run the application as close to customers as possible)
  • Jurisdiction (e.g. required to keep user data in-country)
  • Data gravity (e.g. data exists in one provider)
• Isolation
  • Environment (e.g. development, testing, staging, prod, etc)
  • Performance isolation (teams don't want to feel each other)
  • Security isolation (sensitive data or untrusted code)
  • Organizational isolation (teams have different management domains)
  • Cost isolation (teams want to get different bills)
• Reliability
  • Blast radius (an infra or app problem in one cluster doesn't kill the whole system)
  • Infrastructure diversity (an underlying zone, region, or provider outages does not bring down the whole system)
  • Scale (the app is too big to fit in a single cluster)
  • Upgrade scope (upgrade infra for some parts of your app but not all of it; avoid the need for in-place cluster upgrades)

There is currently no standard way to connect or even think about Kubernetes services beyond the single cluster boundary, and Kubernetes Multicluster SIG has put together a proposal KEP-1645 to extend Kubernetes Service concepts across multiple clusters.

Flomesh team has been spending time tackling the challenge of multicluster communication, integrating north-south traffic management capabilities into osm-edge SMI compatible service mesh, and contributing back to the Open Source community.

In this part of the series, we will be looking into motivation, goals, architecture of FSM multi-cluster support, its components, and how it integrates with osm-edge full-fledged, lightweight, SMI-compatible service mesh suitable for cloud-native and edge.

Flomesh Service Mesh

FSM is an open-source product from Flomesh for Kubernetes north-south traffic, gateway API controller, and multi-cluster management. FSM uses Pipy, a programmable proxy at its core, and provides an Ingress controller, Gateway API controller, load balancer, cross-cluster service registration discovery, and more.

Motivation

During our consultancy and support to the community, commercial clients, and enterprises we have seen multiple requests and desires (a few of which are cited above) on why they want to split their deployments across multiple clusters while maintaining mutual dependencies between workloads operating in those clusters. Currently, the cluster is a hard boundary, and service is opaque to a distant K8s consumer who may otherwise use metadata (e.g. endpoint topology) to better direct traffic. Users may want to use services distributed across clusters to support failover or temporarily during migration, however, this needs non-trivial customized solutions today.

Flomesh team aims to help the community by providing solutions to these problems.

Goals

• Define a minimal API to support service discovery and consumption across clusters.
  • Consume a service in another cluster.
  • Consume a service deployed in multiple clusters as a single service.
• When a service is consumed from another cluster its behavior should be predictable and consistent with how it would be consumed within its cluster.
• Allow gradual rollout of changes in a multi-cluster environment.
• Provide a stand-alone implementation that can be used without any coupling to any product and/or solution.
• Transparent integration with osm-edge service mesh, for users who want to have multi-cluster support with service mesh functionality.
• Fully open source and welcomes the community to participate and contribute.

Architecture

• Control plane

• osm-edge integration (managed cluster)

FSM provides a set of Kubernetes custom resources (CRD) for cluster connector, and make use of KEP-1645 ServiceExport and ServiceImport API for exporting and importing services.

FSM multi-cluster support is already in the alpha stage and is available on FSM Github, you are welcome to give it a try.

In the upcoming part of this series, we will be exploring FSM multi-cluster functionality via hands-on tutorial and demo, so stay tuned.

Summary

Kubernetes is no longer constrained to a single boundary, and use cases of having multiple Kubernetes clusters are becoming quite common. There wasn't a typical or standard way of connecting numerous clusters and providing a transparent means to access services across boundaries. Kubernetes multi-cluster special interest group is working on proposals to provide a common set of APIs that can be implemented by vendors and provide a consistent way for k8s users to access services within or across boundaries.

Flomesh Service Mesh(FSM) from Flomesh is a Kubernetes North-South traffic manager, that provides Ingress controllers, Gateway API, Load Balancer, and cross-cluster service registration and service discovery. FSM uses Pipy - a programmable network proxy, as its data plane and is suitable for cloud, edge, and IoT.