Building a Kubernetes-based Solution in a Hybrid Environment by Using KubeMQ

Introduction

As Kubernetes has become the de-facto standard for deploying, managing and scaling container-based workloads, it has introduced features to allow clusters to exist across multiple clouds, span clouds, and on-premises environments, and even connect clouds and edge computing deployments. Working across clouds and edges has been done via the Kubernetes native confederation project or third-party control planes that allow managing discrete clusters from one dashboard.

However, Kubernetes is a platform for deploying applications and isn’t inherently aware of location, topologies, or architectural distribution. While it can provide scalability and connect clouds to the edge, it falls to the application that is built and deployed on Kubernetes to work within that environment.

Requiring a developer to be aware of that level of detail pushes overhead and complexity too far into the development lifecycle. This is where a Messaging Platform abstraction layer comes into play.

Let’s look at one solution to this problem, KubeMQ.

KubeMQ

KubeMQ is a Kubernetes-native messaging platform that was specifically designed for operating in these various scenarios; it excels at connecting clusters and nodes across different locations. KubeMQ leverages Operators to ensure that deployment is painless and leverages proven technologies, then uses Bridges to connect different Kubernetes environments together, whether they are federations, discrete clusters, or span clouds and edges.

Cross/Hybrid Cloud Bridges

Two of the more common approaches to deploying Kubernetes in hybrid environments are from cloud-to-cloud and cloud to on-prem. Whether this is from using a single control plane like Rancher, Platform9, or Gardener to create multiple clusters that are managed from a single location, or utilizing Kubernetes federation to create a cluster that spans different regions, this model has become a key feature offered by Kubernetes that has helped drive adoption.

However, the ability to create hybrid deployments comes with additional complexity, especially when managing applications that are deployed across these clusters and the need for services to communicate with one another.

For example, imagine a scenario where most of the transactional systems are hosted in AWS. They were originally built utilizing Lambdas and Fargate, and there is no appetite to move them. However, the organization has built out most of the analytical capabilities within Google’s Cloud Platform since the data team was more familiar with BigQuery and Data Studio. This new data platform is built utilizing GKE - Google’s managed Kubernetes offering - and is combined with native ingestion of Google Analytics information.

Modifying the existing systems in such a way that they would be aware of the partition would add complexity and overhead. Instead, it makes more sense to externalize that and build on top of a platform that will distribute messages and data across multiple clouds without changing the system itself, which is where KubeMQ comes into play - helping to bridge clouds without the systems having awareness.

KubeMQ Bridges offer many different ways of transferring information from one cluster to another via topologies. KubeMQ Bridges support 1:1 messaging across clusters, replication for 1:n, aggregation for n:1, and transformation for m:n messaging. Of course, KubeMQ isn’t limited to simply message delivery but also for message transformation as part of cross-cluster communication.

KubeMQ Bridges - Binding Multiple Clusters Together

This allows for the construction of architectures where information can be loaded from databases, caches, and services in one cluster then distributed to another cluster for further processing and storage. This only requires some YAML configuration for the KubeMQ Bridge.

For example, here is the same configuration for a Bridge with the Sources (within the current cluster) and Targets (in external clusters). It takes information from a source RPC query on one cluster and sends it to the target RPC destination for processing:

bindings:
  - name: clusters-sources # unique binding name
    properties: # Bindings properties such middleware configurations
      log_level: error
      retry_attempts: 3
      retry_delay_milliseconds: 1000
      retry_max_jitter_milliseconds: 100
      retry_delay_type: "back-off"
      rate_per_second: 100
    sources:
      kind: source.query # Sources kind
      name: name-of-sources # sources name 
      connections: # Array of connections settings per each source kind
        - .....
    targets:
      kind: target.query # Targets kind
      name: name-of-targets # targets name
      connections: # Array of connections settings per each target kind
        - .....

KubeMQ’s value isn’t limited to connecting large clusters and architectures together, however. It can also be used in environments where data needs to be collected from a variety of devices where computation needs to be performed as close as possible, such as the edge.

Edge Computing

Kubernetes has also been making inroads in connecting cloud environments with the edges, specifically, deployments where computation moves as close to the user as possible.

Typically, edge computing nodes are more resource-constrained, leading to different resource usage patterns where part of the workload occurs as close to the end-user as possible. Heavier or less time-constrained processing happens upstream in the cloud.

A sample edge computing scenario that could benefit from better-connected topologies is modern security cameras like the Eufy Indoor Cam 2k. AI-driven functionality is integrated into the hardware to help classify the different types of action being monitored, such as whether a human or pet is within the frame.

However, the actual video storage and transcoding to different formats cannot take place on the device itself and requires computational resources. Building a stream-based architecture where the edge node is close to the consumer will result in a better user experience, leading to a conversion to a paying subscription.

KubeEdge is a Cloud Native Computing Foundation (CNCF) project that is designed to operate in this space. It helps deploy container-based applications in the cloud and at the edge and also integrates IoT devices that support the MQTT protocol. (Note: MQTT is actually the name of the protocol, not an acronym.)

KubeEdge and MQTT for Cloud-Edge Computing

KubeMQ was also designed to operate in this environment, on both sides of the KubeEdge deployment within the cloud and on the edge nodes. KubeMQ has extremely low resource consumption, which allows it to run on Edge-specific Kubernetes distributions, including K3s and MicroK8s. This allows KubeMQ to create a bridge between edge nodes and more powerful computer nodes running in a cloud, pushing messages upstream as required.

KubeMQ also provides a source to process MQTT messages directly from the broker, allowing IoT devices to participate in a larger microservices-based architecture without additional effort. This allows users to create systems that can collect and aggregate information from tens of thousands of hardware devices (or more, due to the scalable nature of Kubernetes and KubeMQ) and build near real-time capabilities right at the edge.

Here’s an example of creating an MQTT source within KubeMQ that will allow the processing of events from an MQTT broker. In this configuration, messages from the local MQTT broker will be consumed and sent in real-time to an event subscriber for consumption:

bindings:
  - name: mqtt-kubemq-event
    source:
      kind: messaging.mqtt
      name: mqtt-source
      properties:
        host: "localhost:1883"
        dynamic_map: "true"
        topic: "queue"
        username: "username"
        password: "password"
        client_id: "client_id"
        qos: "0"
    target:
      kind: kubemq.events
      name: target-kubemq-events
      properties:
        address: "kubemq-cluster:50000"
        client_id: "kubemq-http-connector"
        channel: "events.mqtt"
    properties:
      log_level: "info"

Using KubeMQ as a messaging platform allows for the seamless creation and management of systems that operate at the edge and helps ensure that projects will not stall when addressing resource-constrained deployments or environments.

Conclusion

With Kubernetes becoming the de facto container deployment system, there has been a shift to hybrid cloud computing. Whether that is implemented as a single control plane managing multiple clusters or creating a federated cluster that operates across regions, many organizations are looking into hybrid clouds for the future.

In fact, at least two-thirds of CIOs have stated an intent to utilize hybrid clouds: to maximize savings or increase resiliency. Other organizations have embraced deployments that span clouds and on-prem hardware to add capacity, handle privacy or security concerns, or deal with scenarios where a core system must remain on-prem, but the rest of the services can be deployed elsewhere.

In addition, there are more organizations where they are shipping devices where computational resources are co-located with the customer, creating scenarios where organizations need to bridge their cloud deployments with their edge computing needs.

Having a messaging platform that was purposefully built utilizing the underlying Kubernetes platform is a crucial component to a successful deployment and operational model in these environments. KubeMQ bridges allow the creation of microservice-based architectures that span on-prem, clouds, and edge devices, helping to create message-driven systems that can scale up reliably and guarantee success.