DEV Community: Michael Laccetti

Building a Kubernetes-based Solution in a Hybrid Environment by Using KubeMQ

Michael Laccetti — Thu, 11 Mar 2021 18:42:02 +0000

Introduction

As Kubernetes has become the de-facto standard for deploying, managing and scaling container-based workloads, it has introduced features to allow clusters to exist across multiple clouds, span clouds, and on-premises environments, and even connect clouds and edge computing deployments. Working across clouds and edges has been done via the Kubernetes native confederation project or third-party control planes that allow managing discrete clusters from one dashboard.

However, Kubernetes is a platform for deploying applications and isn’t inherently aware of location, topologies, or architectural distribution. While it can provide scalability and connect clouds to the edge, it falls to the application that is built and deployed on Kubernetes to work within that environment.

Requiring a developer to be aware of that level of detail pushes overhead and complexity too far into the development lifecycle. This is where a Messaging Platform abstraction layer comes into play.

Let’s look at one solution to this problem, KubeMQ.

KubeMQ

KubeMQ is a Kubernetes-native messaging platform that was specifically designed for operating in these various scenarios; it excels at connecting clusters and nodes across different locations. KubeMQ leverages Operators to ensure that deployment is painless and leverages proven technologies, then uses Bridges to connect different Kubernetes environments together, whether they are federations, discrete clusters, or span clouds and edges.

Cross/Hybrid Cloud Bridges

Two of the more common approaches to deploying Kubernetes in hybrid environments are from cloud-to-cloud and cloud to on-prem. Whether this is from using a single control plane like Rancher, Platform9, or Gardener to create multiple clusters that are managed from a single location, or utilizing Kubernetes federation to create a cluster that spans different regions, this model has become a key feature offered by Kubernetes that has helped drive adoption.

However, the ability to create hybrid deployments comes with additional complexity, especially when managing applications that are deployed across these clusters and the need for services to communicate with one another.

For example, imagine a scenario where most of the transactional systems are hosted in AWS. They were originally built utilizing Lambdas and Fargate, and there is no appetite to move them. However, the organization has built out most of the analytical capabilities within Google’s Cloud Platform since the data team was more familiar with BigQuery and Data Studio. This new data platform is built utilizing GKE - Google’s managed Kubernetes offering - and is combined with native ingestion of Google Analytics information.

Modifying the existing systems in such a way that they would be aware of the partition would add complexity and overhead. Instead, it makes more sense to externalize that and build on top of a platform that will distribute messages and data across multiple clouds without changing the system itself, which is where KubeMQ comes into play - helping to bridge clouds without the systems having awareness.

KubeMQ Bridges offer many different ways of transferring information from one cluster to another via topologies. KubeMQ Bridges support 1:1 messaging across clusters, replication for 1:n, aggregation for n:1, and transformation for m:n messaging. Of course, KubeMQ isn’t limited to simply message delivery but also for message transformation as part of cross-cluster communication.

KubeMQ Bridges - Binding Multiple Clusters Together

This allows for the construction of architectures where information can be loaded from databases, caches, and services in one cluster then distributed to another cluster for further processing and storage. This only requires some YAML configuration for the KubeMQ Bridge.

For example, here is the same configuration for a Bridge with the Sources (within the current cluster) and Targets (in external clusters). It takes information from a source RPC query on one cluster and sends it to the target RPC destination for processing:

bindings:
  - name: clusters-sources # unique binding name
    properties: # Bindings properties such middleware configurations
      log_level: error
      retry_attempts: 3
      retry_delay_milliseconds: 1000
      retry_max_jitter_milliseconds: 100
      retry_delay_type: "back-off"
      rate_per_second: 100
    sources:
      kind: source.query # Sources kind
      name: name-of-sources # sources name 
      connections: # Array of connections settings per each source kind
        - .....
    targets:
      kind: target.query # Targets kind
      name: name-of-targets # targets name
      connections: # Array of connections settings per each target kind
        - .....

KubeMQ’s value isn’t limited to connecting large clusters and architectures together, however. It can also be used in environments where data needs to be collected from a variety of devices where computation needs to be performed as close as possible, such as the edge.

Edge Computing

Kubernetes has also been making inroads in connecting cloud environments with the edges, specifically, deployments where computation moves as close to the user as possible.

Typically, edge computing nodes are more resource-constrained, leading to different resource usage patterns where part of the workload occurs as close to the end-user as possible. Heavier or less time-constrained processing happens upstream in the cloud.

A sample edge computing scenario that could benefit from better-connected topologies is modern security cameras like the Eufy Indoor Cam 2k. AI-driven functionality is integrated into the hardware to help classify the different types of action being monitored, such as whether a human or pet is within the frame.

However, the actual video storage and transcoding to different formats cannot take place on the device itself and requires computational resources. Building a stream-based architecture where the edge node is close to the consumer will result in a better user experience, leading to a conversion to a paying subscription.

KubeEdge is a Cloud Native Computing Foundation (CNCF) project that is designed to operate in this space. It helps deploy container-based applications in the cloud and at the edge and also integrates IoT devices that support the MQTT protocol. (Note: MQTT is actually the name of the protocol, not an acronym.)

KubeEdge and MQTT for Cloud-Edge Computing

KubeMQ was also designed to operate in this environment, on both sides of the KubeEdge deployment within the cloud and on the edge nodes. KubeMQ has extremely low resource consumption, which allows it to run on Edge-specific Kubernetes distributions, including K3s and MicroK8s. This allows KubeMQ to create a bridge between edge nodes and more powerful computer nodes running in a cloud, pushing messages upstream as required.

KubeMQ also provides a source to process MQTT messages directly from the broker, allowing IoT devices to participate in a larger microservices-based architecture without additional effort. This allows users to create systems that can collect and aggregate information from tens of thousands of hardware devices (or more, due to the scalable nature of Kubernetes and KubeMQ) and build near real-time capabilities right at the edge.

Here’s an example of creating an MQTT source within KubeMQ that will allow the processing of events from an MQTT broker. In this configuration, messages from the local MQTT broker will be consumed and sent in real-time to an event subscriber for consumption:

bindings:
  - name: mqtt-kubemq-event
    source:
      kind: messaging.mqtt
      name: mqtt-source
      properties:
        host: "localhost:1883"
        dynamic_map: "true"
        topic: "queue"
        username: "username"
        password: "password"
        client_id: "client_id"
        qos: "0"
    target:
      kind: kubemq.events
      name: target-kubemq-events
      properties:
        address: "kubemq-cluster:50000"
        client_id: "kubemq-http-connector"
        channel: "events.mqtt"
    properties:
      log_level: "info"

Using KubeMQ as a messaging platform allows for the seamless creation and management of systems that operate at the edge and helps ensure that projects will not stall when addressing resource-constrained deployments or environments.

Conclusion

With Kubernetes becoming the de facto container deployment system, there has been a shift to hybrid cloud computing. Whether that is implemented as a single control plane managing multiple clusters or creating a federated cluster that operates across regions, many organizations are looking into hybrid clouds for the future.

In fact, at least two-thirds of CIOs have stated an intent to utilize hybrid clouds: to maximize savings or increase resiliency. Other organizations have embraced deployments that span clouds and on-prem hardware to add capacity, handle privacy or security concerns, or deal with scenarios where a core system must remain on-prem, but the rest of the services can be deployed elsewhere.

In addition, there are more organizations where they are shipping devices where computational resources are co-located with the customer, creating scenarios where organizations need to bridge their cloud deployments with their edge computing needs.

Having a messaging platform that was purposefully built utilizing the underlying Kubernetes platform is a crucial component to a successful deployment and operational model in these environments. KubeMQ bridges allow the creation of microservice-based architectures that span on-prem, clouds, and edge devices, helping to create message-driven systems that can scale up reliably and guarantee success.

How KubeMQ customers build scalable messaging platforms with Kubernetes Operators

Michael Laccetti — Thu, 04 Mar 2021 16:03:11 +0000

Over the last several years, the adoption of Kubernetes has increased tremendously. In fact, according to a Cloud Native Computing Foundation (CNCF) survey, 78% of respondents in late 2019 were using Kubernetes in production. Leveraging Kubernetes allows organizations to create a management layer to commodify clouds themselves and build cross- or hybrid-cloud deployments that hide the provider-specific implementation details from the rest of the team.

One crucial part of the Kubernetes ecosphere is Operators — a tool initially introduced by CoreOS in 2016 to utilize the Kubernetes APIs themselves to deploy and manage the state of applications. Operators are a critical part of the deployment and operation of applications in a cross/hybrid environment. They can help manage and maintain state across a federated Kubernetes deployment (multiple Kubernetes clusters running together) or even across clusters.

But what exactly are Operators, and how do they help manage these stateful applications? Let’s take a look at Operators in detail, how they work within Kubernetes, and how the KubeMQ messaging platform uses Operators to help you build complex and scalable messaging services with minimal coding and overhead.

What are Operators?

At a high level, Operators allow you to automate tasks beyond what Kubernetes natively provides. Operators are software extensions that hook into Kubernetes APIs and the control plane to manage a custom resource (CR) — or an extension to the Kubernetes API. The CR describes the desired state of the application, and the control plane component (the Operator itself) monitors the CR to ensure that the application is running as expected.

For example, an Operator might deploy and scale a pod, take and restore a backup, manage network services and ingresses, or manage a persistent data store.

Since Operators hook into native Kubernetes tools like kubectl, they become a common language for managing complex deployments where state is involved. Using a Helm chart is excellent for deploying and managing a stateless application like a web server. Still, for deploying stateful systems like etcd, PostgreSQL, and KubeMQ, Operators are the key to success.

KubeMQ — Kubernetes-Native Messaging Platform

Before we dive into understanding why Operators are an essential ingredient to KubeMQ’s success, let’s first examine the benefits of KubeMQ.

At a high level, KubeMQ isn’t just a message broker or queue but a messaging platform that can be used to build a message-based architecture that works across multi- or hybrid-clouds and edge computing. KubeMQ allows services in these environments to communicate with each other in any messaging pattern — pub/sub, streams, queues, and so on. KubeMQ is Kubernetes Native and is easy and quick to deploy in less than a minute.

Core components

The KubeMQ messaging platform comprises four core components — server, bridges, sources, and targets.

KubeMQ server is deployed to each Kubernetes cluster, which handles the message processing.
KubeMQ bridges help to build the desired messaging topology across clusters, sending messages between KubeMQ servers. ( KubeMQ sources ingest messages from existing systems like RabbitMQ or Kafka into the KubeMQ platform.
KubeMQ targets send messages to external systems, like Redis, PostgreSQL, S3, or other messaging systems like ActiveMQ.

Combining all of these components allows for creating a cross-cluster messaging and no-code-message-based-microservices architecture on Kubernetes.

How Does KubeMQ Use Operators to Succeed?

So how does KubeMQ work in the world of Operators?

First, KubeMQ deploys as an Operator to ensure that it can operate at a native Kubernetes level. One of the tenets of utilizing KubeMQ is that it is better to create many small clusters and bind them together rather than creating one massive cluster.

This allows for better performance, scalability, and resilience. One key to success with this approach is utilizing Operators as the deployment and management tool for KubeMQ. The Operator deploys the clusters and ensures that the various KubeMQ bridges, sources, and targets are configured correctly for each cluster. This extends to how KubeMQ was written utilizing Go. This makes KubeMQ fast and helps hook KubeMQ into native Kubernetes data models, events, and APIs, making it less complicated to manage the state of the clusters. It also allows for easier configuration validation.

Deploying as an operator also helps KubeMQ keep overhead to a minimum. For example, a large financial company with high volumes of real-time messages for price quotes, transactions, and client funding leverages KubeMQ to decrease the number of servers previously required to fulfill their needs. It has also allowed the company to reallocate the operational overhead to better tasks, rather than monitoring and maintaining messaging infrastructure. Similarly, the company has leveraged the KubeMQ operator to help elastically scale their infrastructure based on the load. For example, when markets close, demand drops, and the clusters can scale down accordingly.

Cross/Hybrid Cloud and Operators

The KubeMQ Operator also helps track state — a key reason for leveraging KubeMQ for reliable cross/hybrid cloud deployments. First, this state can validate that the desired capacity and configuration are in place for each cluster. Comparing the desired state in the CR against the existing state in Kubernetes allows the Operator to ensure that failures are caught and addressed, capacity is added as required, and the various bridges, sources, and targets are configured.

One KubeMQ customer in the agricultural vertical heavily leverages this feature of KubeMQ to run their messaging platform across edge computing systems along with cloud deployments. This provides better performance and reliability and allows them to grow their business with new services without interruptions or downtime. They simply create new clusters and configure them as required. Knowing that the KubeMQ operator validates the CR definition helps to prevent deploying clusters with faulty configurations.

Conclusion

Operators are critical for managing and deploying complex Kubernetes systems.

They operate utilizing Kubernetes-native APIs and systems.
An Operator’s CR definition provides configuration management and validation at deployment time.
They track the desired state against the existing state and act accordingly if there are any deltas between them.
Operators are well-known and understood at most organizations, ensuring that no additional training is required for operational success,

KubeMQ heavily leverages the Operator model to help customers succeed in building out complex and scalable messaging platforms with minimal coding and overhead. This helps to deliver greater business value by allowing organizations to solve business problems quickly and efficiently without spending time and resources on managing and maintaining messaging infrastructure.

Bringing Your (Encryption) Keys to Multi/Hybrid Clouds

Michael Laccetti — Tue, 08 Sep 2020 02:07:52 +0000

Tools and Setup

Before we dive into the fun part of getting keys shared amongst cloud providers, there are a variety of tools required to get this tutorial working. First, you’ll need to download and install Vault, then get it up and running. You will also need to install cURL and OpenSSL — these usually comes pre-installed with most Linux OSs, and are available via most package managers (apt, yum, brew, choco/scoop, etc.). Our examples also use head and diff which are part of the coreutils and diffutils packages under Ubuntu; you can either find a similar package for your OS or find a manual workaround for those portions. Next, install the AWS command line tools (CLI) and make sure you configure the CLI to connect to your account. The last step is to install and configure the Heroku CLI.

One last note — the Heroku feature to utilize keys from AWS requires a private or shield database plan, so please ensure your account has been configured accordingly.

Intro

In today’s hyperconnected world, the former approach of locking services behind Virtual Private Networks (VPNs) or within a demilitarized zone (DMZ) is no longer secure. Instead, we must operate on a zero-trust network model, where every actor must be assumed as malicious. This means that a focus on encryption — both at rest and in transit — along with identity and access management is critical to ensuring that systems can interact with each other.

One of the most important parts of the encryption process is the keys used to encrypt and decrypt information or used to validate identity. A recent approach to this need is called Bring Your Own Key (BYOK) — where you as the customer/end user own and manage your key, and provide it to third parties (notably cloud providers) for usage. However, before we dig into what BYOK is and how we can best leverage it, let’s have a quick recap on key management.

Key Management

At a high level, key management is the mechanism by which keys are generated, validated, and revoked — manually and as part of workflows. Another function of key management is ensuring that the root certificate that is used as a source of all truth is kept protected at a layer below other certificates, since revoking a root certificate would render the entire tree of certificates issued by it invalid.

One of the more popular tools used for key management is HashiCorp’s Vault — specifically designed for a world of low trust and dynamic infrastructure, where key ages can be measured in minutes or hours, rather than years. It includes functionality to manage secrets, encryption, and identity-based access, provides many ways to interact with it (CLI, API, web-based UI), and can connect to many different providers through plugins. This article will not focus on how to deploy Vault in a secure fashion, but the use cases that Vault can offer around BYOK and now to consume the keys in multiple cloud environments.

A key feature of using Vault is that it functions in an infrastructure- and provider-agnostic fashion — it can be used to provision and manage keys across different systems and clouds. At the same time, Vault can be used to encrypt and decrypt information without exposing keys to users, allowing for greater security.

BYOK on Multi Cloud

At this point, we’d like to dive into a specific use-case — demonstrating how you can create and ingest your own keys into multiple clouds, focusing on Amazon Web Services (AWS) and Heroku. For our purposes, we’ll start with uploading our keys to AWS KMS using Amazon’s CLI and demonstrating how the keys can be used within AWS. We will then rotate the keys manually — ideally, this is automated in a production implementation. Finally, we will utilize the generated keys in Heroku to encrypt a Postgres database.

📝 Note

Before we get started, ensure that Vault is installed and running, and you have followed the Vault transit secrets engine setup guide, since we’ll use that to generate the keys we upload to AWS. Once you have validated that Vault is running and the AWS CLI is installed, it is time to get started.

Create Key in Vault

First, we need to create our encryption key within Vault:

vault write -f transit/keys/byok-demo-key exportable=true allow\_plaintext\_backup=true

To export the key, we can use the Vault UI (http://localhost:8200/ui/vault/secrets/transit/actions/byok-demo-key?action=export), or we’ll need to use cURL, since the Vault CLI doesn’t support exporting it directly. The 127.0.0.1 address maps to the Vault server — a production setup would not be localhost, nor unencrypted.

curl — header “X-Vault-Token: <token>” [http://127.0.0.1:8200/v1/transit/export/encryption-key/byok-demo-key/1](http://127.0.0.1:8200/v1/transit/export/encryption-key/byok-demo-key/1)

This command will output a base64 encoded plaintext version of the key which we will upload to AWS KMS. Save the base64 plaintext key in a file — we used vault_key.b64.

Upload Key to AWS KMS

Now, we need to generate a Customer Master Key (CMK) with no key material:

\# create the keyaws kms create-key — origin EXTERNAL — description “BYOK Demo” — key-usage ENCRYPT\_DECRYPT\# give it a nice nameaws kms create-alias — alias-name alias/byok-demo-key — target-key-id <from above>

Copy down the key ID from the output, and download the public key and import token:

aws kms get-parameters-for-import — key-id <from above> — wrapping-algorithm RSAES\_OAEP\_SHA\_1 — wrapping-key-spec RSA\_2048

Copy the public key and import token from the output of the above step into separate files (imaginatively, we used import_token.b64 and public_key.b64 as the filenames), then base64 decode them:

openssl enc -d -base64 -A -in public\_key.b64 -out public\_key.binopenssl enc -d -base64 -A -in import\_token.b64 -out import\_token.bin

With the import token and public key downloaded, we can now use them to wrap the key from vault, first by converting the key to the OpenSSL byte format, then encrypting it using the public key from KMS.

\# convert the vault key to bytesopenssl enc -d -base64 -A -in vault\_key.b64 -out vault\_key.bin\# encrypt the vault key with the KMS keyopenssl rsautl -encrypt -in vault\_key.bin -oaep -inkey public\_key.bin -keyform DER -pubin -out encrypted\_vault\_key.bin\# import the encrypted key into KMSaws kms import-key-material — key-id <from above> — encrypted-key-material fileb://encrypted\_vault\_key.bin — import-token fileb://import\_token.bin — expiration-model KEY\_MATERIAL\_EXPIRES — valid-to $(date — iso-8601=ns — date=’364 days’)

Now that the key has been uploaded, we can quickly encrypt and decrypt via the CLI to validate that the key is functioning properly:

\# generate some random text to a filehead /dev/urandom | tr -dc A-Za-z0–9 | head -c 1024 > encrypt\_me.txt\# encrypt the fileaws kms encrypt — key-id <from above> — plaintext fileb://encrypt\_me.txt — output text — query CiphertextBlob | base64 — decode > encrypted\_file.bin\# decrypt the fileaws kms decrypt — key-id <from above> — ciphertext-blob fileb://encrypted\_file.bin — output text — query Plaintext | base64 — decode > decrypted.txt\# validate they match; should be a blank linediff encrypt\_me.txt decrypted.txt

At this point, the key is in place and can be used to encrypt data at rest or in transit in different parts of AWS. One of the easiest and most important places to encrypt data is in S3; files sitting around in storage mandate being encrypted. When creating a bucket, make sure you enable server-side encryption, select KMS as the key type, then choose the specific key you created from the KMS master key dropdown:

At this point, any objects that are put in the bucket will automatically be encrypted and then decrypted when they are read. This same approach can be used for encrypting databases — for a complete list of the different services that integrate with KMS, you can see the list here.

Use KMS Key in Heroku

Our next step is to use the key we generated and uploaded to AWS in Heroku! For our purposes, we’ll encrypt a Postgres database during creation. To do this, we need to grant Heroku’s AWS account access to the key that we created. This can be done via the AWS UI when creating a key, via the CLI, or via a policy.

During the key creation wizard in AWS, step four will ask to define key usage permissions; there will be a separate section that allows you to add AWS account IDs to the key:

Type in the Heroku AWS account ID (021876802972) and finish the wizard. If you want to use the CLI to achieve this, you need to update the default policy for the key:

\# get the existing key policyaws kms get-key-policy — policy-name default — key-id <from above> — output text

Save the output from the above into a text file called heroku-kms-policy.json and add the following two statements:

{
  "Sid": "Allow use of the key",
  "Effect": "Allow",
  "Principal": {
    "AWS": "arn:aws:iam::021876802972:root"
  },
  "Action": [
    "kms:Encrypt",
    "kms:Decrypt",
    "kms:ReEncrypt\*",
    "kms:GenerateDataKey\*",
    "kms:DescribeKey"
  ],
  "Resource": "\*"
},
{
  "Sid": "Allow attachment of persistent resources",
  "Effect": "Allow",
  "Principal": {
    "AWS": "arn:aws:iam::021876802972:root"
  },
  "Action": [
    "kms:CreateGrant",
    "kms:ListGrants",
    "kms:RevokeGrant"
  ],
  "Resource": "*",
  "Condition": {
    "Bool": {
      "kms:GrantIsForAWSResource": "true"
    }
  }
}

Now, update the existing policy with the new statements:

aws kms put-key-policy — key-id <from above> — policy-name default — policy file://heroku-kms-policy.json

There is no output from the above, so re-run the get-key-policy command to validate that it worked.

To update the policy via the UI, browse to the key under the “Customer managed keys” section in the AWS KMS console and edit to add the two statements from above. When switching from policy to default view, the “Other AWS accounts” section should show up:

With Heroku granted access to the key, we now can use the Heroku CLI to attach a Postgres database to an app, specifying the key ARN (which should look something like arn:aws:kms:region:account ID:key/key ID):

heroku addons:create heroku-postgresql:private-7 — encryption-key CMK\_ARN — app your-app-name

You can find a full set of documentation on encrypting your Postgres database on Heroku, including information on how to encrypt an already existing database.

Cleaning Up

At this point, given the cost of running a private or shield database in Heroku, you may want to delete any resources you have created. Similarly, removing the keys from AWS is suggested, since it will take seven days for them to be fully deprovisioned and deleted. Quitting the Vault dev server will remove all information, since the dev instance is ephemeral.

Considerations

While there is a lot to like about the BYOK approach to AWS and Heroku, there are a few considerations that need to be highlighted:

While Vault allows for key rotation, it requires external automation — cron, a CI pipeline, Nomad/Kubernetes jobs, etc.
Similarly, customer managed keys in AWS also require manual rotation.
If there is a catastrophic failure in AWS, it is possible that the key will need to be re-imported; automating, testing, and validating this should be part of any production use case.
If keys are deleted in AWS while being used in Heroku, all services and servers that depend on that key will be shutdown.

Conclusion

With all of this in place, we have demonstrated that maintaining local ownership of your encryption keys is both possible and desirable, and that ownership can then be extended into various cloud providers. This article only scratched the surface of the functionality offered by Vault, KMS, and Heroku — Vault itself can be used to manage asymmetric encryption as well, helping to protect data in transit along with identity validation. KMS is a cornerstone for encryption in AWS, and hooks into most services, allowing for an effortless way to ensure data is kept secure. Finally, Heroku’s ability to consume KMS keys directly allows for an additional level of security without adding management overhead.