DEV Community: Benjamen Pyle

Linkerd Service Mesh on AWS EKS

Benjamen Pyle — Sun, 31 Aug 2025 14:39:29 +0000

I've been writing for a while now on setting up Kubernetes specifically with AWS' EKS. I love how EKS gives me the flexibility to install the standard and custom Kubernetes resources that I need and want for my projects. In this article, I want to explore setting up Linkerd, a service mesh for Kubernetes, on EKS. Linkerd is a popular service mesh that provides a set of features such as service discovery, load balancing, and traffic management. By using Linkerd, we can easily manage and monitor our Kubernetes applications and ensure that they are running smoothly. I did some light comparisons a while back between Istio and Linkerd in this article. But let's dive in a little bit deeper and setup Linkerd on EKS!

Setting up Linkerd on EKS

Before I dig into the article, everything that's in here is stored in this GitHub repository. Feel free to clone it and follow along!

Configuring the EKS Cluster

I love running EKS. It takes care of all the heavy lifting for me, allowing me to focus on building and deploying my applications. My preferred way of configuring the EKS cluster is to use the AWS CLI and the eksctl tool. I will supply a configuration file to the eksctl create cluster command so that I can manage the settings of the cluster in file, but update via the CLI.

The below YAML file will create a cluster called sandbox in the us-west-2 region with a managed Node Group that runs Amazon Linux 2 on Graviton and has 6 vCPUs and 16 GB of memory.

---
apiVersion: eksctl.io/v1alpha5
kind: ClusterConfig

metadata:
  name: sandbox
  region: us-west-2

managedNodeGroups:
  - name: mng-arm
    instanceType: m6g.large
    desiredCapacity: 2

I can then run this command to create the cluster:

eksctl create cluster -f cluster.yaml

Deploying Linkerd

I love deploying applications in Kubernetes either software I've produced or community applications such as Linkerd. For convenience, I've created a shell script for deploying Linkerd. The script includes:

Installing the Linkerd tooling
Deploying the Kubernetes CRDs
Installing the Linkerd control plane
Installing the Linkerd visualization dashboard

Launching the script looks like this:

./deploy-linkerd.sh

Once the script is complete, I can verify that Linkerd is running, including the Viz Dashboard:

Deploying Applications

For my example services, I've built 2 APIs that are running as Kubernetes ReplicaSets. They are called the Comms Service and the Greeter Service. To get this working, I'm opting to just forward the port of the Comms Service instead of setting up a full ingress with public availability. I wanted to be able to just get testing with Linkerd vs building out something a little more complex.

Adding the Linkerd proxy to my services is a simple as adding an annotation on the Service itself.

apiVersion: v1
kind: Service
metadata:
  name: comms-service
  annotations:
  # Adding in Linkerd
    linkerd.io/inject: enabled
spec:
  selector:
    app: comms-service
  ports:
    - protocol: TCP
      port: 3000
      targetPort: 3000

As a visual reference, here's a diagram showing how the services relate and bring Linkerd into the mix.

With everything in place, I can now set up the port forward and then execute a cURL command to run the Comms Service API.

kubectl port-forward comms-service 3000:3000 -n greeter

> curl -v http://localhost:3000/say-hello\?name\=Benjamen | json_pp
* Host localhost:3000 was resolved.
* IPv6: ::1
* IPv4: 127.0.0.1
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0*   Trying [::1]:3000...
* Connected to localhost (::1) port 3000
> GET /say-hello?name=Benjamen HTTP/1.1
> Host: localhost:3000
> User-Agent: curl/8.7.1
> Accept: */*
>
* Request completely sent off
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0< HTTP/1.1 200 OK
< content-type: application/json
< vary: origin, access-control-request-method, access-control-request-headers
< access-control-allow-origin: *
< access-control-expose-headers: *
< content-length: 29
< date: Sun, 31 Aug 2025 14:11:15 GMT
<
{ [29 bytes data]
100    29  100    29    0     0    110      0 --:--:-- --:--:-- --:--:--   110
* Connection #0 to host localhost left intact
{
   "message" : "Hello, Benjamen"
}

I have the Access Logs for Linkerd turned on and emitting them in JSON format. I'm then able to run kubectl to fetch logs on the Greeter Service to see what I'm getting.

kubectl logs -n greeter -l app=greeter-service -c linkerd-proxy --tail=1 | jq .
{
  "client.addr": "192.168.36.87:36930",
  "client.id": "default.greeter.serviceaccount.identity.linkerd.cluster.local",
  "host": "greeter-service.greeter.svc.cluster.local:3000",
  "method": "GET",
  "processing_ns": "270323",
  "request_bytes": "",
  "response_bytes": "29",
  "status": 200,
  "timestamp": "2025-08-31T14:10:58.256786757Z",
  "total_ns": "1534820",
  "trace_id": "",
  "uri": "http://greeter-service.greeter.svc.cluster.local:3000/hello?name=Benjamen",
  "user_agent": "",
  "version": "HTTP/2.0"
}

All of that looks amazing!

Next Steps

First, I'd recommend that if you followed along, you should clean up the resources you created. You can do this by running the following command:

./script/cleanup-all.sh

As for Linkerd, I've barely scratched the surface of what I can accomplish with it. The following capabilities are what I'd recommend exploring next (and I just might in future posts):

These 3 topics would be a great next step for adding robustness to a Kubernetes managed application. What I love about using a Service Mesh is that I don't have to hand code or pick libraries to handle additional features. I get this at the infrastructure level so developers (like myself) can focus on building great applications.

Wrapping Up

Linkerd is a powerful Service Mesh that comes in a very simple and non-complex package. Some of the knocks on Meshes are that they have this inherent level of complexity to configure and just get running. I hope you've seen from this more bite-sized article that setting up Linkerd is extremely straightforward and can have minimal configuration. Sure, things will get more in-depth as you add more features, but that's how I'd like it to be. Easy to setup. Advanced when I need it.

In future articles, I hope to explore some of the more advanced features and how I can add additional capabilities to my Kubernetes applications without having to write any more code.

Thanks for reading and Happy Building!

KEDA to build Event-Driven Applications on EKS

Benjamen Pyle — Wed, 20 Aug 2025 00:16:00 +0000

Event-driven applications aren't new, but the patterns and discussion in the context of the cloud are hard to miss these days. It's hard to argue with the patterns and practices because with events, I can build systems that are more reliable, available, and tolerant of fault and issues that arise both in and out of my control. I've actually written quite a bit about building event-driven systems with serverless technologies like Lambda, SQS, SNS, EventBridge, and DynamoDB, but I don't want to explore familiar topics here. What if I took that a different direction and built an event-driven consumer inside a Kubernetes and more specifically Amazon Elastic Kubernetes Service (EKS) deployment? The building blocks would be the same, SQS for instance, but can I replace Lambda and how would that work? Let's dive into Event-Driven Pods with KEDA and EKS.

Architecture and Design

If you want to follow along or just want to get right to the details, here is the Github repository that has the working implementation that I'm going to go through below. There's a helpful README and a QuickStart so if you want to get going.

I'm doing something a little different for this article and using Claude Code to help me generate code and diagrams. I assure you the writing is still 100% mine, but spell- and grammar-checked again with my friend Claude. In that spirit, here's a diagram showing how the architecture I'm working on comes together.

┌───────────────────────────────────────────────────────────────────────────────┐
│                                    AWS Cloud                                  │
│                                                                               │
│  ┌────────────────────────────────────────────────────────────────────────┐   │
│  │                              Amazon EKS Cluster                        │   │
│  │                                                                        │   │
│  │  ┌──────────────────────────────────────────────────────────────────┐  │   │
│  │  │                        Kubernetes Control Plane                  │  │   │
│  │  │                         (Managed by AWS)                         │  │   │
│  │  └──────────────────────────────────────────────────────────────────┘  │   │
│  │                                    │                                   │   │
│  │  ┌──────────────────────────────────────────────────────────────────┐  │   │
│  │  │                           Worker Nodes                           │  │   │
│  │  │                                                                  │  │   │
│  │  │  ┌─────────────────┐    ┌─────────────────┐    ┌──────────────┐  │  │   │
│  │  │  │   keda namespace│    │ default         │    │ System Pods  │  │  │   │
│  │  │  │                 │    │                 │    │              │  │  │   │
│  │  │  │ ┌─────────────┐ │    │ ┌─────────────┐ │    │ • CoreDNS    │  │  │   │
│  │  │  │ │ KEDA        │ │    │ │   Rust SQS  │ │    │ • kube-proxy │  │  │   │
│  │  │  │ │ Operator    │ │◄───┤ │ Processor   │ │    │ • AWS Node   │  │  │   │
│  │  │  │ │             │ │    │ │ Deployment  │ │    │   Daemonset  │  │  │   │
│  │  │  │ └─────────────┘ │    │ │             │ │    └──────────────┘  │  │   │
│  │  │  │                 │    │ │ ┌─────────┐ │ │                      │  │   │
│  │  │  │ ┌─────────────┐ │    │ │ │ Pod 1   │ │ │                      │  │   │
│  │  │  │ │ Metrics     │ │    │ │ └─────────┘ │ │                      │  │   │
│  │  │  │ │ Server      │ │    │ │ ┌─────────┐ │ │                      │  │   │
│  │  │  │ └─────────────┘ │    │ │ │ Pod 2   │ │ │                      │  │   │
│  │  │  │                 │    │ │ └─────────┘ │ │                      │  │   │
│  │  │  │ ┌─────────────┐ │    │ │     ...     │ │                      │  │   │
│  │  │  │ │ Admission   │ │    │ │ ┌─────────┐ │ │                      │  │   │
│  │  │  │ │ Webhooks    │ │    │ │ │ Pod N   │ │ │                      │  │   │
│  │  │  │ └─────────────┘ │    │ │ └─────────┘ │ │                      │  │   │
│  │  │  └─────────────────┘    │ │             │ │                      │  │   │
│  │  │                         │ │ ┌─────────┐ │ │                      │  │   │
│  │  │                         │ │ │   HPA   │ │ │                      │  │   │
│  │  │                         │ │ └─────────┘ │ │                      │  │   │
│  │  │                         │ │             │ │                      │  │   │
│  │  │                         │ │ ┌─────────┐ │ │                      │  │   │
│  │  │                         │ │ │Scaled   │ │ │                      │  │   │
│  │  │                         │ │ │Object   │ │ │                      │  │   │
│  │  │                         │ │ └─────────┘ │ │                      │  │   │
│  │  │                         │ └─────────────┘ │                      │  │   │
│  │  │                         └─────────────────┘                      │  │   │
│  │  └──────────────────────────────────────────────────────────────────┘  │   │
│  └────────────────────────────────────────────────────────────────────────┘   │
└───────────────────────────────────────────────────────────────────────────────┘

The key elements of this solution to dig through and configure:

Kubernetes (EKS)
- Deployment with Docker Image built from source
KEDA
- Includes the KEDA Operator
- ScaledObject which is the resource that handles interacting with Metrics and Horizontal Pod Autoscaler
- TriggerAuthentication is a resource that allows our scaler to use the Pod's identity
IRSA which is IAM Roles for Service Accounts which let's our pods assume an IAM role and the policies attached.

Once deployed, I'm going to use the KEDA ScaledObject (Scaler) to monitor an AWS SQS for queue depth and then spin up pods in my Deployment that can handle those messages and do something with each payload. A very traditional competing consumers pattern implemented as a scale-to-zero deployment with Kubernetes.

SQS Queue ──► KEDA Operator ──► ScaledObject ──► HPA ──► Deployment ──► Pods
    │              │                │            │           │          │
    │              │                │            │           │          │
    └──────────────┼────────────────┼────────────┼───────────┼──────────┘
                   │                │            │           │
                   ▼                ▼            ▼           ▼
             Metrics Query    Scale Decision   Pod Count   Message 
            (Every 15s)      (Queue ÷ 5)     Adjustment  Processing

Timeline:
1. KEDA queries SQS queue depth every 15 seconds
2. If queue has messages: KEDA calculates desired pods = ceiling(messages ÷ 5)
3. KEDA updates HPA with external metrics
4. HPA scales deployment up/down
5. New pods start and begin processing messages
6. When queue is empty, pods scale down after 300s cooldown

Event-Driven Pods with KEDA and EKS

Foundationally, I could start with the cluster or with the SQS. I tried to keep things granular so they could be run separately but not so much that they didn't make sense. One big script to do the whole deploy would be nice, but it wouldn't highlight the pieces quite as well. Perhaps that's a future enhancement.

Here are the dependencies you'll need to follow along:

Rust 1.80 or later
Docker
AWS account
AWS CLI configured
eksctl installed (for cluster creation)

EKS Cluster

The project repository has a directory called kubernetes which houses the things I'm going to use to build and configure the cluster. A directory underneath that is cluster which is where my cluster configuration is located.

---
apiVersion: eksctl.io/v1alpha5
kind: ClusterConfig

metadata:
  name: sandbox
  region: us-west-2

managedNodeGroups:
  - name: mng-arm
    instanceType: m6g.large
    desiredCapacity: 2

This file will be fed into the eksctl CLI that will build my EKS cluster. One thing to note is that I'm using m6g instances for my node pool. Those are ARM chips running AWS Graviton.

SQS

I could use something like CDK or Terraform, but in this case I'm just building a simple SQS standard queue so I'm using CloudFormation wrapped in a script.

The queue has a few properties set on it, but overall it's not exotic.

# Snippet
Resources:
  SQSQueue:
    Type: AWS::SQS::Queue
    Properties:
      QueueName: !Ref QueueName
      VisibilityTimeout: !Ref VisibilityTimeout
      MessageRetentionPeriod: !Ref MessageRetentionPeriod
      DelaySeconds: !Ref DelaySeconds

There is a script in the ./scripts directory for running this YAML file and it'll output the ARN of the queue for use in other parts of the applications.

Now having a cluster and queue, I can start applying KEDA and the application.

KEDA

KEDA stands for Kubernetes Event-Driven Autoscaling. Over the past few years, the Serverless world has taken advantage of the wonderful integrations that Lambda provides when connecting to DynamoDB, SQS, EventBridge, and many other elastic services. KEDA brings a lot of the configurability that comes with Lambda into Kubernetes in the way of curated "scalers". And then if you don't see what you like, you can always build your own.

For a list of scalers, start here. And then more specifically, the SQS Scaler that I'll be using.

What KEDA will do for me is monitor the resource via the scaler definition and then expand or collapse my pod count so that I have enough compute to handle the demand. But before I can configure the scaler, I need to install KEDA. To do that, I run this script.

./kubernetes/scripts/install-keda.sh. This script will update Helm and then run a Helm install to create the namespace and KEDA's required components. Once I've run this script. I'll see three KEDA pods.

> kubectl get pods -n keda
NAME                                              READY   STATUS    RESTARTS   AGE
keda-admission-webhooks-7fc99cdd4d-ncbzn          1/1     Running   0          10h
keda-operator-75bc596ffb-wrqsm                    1/1     Running   0          9h
keda-operator-metrics-apiserver-c5b6f8b88-278ws   1/1     Running   0          10h

Now that I have KEDA installed, I want to have the ability for KEDA to use IRSA when working with SQS. You can read more from AWS on IRSA

I've scripted this configuration ./kubernetes/setup-keda-irsa.sh. The script will verify that the OIDC provider has been associated with the cluster, create a service account, and build a role for working with SQS. You'll see further down that I opt to use the application's pod identity as opposed to the KEDA operator identity, so this step is not truly needed, but it put it in for flexibility.

Deploying the Application

IRSA is 100% required as I look to deploy the actual application. Since I'm using the KEDA scaler with the pod's identity, then my pod needs IRSA to communicate with the resources that I require. In this case, the SQS.

Setting up IRSA is done with a script I created called ./kubernetes/setup-app-irsa.sh. It does very similar things to the KEDA version where it checks for the OIDC provider, builds a service account, and then creates a role.

The next piece of the puzzle is to configure some environment variables so that my deployment resources are more dynamic. That script is called ./scripts/configure-manifests.sh. The SQS has an ARN and a QueueURL that I didn't want to embed in the final solution so this step squares that way before deploying.

And lastly, running kubectl apply -f kubernetes/keda-service/ will launch all of the resources required to run. Let's take a quick tour.

Deployment

I won't dive through all the resources but just point out highlights that I think are important. For the deployment, the spec section has my service account I created, the Docker image (which I've prebuilt and published), and ConfigMap which holds some environment variables that help me configure the service.

spec:
  serviceAccountName: sqs-processor-sa
  containers:
  - name: sqs-processor
    image: public.ecr.aws/f8u4w2p3/rust-sqs-processor:1.0.0
    imagePullPolicy: IfNotPresent
    envFrom:
    - configMapRef:
        name: sqs-processor-config

This article is more about KEDA than it is the Rust code, but I do want to show that my Rust code is going to poll and print what it finds. The Rust code has an infinite loop that waits for signals that pod is being terminated. It also includes a server that has a healthcheck that Kubernetes can poll to make sure the container is still responding correctly.

let receive_result = self
    .client
    .receive_message()
    .queue_url(&self.config.queue_url)
    .max_number_of_messages(self.config.max_messages)
    .wait_time_seconds(self.config.wait_time_seconds)
    .visibility_timeout(self.config.visibility_timeout)
    .send()
    .await
    .context("Failed to receive messages from SQS")?;

let messages = receive_result
    .messages()
    .unwrap_or_default()
    .iter()
    .map(|msg| Message::from(msg.clone()))
    .collect::<Vec<Message>>();

info!("Received {} messages from SQS", messages.len());

KEDA Configuration

Configuring KEDA for this deployment is done through a TriggerAuthentication and ScaledObject resource.

The TriggerAuthentication expresses that KEDA when watching the resource (SQS) to scale will use the pod's identity, which is done with IRSA and the role I defined early up.

apiVersion: keda.sh/v1alpha1
kind: TriggerAuthentication
metadata:
  name: aws-pod-identity
  namespace: default
spec:
  podIdentity:
    provider: aws-eks

This resource pairs with the ScaledObject to give KEDA the authority and configuration it needs. This is the full spec. I am required to provide a namespace and then some other values in the spec. Notice i can cap the replicas, how many messages each pod can handle, how long to wait before scaling down, and then specifics about the SQS and AWS itself.

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: sqs-processor-scaler
  namespace: default
spec:
  scaleTargetRef:
    name: sqs-processor  # Name of the deployment to scale
  minReplicaCount: 0     # Minimum number of replicas
  maxReplicaCount: 10    # Maximum number of replicas
  pollingInterval: 15    # How frequently to check the metrics (in seconds)
  cooldownPeriod: 300    # Period to wait after last trigger before scaling down
  triggers:
  - type: aws-sqs-queue
    metadata:
      queueURL: https://sqs.us-west-2.amazonaws.com/{AWS_ACCOUNT_ID}/sqs-processor-queue  # Replace with your SQS queue URL
      queueLength: "5"   # Target messages per pod
      awsRegion: "us-west-2"  # Replace with your AWS region
      identityOwner: "pod"  # Use IAM role attached to the application pod's service account
    authenticationRef:
      name: aws-pod-identity  # Reference to the TriggerAuthentication resource

Testing the Solution

Everything is installed, deployed, and configured at this point and now it's time to test things out. Fortunately, I asked Claude Code to build me a test script that publishes messages onto my SQS for my pod(s) to pick up and process.

I'm going to make that happen like this:

./scripts/send-test-messages.sh

I now wait 15 seconds (because that's what I configured in my ScaledObject) to get my new pod and see what happens.

My pod appeared, worked the queue, and then shut itself down. Pretty neat and not that hard to put together!

Wrapping Up

This is a great stopping point for you to pick up and extend and take further. It's also good building block should you want to explore other scalers. But if this is the end of your journey for now, make sure to clean up your resources and save some $$$.

./scripts/cleanup-resources.sh

Now that everything has been destroyed, I feel like this literally is barely scratching the surface on what you can do with KEDA. One of my favorite ways to use Lambda functions is for event-driven processing. However, if your team is heavily invested in Kubernetes and/or containers, this is a nice alternative and perhaps the first thing you reach for instead of Lambda in the future. Additionally, you can replace the traditional HPA metrics for scaling your pods that are always on. For instance, use KEDA to grow your replicas running an API based upon a metric that you are tracking in Prometheus or Datadog perhaps. It doesn't have to be "events" in the asynchronous way, it can be other metrics that trigger an event. Again, barely scratching the surface.

I've been writing more about Kubernetes lately and expect that to continue over the coming months. I could even make this solution "serverless" by running it on Fargate as opposed to a dedicated node pool. Again, the options and the flexibility that Kubernetes gives me is highly appealing and I know that many who work primarily in the serverless world often shy away from it due to complexity. And I know that many in the Kubernetes world shy away from serverless due to lack of control and vendor lock in. Both can be true, but as a developer and a builder, having more tools on your belt gives you more ways to bring value to your customers. I also find Kubernetes allows me to utilize the Linux and Networking skills that I've acquired and honed throughout the years which provides a great base for better understanding how and when to apply serverless technologies.

With all that said, I hope you've enjoyed the article and see how you can build Event-Driven applications with KEDA and EKS.

Thanks for reading and happy building!

Establishing Datadog on Kubernetes with EKS

Benjamen Pyle — Sat, 19 Jul 2025 14:38:57 +0000

Over the past few years I've spent a great deal of time writing and building with Datadog. I find that their platform gives me as a builder the right insight and tools to diagnose things quickly, make adjustments when things run out of resources, and observe my software's behavior in test and at scale. During this past year or two, I've been expanding my skills into the Kubernetes ecosystem and was so pleased to find that my Datadog experience is valuable there as well. So, from Serverless to Kubernetes, Datadog has me covered. Let's explore what establishing Datadog on Kubernetes means for me as a developer.

Datadog on Kubernetes

Let's start out by exploring what the ecosystem looks like when deploying Datadog on Kubernetes. The image below is from a wonderful article on the Datadog blog which shows that there are two agents I will be running.

Node Agent -- handles node level APM and metrics collection
Cluster Agent -- runs on a Node and deals with aggregating data from the various nodes to ship to the Datadog platform

Credit to Datadog on the above image from this blog article

For the balance of this article, I'm going to explore how to make this a reality on a fresh Kubernetes cluster hosted on AWS' EKS (Elastic Kubernetes Service)

Working through the Project

If you want to follow along, the repository backing this article can be found at this link to GitHub

Creating the Cluster

To get the cluster up and running, I've created an eksctl configuration that will help with that.

---
apiVersion: eksctl.io/v1alpha5
kind: ClusterConfig

metadata:
  name: sandbox
  region: us-west-2

managedNodeGroups:
  - name: mng-arm
    instanceType: m6g.large
    desiredCapacity: 2

This file sets up an EKS cluster running 2 nodes running AWS' Graviton processors. I just like running Graviton over x86. Better, cheaper, faster, and all of that. These nodes and the cluster are in us-west-2. Basic setup.

eksctl create cluster -f cluster/cluster-config.yaml

Give this a little bit of time to run. There's a CloudFormation stack that'll be generated and execute in your AWS account. Once finished, updated your kubectl and check the status of the nodes.

aws eks update-kubeconfig --name sandbox --region us-west-2
kubectl get nodes

Installing Datadog

There are a couple of ways to install Datadog into a Kubernetes cluster. The approach I like to take is to use Helm. With Helm, I can keep all of my values for the Datadog agents in the proper area and update my releases as needed. I find the Helm values file looks pretty close to an agent configuration file.

Configure Datadog Secret

The agent can access the secrets store for things like the API Key which needs to stay private. My first order for configuration is to get the secret loaded up.

kubectl create secret generic datadog-secret --from-literal api-key=<YOUR_DATADOG_API_KEY>

Working with Helm

Installing Datadog with Helm is quite easy. I need to add the repository that Datadog is stored in and then run the installation. I can also verify that everything looks good.

helm repo add datadog https://helm.datadoghq.com
helm repo update

helm install datadog-agent -f datadog/datadog-values.yaml datadog/datadog

kubectl get pods -l app.kubernetes.io/name=datadog

Deploying Applications

As a part of this walkthrough, I'm going to deploy a couple of services so that I can show how Datadog blends Kubernetes with APM (Application Performance Monitoring) so easily. And of course, I'm going to use some code written in Rust, packaged with Docker, and hosted in AWS ECR (Elastic Container Registry). For this sample, I'll have "service-a" and "service-b". I'll make requests to "service-b" which will call "service-a" and those traces will be emitted via OpenTelemetry and shipped to Datadog via the node and cluster agent.

# create the namespace first
kubectl apply -f namespaces/rust-services-namespace.yaml
kubectl get namespaces

Time to install the services.

kubectl apply -f services/service-a.yaml
kubectl apply -f services/service-b.yaml

Before moving on to showing what this all produces, I need to point out how I'm connecting the services to Datadog via the node agent. This is a snippet from the service definition.

  env:
    # this allows me to get the Agent Address which the OTEL tracer must have
    - name: HOST_IP
      valueFrom:
        fieldRef:
          fieldPath: status.hostIP
    - name: BIND_ADDRESS
      value: "0.0.0.0:3000"
    - name: DD_TRACING_ENABLED
      value: "true"
    # notice that I'm using HOST_IP defined above
    - name: AGENT_ADDRESS
      value: $(HOST_IP)
    - name: RUST_LOG
      value: "info"
    - name: SERVICE_A_URL
      value: http://service-a

Putting the Pieces Together

I find that the more I consume from the Datadog ecosystem the more value I find. I've done so much with Serverless, logs, APM, and databases, but being able to bring it all together to observe my infrastructure as well is just so powerful. I'm going to make a quick tour (as this is an introduction) through some of the parts I like and use the most. This isn't exhaustive, but it should be enough to get you going.

Cluster(s) Overview

This initial view is just a nice overview of the cluster(s) that I'm managing. I can jump into specifics by clicking on any one of those tiles, but on the surface it's just a great top-level visual. You can see at this point, I've just got the one cluster, and the other tiles are representing the items/resources we've deployed together in this article.

Once I drill into any of those, the navigation options open up tremendously. All of these options can be scoped by any tag that I've create, labels defined in Kubernetes, or the DD_ENV (Datadog environment) that I've scoped my resources down to.

Nodes, Deployments, Containers, and Processes

Diving deeper, I can drill as far down the rabbit hole as I'd like. Starting out with nodes, I can see all of my nodes in whatever state they are in, including all the various elements that make them unique. IP address, chip architecture, and so on and so forth.

Diving further, my Deployments are also exposed via Datadog. If I have two nodes running, I've got some number of pods using the resources I have deployed up to this point. I could further view these by pod or if I'd deployed as ReplicaSets or StatefulSets, they'd be exposed that way too.

The last level of depth is the actual process itself. I find this pretty fascinating as I can explore the physical process and the user that's running the process. Datadog gives me that which I find incredibly powerful.

Traces

As a long time Datadog user, traces are where I've spent most of my time. Traces in containers. Traces in Lambda Functions. Traces between Functions and Containers. What I really enjoy is that I can see the traces on the individual node, deployment, container, or process just like I showed up above. I don't need to visit the specific APM section of the Datadog UI, it's just right there for me to access.

Wrapping Up

From this point, there are so many places to jump off and explore, but you should have a working Kubernetes cluster running on EKS with Datadog correctly installed. If at this point, you've gone as far as you want, and choose to cleanup your resources, here's how you delete the cluster.

eksctl delete cluster -f cluster/cluster-config.yaml

However, should you want to continue, I encourage you to explore this expansive documentation at Datadog in relation to Kubernetes. There's so much to dive into and stay tuned, as I'll have more walkthroughs and opinions in the future as well.

At the beginning of this article, I linked out to the GitHub repository. In case you missed it, here it is again. I encourage you to run through this in your own environment as it'll help the information stick.

My wrap-up thoughts on all of this can simply be summed up as this. Do not build distributed and cloud native software without a solid plan for observability. It is not a smart strategy to try and "add as you go". Do it right from the start and you won't regret it. And when it comes to tooling around observability, Datadog is the top of the stack. Use it. Learn it. And you won't regret it.

Thanks for reading and happy building!

Building an API Gateway with Istio and EKS

Benjamen Pyle — Mon, 07 Jul 2025 14:53:37 +0000

Working with Kubernetes opens a world of possibilities in a software project. That's one of its biggest appeals to me as a developer. And when building with distributed systems, one of the components that often gets added into the mix is a service mesh. If you're not familiar with the benefits of using a service mesh, head over there first. It provides a nice overview of the benefits of using a mesh and how it fits into a Kubernetes deployment.

However, a limitation of a mesh is that it deals with east/west traffic. That means service to service communication. But what about that critical north/south traffic? The kind that comes from users? Can my mesh provider in Istio play a role in this as well? And how about pairing with Amazon's EKS? Let's dive in!

API Gateway

What is an API Gateway

First off, let's define what an API Gateway is in the context of Kubernetes.

An API Gateway acts as a reverse proxy, routing external traffic to your internal services while providing essential features like authentication, rate limiting, and request transformation. In Kubernetes environments, it serves as the entry point for all external traffic to your cluster's services.

Here are key reasons to implement an API Gateway in your Kubernetes setup:

Traffic Management: Centralized control over routing, load balancing, and traffic splitting between different service versions
Security: Consolidated authentication, authorization, and SSL/TLS termination
Monitoring: Single point for collecting metrics, logging, and tracing data
API Composition: Ability to aggregate multiple backend services into a single API
Rate Limiting: Protection against abuse through request throttling
Protocol Translation: Converting between different protocols (HTTP/1.1, HTTP/2, gRPC) as needed

Istio Platform Install

All of these parts and pieces are nice, but how do they connect with AWS EKS and can I produce a working solution? This walkthrough will be fairly in-depth so starting off with the cluster build.

EKS Cluster

Throughout this build, I'm going to favor simplicity in terms of using eksctl, bare YAML for my Kubernetes resources, and shell scripts where needed. This could be taken further with CDK, CloudFormation, Terraform, and other tools, but in examples that I tend to get the most out of, unless it's around that abstraction purpose, I like to see and do things vanilla.

If you've cloned the repository https://github.com/benbpyle/eks-istio-ingress and are working from it, there's a file call cluster-config.yaml. In that file, the cluster's configuration as well the node group requirements are defined.

---
apiVersion: eksctl.io/v1alpha5
kind: ClusterConfig

metadata:
  name: sandbox
  region: us-west-2

managedNodeGroups:
  - name: mng-arm
    instanceType: m6g.large
    desiredCapacity: 2

When I run eksctl create cluster -f cluster-config.yaml everything gets going. My VPC is configured, I get some subnets, a nodegroup for my cluster to work with as well as a functioning Kubernetes install.

AWS Gateway Controller

With Istio installed, I'm marching my cluster toward allowing ingress traffic through my Istio gateway. The next step in the journey is to install an AWS Gateway Controller. This gateway controller will launch an AWS Load Balancer that can connect the outside world to my cluster. I also get a couple of pods that make that connection more physical be bridging the cloud load balancer to a resource inside my cluster.

I've created a shell script that encapsulates some of the tasks that are needed to get the controller in my cluster.

# Associate IAM OIDC provider with the cluster
eksctl utils associate-iam-oidc-provider \
    --region us-west-2 \
    --cluster sandbox \
    --approve

# Download IAM policy for the AWS Load Balancer Controller
curl -o iam-policy.json https://raw.githubusercontent.com/kubernetes-sigs/aws-load-balancer-controller/v2.4.3/docs/install/iam_policy.json

# Create IAM policy using the downloaded policy document
aws iam create-policy \
    --policy-name AWSLoadBalancerControllerIAMPolicy \
    --policy-document file://iam-policy.json

# Create service account and attach IAM policy
eksctl create iamserviceaccount \
--cluster=sandbox \
--namespace=kube-system \
--name=aws-load-balancer-controller \
--attach-policy-arn=arn:aws:iam::252703795646:policy/AWSLoadBalancerControllerIAMPolicy \
--override-existing-serviceaccounts \
--region us-west-2 \
--approve

# Add AWS EKS Helm repository
helm repo add eks https://aws.github.io/x-charts

# Apply AWS Load Balancer Controller CRDs 
kubectl apply -k "github.com/aws/eks-charts/stable/aws-load-balancer-controller/crds?ref=master"

# Install AWS Load Balancer Controller using Helm
helm install aws-load-balancer-controller eks/aws-load-balancer-controller -n kube-system --set clusterName=sandbox --set serviceAccount.create=false --set serviceAccount.name=aws-load-balancer-controller

Running the gateway_controller.sh script is as follows:

chmod 755 gateway-controller.sh
./kubernetes/alb/install-gateway-controller.sh

A glimpse into the new cluster with Istio and the gateway controller looks
like this.

Kubernetes Gateway CRDs

Resources in Kubernetes need definition files so that the server can
validate that my resource configuration is applied correctly. Some
Kubernetes servers don't come prepared with the correct Gateway Resource
Definitions. Let's get that taken care of.

kubectl get crd gateways.gateway.networking.k8s.io &> /dev/null || { kubectl kustomize "github.com/kubernetes-sigs/gateway-api/config/crd?ref=v1.2.1" | kubectl apply -f -; }

Installing Istio

I need to get Istio installed and into my cluster if I want to have it
control my ingress traffic.

Up first, download and unzip the Istio binaries
Run the istioctl command.

# sets up some namespaces
kubectl apply -f kubernetes/namespaces.yaml
istioctl install -f kubernetes/istio/istio-installation.yaml

Let's recap so far!

✓ Created an EKS cluster with managed node groups using eksctl
✓ Associated IAM OIDC provider with the cluster
✓ Created and attached IAM policy for AWS Load Balancer Controller
✓ Installed AWS Load Balancer Controller via Helm
✓ Added Kubernetes Gateway CRDs
✓ Downloaded and installed Istio using istioctl
✓ Configured Istio with custom installation settings

Now onto my application!

Deploying Rust Code

To demonstrate this solution, I've built two small Rust Web APIs powered by
Axum. Service-B will host the public endpoint that has a dependency on
Service-A and will use a consumer's input to build a final payload.

Deploy Service-A

I'm going to deploy Service-A first because it has no dependencies. But
first, I'm creating my namespace to house this code.

kubectl apply -f service-a-service.yaml

I've included a few other Istio resources in this build below.

Deployment has the pod definition and spec
Service is a standard Kubernetes resource that sets up the port
DestinationRule is where I'm building some outlier configuration. I'll address this in a future article where I talk about Service Mesh Circuit Breaking

apiVersion: apps/v1
kind: Deployment
metadata:
  namespace: rust-services
  labels:
    app: service-a
  name: service-a
spec:
  selector:
    matchLabels:
      app: service-a
  replicas: 1
  template:
    metadata:
      annotations:
        # Used in other solutions for Datadog
        traffic.sidecar.istio.io/excludeOutboundPorts: "8126"
      name: service-a
      namespace: rust-services
      labels:
        app: service-a
        sidecar.istio.io/inject: "true"
    spec:
      nodeSelector:
        eks.amazonaws.com/nodegroup: mng-arm
      containers:
        - name: service-a
          image: public.ecr.aws/f8u4w2p3/rust-service-a:latest
          ports:
            - containerPort: 3000
              name: http-web-svc
          env:
            - name: HOST_IP
              valueFrom:
                fieldRef:
                  fieldPath: status.hostIP
            - name: BIND_ADDRESS
              value: "0.0.0.0:3000"
            - name: DD_TRACING_ENABLED
              value: "true"
            - name: AGENT_ADDRESS
              value: $(HOST_IP)
            - name: RUST_LOG
              value: "info"
---
apiVersion: v1
kind: Service
metadata:
  namespace: rust-services
  labels:
    app: service-a
  name: service-a
spec:
  selector:
    app: service-a
  ports:
    - name: default
      protocol: TCP
      port: 80
      targetPort: 3000

---
apiVersion: networking.istio.io/v1
kind: DestinationRule
metadata:
  name: service-a-destination-rule
  namespace: rust-services
spec:
  host: service-a
  trafficPolicy:
    outlierDetection:
      consecutive5xxErrors: 0
      consecutiveGatewayErrors: 5
      interval: 15s
      baseEjectionTime: 30s
      maxEjectionPercent: 100
      minHealthPercent: 50

Deploy Service-B

This service is the receiver of outside traffic (via Istio) and will make calls into Service-A and communicate over the service mesh (Istio). An all-Istio managed solution! Let's have a look at the service first and then see about connecting the new Gateway to this inbound traffic handling service. Notice it'll look very much like the Service-A definition above.

kubectl apply -f service-b-service.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  namespace: rust-services
  labels:
    app: service-b
  name: service-b
spec:
  selector:
    matchLabels:
      app: service-b
  replicas: 1
  template:
    metadata:
      annotations:
        traffic.sidecar.istio.io/excludeOutboundPorts: "8126"
      namespace: rust-services
      labels:
        app: service-b
        sidecar.istio.io/inject: "true"
    spec:
      nodeSelector:
        eks.amazonaws.com/nodegroup: mng-arm
      containers:
        - name: service-b
          image: public.ecr.aws/f8u4w2p3/rust-service-b:2-routes
          ports:
            - containerPort: 3000
              name: http-web-svc
          env:
            - name: HOST_IP
              valueFrom:
                fieldRef:
                  fieldPath: status.hostIP
            - name: BIND_ADDRESS
              value: "0.0.0.0:3000"
            - name: DD_TRACING_ENABLED
              value: "false"
            - name: AGENT_ADDRESS
              value: $(HOST_IP)
            - name: RUST_LOG
              value: "info"
            - name: SERVICE_A_URL
              value: http://service-a
---
apiVersion: v1
kind: Service
metadata:
  namespace: rust-services
  labels:
    app: service-b
  name: service-b
spec:
  selector:
    app: service-b
  ports:
    - name: default
      protocol: TCP
      port: 80
      targetPort: 3000
---
apiVersion: networking.istio.io/v1
kind: DestinationRule
metadata:
  name: service-b-destination-rule
  namespace: rust-services
spec:
  host: service-b
  trafficPolicy:
    outlierDetection:
      consecutive5xxErrors: 0
      consecutiveGatewayErrors: 5
      interval: 15s
      baseEjectionTime: 30s
      maxEjectionPercent: 100
      minHealthPercent: 50

Connecting Service-B to the Gateway

The last 2 pieces of the puzzle include adding the actual Gateway itself and connecting the Gateway to the Application Load Balancer defined in AWS. What I really enjoy about using Istio as an Ingress/API Gateway is that I use constructs like VirtualService and HTTPRoute to manage my traffic flow. These are the same resources I use when working with the Service Mesh aspects, so I just get the same feelings. It also means I know how to troubleshoot problems with these resources. And I can use my normal observability tools like Kiali and Datadog. More on both of those in a future article.

So let's get to configuring the Istio Gateway.

kubectl apply -f kubernetes/istio/gateway.yaml

The contents of the gateway.yaml file include a Gateway resource and a VirtualService. If you've used Istio before, you'll notice most of the VirtualService looks like what you've seen before. But pay special attention to the gateways section. This is where I make the connection the Gateway I defined right above it. And in the Gateway resource, notice that it's in the same namespace as the rest of the application services. rust-services. Those two details tripped me up as I was first working with Istio as an Ingress Gateway.

apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: application-gateway
  namespace: rust-services
spec:
  selector:
    istio: ingressgateway
  servers:
    - port:
        number: 80
        name: http
        protocol: HTTP
      hosts:
        - "*"
---
apiVersion: networking.istio.io/v1
kind: VirtualService
metadata:
  name: service-b
  namespace: rust-services
spec:
  hosts:
    - "*"
  gateways:
    - application-gateway
  http:
    - match:
        - uri:
            prefix: /
      route:
        - destination:
            host: service-b

Putting it All Together

What would a walkthrough be without highlighting how to test our work? Being that this is an API focused article, and a simple one at that, I've got a HealthCheck and an actual endpoint to run through. To make this happen, you'll need the DNS name for the Application Load Balancer you created earlier. You can grab that from the AWS Console.

For the HealthCheck:

GET http://<ALB URL>/health

HTTP/1.1 200 OK
Date: Mon, 07 Jul 2025 11:53:17 GMT
Content-Type: application/json
Content-Length: 20
Connection: keep-alive
x-envoy-upstream-service-time: 1
server: istio-envoy

{
  "status": "Healthy"
}
Response file saved.
> 2025-07-07T065317.200.json

Response code: 200 (OK); Time: 184ms (184 ms); Content length: 20 bytes (20 B)

Now let's do the same thing for the actual endpoint>

GET http://<ALB URL>>?name=A+Field

HTTP/1.1 200 OK
Date: Mon, 07 Jul 2025 11:54:47 GMT
Content-Type: application/json
Content-Length: 59
Connection: keep-alive
x-envoy-upstream-service-time: 3
server: istio-envoy

{
  "key_one": "(A Field)Field 1",
  "key_two": "(A Field)Field 2"
}
Response file saved.
> 2025-07-07T065447.200.json

Response code: 200 (OK); Time: 186ms (186 ms); Content length: 59 bytes (59 B)

Cleaning Up

One thing about working with Kubernetes vs Serverless is that my resources are going to cost me dollars even when I'm not using them. This is because the EKS control plane is billing hourly and I've allocated two EC2 instances that also are running full time to host my pods. Being cost conscious is important, so here's how to clean up the resources created above.

kubectl delete -f kubernetes/alb/alb-ingress.yaml
kubectl delete -f kubernetes/istio/gateway.yaml
kubectl delete -f kubernetes/service/service-b.yaml
istioctl uninstall --purge
kubectl delete -f kubernetes/namespaces.yaml

Wrapping Up

I hope this article gives you the confidence to dive into connection Istio and EKS when building an API Gateway for your APIs. The Gateway spec offers much more control over the flow of your traffic and is the preferred way going forward in the Kubernetes ecosystem. However, there's not a ton of documentation out there and I could find little in the way of plain YAML the showed how these pieces come together. With that said, I barely scratched the surface of the power and flexibility this solution provides.

Kubernetes gets the rap of being complex and difficult to manage but one of the goals of this article was to show that you can achieve a powerful Gateway with very minimal setup and configuration. I don't believe there's more YAML in this solution than what I write in a Serverless build. I also don't believe that there are more "parts". I'm just more in control of them.

The last bit I'll say is that I'd love for you to try out the solution yourself. Get your hands dirty so to speak and run through the different operations required to make this happen. Feel free to clone this GitHub Repository and get started. The README in that repos will help you get going as well.

Thanks for reading and happy building!

4 Benefits to using a Service Mesh

Benjamen Pyle — Sun, 20 Apr 2025 16:20:32 +0000

I've been spending a great deal of time lately working with Service Meshes and after having a few of the same conversations over and over (in a good way), I wanted to codify some of the reasons why they exist and when I think as a developer they come in useful. I've written before about comparing Istio and Linkerd where I touch upon the concepts of a mesh but in this article, I want to go a little deeper and break my thoughts up into categories of problems that they help solve. With that stated, let's dive into what problems a service mesh solves and when you should be putting one in your architecture.

Defining a Service Mesh

Before jumping in, I want to define a service mesh so there's something to point back to along the way. I'm a big fan of the way Linkerd succinctly states it.

A service mesh is a tool for adding security, reliability, and observability features to cloud native applications by transparently inserting this functionality at the platform layer rather than the application layer. -- Linkerd

I could almost end the article right here because it lists 3 of the 4 main things that a mesh does in an architecture. The big draw to focus on is that I can get all of these benefits at the platform level vs doing it in the application level. From experience, this is a huge draw for architects and CTOs that want these benefits but find the challenge of having teams implement these capabilities highly difficult to coordinate and do correctly. Even with libraries, frameworks, and other tools, having this functionality sitting on top of custom service code makes it transparent to the developer and immediately available to the platform team. And if done right, still puts the developer in control, but with configuration instead of application code.

Why a Service Mesh

A service mesh isn't a silver bullet. Truthfully nothing is in tech and while I have my favorite approaches and tools, I'm always careful to not try and solve things the way I've always done them. But to narrow down when I do look to bring a service mesh into my solution, here are some of the pieces of criteria that I evaluate.

1) Does my system have distributed APIs that are deployed as small and independent parts that work together to solve a user's problem? Specifically am I working with APIs and HTTP or TCP requests.
2) Is there a strong desire in service resiliency coupled with an obsession around observability of the service interactions?
3) Are the developers platform and non-functionally focused or is there a platform team that has a focus to working as a safety net for the teams they support?
4) Does the team favor a configuration style approach over investing doing similar practices in their application level code?

The last bullet point is tricky, because with all of the benefits of service mesh, the same could be accomplished by adding it into my application code. But by adding it directly into code, I'm tightly coupling my desired outcomes into my product value implementation. There's a lot of code I must produce that doesn't directly add bottom line value into my product. That's because a mesh or the resiliency that it provides solves many of the non-functional "ilities" that often come late in a software build.

The other thing to keep in mind is that in a polyglot type environment where different frameworks and languages are used to solve different feature problems, reusable libraries must be available for each of these combinations. It's not to say that this impossible but it really does become impractical the larger the solution gets. Not to mention, many of these operations are easy to discuss but can be tough to execute well.

With all of that said, here's the reasons that I lean into a mesh when I have the need and enjoy the benefits that it will provide me.

Resiliency

It's hard to not put these in order of usage or importance to me, so I'll try and remain unbiased as I navigate through my breakdowns. However, many people come to a service mesh to bring more resiliency into their solutions. When I talk about resiliency, I mean how well does my application handle failure and latency.

Software systems are always going to encounter failure. That failure might be due to network packet loss (which happens), a bad deployment leaving a service in a bad state, an application error that is unfixed, and many other scenarios. The fact is, everything fails at one point or another and failure can create all kinds of bad outcomes.

Specifically, a mesh can help with resiliency by providing the following capabilities. All of these are usually very configurable.

Retries: the ability to have specific requests to specific services retry in case of issues that are not persistent. Think about a faulty HTTP request that if just retried would be 100% OK. This failure effectively goes away for the end user.
Timeouts; hanging onto a connection longer than it should will cause traffic and load to build up and can be harmful to your application. Most HTTP clients in programming languages allow the developer to set a timeout on their request. So many times, this is just left at the default. Maybe that works and maybe it doesn't. But combining timeout management with retries is powerful.
Circuit Breaking: this is something that not a lot of developers think about but it's actually a really cool concept. Think about having a closed circuit where traffic is flowing and behaving as expected. Now, imagine some failure happens, and the circuit is open, thus limiting traffic from reaching its destination. Now further build upon that and make it configurable. How many times does code just call hosts that are in a failure state only to continue piling up load and making things worse. Circuit breaking is a pattern for shedding load and protecting the ecosystem.
Rate Limiting: sometimes I have a component that can only handle so much traffic. Maybe it has a dependency on something that is limited downstream. Maybe I can only run a limited number of instances. Regardless of why, limiting the traffic that the service can handle without it having to fail over to prove it's exhausted is a wonderful feature.

Traffic Management/Shaping

If you've ever used Nginx or even an AWS Application Load Balancer, you've got some familiarity with traffic shaping. If you haven't used software like those two, think of being a train tracker switch operator that can move requests (trains) from one track to another all with configuration.

Shaping traffic can be as simple as this:

Take all requests that have this path and this header and route it to this service.

That's a very basic version of what can be done with a mesh. However, it can get more complex such as:

Take all requests with this path and this header and weight 35% to a subset of a service and route another 65% to another subset of the service.

I can also route top-level paths to different service.s. I can return direct or static responses. Paths can be prefixes, exacts, or regexes. And at that point, the rabbit hole can get deep.

The best way I can describe shaping traffic is with the Nginx reference. I've got the power to move traffic around in a highly configurable way that can also take into account load balancing and make use of the other pieces of my mesh. Resiliency, observability, and security.

Observability

I've seen cloud native systems skip this core piece more times than I'd like. Small aside here, but if you are working in a cloud native and distributed system without observability, stop adding features right now and go get this solved.

Observability is the super power that allows me to visibility inspect the healthy and wellness of my services and the health of their interactions with other services. By using a service mesh, I'll gain the ability to see metrics like:

Throughput
Latency
Is my circuit open/configured
Topology
And others

In the case of Istio, Kiali is the tool of choice. Linkerd also has a dashboard as well that exposes this information.

I cannot stress enough how much power observability gives you as a developer or platform engineer. And with a mesh, I can include its capabilities into my overall observability strategy to gain a very complete picture of my application's health and performance.

Security

This is a topic that most developers don't usually dive into first, but a service mesh's ability to create more secure systems is a strong argument for introducing one into your application.

What I mean by security is two fold.

It means that I can protect traffic that is between my application services. Most of the time, intra-system calls happen over non-TLS channels. With a service mesh, if the requirement exists to have that traffic encrypted from end to end the proxy which is implementing the mesh will terminate the encrypted traffic and forward it onto application container. This keeps the application code from not knowing or caring about the traffic having been encrypted, and only focusing on its operations.

The second part of security is preventing the traffic from ever occurring. A service mesh can do just that. It means that I can set up a zero-trust environment where service to service communication must be vetted and approved before I allow it. As an added bonus, I can do this on egress of my mesh as well to external services.

Again, I can do these things without having to put anything different into my code.

Final Thoughts

I'm at the point in my journey where if I have containerized workloads that communicate with each other, I'm almost always going to reach for a service mesh to enhance my applications capabilities. There are many options out on the market which will have varying degrees of configurability, complexity, and implementations, but with anything, find the one you feel most comfortable with and invest some time into learning how it works.

If you are considering a service mesh in your next project, here's the list of the contenders I'd recommend you looking into and in no order.

For Kubernetes

Istio
Linkerd
Kuma
Consul

AWS Specific

ServiceConnect
VPC Lattice - not a service mesh press but has capabilities that will accomplish most of the above

My parting words are, plan for resilience and observe by default. These are things that a service mesh will help you with and improve your user experience.

Thanks for reading and happy building!

Chart an Extensible Course with Helm

Benjamen Pyle — Sat, 29 Mar 2025 15:36:43 +0000

Clicks, copies, and pasting. That's an approach to deploying your applications in Kubernetes. Anyone who's worked with Kubernetes for more than 5 minutes knows that this is not a recipe for repeatability and confidence in your setup. Good news is, you've got options when tackling this problem. The option I'm going to present below is using Helm.

Helm describes itself as:

The package manager for Kubernetes -- https://helm.sh

But what exactly does that mean for the developer or administrator of a Kubernetes cluster? And how does one package an API into common Kubernetes resources like Services, Deployments, ReplicaSets, and others? The example project below will explore some of the answers to these questions and give a great starting point for taking it further.

What Problem Does this Solve?

Going back to deploying an API application into a cluster, there are a number of different resources needed to make this happen. For a simple use case, I'd need the following:

Service
ReplicaSet
ConfigMap
Deployment

Those four resources would give me a starting point to extend from but at a minimum, I'd have my service running and serving traffic. I could of course hard code all of my values directly into my resource definitions. Again, that's an approach. But the minute that I need to have another environment, such as QA, I'm going to end up duplicating my work and having 2 copies of my full resources. This introduces waste and opportunities for errors.

What Helm allows me to do is define templates that I then can supply values into. Meaning that I will have values files per environment but only one copy of the actual set of resources that I'm going to need. Less waste. Less room for error. And one version of truth.

Coding Example

Consider this an introduction to Helm with a specific project. I'm by no means trying to show everything about Helm or dig into specifics about the 4 resources I'm going to work with. My goal is for you to have some exposure to this approach to building Kubernetes resources and feel a little more comfortable to try Helm in your own projects.

Application Code

As always, there will be a link to the GitHub Repository at the end of this article and I've published this Docker Image into my AWS ECR Public Repository if you'd like to run things as well. The implementation of what's behind that repository is a Rust program that is running an Axum server listening to serve two routes. / and /second

use axum::{routing::get, Router};

async fn root_handler() -> &'static str {
    tracing::info!("Hello, Axum!");
    "Hello, Axum!"
}
async fn second_handler() -> &'static str {
    tracing::info!("Second handler");
    "Second handler"
}

#[tokio::main]
async fn main() {
    let bind_address = std::env::var("BIND_ADDRESS").expect("BIND_ADDRESS is required");
    let app = Router::new()
        .route("/", get(root_handler))
        .route("/second", get(second_handler));
    let listener = tokio::net::TcpListener::bind(bind_address.clone())
        .await
        .unwrap();
    tracing::info!("Up and running ... listening on {}", bind_address);
    axum::serve(listener, app).await.unwrap();
}

Tour of Helm

I've written before about my preference for Minikube to run my clusters locally and I'm going to do the same thing here. I love running AWS Elastic Kubernetes Service but the cost of the cluster and the nodes just doesn't lend itself to what I wanted to do with this example. If you've never run Minikube before, read that article first, and then jump back over here when you are ready.

The first step in the process here is to install Helm. Depending upon your system, there are plenty of options to make the happen. With that step out of the way, it's time to talk Charts.

Chart

Helm orients itself around the concept of a Chart. Think of this as your application definition. Whatever is defined in your project, it happens under the umbrella of a chart. When creating a chart with helm create my-chart, you'll be treated to a structure that looks like this:

my-chart/
  Chart.yaml
  values.yaml
  charts/

The Chart.yaml file that gets created is below.

apiVersion: v2
name: chart
description: A Helm chart for Kubernetes

# A chart can be either an 'application' or a 'library' chart.
#
# Application charts are a collection of templates that can be packaged into versioned archives
# to be deployed.
#
# Library charts provide useful utilities or functions for the chart developer. They're included as
# a dependency of application charts to inject those utilities and functions into the rendering
# pipeline. Library charts do not define any templates and therefore cannot be deployed.
type: application

# This is the chart version. This version number should be incremented each time you make changes
# to the chart and its templates, including the app version.
# Versions are expected to follow Semantic Versioning (https://semver.org/)
version: 0.1.0

# This is the version number of the application being deployed. This version number should be
# incremented each time you make changes to the application. Versions are not expected to
# follow Semantic Versioning. They should reflect the version the application is using.
# It is recommended to use it with quotes.
appVersion: "1.16.0"

Templates

Templates are at the center of what I'm planning on building. Think of the templates as the Kubernetes resource because that's what they are. But instead of having values filled in, I'm going to use the Go Template Language to put placeholders that will be filled in from the values file that I'll show below.

Take this ConfigMap file. I've got the double curly braces {{ }} which tells Helm that this is where a value needs to be inserted. And then I'm taking advantage of built-in Helm objects such as Release and Values. Built-in Objects are how you'll work with filling in values in your templates.

apiVersion: v1
kind: ConfigMap
metadata:
  name: {{ .Release.Name }}-configmap
data:
  BIND_ADDRESS: {{ .Values.container.environmentVariables.bindAddress | quote }}
  RUST_LOG: {{ .Values.container.environmentVariables.rustLog | quote }}

Notice as well that I'm using a | after my .Values and sending it to the quote function. Helm supports piping which can chain outputs together to make the file value that I'm looking for. And in this case, I'm piping my value to the quote function which will put " " around my output.

Exploring the Deployment resource shows a little more of the power of what I can fill in with templates.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: {{ .Release.Name}}-deployment
  labels:
    app: rust-service-1
spec:
  replicas: {{ .Values.replicas }}
  selector:
    matchLabels:
      app: rust-service-1
  template:
    metadata:
      labels:
        app: rust-service-1
    spec:
      containers:
        - name: web-api
          image: {{ .Values.container.image.path }}
          ports:
            - containerPort: {{ .Values.container.containerPorts.targetPort }}
              name: web-port
              protocol: {{ .Values.container.containerPorts.protocol }}
          envFrom:
            - configMapRef:
                name: {{ .Release.Name }}-configmap

The templating language is extremely powerful. The above is just a basic example, but I could interject loops and conditionals to alter the flow of what is generated. Having the ability to package my deployments like this makes having multiple environments a piece of cake.

Values

If Templates are the blueprint for our resources, then values fill in those blueprints with concrete implementations. Values are also YAML resources which have a hierarchy of application. Every project should have a default values.yaml file. This values file can be overridden by a supplied values file and then that file can be override by command line set arguments. The most common way is to create a values file per environment or per version that you are looking to maintain.

In this example, I'm not showing an override file, but if you want to play around with the code, just copy the values.yaml and name the new file second-values.yaml and when you go to run the chart, helm install your-release ./chart -f second-values.yaml and you'll see how you can have 2 deployments managed by different releases with different values.

The values file can have the structure that you choose. Think of it a little like the API you are giving the template builders to fill in what they need when setting up resources. The file can be nested or flattened or a bit of both. I tend to favor a nested approach, but only 3 levels deep. This is really just my preference, so feel free to explore here.

In my values.yaml file, you'll notice the fields I've used in my templates to fill in what matters.

replicas: 2
container:
  containerPorts:
    port: 80
    protocol: TCP
    targetPort: 3000
  image:
    path: public.ecr.aws/f8u4w2p3/rust/rust-service-1:latest
  environmentVariables:
    bindAddress: 0.0.0.0:3000
    rustLog: "INFO"

Again, simple example, but you can take this much further to build out some very extensible resource definitions.

Installing the Chart

Helm charts are either installed, upgraded, or deleted. The CLI is plenty powerful, but these are the 3 commands that I want to explore here.

Install

Installing my chart takes this shape from the terminal.

helm install first-release ./chart

The first-release is the way Helm manages the resources by release. Again, I could have many releases or I could use just one. The flexibility here provides a great deal of power. When I run that command, all 3 of the templates I have which define resources are created.

ConfigMap

Deployment

Pods

Upgrade

I've got a chart now with resources deployed, but how do I modify it? If I changed some values in my values.yaml file, I can modify the release by running upgrade.

helm upgrade first-release ./chart

Delete

And when you are all done, just run delete.

helm delete first-release

Wrapping Up

I feel like I barely scratched the Helm surface here but hopefully it was just enough to show you the power and customization that can be had when using Helm for your Kubernetes resources. I've used the popular Kustomize in other projects and while I do like that approach, Helm just feels like I have more control and I write less code. I put my resources together once and then let Helm fill in my values.

If you are interested in trying out the example above, here is the GitHub Repository that has a full working solution. Clone it and adjust it to your needs to see how Helm works. Happy Charting!

Thanks for reading and happy building!

Connecting Rust Lambda Functions with OpenTelemetry and Datadog

Benjamen Pyle — Tue, 25 Feb 2025 18:17:31 +0000

Tracing Lambda Functions with observability code is the basement level of instrumentation that should exist when writing Serverless Applications. So many times, even in Lambda Functions, there are time bombs that will cause major delays or system problems triggered by "exceptional" payloads or even poorly coded SQL statements. But in my experience, I've seen developers with down the middle of the fairway type payloads that should "just work", yield latencies that are outside of acceptable when it comes to user's expectations.

Now, let's go a bit further and imagine that there's a Lambda Function that needs data from another Lambda Function and these operations might produce an event that ends up on a Queue. That event could then be processed by another Lambda Function resulting in 3 separate code executions on 3 different runtimes and physical infrastructure. If I've only instrumented at the bare minimum of "function level", then I get 3 pictures, but those pictures don't tell the story that actually happened.

In this article, I'm going to dive in on how to instrument Rust Lambda Functions with OpenTelemetry so that Datadog can then visualize the relationships between Spans and the single parent Trace. Here we go, connecting Rust Lambda Functions with OpenTelemetry and Datadog.

Architecture

I've shared many times here on my site about OpenTelemetry, Rust, and Datadog but everything up to this point has been as I described it above. Single function tracing. There's a large world out there that for some API requests, multiple Lambda Function Invocations might happen. And every time those functions get invoked without the context of the original invocation, the story gets fractured. This article will show you another way.

Taking a quick tour of that image, the request from the outside is going to take the following paths.

1) API Request lands in API Gateway triggering a Lambda Function Invocation
2) The initial Lambda Function will make an HTTP request back through the same API Gateway into another Lambda Function
3) When the request returns, the original Lambda Function puts a message onto a Simple Queue Service queue
4) And finally, a Lambda Function reads that queue and processes the message

So 3 Lambda Function invocations, that all need to be stitched together to tell the same story.

And as always, at the end of the article, there's a GitHub repository that can be cloned to get started.

Rust Lambda Function 1

The infrastructure as code is written in TypeScript using the AWS CDK but instead of carving that out in its own section, I'll just reference it as I walk through code. And with the CDK, I'm using Cargo Lambda for the Rust builds and the CDK Construct that includes RustFunction.

To get my first Lambda Function up and running, I need that API Gateway and the Datadog Lambda Extension for tracing.

let api = new RestApi(this, "RestApi", {
  description: 'Sample API',
  restApiName: 'Sample API',
  disableExecuteApiEndpoint: false,
  deployOptions: {
    stageName: 'demo',
  },
});

const layer = LayerVersion.fromLayerVersionArn(
  this,
  'DatadogExtension',
  'arn:aws:lambda:us-east-1:464622532012:layer:Datadog-Extension-ARM:68'
)

const postFunction = new RustFunction(this, 'PostFunction', {
  architecture: Architecture.ARM_64,
  functionName: "sample-post-function",
  manifestPath: path.join(__dirname, `../../lambdas/post-function`),
  memorySize: 256,
  environment: {
    RUST_LOG: 'info',
    FUNCTION_NAME: "post-function",
    DD_API_KEY: process.env.DD_API_KEY!,
    DD_SITE: process.env.DD_SITE!,
    AGENT_ADDRESS: '127.0.0.1'
  },
  layers: [layer]
});

  api.root.addMethod("POST", new LambdaIntegration(postFunction))

With the Function and the API Gateway in place, let's have a look at the Rust code. My main function does what a main always does. Initialize clients, parse environment variables, and build references to things I want to reuse in future invocations. It also establishes the linkage between the Lambda Runtime and my Function Handler. However, I do want to show the init_datadog_pipeline function. This initializes an OpenTelemetry pipeline from a community manage project that setups the OpenTelemetry endpoints and does some resource mapping behind the scenes between OpenTelemetry and Datadog.

fn init_datadog_pipeline() -> opentelemetry_sdk::trace::Tracer {
    let agent_address = env::var("AGENT_ADDRESS").expect("AGENT_ADDRESS is required");
    match new_pipeline()
        .with_service_name(env::var("FUNCTION_NAME").expect("FUNCTION_NAME is required"))
        .with_agent_endpoint(format!("http://{}:8126", agent_address))
        .with_api_version(ApiVersion::Version05)
        .install_simple()
    {
        Ok(a) => a,
        Err(e) => {
            panic!("error starting! {}", e);
        }
    }
}

How do I take advantage of this pipeline though? I can use the instrument macro and I can manually create spans. I've shown both of those before, but how do I connect my span between an HTTP request?

The first thing I want to do, is make sure that my HTTP Client automatically instruments my API requests. In my main function, I use the ClientBuilder to establish this connection. I like the Reqwest crate for this type of work

let client = aws_sdk_sqs::Client::from_conf(config);
let reqwest_client = reqwest::Client::builder().build().unwrap();
let http_client = ClientBuilder::new(reqwest_client)
        // Added in the tracing middleware
    .with(TracingMiddleware::default())
    .build();

Now inside of the handler, I'm going to add the trace context to the headers of the request via context propagation

let ctx = Span::current().context();
let propagator = TraceContextPropagator::new();
let mut fields = HashMap::new();

let mut trace_parent: Option<String> = None;

propagator.inject_context(&ctx, &mut fields);
let headers = fields
    .into_iter()
    .map(|(k, v)| {
        if k == "traceparent" {
            trace_parent = Some(v.clone());
        }
        return (
            HeaderName::try_from(k).unwrap(),
            HeaderValue::try_from(v).unwrap(),
        );
    })
    .collect();

let response = http_client
    .get("<url>/demo")
    .headers(headers)
    .send()
    .await;

Even though the code to propagate to SQS is right below this, I'll touch on that when I get to Lambda Function 3. For now, I've got my context being sent over to the next Lambda Function via HTTP Headers.

Rust Lambda Function 2

Digging into the Read Function will show a very simple JSON response with no processing. I'm going to first create the function and then assign it to the GET verb on the default / endpoint.

const readFunction = new RustFunction(this, 'ReadFunction', {
  architecture: Architecture.ARM_64,
  functionName: "sample-read-function",
  manifestPath: path.join(__dirname, `../../lambdas/read-function`),
  memorySize: 256,
  environment: {
    RUST_LOG: 'info',
    FUNCTION_NAME: "read-function",
    DD_API_KEY: process.env.DD_API_KEY!,
    DD_SITE: process.env.DD_SITE!,
    AGENT_ADDRESS: '127.0.0.1'
  },
  layers: [layer]
});

api.root.addMethod("GET", new LambdaIntegration(readFunction))

Unlike the Lambda Function 1, I am going to dig into main because I did something a little different here. Deep in the initialization of the Lambda Runtime, there is a span that is emitted from the Lambda Runtime. It has the simple name of "Lambda Runtime Invocation". Now normally, this might not seem like a big deal. However, if I want connected spans, I don't have access into this particular span's parent trace. Therefore, I end up with all of my spans connected, but this one lone span with its own trace parent.

I ended up getting around this but creating my own instance of the Lambda Runtime which allows me to leave this span out of the my call chain. I got the idea from reading through the Lambda Runtime GitHub Issues and codebase.

Here's a look at the main for that Function.

#[tokio::main]
async fn main() -> Result<(), Error> {
    let telemetry_layer = tracing_opentelemetry::layer().with_tracer(init_datadog_pipeline());
    let fmt_layer = tracing_subscriber::fmt::layer()
        .json()
        .with_target(false)
        .with_current_span(false)
        .without_time();

    Registry::default()
        .with(telemetry_layer)
        .with(fmt_layer)
        .with(tracing_subscriber::EnvFilter::from_default_env())
        .init();
    // Initialize the Lambda runtime
    let runtime = Runtime::new(service_fn(handler));
    runtime.run().await?;
    Ok(())
}

What that produces is no spans for the initial Lambda Invocation and allows me to start tracing in the handler function. I of course could have added a span in here, but I wouldn't have context because my main function doesn't know about the web API request.

However, when I get into my handler code, I do have the context because I have access to the HTTP Headers for the request. I can then use those to set the parent of the span, which is the traceparent that is my propagation connector.

let mut fields: HashMap<String, String> = HashMap::new();
fields.insert(
    "traceparent".to_string(),
    String::from(
        request
            .payload
            .headers
            .get("traceparent")
            .unwrap()
            .to_str()
            .unwrap(),
    ),
);

let propagator = TraceContextPropagator::new();
let context = propagator.extract(&fields);
let span = tracing::Span::current();
span.set_parent(context);

And now in any subsequent things I do, I have lineage. For instance, I have a function called generate_context that will get included in this chain. I'll show what this looks like when I get into the Datadog section.

#[instrument(name = "AddContext")]
fn generate_context() -> AddedContext {
    AddedContext {
        timestamp: Utc::now().timestamp_millis(),
        description: "From Read".to_string(),
    }
}

Rust Lambda Function 3

To complete the sequence, I need to build the Lambda Function that responds to the event that is posted on the SQS. With CDK, I first build the function like I did above and then I'll attach the correct permissions to read from the queue.

const changeFunction = new RustFunction(this, 'ChangeFunction', {
  architecture: Architecture.ARM_64,
  functionName: "sample-handle-change-function",
  manifestPath: path.join(__dirname, `../../lambdas/handle-change-function`),
  memorySize: 256,
  environment: {
    RUST_LOG: 'info',
    FUNCTION_NAME: "handle-change-function",
    DD_API_KEY: process.env.DD_API_KEY!,
    DD_SITE: process.env.DD_SITE!,
    AGENT_ADDRESS: '127.0.0.1'
  },
  layers: [layer]
});

const queue = new Queue(this, 'PostQueue', {
  queueName: 'sample-post-queue'
});

queue.grantConsumeMessages(changeFunction);
changeFunction.addEventSource(new SqsEventSource(queue, {
  batchSize: 10,
}))

Before diving into the Function, I want to circle back to the originating Lambda Function to show how I'm connecting the traceparent back into this function. I'm going to carry the context in the correlation_id field.

#[derive(Serialize, Debug)]
struct MessageBody {
    timestamp: i64,
    description: String,
    id: String,
    correlation_id: String,
}

I'm then populating the value from the trace context that I'm sending in the web request so I know that each of these will have the same trace parent id.

let mut trace_parent: Option<String> = None;

propagator.inject_context(&ctx, &mut fields);
let headers = fields
    .into_iter()
    .map(|(k, v)| {
        if k == "traceparent" {
            trace_parent = Some(v.clone());
        }
        return (
            HeaderName::try_from(k).unwrap(),
            HeaderValue::try_from(v).unwrap(),
        );
    })
    .collect();

And finally, I'm going to send this all into the function that puts the payload on the SQS queue.

#[instrument(name = "Post Message")]
async fn post_message(
    client: &aws_sdk_sqs::Client,
    mut payload: MessageBody,
    trace_parent: Option<String>,
) -> Result<(), aws_sdk_sqs::error::SdkError<SendMessageError>> {
    match trace_parent {
        Some(x) => {
            payload.correlation_id = x;
        }
        None => payload.correlation_id = "".to_string(),
    }
    let span = tracing::info_span!("SQS");
    let message = serde_json::to_string(&payload).unwrap();
    client
        .send_message()
        .queue_url("<QUEUE_URL>")
        .message_body(&message)
        .send()
        .instrument(span)
        .await?;

    Ok(())
}

And the last piece of the puzzle is to read the message from SQS and set the parent of the span to the trace id for the incoming message.

#[instrument(name = "Handler")]
async fn handler(event: LambdaEvent<SqsEvent>) -> Result<(), &'static str> {
    event.payload.records.into_iter().for_each(|record| {
        let r: MessageBody = serde_json::from_str(record.body.unwrap().as_ref()).unwrap();
        let mut fields: HashMap<String, String> = HashMap::new();
        // find the trace parent id
        fields.insert("traceparent".to_string(), r.correlation_id.clone());

        let propagator = TraceContextPropagator::new();
        let context = propagator.extract(&fields);
        let span = tracing::Span::current();
        // set the parent of the span
        span.set_parent(context);
        span.record("otel.kind", "SERVER");
        tracing::info!("(Body)={}", r.clone());
        tracing::info_span!("Processing Record");
    });
    Ok(())
}

And that is how to connect context from one function to another function when passed through SQS.

Datadog to Bring it Together

I know that standards are a good thing, but I do hope one day to have a native Datadog SDK for Rust. I'm a huge fan of using their SDKs with other languages like Java, C#, and Go, and I'm hopeful at some point we get one for us Rustaceans. However, until then, I'm grateful that the Datadog Lambda Extension will handle OpenTelemetry traces and spans like what I showed you above. To reiterate that, I'm adding the Extension to each of my functions, and setting a few environment variables.

const layer = LayerVersion.fromLayerVersionArn(
  this,
  'DatadogExtension',
  'arn:aws:lambda:us-east-1:464622532012:layer:Datadog-Extension-ARM:68'
)

// in the function definition
environment: {
  DD_API_KEY: process.env.DD_API_KEY!,
  DD_SITE: process.env.DD_SITE!,
  AGENT_ADDRESS: '127.0.0.1'
},

The extension needs to match the runtime architecture that I've chosen. I almost always go ARM (Graviton), so I do the same with the extension. And then I had the Datadog API Key, Site, and an agent address. The agent address is just so that I can tell the OpenTelemetry exporter where to find the OpenTelemetry collector. In my case, the Datadog extension IS the collector.

Trace and Span Graphs

Starting off the visuals, Datadog gives me a nice map to show what my workflow in the Post Function looks like. The visual represents the flow I've been demonstrating in the paragraphs above. Not surprising to see 98% percent of the execution time is attributed to the post-function

Looking deeper into an individual trace which originates with an API request, a better view of all of the work above comes into focus. I can see that all of the instrumentation work shows up in this waterfall graphic.

And lastly, a span list view will show a different picture of the waterfall above.

Tracing Thoughts and Comments

I continue to be blown away by how well Datadog works with my OpenTelemetry traces and spans. The instrumentation is just so required in a modern application. Even if I was building a more monolithic system, the tracing of the logic through that larger application would be a requirement to ship to production. Visualizing, searching, dashboarding and whatnot off of this rich user data will not only help build confidence and trust from your users, but empower teams to tackle new features and deploy more often with safety. It's just a requirement for anything I'm working on. And Datadog is at the center of it all for me.

One thing I didn't mention as a contra thought, is that I'm using child spans to relate to other spans and this works fine in this example. However, there are times where I want to imply a more casual or loose relationship between spans. Enter a Span Link. Span links would be perfect for queue reading because the producer shouldn't know much if any about its consumers. So I could argue that the consumers are linked spans and not necessarily child spans. I haven't done this with Datadog before, so I need further, but I wanted to include these thoughts here to help you think as well about how systems can be shown to have connection through OpenTelemetry.

Wrapping Up

I started this article thinking it would be a quick breeze through Rust, Lambda, OpenTelemetry, and how to connect Lambda Functions with Datadog. I hope you are still with me and managed to get through it all. It took me a bit of time to get through some of the Rust Lambda Runtime pieces but once I got it figured out, everything came together nicely.

I seriously can't stress enough, Observability is so often compared in my mind to Caching. People either cache early or cache late. They either plan to observe early, or are forced to observe late due to fires and problems. Get out ahead of the curve and build observability into your projects from day 1. Trust me on this one.

And as promised, here's the GitHub repository that contains the full working source code that I highlighted in the article. Feel free to clone, fork, or submit a PR. Enjoy!

Thanks for reading and happy building!

Generate a Momento Disposable Token with Rust and Lambda

Benjamen Pyle — Fri, 14 Feb 2025 16:57:29 +0000

Working with browser hosted code (UI) requires a developer to be cautious about exposing secrets and tokens. A less than trustworthy person could take these secrets and do things that the user doesn't intend. And while we are all responsible for our internet usage, token and secrets security from an application standpoint falls squarely on a developer's shoulders. This is why when using Momento, I like to take advantage of the Authorization API. What the Authorization API allows me to do is create a disposable token from a secure location, so that my UI clients can just refresh them as needed to work with Topics or Caches. Thus, not having the credential leak up into the "easy to see" JavaScript code. Let's dive into a Lambda Function coded in Rust that implements this Token Vending Machine concept with Momento.

Article Architecture

I usually like to work backwards to forwards, meaning I establish what I want in the end and then build from there. When looking at a sample implementation, that means starting from the diagram and walking through what I'm building.

A user's session will need to establish an authenticated and authorized connection to Momento by way of the JavaScript client SDK. Every call to Momento is over an HTTP API request so it's going to get authenticated and authorized. Which is a good thing! However, doing this, requires a token which is what I'll be fetching from the Rust Lambda Function that will be demonstrated throughout the article. The flow goes like this:

1) User requires a token to connect to Momento
2) Browser makes a request to an endpoint backed by a Lambda Function
3) Rust Lambda Function uses a long-lived and secure API Token that has permissions to create short-lived disposable tokens
4) Rust Function uses the Momento SDK to request a token with the supplied Topic and Cache names with scopes to publish and subscribe
5) A token is returned from the Lambda Function where the client code can use to subscribe to a Momento topic.
6) The token has an expiration timestamp represented as a Unix Epoch so that the client can refresh before the token has a chance to expire

So let's walk through those steps above and explore the implementation.

Implementing a Momento Token Vending Machine with Rust

I know I'm focusing on Lambda, Momento, and Rust, but there are many other components that go into what I'd consider a quality Lambda Function build. To address those, let's have a look at the CDK code and what all gets shipped to AWS.

AWS CDK Code

TypeScript has become my goto when it comes to creating AWS infrastructure. I like the CDK, and I especially like having the ability to use the Cargo Lambda CDK Construct. If you haven't used it before, check out the repository and jump into the documentation. It's straightforward and the classes inherit from AWS bases. In addition to Cargo Lambda, I like to include the Datadog Lambda Extension. This piece of goodness allows me to collect my OpenTelemetry traces into the Datadog UI for easy assessment of performance and any latency or error issues. I'll highlight further as the article evolves.

Here we go! The below is the CDK code that brings the above together.

Adding the Datadog Extension

Pay special attention to the following when adding the Datadog extension.

Region: I'm using the region my Lambda function is hosted in
ARM/x64: I'm picking the chip architecture that my Lambda Function is compiled for.
Version: 68 in this case, but :latest can also be used.

const layer = LayerVersion.fromLayerVersionArn(
  scope,
  'DatadogExtension',
  'arn:aws:lambda:us-east-1:464622532012:layer:Datadog-Extension-ARM:68'
)

Long-Lived API Key

I'm going to use a long-lived API key with Momento so that this Lambda Function can make requests without worrying about expiration. This is completely acceptable solution. Think of it like a scoped API key essentially. To set that up, I'm using AWS SecretsManager.

const secret = new Secret(scope, 'MomentoKeySecret', {
  secretName: 'MomentoApiKeySecret',
  secretObjectValue: {
    momentoSecret: SecretValue.unsafePlainText(process.env.MOMENTO_API_KEY!)
  }
});

Cargo Lambda Rust Function

Wrapping up the infrastructure components is the definition of the Rust Lambda Function and granting its ability to read from the secret defined above in SecretsManager. Additionally, I'm exposing the function over a FunctionURL. This of course could be internal behind an Application Load Balancer or exposed behind a variety of API Gateway setups. The FunctionURL just makes this example simple to pull together.

Key things to point out in the RustFunction are:

Architecture: set to ARM because I prefer to run on the AWS Graviton chips
Environment:
- Setting RUST_LOG allows me to control crate log levels (this is a convention)

const vendingMachine = new RustFunction(scope, 'TokenVendingMachineFunction', {
  architecture: Architecture.ARM_64,
  functionName: "momento-token-vending-machine",
  manifestPath: path.join(__dirname, `../../../lambdas/`),
  memorySize: 256,
  environment: {
    RUST_LOG: 'info',
    FUNCTION_NAME: "token-vending-machine",
    DD_API_KEY: process.env.DD_API_KEY!,
    DD_SITE: process.env.DD_SITE!,
    AGENT_ADDRESS: '127.0.0.1'
  },
  layers: [layer]
});

new FunctionUrl(scope, 'AuthUrl', {
  function: vendingMachine,
  authType: FunctionUrlAuthType.NONE,
  cors: {
    allowedOrigins: ["*"],
    allowedHeaders: ["*"]
  }
})

secret.grantRead(vendingMachine);

Rust Lambda Function

At this point, using CDK, I can easily run a cdk deploy and my code will be live in AWS in just a couple of minutes. However, I'd like to dive in further on the Rust and Lambda code, specifically addressing the Momento Auth pieces

Main and Initializing

All Rust code (unless it's a lib) starts out with a main function. Even Lambda Functions must have a main. In my main below, I'm setting up Momento, Datadog, OpenTelemetry, and other reusable components. Since my handler is what is called over and over, I want to have things warm and in memory, ready to use as events come in.

To initialize the OpenTelemetry, I'm establishing a telemetry layer which I'm registering.

let telemetry_layer = tracing_opentelemetry::layer().with_tracer(init_datadog_pipeline());
let fmt_layer = tracing_subscriber::fmt::layer()
    .json()
    .with_target(false)
    .without_time();

Registry::default()
    .with(telemetry_layer)
    .with(fmt_layer)
    .with(tracing_subscriber::EnvFilter::from_default_env())
    .init();

The next pieces of main are about fetching the Momento API key from the AWS secret I defined in the infrastructure. And with that secret, I'll construct an instance of the Momento Auth client so that I can communicate with the Auth API and create the disposable tokens.

let config = aws_config::load_defaults(BehaviorVersion::latest()).await;
let secrets_client = aws_sdk_secretsmanager::Client::new(&config);

// this is how long the token has before expiring.  If no value is supplied, then the
// default of 60 seconds is used
let expires_duration_minutes = u64::try_from(
    env::var("KEY_EXPIRES_DURATION")
        .as_deref()
        .unwrap_or("")
        .parse()
        .unwrap_or(60),
)?;
let resp = secrets_client
    .get_secret_value()
    .secret_id("MomentoApiKeySecret")
    .send()
    .await?;
let string_field = resp
    .secret_string()
    .expect("Secret string must have a value");
let secret_string: MomentoSecretString = serde_json::from_str(string_field)
    .expect("Secret string must serde into the correct type");
let cache_client = AuthClient::builder()
    .credential_provider(CredentialProvider::from_string(
        secret_string.momento_secret,
    )?)
    .build()?;
let shared_cache_client = &cache_client;

Function Handler

With all Lambda Functions, I need to define a function that will be called when the Lambda Function is supplied events. For web APIs, that event is a request from an external client. My main establishes this connection by the following code.

run(service_fn(move |event: Request| async move {
    handler(shared_cache_client, expires_duration_minutes, event).await
}))
.await

As exposed, I need to send a Momento client, the expiration in minutes I want to let the token be valid for, and the event which is the web request.

async fn handler(
    client: &AuthClient,
    token_expires_in_minutes: u64,
    request: Request,
) -> Result<impl IntoResponse, Error> {
  let body = request.body();
  let body_string = std::str::from_utf8(body)?;
  let parsed_body = serde_json::from_str::<TokenRequest>(body_string);

  match parsed_body {
      Ok(token_request) => {
          let token = generate_token(
              client,
              token_expires_in_minutes,
              token_request.cache_name,
              token_request.topic_name,
          )
          .await?;
          let token_body = serde_json::to_string(&token)?;
          let response = Response::builder()
              .status(200)
              .header("Content-Type", "application/json")
              .body(token_body)
              .map_err(Box::new)?;
          Ok(response)
      }
      Err(e) => {
          println!("(Error)={}", e);
          let response = Response::builder()
              .status(400)
              .header("Content-Type", "application/json")
              .body("Bad request".to_string())
              .map_err(Box::new)?;
          Ok(response)
      }
  }
}

The Lambda Function handler does the following.

Take in the request
Parse the body of the request
- Body in the correct format then generate the token
- If not, return a 400 BAD REQUEST

For the request body, I'm expecting it to look like this.

{
    "cacheName": "SampleCache",
    "topicName": "SampleTopic"
}

The Rust structure that this serializes into has the following definition.

#[derive(Deserialize, Debug)]
pub struct TokenRequest {
    #[serde(rename = "cacheName")]
    pub cache_name: String,
    #[serde(rename = "topicName")]
    pub topic_name: String,
}

Now with a struct populated with my request data, I can look at how to generate the disposable token. It's much easier than I thought it might be.

Generating the Disposable Token

This disposable token logic is the heart of this Lambda Function's existence. Remember, Client code or the UI is going to request a token that I want to scope down to the cache and topic supplied in the payload. This will guarantee that the client has access to what's needed for the duration defined the environment variable discussed above.

async fn generate_token(
    client: &AuthClient,
    expires_in_minutes: u64,
    cache_name: String,
    topic_name: String,
) -> Result<VendedToken, Error> {
    let query_span = tracing::info_span!("Momento generate token");
    let expires_in = ExpiresIn::minutes(expires_in_minutes);
    let scopes = PermissionScopes::topic_publish_subscribe(
        CacheSelector::CacheName { name: cache_name },
        TopicSelector::TopicName { name: topic_name }
    ) .into();
    let token = client
        .generate_disposable_token(scopes, expires_in)
        .instrument(query_span)
        .await?;
    let expires_at = token.clone().expires_at();
    let vended_token = VendedToken {
        auth_token: token.auth_token(),
        expires_at: expires_at.epoch(),
    };

    Ok(vended_token)
}

Let's break the above down just a little. First up is the query_span and expires_in.

The query_span plugs into OpenTelemetry that allows me to time the Momento operations by way of the Rust Instrument trait. I highly recommend any Rust code you write take advantage of these opportunities. Tracing in the spirit of observability will make finding errors and poor user experiences so much easier when you start to get some volume.

let query_span = tracing::info_span!("Momento generate token");
let expires_in = ExpiresIn::minutes(expires_in_minutes);

The next piece of this function is to create the Disposable token. Scopes are a required parameter to the generate_disposable_token function. For my example, I'm giving the token access to Publish and Subscribe to the Cache/Topic combination. And notice that the expires_at parameter is finally being used to round out the function call.

let scopes = PermissionScopes::topic_publish_subscribe(
    CacheSelector::CacheName { name: cache_name },
    TopicSelector::TopicName { name: topic_name }
) .into();
let token = client
    .generate_disposable_token(scopes, expires_in)
    .instrument(query_span)
    .await?;
let expires_at = token.clone().expires_at();
let vended_token = VendedToken {
    auth_token: token.auth_token(),
    expires_at: expires_at.epoch(),
};

The last piece of the function is to create the VendedToken. The values returned from the Momento function call are used to populate the struct.

#[derive(Serialize, Debug)]
#[serde(rename = "camelCase")]
pub struct VendedToken {
    pub auth_token: String,
    pub expires_at: u64,
}

Measuring Performance with Datadog and OpenTelemetry

So I can't end an article just demonstrating how to fetch disposable tokens written in Rust and deployed in a Lambda Function without talking about performance. I am always blown away at the speed of Momento's services. I hadn't done much work with the Auth API so I wanted to see if the timings that I've been accustomed to with Cache and Topics also held true with Auth.

With the observability code using OpenTelemetry that I've shown above, I'm able to not only track the Lambda Function's execution timings, but also the Momento specific API calls via the Instrument trait that I showed above. I bring this metrics and traces together via Datadog because there isn't a better tool on the market to help me observe my Lambda Functions as well as other cloud resources.

High Level Function Latency

First up is looking at the high level Lambda Function latency. I'm graphing the 50th, 75th, 90th, and 95th percentiles with this Datadog line graph.

I've written about Rust and Lambda performance quite a bit over the past 18 months, but I'm always amazed at how quickly and consistently my function code performs with Rust. I can also make the same statements when it comes to pairing Rust with Momento. Time and time again, their platform performs consistently, regardless of the load and requests I throw at it. The same can be said about the Auth API that I'm exercising here. Consistent p95 latency at the 3ms is just fantastic and not going to be noticeable by an end user.

Breaking it Down Further

High level tracing is great and something that I love about using Datadog, but since I took advantage of the Instrument trait further up, let's have a look at exactly how the Momento Auth operations play into the overall function latency.

This table shows the two spans that are included in the overall latency of the Lambda Functions execution. If you remember from the code well above, I called the Momento Auth span Momento generate token. I'm happy all day long with an average latency of 1.25ms and a tail p99 latency of 2.19ms. I can't recommend their Rust SDK enough. It is my first and preferred way to work with Momento.

Wrapping Up

Working with client code that is insecure by nature that also needs to authenticate with the Momento API for things like Topic subscriptions can be a challenge. However, by implementing a token vending machine with Rust, deployed with Lambda, and monitored with Datadog produces a solution that is fast, reliable, and observable.

I've been saying this for a while, but I truly believe that building Lambda Functions with Rust is the way to go. And I love seeing companies like Momento invest in Rust specific SDKs. This feature to build disposable tokens was just added in 2025 and will unlock developers to implement this vending machine pattern in Rust like I've shown the article.

I included a bunch of code snippets throughout, but if you want the full repository, here is the Github repository

Thanks for reading and happy building!

DSQL Part 2 – More Rust and a Momento Cache

Benjamen Pyle — Sat, 14 Dec 2024 17:30:15 +0000

DSQL Part 2 - More Rust and a Momento Cache

It's been two weeks since the launch of AWS DSQL and I'm still excited about where they are heading with this product. If you want to read about my first impressions, check out that article first and this one will be waiting for you when you return.

As I let that first article settle, I started thinking about single read item performance and as you know, with Lambda, performance affects cost. Lambda cost can summed up as the product of Total Compute and Memory Allocated with total compute being more than just clock time. Lambda is charging per wall time which comes into play when I have I/O bound operations such as SQL queries. The focus of this article is to see how well a single query on a key performs and what that does to overall Lambda performance. And then by adding Momento as a read aside cache, does that have even further benefits in terms of cost and performance.

Let's dive in!

The Solution

The repository that I'll share at the end of the article has 3 binaries in it.

A Cache builder - Rust project for creating a Cache in Momento
Data "Seeder"- Rust project for loading 100K records into a DSQL Table
Lambda Get - Rust project that is an AWS Lambda Function that fetches from the Cache first and then DSQL if not found before writing the data back into the Cache

This is a fairly common scenario and use case when building high traffic APIs. In general, the latency incurred is in waiting on I/O operations. So by leveraging a cache the idea is that the user waits less and by waiting less, I'm charged less by AWS and the other serverless pieces in my application. Let's see if that holds true.

Digging In

There are a couple of prerequisites that if you want to follow along you'll need to take care of.

First, you'll need an account at Momento. It's free to get started, so shuttle on over there and get that taken care of. With an account, you'll then need an API Key. The Momento Docs do a better job than I explaining how this works. Make sure to grant the key admin permissions for future tasks so that you can run the Cache Builder binary.

Second, create a cluster in DSQL. I explained how to do this in my First Impressions article. You'll need to click your way through the AWS Console because CloudFormation and CDK support isn't there yet. It is still only in Preview, so it'll be coming!

Cache Builder

I like to think of caching as something that is often only done either early or late in a project. And to me, early is really the only correct choice, but by doing so early, I incur costly over provisioned resources that eat into budget. This is why most don't cache until later. Either cost or just sheer dependencies on other teams brings developers to wait until it's absolutely necessary to make the leap. And usually at that point, the developer isn't the one making the decision.

With a cache like Momento, I can bring it in whenever I want because it's serverless and I don't incur costs when I'm not using the software. It's a perfect fit for highly available production environments but it also is just what I need in development and QA stacks when traffic is significantly less. Which is why I reach for it as my cache of choice.

Building a cache to be used for my Lambda function is done through the Momento SDK. It covers administration operations as well as application operations. The Cache Builder below follows into the admin or control plane space.

The main thing to point out is that the API Key created in the previous step is used here in the Cache Builder. I'm using an environment variable called MOMENTO_API_KEY and creating a cache called CacheableTable

Run that with:

cargo run

#[tokio::main]
async fn main() -> Result<(), MomentoError> {
    let cache_client = CacheClient::builder()
        .default_ttl(Duration::from_secs(60))
        .configuration(configurations::Laptop::latest())
        .credential_provider(CredentialProvider::from_env_var(
            "MOMENTO_API_KEY".to_string(),
        )?)
        .build()?;
    let cache_name = "CacheableTable";
    match cache_client.create_cache(cache_name).await? {
        CreateCacheResponse::Created => println!("Cache {} created", cache_name),
        CreateCacheResponse::AlreadyExists => println!("Cache {} already exists", cache_name),
    }
    Ok(())
}

I'm all set at this point and ready to move onto step 2. Seeding my table.

Seeding the Table

I toyed with the idea of just doing a few records but ended up moving off of that thought because I wanted to see how well a get by index performed when navigating 100K records. This data seeder project can be adapted to your needs, but right now, it stands up some threads and then each thread loops to 1000 and creates a record. Feel free to tweak this to your needs.

async fn load_data(pool: &PgPool) {
    let mut children = vec![];

    for _ in 0..100 {
        let clone_pool = pool.clone();
        let handle = tokio::spawn(async move {
            for j in 0..1000 {
                let i = CacheableItem::default();

                let result = sqlx::query("INSERT INTO CacheableTable (id, first_name, last_name, created_at, updated_at) VALUES ($1, $2, $3, $4, $5)")
            .bind(i.id.to_owned())
            .bind(i.first_name.clone())
            .bind(i.last_name.clone())
            .bind(i.created_at)
            .bind(i.updated_at)
            .execute(&clone_pool)
            .await;

                match result {
                    Ok(_) => {
                        println!("(Item)={:?}", i);
                    }
                    Err(e) => {
                        println!("Error saving entity: {}", e);
                        break;
                    }
                }
            }
        });
        children.push(handle);
    }

    for t in children {
        t.await.unwrap();
    }
}

With sufficient data, it's time to dig into the Lambda Function!

Get Lambda

This is where the meat of the fun starts happening. The first two projects are just about setting the table. My Lambda function is where the code gets real and I can start measuring performance. I'm going to walk through the code and how it works and then I'll tackle performance and my takeaways.

Main

This Rust Lambda function will look like so many that I've written and you've read. I'm creating my tracing setup, creating a PgPool for DSQL and then setting up the CacheClient which will interact with Momento.

I am going to use my favorite APM and tracing tool in Datadog to bring together my Tokio and OpenTelemetry tracing. The graphs will make so much of this very real.

One thing to note in all of this below is that I'm setting up 3 libraries. And my favorite in the below is in the Cache Client because of the simplicity of it. I could make the argument that both the tracing and DSQL require more dependencies, but there's something elegant in the Momento one. Developer experience matters. If you are building crates, keep that in mind.

async fn main() -> Result<(), Error> {
    // Create the tracer and establish OTEL pieces
    let tracer = opentelemetry_datadog::new_pipeline()
        .with_service_name("get-lambda")
        .with_agent_endpoint("http://127.0.0.1:8126")
        .with_api_version(opentelemetry_datadog::ApiVersion::Version05)
        .with_trace_config(
            opentelemetry_sdk::trace::config()
                .with_sampler(opentelemetry_sdk::trace::Sampler::AlwaysOn)
                .with_id_generator(opentelemetry_sdk::trace::RandomIdGenerator::default()),
        )
        .install_simple()
        .unwrap();
    let telemetry_layer = tracing_opentelemetry::layer().with_tracer(tracer);
    let logger = tracing_subscriber::fmt::layer().json().flatten_event(true);
    let fmt_layer = tracing_subscriber::fmt::layer()
        .with_target(false)
        .without_time();

    Registry::default()
        .with(fmt_layer)
        .with(telemetry_layer)
        .with(logger)
        .with(tracing_subscriber::EnvFilter::from_default_env())
        .init();

    // DSQL and AWS Config
    let region = "us-east-1";
    let cluster_endpoint = env::var("CLUSTER_ENDPOINT").expect("CLUSTER_ENDPOINT required");
    let momento_key = env::var("MOMENTO_API_KEY").expect("MOMENTO_API_KEY required");
    let cache_name = env::var("CACHE_NAME").expect("CACHE_NAME required");

    // Generate auth token
    let sdk_config = aws_config::load_defaults(BehaviorVersion::latest()).await;
    let signer = AuthTokenGenerator::new(
        Config::builder()
            .hostname(&cluster_endpoint)
            .region(Region::new(region))
            .build()
            .unwrap(),
    );
    let password_token = signer
        .db_connect_admin_auth_token(&sdk_config)
        .await
        .unwrap();

    // Setup connections
    let connection_options = PgConnectOptions::new()
        .host(cluster_endpoint.as_str())
        .port(5432)
        .database("postgres")
        .username("admin")
        .password(password_token.as_str())
        .ssl_mode(sqlx::postgres::PgSslMode::VerifyFull);

    let pool = PgPoolOptions::new()
        .max_connections(10)
        .connect_with(connection_options.clone())
        .await?;
    let shared_pool = &pool;

    // Momento Cache Setup
    let cache_client = CacheClient::builder()
        .default_ttl(Duration::from_secs(5))
        .configuration(configurations::Lambda::latest())
        .credential_provider(CredentialProvider::from_string(momento_key).unwrap())
        .build()?;

    let shared_cache_client = &cache_client;
    let shared_cache_name = &cache_name;

    run(service_fn(move |event: Request| async move {
        function_handler(shared_pool, shared_cache_client, shared_cache_name, event).await
    }))
    .await
}

Function Handler

The handler code is the entry point for my logic. It's where the Rust Runtime stops and I begin. I'm going to share it in snippets to demonstrate the read aside pieces of the caching.

The HTTP Part

The first part of the handler fetches the Item ID that is supplied in the ?id= part of the query string. That ID then allows me to first lookup the item in the CacheableItem cache that I created in Step 1.

let id = request
    .query_string_parameters_ref()
    .and_then(|params| params.first("id"))
    .unwrap();

let mut body = json!("").to_string();
let mut status_code = StatusCode::OK;
let u = Uuid::from_str(id).unwrap();

Query the Cache

With an Item ID, I can then peek at my cache via the CacheClient. In my handler, I'm calling the function with the client, the cache name and the id that I picked up from the query string.

let cache_item = query_cache(cache_client, cache_name.to_owned(), id.to_string()).await;

What's to note here is that the Momento CacheClient abstracts the working with the cache itself. I could easily be fetching from a HashMap and looking for a key in a bag. The bulk of the below is just safe Rust code when working with a Result instead of unwrapping. I could have simplified even more with the ? Operator but the error just works as a MISS and not something that I want to return back to handler function that muddies it up.

Also pay attention to two parts of the function that'll show up in the Datadog bits toward the bottom.

The #[instrument] macro. This will create a span called Query Cache which will be the function span.
The Momento SDK allows me to instrument the query to the cache via the instrument method where I pass in the query_span. This gives me the ability to isolate just the Momento code when looking at performance and success.

#[instrument(name = "Query Cache")]
async fn query_cache(
    client: &CacheClient,
    cache_name: String,
    id: String,
) -> Option<CacheableItem> {
    let query_span = tracing::info_span!("Momento GET");
    let response = client.get(cache_name, id).instrument(query_span).await;

    match response {
        Ok(r) => {
            let item: Result<String, MomentoError> = r.try_into();

            match item {
                Ok(i) => {
                    let o: CacheableItem = serde_json::from_str(i.as_str()).unwrap();
                    tracing::info!("(CacheItem)={:?}", o);
                    Some(o)
                }
                Err(e) => {
                    tracing::info!("(Cache MISS)={}", e);
                    None
                }
            }
        }
        Err(e) => {
            tracing::error!("(GetResponseError)={}", e);
            None
        }
    }
}

Cache HIT or Miss

In the even of a HIT or MISS from the cache, one of two paths will be taken.

A HIT will return the object back and that's the end of the request
A MISS will look for the item in the database and then write the item into the cache

match cache_item {
    Some(i) => {
        tracing::info!("Cache HIT!");
        body = serde_json::to_string(&i).unwrap();
    }
    None => {
        tracing::info!("Cache MISS!");
        let item = query_row(pool, u).await;
        match item {
            Some(i) => {
                write_to_cache(cache_client, cache_name.to_owned(), i.clone()).await;
                body = serde_json::to_string(&i).unwrap();
            }
            None => {
                status_code = StatusCode::NOT_FOUND;
            }
        }
    }
}

Querying the table is a beautiful block of code to me. It shows how well AWS has hidden the DQL implementation because if you didn't know I was using DSQL, you'd think I was querying any normal Postgres database.

Note that this code as well has the instrument macro and the SQLx library allows me to instrument to the SELECT query as well to isolate its peformance.

#[instrument(name = "DSQL Query")]
async fn query_row(pool: &PgPool, u: Uuid) -> Option<CacheableItem> {
    let query_span = tracing::info_span!("DSQL Read");
    let item = query_as!(
        CacheableItem,
        "select id, first_name, last_name, created_at, updated_at from CacheableTable where id = $1",
        u
    )
    .fetch_optional(pool)
        .instrument(query_span)
    .await;

    item.unwrap_or_default()
}

And when the query returns a result, I'm going to write that item back into the cache for next time. Again, same thing on the instrumentation.

#[instrument(name = "Write Cache")]
async fn write_to_cache(client: &CacheClient, cache_name: String, item: CacheableItem) {
    let query_span = tracing::info_span!("Momento SET");

    let value = serde_json::to_string(&item).unwrap();
    let result = client
        .set(cache_name, item.id.to_string(), value.clone())
        .instrument(query_span)
        .await;

    match result {
        Ok(_) => {
            tracing::info!("Cache item set");
            tracing::info!("(Item)={:?}", value);
        }
        Err(e) => {
            tracing::error!("(CacheWriteError)={}", e);
        }
    }
}

Quick Thoughts

Before digging into how this comes together and looking at performance, I wanted to touch upon how simple it was to implement this powerful pattern.

A read aside caching strategy is a straightforward approach to boosting performance. It acts like this.

Get a request from a client
Look for the item in cache
If found
- Return the item
If not found
- Read from durable storage
- Write the item to cache
- Return the item

With Momento, I can set the duration on the item depending upon how often my data changes. Read aside works really well for times when the data doesn't change very often. And if the data does change, you can initiate a cache bust to force a reload via the read aside. It's not ideal in highly volatile data and a write through approach might be a better fit. I'll tackle that in a future post!

But to tie it back to my implementation, this Rust code is very fast, very safe, and honestly not that much to pull together. It turns this Lambda function into a powerhouse.

Instrumentation and Performance

The time to think about instrumentation starts at the first line of code. I don't build any Lambda Functions without instrumentation. Or any event-driven system for that matter. It's just too hard to debug and improve without it. I love leaning on Datadog to bring me powerful visuals and insights into the performance of my functions and systems. And with the latest Lambda Extension coded purely in Rust, the performance makes it a no brainer for me.

It took me a little while in my Rust and Lambda journey to get this right, but the tracing setup I showed you above is rock solid and will yield great results. It can also be easily adapted should you not want to use Datadog. But why I'd ask?

Function Performance

Starting at the top, I ran 30 virtual users through my function for a duration of 15 minutes. I used Postman to run the API request and I also put this snippet in front of it so that I know the cache will get hit consistently. Lastly, I'm using a FunctionURL not APIGW, but I want to focus on the Lambda metrics, not the TTFB, TLS negotiation, DNS resolution and other things not in my control.

var ids = [
    "1340d27f-c5fa-45d1-93ec-91b8465bce4e",
    "12bc9d0a-3e53-45f1-9186-4d3908c5230b",
    "26f1cb7f-94ee-46e6-b1ec-1eeca5ed35b6",
    "cb92d622-eff1-47bc-bd5d-5446664114bc",
    "0ace71c4-0983-453c-8932-265cec7231e2",
    "8e1ded56-ccfb-460c-a301-a830a8d2ef9e",
    "1340d27f-c5fa-45d1-93ec-91b8465bce4e",
    "374fb037-12d9-430a-8fd5-dd6c538774b3",
    "4881a44a-21be-4f93-9533-0995a4ce980a",
    "2b174dc9-c836-441e-84c4-9e2133f2d50d",
    "031b9117-1df6-4f3b-aac2-957ea9d57e3b",
    "bc92ea92-17f6-4805-898f-63bcded8d853",
    "fcbc51f0-ef79-4215-a7a7-2366a093fcf2",
    "1bdb2581-b449-42a1-ae49-37e2e6ff4374",
    "c2e9bd25-bc12-4eec-bd10-936e0c8ead0f",
    "5bf7cbaf-32db-4051-9057-fee0cf4aefca",
    "0d350981-26b5-4998-95ff-1a76b20909df",
    "07a04b77-2010-4b42-a85f-a7a5cd4a9cb9",
    "26740679-1a26-493e-becf-125c3611ad61",
    "971a9f84-da91-4156-8276-5a94e6f14dca",
    "b4c63a1d-8e2f-4589-ae32-670ab999e60d"
];

var i = Math.floor(Math.random() * 20);
pm.collectionVariables.set("ID", ids[i]);

I first want to look at the average function latency. This is mind boggling to me. Remember, my function does this

Handles a request
Deserializes the payload
Does the cache/DSQL pieces
Serializes the result and returns

That to me is an absurdly low latency. I think any user that encounters this GET operation is going to be happy with the results of their query. But let's dig a little deeper at the p90 and see what it looks like.

As I start to bring in more of the sample, I see begin to see the executions that didn't use the cache as well as any cold starts that I might have encountered. And outside of that spike to 40ms, most of the requests remain under 15ms. Still more than acceptable for a GET operation when coming from a browser or another API client. Remember, I'm removing the things I can't control from the discussion here. Optimizing for those is another discussion.

Component Performance

Going a touch deeper, I want to explore what a single trace might look like and breakdown down a cache hit vs a cache miss and see where things stack up. Does adding the cache in Momento make a difference? And do I think the difference is meaningful.

Cache Miss

A cache miss as defined above is when I query Momento and don't find the item I'm looking for. That invocation of the handler will then query DSQL. And by laying the foundation with the instrumentation code, I get full visibility into these operations.

I like looking at these trace graphs in both flame and waterfall. What the below highlights are all of my available spans since the Miss runs all paths. Things to note here.

A Momento read is amazingly fast. 1.89ms is nuts
A DSQL query, is also super fast. It's just under 4 times slower on this particular request
A write to Momento looks just the same as a read at 1.90ms. That might be the most impressive and understated piece of this
Looking at the wrapping function which is the DSQL Query, I'm going to save that entire block plus the Write Cache block the next time I read this key

Cache Hit

Let's take a look at the happy path and a hit. Same two graphs. Essentially tells the story you think it would. Fewer steps equals better performance. Single digit millisecond performance is relative but it's still a boost. And remember, part of the calculation of Lambda's cost is compute time. So waiting on I/O might matter at volume.

Consistency Story

The function performance shows good consistency but I do see some spikes in the p90. And not shown is the p99 here, but there are more spikes that pop up as well. What can that be attributed to? Well, it's the consistent performance of the cache hit vs the spikiness of the cache miss. What I've observed so far is that while DSQL performance is amazing, I do get spots in my calls that latency isn't as smooth. But then also where I see smoothness, I see a more consistent 20ms performance vs the single digit that I see in the trace above. Which furthers leans me to think that performance will get better as this goes GA, but also that you can't underestimate the benefits of putting a cache in place where you want to squeeze that last drop of cost, utilization, and performance out of your application. Caching early is almost always best.

With the proper instrumentation, I can further isolate individual resources in my requests. Datadog does this for me which makes highlighting trouble spots super easy

A table view of the resource breakdown yields this. Each and ever span in all of my traces is represented below. Everything from the cold start load_region to all of the operations I've show above. A couple of things standout

DSQL performance is not bad on average. A p95 latency on a primary key column yields 23ms on average. Considering early on in public preview and all of the work that it does, I'm not disappointed by that. If it never gets any better, I think I'm still OK with it honestly. And the p99 tail at 229ms is not going to impact most. I do need to look more at multiple queries, and building more complex things. But again, this is a start.
Momento's cache is stupid fast and consistent. I ran 21K GETs vs 1.74K DSQL SELECTS and the total time in Momento was less than the DSQL. P99 latency of 3ms and an average of 1.7ms is amazing to me. That's an average 12x improvement when getting a hit vs a miss on my cache when it comes to performance.

And a couple of more visuals that show the consistency of each of these databases.

This is the Momento GET consistency that is in that table graphed over time.

And here is the DSQL Select operation graphed over time as well

Takeways

I'm not sure where to start here, so I'm just going to plow through my thinking.

Rust Continues to Amaze Me

Rust and Lambda still continue to blow me away. The code comes out so clean and it's defect free. I know in spots it might look verbose, but by correctly handling Result and Option, my code is readable and doesn't fail under bad scenarios. I feel like a year into this journey, I'm starting to feel better about my ability to pull things like this together.

Momento

I've said quite a bit about Momento throughout this article. But here's my top 3 things I love about it working with it.

Developer experience. The SDK feels lightweight, yet powerful. They make solid use of the builder pattern. Things are in the right place. I do wish they would add some feature flags so that I could remove the control plane APIs from the data plane APIs but I'm nitpicking there.
They have focused so heavily on the performance and it shows. The best case and worst case scenarios are so close in duration. And then just the overall numbers I see blow me away.
Serverless for the win! Pay as I go? Yes please. This allows me to cache early and not late when things start to hurt. I can leverage my tooling here to delight my customers from day 1. Not on day "it hurts"

DSQL

I'm so excited for DSQL to go GA and get the opportunity to use it in production. I've said before, we took NoSQL too far in Serverless because we had to. I see a future where I can build more with SQL in the future because I miss it. My takeaways are this.

Performance is a little spikey but if they settle in the low 10ms, I'm 100% good with that. It doesn't have to be DynamoDB fast. And I surely don't expect it to be Momento fast. I'd like to see things smooth a little over time too. But for my second pass through, I'm very impressed and encouraged.
No leaky DSQL code in my code. I'm SO glad that AWS leaned into this just being SQL and let me use the tooling and libraries I'm used to when working with SQL. I don't personally like working with Data APIs. I just want to query with SQLx and move on. They delivered on this and again, Developer Experience is so important!
The same as #3 in the Momento category. From what I can tell, this is going to be a Serverless offering. So many new use cases are going to be unlocked and design patterns around the constraints that serverless brings makes me happy. I've been hoping for this for quite some time.

Affects on Cost

I keep going back to this. Serverless compute has a cost component wrapped around your execution. I believe that by adding a cache to prevent reads against a SQL database is super useful. Now if I was having to query against an always own Redis cluster in ElastiCache, I might feel different. But with Momento, I only pay for what I use so I get the best of cost and the best for my users. More developers should be looking into read aside and write through caching options by taking advantage of this approach.

And I can't underscore enough, if you stack this up against one of the more common languages that Lambda functions are built with, Rust will yield you the best bang for your buck. It's going to outperform TypeScript, Python, Dotnet, Java, and even Go. And it generally won't be close.

Observability

None of this analysis would have been possible without tracing. Sure, I could have used println and stamped out some log statements. But when building anything in the Cloud, I'm building observability into my code. OpenTelemetry makes this easy and Datadog brings it together for me.

Wrap Up

Thanks for sticking through this one. I know it was long, dense, and information packed but you made it!

The future is so bright when it comes to Serverless that I can't contain my enthusiasm. I hope that you've seen that DSQL is going to be able to play a big part of your designs going forward. It's the right level of abstraction and will be in some cases an easy swap from what you are doing.

And even though it's amazing, pairing it with Momento can turbocharge your users experiences. It's a game changer when building in the cloud.

As always, here's the Github repository for this article. Clone it, use it, and if you find issues, create a PR.

Thanks for reading and happy building!

First Impressions of AWS DSQL with Lambda and Rust

Benjamen Pyle — Sat, 07 Dec 2024 14:47:01 +0000

Serverless developers that use Lambda as their compute of choice have long had to make a trade-off in AWS when it comes to storing their application data. Do I use a purely serverless database in DynamoDB that doesn't require any special networking or infrastructure provisioning? Or do I choose to run a dedicated and not serverless database in RDS and incur the additional cost and be forced to attach my Lambda Function to a VPC so that I can leverage AWS RDS Proxy. For disclosure, I love DynamoDB and generally reach for it over an RDS solution because I'll take its serverless properties and its limitations around things like reporting and broader searching over sticking to tried and true SQL. However, my opinion is that serverless developers reach too often for DynamoDB and would be better suited using RDS or a SQL-based system to service their application data.

But maybe that choice doesn't have to be made going forward. With the release of AWS Aurora Distributed SQL at re:Invent 2024, developers can take advantage of a SQL-based system AND enjoy the benefits of a serverless database while also no longer requiring the Lambda Function to be attached to a VPC to leverage RDS Proxy.

Let's take a look at how this comes together with AWS DSQL, Lambda, and Rust.

AWS DSQL

AWS Distributed SQL (DSQL) is a very recent launch that occurred at re:Invent 2024 rolling in a new option for serverless developers to enjoy SQL with a purely serverless option. DSQL is described by AWS like this:

Amazon Aurora DSQL is a serverless distributed SQL database with virtually unlimited scale, the highest availability, and zero infrastructure management. Aurora DSQL offers the fastest distributed SQL reads and writes and makes it effortless for you to scale to meet any workload demand without database sharding or instance upgrades. With its active-active distributed architecture, Aurora DSQL ensures strong data consistency designed for 99.99% single-Region and 99.999% multi-Region availability. Its serverless design eliminates the operational burden of patching, upgrades, and maintenance downtime. Aurora DSQL is PostgreSQL-compatible and provides an easy-to-use developer experience. -- AWS

And it further makes these claims about what DSQL promises to deliver for developers.

Virtually unlimited scale
Build always available apps
No infrastructure to manage
Easy to use

The service is still in public preview, but I couldn't wait get my hands on it and measure developer experience, ease of use, and some performance.

We went too far with NoSQL database usage in the serverless community and we did so because the tools didn't exist for us to leverage the broader covering features of traditional SQL in serverless builds. - Benjamen Pyle

DSQL Rust and Lambda

I've written about Rust and Lambda extensively, so this article won't try and convince you that you should be doing all of your Lambda code in Rust. But if you are keeping score, I do like Rust for its

Developer experience
Aversion to bugs
Almost complete removal of exceptions
Package management
And oh, it's kinda fast

But enough of the Rust benefits, how am I pulling these three bits together and what did I do with them? Walking through a simple example, I have a Todo model that is stored and retrieved from a Todos table in DSQL. Then I've built two Lambda Functions. One to handle POST (Insert) and one to handle GET (select). Outside of the DSQL instance (which is only ClickOps at this point), I'm provisioning via CDK

CDK Rust Function

In order to track performance which I'll show later on, I'm using the Datadog Lambda Extension to collect my OpenTelemetry Traces.

const layer = LayerVersion.fromLayerVersionArn(
  this,
  'DatadogExtension',
  'arn:aws:lambda:us-east-1:464622532012:layer:Datadog-Extension-ARM:67'
)

Next, I'm adding my Lambda Function and attaching new IAM permissions so that it can execute DSQL operations. Notice I'm granting dsql:* as this is a just a demo. I normally would be much more granular in what I'm assigning. And lastly, I'm using a FunctionURL to expose the Lambda over an HTTP Endpoint. Super simple and great for what I'm doing here.

const insert = new RustFunction(this, 'InsertFunction', {
  architecture: Architecture.ARM_64,
  functionName: "dsql-insert",
  manifestPath: 'lambdas/insert',
  memorySize: 256,
  environment: {
    CLUSTER_ENDPOINT: process.env.CLUSTER_ENDPOINT!,
    DD_API_KEY: process.env.DD_API_KEY!,
    DD_SERVICE: 'dsql-insert',
    DD_SITE: process.env.DD_SITE!,
    RUST_LOG: 'info',
  },
  layers: [layer]
})

insert.addToRolePolicy(new PolicyStatement({
  effect: Effect.ALLOW,
  actions: ["dsql:*"],
  resources: ["*"]
}))

insert.addFunctionUrl({
  authType: FunctionUrlAuthType.NONE,
  cors: {
    allowedOrigins: ["*"]
  }
})

Creating the DSQL Instance

As I mentioned above, there isn't Cloudformation and thus CDK support for DSQL yet. Being in public preview, there's no doubt that this support will follow soon so I'm not worried at all right now about it. However, I do want to show that setting up an instance is amazingly simple.

From the Console, navigate to the Aurora DSQL section and click Create a New Cluster.

And with a Cluster created, I know have access to a host endpoint which I'll use to connect to in my Rust code.

Rust Code

There is so much choice when working with SQL databases. Do I leverage something simple that allows me to write my own SQL and execute it against the database? Do I choose to go with an Object Relational Mapper (ORM)? And further, do I choose to use an ORM that generates code on my behalf? It all depends on the level of abstraction I'm looking for.

I tend to default to as few as possible, which is why when working with SQL and Rust, I almost always reach for SQLx. Setting up SQLx with AWS DSQL requires using the v4 Signature signing of my credentials as fetched from my AWS configuration. I do this work in my main function so that I can reuse the Postgres Pool in my handler without having to establish this connection outside of the Cold Start initializing cycle. That setup is reflected in the below code.

let region = "us-east-1";
let cluster_endpoint = env::var("CLUSTER_ENDPOINT").expect("CLUSTER_ENDPOINT required");
// Generate auth token
let sdk_config = aws_config::load_defaults(BehaviorVersion::latest()).await;
let signer = AuthTokenGenerator::new(
    Config::builder()
        .hostname(&cluster_endpoint)
        .region(Region::new(region))
        .build()
        .unwrap(),
);
let password_token = signer
    .db_connect_admin_auth_token(&sdk_config)
    .await
    .unwrap();

// Setup connections
let connection_options = PgConnectOptions::new()
    .host(cluster_endpoint.as_str())
    .port(5432)
    .database("postgres")
    .username("admin")
    .password(password_token.as_str())
    .ssl_mode(sqlx::postgres::PgSslMode::VerifyFull);

let pool = PgPoolOptions::new()
    .max_connections(10)
    .connect_with(connection_options.clone())
    .await?;

With a connection established, working with SQLx, I'd never know that it was querying to DSQL. I'm simply taking a payload from the JSON body, converting it into a struct and persisting it in DSQL with parameter values that are bound to the query.

#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct Todo {
    id: String,
    name: String,
    description: String,
    created_at: chrono::DateTime<chrono::Utc>,
    updated_at: chrono::DateTime<chrono::Utc>,
}

impl From<TodoCreate> for Todo {
    fn from(value: TodoCreate) -> Self {
        Todo {
            id: Uuid::new_v4().to_string(),
            description: value.description,
            name: value.name,
            created_at: chrono::Utc::now(),
            updated_at: chrono::Utc::now(),
        }
    }
}

#[instrument(name = "Handler")]
async fn function_handler(
    pool: &Pool<Postgres>,
    event: Request,
) -> Result<Response<String>, Error> {
    let body = event.payload::<TodoCreate>()?;
    let mut return_body = json!("").to_string();
    let mut status_code = StatusCode::OK;

    match body {
        Some(v) => {
            let e: Todo = v.into();
            let query_span =
                tracing::info_span!("Save Todo");
            let result = sqlx::query("INSERT INTO Todos (id, name, description, created_at, update_at) VALUES ($1, $2, $3, $4, $5)")
                .bind(e.id.clone())
                .bind(e.name.clone())
                .bind(e.description.clone())
                .bind(e.created_at) 
                .bind(e.updated_at)
                .execute(pool)
                .instrument(query_span)
                .await;

            match result {
                Ok(_) => {
                    let o = e.clone();
                    info!("(Todo)={:?}", o);
                    return_body = serde_json::to_string(&o).unwrap();
                }
                Err(e) => {
                    tracing::error!("Error saving entity: {}", e);
                    status_code = StatusCode::BAD_REQUEST;
                    return_body = serde_json::to_string("Error saving entity").unwrap()
                }
            }
        }
        None => {
            status_code = StatusCode::BAD_REQUEST;
        }
    }

    let response = Response::builder()
        .status(status_code)
        .header("Content-Type", "application/json")
        .body(return_body)
        .map_err(Box::new)?;
    Ok(response)
}

SQLx Aside

When using SQLx, it often helps to work with Macros that help to convert queries to structs in addition to checking the query against the schema I'm running. This can be done via a Cargo subcommand or it can be done live in the editor. Either way, it requires me to set a DATABASE_URL for SQLx to work with. I wasn't able to get the Postgres connection string with DSQL to work just right, so I ended up cloning my schema into a local instance and running it that way. What happens is, SQLx will create query files in a .sqlx director for its use. This is a small annoyance and might be on me, but I've heard of another developer having the same challenge, so I thought I'd call it out here.

That work can be seen here in the Select Lambda Function code. Also note, I'm injecting a span into the instrument method so I can see how long the query portion of the handler is taking.

let query_span = tracing::info_span!("Query Todos");

let rows = sqlx::query_as!(
    Todo,
    r#"
    select id, name, description, created_at, updated_at from Todos limit 10;
    "#,
)
.fetch_all(pool)
.instrument(query_span)
.await;

Exercising DSQL and Lambda

For measurement of how these pieces work together, I'm leaning on Postman to run some virtual users and then the Datadog Lambda Extension which ships my OpenTelemetry traces.

OpenTelemetry Rust Setup

To highlight how this is setup, here's the code as part of my main function.

let tracer = opentelemetry_datadog::new_pipeline()
    .with_service_name("dsql-insert")
    .with_agent_endpoint("http://127.0.0.1:8126")
    .with_api_version(opentelemetry_datadog::ApiVersion::Version05)
    .with_trace_config(
        opentelemetry_sdk::trace::config()
            .with_sampler(opentelemetry_sdk::trace::Sampler::AlwaysOn)
            .with_id_generator(opentelemetry_sdk::trace::RandomIdGenerator::default()),
    )
    .install_simple()
    .unwrap();
let telemetry_layer = tracing_opentelemetry::layer().with_tracer(tracer);
let logger = tracing_subscriber::fmt::layer().json().flatten_event(true);
let fmt_layer = tracing_subscriber::fmt::layer()
    .with_target(false)
    .without_time();

Registry::default()
    .with(fmt_layer)
    .with(telemetry_layer)
    .with(logger)
    .with(tracing_subscriber::EnvFilter::from_default_env())
    .init();

Tracing Execution

At this point, I have a DSQL cluster, a couple of Lambda Functions, and Datadog ready to capture traces. Generating traffic then comes down to using a Postman Runner which in this case I simulated 30 users. An example of one of those POST requests looks like this.

curl --location 'https://<location>lambda-url.us-east-1.on.aws/' \
--header 'Content-Type: application/json' \
--data '{
    "name": "First",
    "description": "Some description"
}'

The result of this execution on the POST in Datadog shows fairly consistent latency patterns. The spikiness indicates those cold starts grabbing a new database connection.

To wrap up the execution review, I want to look at performance.

DSL Performance

I first want to make this upfront statement. This service (DSQL) is in public preview. It's not GA and ready for live production traffic. I 100% believe that things will get better, faster, and cheaper as time goes on. So I'm not overly concerned with what I'm about to show you. But if I was comparing to working with DynamoDB, there is a clear winner in DDB if I was just going on performance.

Cold Start Latency

The initial thing I noticed is that the first connection from Lambda into DSQL can take up to 200 or 300 ms. That seems fairly comparable to building against other AWS services because I have to establish my credentials from the credential provider and initialize my SDKs. I need to do some more investigation about timing the actual connection initialization and break apart the AWS SDK bits to see exactly where the time is being spent. I don't think this is critical though at this point as 300ms is more than satisfactory for < 5% of executions that fall into cold starts.

Insert Latency

When it comes to just running inserts, thanks to Datadog, I've got a waterfall, flame, and overall latency graph. Let's take a look at what those show.

Flame

Waterfall

Overall Latency

Thoughts on Insert Latency

A few thoughts on this small sample.

The performance seems smooth at the p95 mark with spikes showing in the p99 subset. That's fairly normal from my experience when working with other AWS services.
Being that I didn't introduce into jitter into the virtual users, early bursts in latency are just Lambda Functions spinning up to handle the 30 virtual users
The p95 on just saving the Todo was 19.8ms. That's over double what I normally see working with DynamoDB. Again, this is an early preview.

Select Latency

With inserts covered, what does it look like to run a GET operation? For the below data, I'm running the same query over and over. Pretend this is a grid listing Todos on a page.

let query_span = tracing::info_span!("Query Todos");
let mut return_body = json!("Error Fetching Rows").to_string();
let mut status_code = StatusCode::OK;
let rows = sqlx::query_as!(
    Todo,
    r#"
    select id, name, description, created_at, updated_at from Todos limit 10;
    "#,
)
.fetch_all(pool)
.instrument(query_span)
.await;

And to be fair, I'm going to show the same 3 graphs and give you some thoughts.

Flame

Waterfall

Overall

Thoughts on Insert Latency

A few thoughts on this small sample. It boils down to me being suprised at the performance on these reads.

There are significant differences in the sample sizes. I honestly haven't seen performance like this in any other AWS service that I've worked with. Again, preview, but I'm not excited about this early on. I also need to do more digging.
Normally, I tend to see higher initial p99 latency to account for cold starts, but this operation performed the same way throughout the duration of the run. I need to do some more investigation in future articles on this.
The p95 on selecting the top 10 Todos was 193ms. That's a significant bump from what I'm used to seeing when working with DynamoDB

Impressions and Thoughts

I've said this a few times in this article, but this is an early preview version and a very small sample size. So please take my thoughts and opinions with that disclosure in mind.

The Nice List

First off, I'm very impressed with DSQL. If I could sum up the two things that excite me the most, they'd fall in these ways.

The developer experience feels just like working with SQL. Now I understand this is a Postgres-compatible database, so it's not full Postgres. However, when working with libraries like SQLx which I've used with a traditional serverfull RDS, it worked just the same. At the library level. Binding query parameters, establishing a connection, working with serializing and deserializing structs. All felt the same. I mentioned about the query macro hack, but I won't hold this against DSQL at the moment. Overall, solid A for developer experience.
Serverless experience was also top notch. We went too far with NoSQL database usage in the serverless community and we did so because the tools didn't exist for us to leverage the broader covering features of traditional SQL in serverless builds. It was a delight to not have to connect to a VPC, setup and RDS Proxy and all of those others I've had to do before when working with Lambda and RDS. So again, solid A here for serverless experience.

The Naughty List

(This article was written post re:Invent which makes it the holiday season)

The below at this point to me can be chalked up to being an early preview as well as AWS pattern of shipping rock solid 60% complete features to gain customer feedback to drive more innovation. I'm not in the least surprised at the moment on any of the items on the below list (except read performance).

There are a list of missing capabilities at the moment. This is Postgres-compliant, which covers a large area of what that engine does, but it's not 100%. Here are the known limitations.
As with any cloud offering, there are quotas. I'm less concerned with the Cluster quotas as they can be configured. At this point, I'd be paying attention to the database limits as they aren't configurable. I'd need to compare to a standard Postgres instance to see how these shakeout, but they need to be designed around. DynamoDB has the same type of limits. So again, not surprising. The limits
No support for foreign keys. Part of this is kind of comical to me. I've seen some up in arms comments about how could they release without this? Being that many of us rode the NoSQL wave for years, we abandoned foreign keys for code defined schemas (with some exceptions). So to be shocked that a relational system launched without. I personally am not bothered by it because I don't think people are reaching for DSQL if you are comparing it against SQL Server or Oracle. I also am probably not reaching for it if I'm running containers. My head at the moment is this is a perfect complement to Lambda-based compute operations. But I might change my mind as more features come online.
Performance. I can't leave without saying this. I need to dig more into why and if I can do things to improve. But if performance doesn't improve, especially on reads, this might not be a fit for how I want to build. I fully expect this to get better, but I need to call it out as the numbers and graphs above highlight it.

Wrapping Up

For far too long, developers building applications with their compute running on serverless platforms like Lambda have had limited options for using SQL-based data storage. Especially if they wanted to run a pure serverless stack. Services like SQS, Kinesis, EventBridge, DynamoDB, and StepFunctions complimented Lambda nicely but it always felt a touch odd to not have serverless SQL. AWS has done something about that with the launch of Aurora DSQL.

I'm happy to see this gap addressed. I don't just Believe in Serverless, I know that serverless works at low scale and at high scale. It really isn't debatable at this point. However it's not a silver bullet and I'm always evaluating before starting a new project if it's the right fit for the characteristics required in the final value. Having a pure serverless implementation of SQL gives me a new wrinkle in my architecture game that'll improve delivery. I'm sure of that. I'm just going to be waiting a little while to see how things improve on the performance front. I'm OK with the other limitations because with anyone cloud hosted, it has knobs that only turn so far. I give the ability to "Turn it up to an 11" when I go managed. And I'm 100% OK with that. Because the value I get in the 1 - 10 range is usually overwhelmingly worth it.

If you were following along with the code, here is the Github Repository/

I hope you've enjoyed this early view into using DQL with Rust and Lambda. I'll be doing more dives in this area as the new year rolls in. So stay tuned!

Thanks for reading and happy building!

Evaluating 2 Popular Service Meshes

Benjamen Pyle — Sun, 20 Oct 2024 15:41:26 +0000

The decision to add a Service Mesh to an application comes down to how your application communicates between itself. If for instance your design is heavily asynchronous and relies on events and messages, then a service mesh isn't going to make a lot of sense. If however, you've built an application that is heavily reliant on APIs between itself, then a service mesh is a great piece of technology that can make this communication simpler, safer, more consistent, and observable. I want to explore to very popular implementations in the Kubernetes ecosystem which are Istio and Linkerd.

Why a Service Mesh?

Before jumping into the comparison between these two great products, let's back up one step and define what a service mesh is. I believe Linkerd defines it well.

A service mesh is a tool for adding security, reliability, and observability features to cloud native applications by transparently inserting this functionality at the platform layer rather than the application layer. -- Linkerd

What does that mean though? In plain English, if an application needs to make an HTTP, gRPC, or TCP request to another application, it's the responsibility of the client to interact safely with the API that it is requesting. Meaning, that if a timeout should be set on the request, it's on the client. If the service being requested fails, it's on the client to perform the retry. And if the client isn't allowed to communicate with the service, then that might even fall outside into networking to limit this traffic.

Now imagine your client needs to communicate with 3 or more APIs to perform an operation. This could get tricky and create support and troubleshooting challenges. A service mesh looks to insert itself into
this request pipeline to make it a provider problem and not an application problem anymore. It also gives governance and control at the macro level so that all requests across your application get treated similarly. The larger your application gets, the more helpful this technology will become.

Common Implementation Details

With a why established, how does a service mesh accomplish those above-described benefits and features? A service mesh is implemented as a "sidecar" to the actual application service code that you wish to deploy. This sidecar functions as a proxy which handles all incoming and outgoing requests and then applies the configuration rules defined for that proxy. The diagram below is from the Cloud Native Computing
Foundation and is a picture of the Linkerd implementation, but Istio functions similarly.

What happens when each of your pods is deployed is that the service mesh proxy gets launched right next to your application code. The service mesh will alter the container's IP Tables to route all traffic in and
out through this proxy. Therefore, seamlessly intercepting the communication with your code and allowing you to gain the benefits without making a modification to your application.

With the traffic controlled, the service mesh sidecar can then can be configured with the available options as per the implementation.

Linkerd vs Istio

Being that this is more of an intro to service mesh by highlighting opinions on these two products, I'm not going to give a side by side comparison. However, I am going to offer my opinions and thoughts on when I might pick one vs the other. I do have some strong yet loosely held opinions on the products, but my general feeling is that I'm a fan of both and would recommend either for someone looking to add a service
mesh to their architecture.

Linkerd

Launched in 2015, it is considered the first of the service meshes. In 2021, it graduated the Cloud Native Computing Foundation and is deployed in many large customers making it a battle-tested and reliable option.
Some notable customers include X-Box, HEB, Adidas, Microsoft, Chase, and GEICO.

Many of the service mesh options on the market share their base from the Envoy proxy which was originally created by Lyft. Linkerd however is not one of them. They recently went through a full rewrite and choose Rust as their language to build the next generation version on. Now remove my love from Rust aside, this was done on purpose to provide a highly performant and low memory usage implementation. Even though the Envoy proxy is written in C++, what you'll find with the Linkerd Rust proxy is that it's slimmer in features and scope than anything built upon Envoy. And that's on purpose.

Linkerd prides itself as being simple to set up and simple to configure and that it just "works". In my experience, I affirm this belief in their product. Setting up Linkerd is as simple as running a couple of
commands against your Kubernetes cluster and annotating your Deployment resource with Linkerd enabled.

apiVersion: v1
kind: Pod
metadata:
    name: nginx-pod
    annotations:
        linkerd.io/inject: enabled # Enable injection
    spec:
        containers:
        - name: nginx
            image: nginx:latest
            ports:
                - containerPort: 80

The second point that sets Linkerd apart is performance. By being simple and singular focused on providing just a proxy, the product leverages Rust appropriately and can perform more proxy requests with less resources. This is surely a bonus because as your scale grows, so does resource consumption and node requirements to satisfy your pods. Pods need memory and compute, and when implementing a service mesh, you must include these sidecar requirements into your overall pod requirements.

Drawbacks

Nothing is a silver bullet and will solve all your problems for less. That rarely ever works out. Before I get into what I find is missing from Linkerd, let me say this. If your system is built from the ground up on Kubernetes and all of your traffic is contained within your cluster or clusters, then these might not apply to you. But if you have external dependencies or perhaps have irresponsible consumers inside or
outside your cluster, take note of these points.

External Service Configuration

When working with code or APIs outside your cluster, Linkerd (at present) does not provide the option to configure timeouts and retries for managing connections. This might not seem like a big deal, but
timeout and retry management are essential to building reliable systems. For contrast, inside the cluster, these settings work amazing and just as expected. This becomes even more critical if the code you are communicating with is outside your control.

The workarounds at this point would be to build some of this into your application code and its HttpClient. Not ideal, but it is a way to get around this limitation.

The Linkerd community is aware of this gap, and they are currently working on providing support through a new Egress feature. I'd expect to see something in the 2.17 or 2.18 version which is just 1 or 2 revisions
from where things sit today at 2.16.

Rate Limiting

This is another missing feature from Linkerd. The idea of rate limiting is that you can protect your service code by being able to shed load in an event that you have a runaway consumer or load that is greater than
what your system is designed to handle. This is more of a remediation feature that you would enact should something start to get out awry.

Linkerd does support the feature of circuit breaking that will reduce traffic to an unhealthy pod should that pod get into a troublesome state. This isn't the same thing as rate limiting, but it is a feature that will help protect the pod fleet should things get overwhelmed.

Istio

Istio on the other hand was launch just two years behind Linkerd in 2017, and it also graduated the CNCF in 2023. Istio was born out of work by Google and others and is built upon the Envoy proxy.

Istio can be thought of as the full-featured service mesh. It includes everything that Linkerd does and then more. However, these features come with a reputation of being complex and harder to manage. Istio, like
Linkerd, has an impressive list of customer case studies. It is being used at Airbnb, FICO, Atlassian, Ebay, Salesforce, Splunk and Google. That list demonstrates that both meshes have been used at higher scale
than most anything that you are building unless you are riding on a SaaS rocket ship on track to join the ranks of some of the larger businesses in the market.

The things I like about Istio are specifically in that it's easy to get started, and it contains every feature and option that I can think of needing. For instance, if you need external service configuration which is missing from Linkerd, it's already built in. External service management is configured just like a normal VirtualService resource which is Istio's way of configuring things like timeouts, retries, and
rate limiting.

I didn't mention this when talking about Linkerd, but it and Istio support the concept of route-based control. This feature shines in Istio due to the additional features, but in both meshes, this is a great
lever to pull should you need to define route-specific controls. But where this goes even further with Linkerd is that you can establish VirtualServices for any of the services your code requires and can have
even more control over the reliability settings when working with other services.

Think of Istio has having all the knobs you require to fine tune your specific needs in an implementation.

Drawbacks

All of this control and fine tuning of settings does come at a cost. That cost shows up in two places.

First off, Istio is more complex and requires more Kubernetes resources to make things come together. That can be offset with the fact that if you don't need all the features, you won't have all of that configuration. But it must be said that nothing comes for free, and that additional configuration might feel like overhead. My opinion though is that once you nail the patterns down, you won't notice the additional
YAML you need to produce.

The second cost is in runtime and compute. Istio is a much heavier service mesh implementation when compared to Linkerd. This will show up as more resources are required in your pods to support the sidecar. This
shows up even further the larger your mesh gets as those routes and proxy rules will be loaded into the sidecar's memory. All of this can be mitigated of course by paying attention to metrics and adjusting your
pod's resource limits. But again, it's a cost to monitor.

Concluding Thoughts

If I was building an application from scratch or had all my application inside of Kubernetes, Linkerd is my choice for a service mesh. Its simplicity and speed win me out as compared the feature-richness of Istio. And Istio is more feature rich. I didn't touch on Ingress and Egress or API Gateways, but Istio supports and has resources for those as well. Whereas Linkerd simply integrates with those other components of a cluster.

Running an application in Kubernetes is like anything else. I tend to prefer fewer dependencies and having just the right amount to get the job done. And Linkerd fits that bill. I find in most solutions; rate limiting isn't such a huge deal for me, and you can manage load shedding many times in the ingress controller. If I need to shed load between my services, I'm doing something wrong as there are other approaches to managing this problem.

Now I do want to mention that AKS and GKE (Azure and Google's Cloud Kubernetes) are leaning into managed Istio which I must do some more research on. If the management of Istio was taken by my cloud provider,
this might swing my opinion a little, but probably not enough to move me off of Linkerd.

Bottom line though, you can't go wrong with either of these solutions. If you need a service mesh, I'd start with these two and make my selections from here. In future articles I might review some other options like Consul, Kong, and Cillium.

Wrapping Up

I hope you've found this to be helpful if you are starting to look at implementing a service mesh into your application. I'm not a Kubernetes expert by any means, but what I'm trying to accomplish is to bring you a
developer's viewpoint on making the best use of this amazing piece of technology which is Kubernetes.

There's a great deal to learn about the ecosystem, but I can attest from experience, including a service mesh into your application will increase reliability, security, observability, and resiliency. And with either of these products, you don't have to start all-in. You can just inject them
into your pods and let them proxy requests. From there, you can begin adding timeouts, retries, mTLS, and many of the other features they provide.

Thanks for reading and happy building!

Evaluating 2 Popular Service Meshes

Benjamen Pyle — Sun, 20 Oct 2024 15:41:26 +0000

Why a Service Mesh?

Before jumping into the comparison between these two great products, let's back up one step and define what a service mesh is. I believe Linkerd defines it well.

A service mesh is a tool for adding security, reliability, and observability features to cloud native applications by transparently inserting this functionality at the platform layer rather than the application layer. -- Linkerd

Common Implementation Details

With the traffic controlled, the service mesh sidecar can then can be configured with the available options as per the implementation.

Linkerd vs Istio

Linkerd

apiVersion: v1
kind: Pod
metadata:
    name: nginx-pod
    annotations:
        linkerd.io/inject: enabled # Enable injection
    spec:
        containers:
        - name: nginx
            image: nginx:latest
            ports:
                - containerPort: 80

Drawbacks

External Service Configuration

The workarounds at this point would be to build some of this into your application code and its HttpClient. Not ideal, but it is a way to get around this limitation.

Rate Limiting

Istio

Istio on the other hand was launch just two years behind Linkerd in 2017, and it also graduated the CNCF in 2023. Istio was born out of work by Google and others and is built upon the Envoy proxy.

Think of Istio has having all the knobs you require to fine tune your specific needs in an implementation.

Drawbacks

All of this control and fine tuning of settings does come at a cost. That cost shows up in two places.

Concluding Thoughts

Wrapping Up

Thanks for reading and happy building!