DEV Community: Vinod Kumar

Check out my new blog on DynamoDB vector search

Vinod Kumar — Sun, 16 Aug 2026 10:16:16 +0000

Vinod Kumar for AWS Community Builders

Aug 16

Build an AI Agent That Searches Your DynamoDB Data Semantically

#ai #vectordatabase #bedrock #awscommunitybuilders

5 min read

Build an AI Agent That Searches Your DynamoDB Data Semantically

Vinod Kumar — Sun, 16 Aug 2026 10:09:50 +0000

Your application data lives in DynamoDB. Orders, products, customers all of it. Now you want users to search it by meaning rather than exact keywords: "waterproof jacket for hiking" should find the right product even if those exact words never appear in the description.

Until recently this meant adding a second database. You would copy your data into a dedicated vector store, embed it there, and run a pipeline to keep the two in sync. That brought a few costs:

Another system to run, secure, and pay for
Sync lag and drift. Update a product description in DynamoDB and the embedding in the vector store is now stale. Search keeps returning confident, well-ranked, wrong results, and nothing errors
An extra round trip. Vector stores return IDs and similarity scores, so every query needs a second lookup against DynamoDB to fetch the actual records

The solution

DynamoDB now supports vector search natively. Embeddings are stored as an attribute on the same item as your operational data, and you query them with a SearchVectors call against a vector index on that attribute. AWS made this generally available on 5 August 2026, quoting single-digit millisecond latency at over 99% recall.

One table. No pipeline. No drift.

This post walks through building it: a retail transactions table, embeddings from Amazon Bedrock, and an AI agent on Lambda that queries it in natural language. Full code is at github.com/vinod827/dynamodb-vector-agent.

The table is just a table

There is no special vector table. It is a normal DynamoDB table holding order lines with all the usual attributes like product name, description, price, brand, category, country of origin, shipment details. The embedding is one more column on the same item.

Vectors use the existing List type, so there is no new data type and no migration. For an existing table, you add embeddings to your items with UpdateItem and create the index afterwards.

The index definition is short (infra/setup_table.py in the repo), but three choices in it matter.

The partition key. This example uses marketplace. Every search must specify exactly one value for it. That sounds restrictive, but it is how the index scales, a search in the US never touches UK data.

Inline filters are equality-only. No ranges, no BETWEEN. To filter on price, store a bucketed string like priceBand: "50-150" alongside the raw number. That is a modeling decision to make before loading anything.

Cosine, for text. DynamoDB also supports Euclidean, for when vector magnitude is meaningful, and Dot product, for cases like recommendations where direction and magnitude both matter. The rule of thumb is to match whatever your embedding model was trained with.

One thing to watch: TableStatus reaches ACTIVE while the vector index is still building. Poll DescribeTable until the index reports ACTIVE too.

Embed once per product, not once per row

Transactional data repeats. In this dataset, 10,000 order lines reference only 800 distinct products.

Embedding row by row means 10,000 Bedrock calls producing 800 different vectors. Caching by product ID makes it 800 calls twelve times fewer here, and more than a thousandfold over a million rows against the same catalogue.

It is just a dictionary, but it is easy to miss while writing the loop.

The search understands the question

The embedding model here is Titan Text Embeddings V2. Worth being clear that this is not a chat model, it takes text and returns 1024 numbers, deterministically, and nothing else.

Searching for "wireless headphones for commuting" in the US marketplace:

Rows 2 and 3 are the interesting part. The Soundcore keyboard description literally contains the words "built for daily commuting". The Logitech headphones say "travel and field work" and never mention commuting at all. Semantic search ranked the headphones higher anyway, because it identified that the question was fundamentally about headphones.

A keyword search would have ranked them the other way round.

Every hit also came back with brand, price, country of origin and order status attached which solves the extra round trip from the problem statement. The match and the record arrive together.

(The sample data is generated from a handful of description templates, so scores cluster more tightly than real product copy would. The ranking behavior is real; the absolute numbers are not meaningful.)

The agent

The agent is small: one Lambda, one tool (agent/handler.py). What makes it an agent rather than a search box is that the model decides what to search for and how to phrase it — users rarely phrase questions the way a product description is written.

Asked "show me winter gear people bought in the UK":

Here are some examples of winter gear purchased in the UK:
Vango Ripstop Nylon Hiking Pack — Outdoor, £60.90. A ripstop nylon hiking pack built for cold conditions, ultralight and packable.
Craghoppers Windproof Accessory — Apparel, £99.49–£111.26. Windproof accessory for cold-weather running, with a breathable mesh design.

Neither description contains the word "winter". The embedding matched "cold conditions" and "cold-weather running" to the idea, and the agent read the rows and wrote them up.

Where it falls short

The handler also logs what the agent searched for:

--- searches the agent ran ---
  {'marketplace': 'UK', 'query': 'winter gear'}  -> 5 hits

It passed the user's words straight through. The system prompt asks it to rewrite the question into product-description language — something like "insulated thermal layers waterproof outerwear" — which would have retrieved a wider set. It skipped that step.

This run used Amazon Nova 2 Lite. Dropping instructions that require an extra reasoning step is common in smaller models, and the failure is quiet: the answer looks fine, and only the trace shows the interesting part did not happen.

Two takeaways. Log what your agent searched for, not just what it answered. And because the Converse API is model agnostic for tool use, switching models is one environment variable and no code change:

export AGENT_MODEL_ID=us.anthropic.claude-sonnet-4-6

Things that will trip you up

The SDK. You need botocore >= 1.43.64. Lambda's bundled version lags, so bundle your own.

Bedrock model access. The Model access console page is retired and models auto-enable on first invocation, but an account can sit at NOT_AUTHORIZED until the model is invoked once from the playground. Anthropic models additionally require a one time use case form.

A missing partition key fails silently. If an item lacks the vector index partition key attribute, the write to the table succeeds and the item is left out of the index. No error, it simply never appears in search results.

Summary

The problem was maintaining two copies of the same data in two systems, and paying for it in cost, complexity, drift and an extra round trip on every query.

The solution removes one of the copies. The embedding is a column, the search is an index, and both live in the same table under the same permissions and the same pay-per-request billing as everything else.

If your data already lives in DynamoDB, semantic search is now a column and an index away.

Code: github.com/vinod827/dynamodb-vector-agent

[Boost]

Vinod Kumar — Tue, 01 Jul 2025 01:21:35 +0000

Vinod Kumar

Jun 30 '25

🧠 Getting Started with Amazon EKS Auto Mode: Simplifying Kubernetes Node Management

#aws #eks #containers #fargate

2 min read

🧠 Getting Started with Amazon EKS Auto Mode: Simplifying Kubernetes Node Management

Vinod Kumar — Mon, 30 Jun 2025 13:56:50 +0000

With Kubernetes adoption growing fast, managing worker nodes, capacity scaling, and right-sizing clusters is becoming increasingly complex. Enter Amazon EKS Auto Mode – a powerful new capability designed to simplify cluster operations by automating infrastructure provisioning and scaling.

In this blog, we'll dive into:

What is EKS Auto Mode?
How it works under the hood
How to create an EKS Auto Mode cluster
Deploying workloads on it

🚀 What is Amazon EKS Auto Mode?

Amazon EKS Auto Mode is a fully managed option to provision Kubernetes clusters without having to manage EC2 instances or Fargate profiles yourself. It automatically provisions serverless compute capacity that matches your pod specifications using AWS-managed node groups under the hood.

It’s ideal for:

Teams looking to reduce infrastructure overhead
Rapid autoscaling workloads
Cost-efficient, just-in-time compute

🔧 How EKS Auto Mode Works

Behind the scenes:

EKS Auto Mode leverages Karpenter, AWS's open-source autoscaler.
It provisions compute (EC2 or Fargate) based on pod resource requests.
It automatically handles zone selection, instance types, and right-sizing.
You get an ephemeral node group with no configuration overhead.

🛠️ Prerequisites

AWS CLI configured
eksctl installed (brew install eksctl)
IAM role with EKS and EC2 permissions
kubectl installed and configured

🧪 Creating an EKS Cluster in Auto Mode
Amazon introduced a new --auto-mode flag in eksctl. Here’s how to launch a cluster:

eksctl create cluster --name eks-auto-cluster \
  --region us-west-2 \
  --auto-mode \
  --kubernetes-version 1.29

This creates:

A control plane
VPC, subnets, security groups
Auto-managed capacity that scales with your workloads

No need to define node groups!

📦 Deploying a Sample App

Let’s deploy a basic NGINX app to test Auto Mode scheduling:

Create a Deployment YAML:

# nginx-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx
spec:
  replicas: 3
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx:latest
        resources:
          requests:
            cpu: "250m"
            memory: "256Mi"
          limits:
            cpu: "500m"
            memory: "512Mi"

Apply the Deployment:

kubectl apply -f nginx-deployment.yaml

💡 As soon as you apply this, EKS Auto Mode will dynamically provision compute based on your resource requests. It can choose optimized EC2 instances behind the scenes.

📈 Observe Auto Scaling in Action

You can watch pods being scheduled and nodes being created:

kubectl get pods -w
kubectl get nodes -w

You can also use the EKS console to view events and node activity.

💸 Pricing Note

You're billed only for the actual compute (EC2 instances) that Auto Mode provisions. There’s no separate charge for Auto Mode itself. Spot instances may also be used to optimize cost.

📌 Limitations (as of 2025)

Only available in select AWS regions
Some custom scheduling features (like affinity) might need tweaks
Works best with recent Kubernetes versions (>=1.28)

🧹 Clean Up

To delete the cluster:
eksctl delete cluster --name eks-auto-cluster --region us-west-2

🎉 Conclusion

Amazon EKS Auto Mode is a game-changer for teams who want to focus on workloads, not infrastructure. It delivers Kubernetes scalability with zero node group management. Just define your pods and let AWS handle the rest.

If you're deploying microservices, running dev/test environments, or just starting with Kubernetes – EKS Auto Mode is worth trying out.

Securing Kubernetes Workloads with Falco

Vinod Kumar — Sun, 11 May 2025 06:49:20 +0000

In today’s cloud-native world, security is paramount. It’s not enough to secure applications only during deployment; runtime security is crucial. Falco, a cloud-native security tool, helps detect threats in real-time across hosts, containers, Kubernetes, and cloud environments.

What is Falco?

Falco is a cloud-native security tool that helps in detecting the threats in real time. It can detect threats across hosts, containers, Kubernetes & cloud environments.

Falco uses the eBPF technology that continuously monitors the events (such as Linux syscalls), and reports for any suspicious activities like abnormal behaviours, potential security threats and compliance violations from apps by triggering the alerts to the team. It’s an open-source CNCF project originally developed by Sysdig.

Why Use Falco?

Falco offers several advantages for runtime security:

Real-time threat detection: It can detect threats like reverse shells, RBAC abuse, and file access.
Lightweight and powerful: Falco leverages eBPF for efficient monitoring.
Highly customizable: You can tailor Falco rules to your specific needs.
Versatile: It supports various event sources.
Easy integration: Falco integrates seamlessly with alerting systems like Slack and webhooks.

How does Falco Works

Falco continuously monitors the event sources, reporting any suspicious activities or potential security threats based on the defined Falco rule. If any of the Falco rule is evaluated to true by the Falco Engine then the event is send as the Falco outputs.

By default, Falco performs a standard output of such events which can then be configured to send it to Slack (over HTTP/S), gRPC protocol, file outputs, etc.

Falco Event Sources

Falco intelligently monitors events from various sources:

🧠 Linux SysCalls: Observes low-level system activity using Kernel Modules or eBPF.
☸️ Kubernetes Audit Logs: Detects unauthorized access attempts and resource manipulation.
📦 Container Lifecycle: Monitors container start, stop, and execution events.
☁️ Cloud Provider Logs: Ingests logs from cloud providers like AWS CloudTrail.
🔧 Custom Event Sources: Supports additional custom event sources.

Falco Rules

Falco Rules are the core of its detection mechanism. These YAML-based rules define the suspicious anomalies to detect. The Falco Rule Engine evaluates these rules, raising alerts based on syscalls and metadata. You can use predefined rules or customize them to meet your specific security requirements.

A Falco rule consists of several components:

rule: Name of the rule
desc: Description of the behavior
condition: Logical expression based on syscalls and fields
output: Alert message format
priority: Severity Level
tags: For organizing the rules

Sample Falco rule:

Falco Outputs

Falco generates outputs that can be used for alerting and further analysis. By default, Falco sends as a standard output.

Falcosidekick

Falcosidekick is a companion tool that acts as a central endpoint for Falco instances, forwarding events to external systems. It can run as daemon process, or can be deployed in Kubernetes cluster.

Falcosidekickui

Falcosidekick UI provides visualization for real-time event monitoring and analysis to the Falco Events using a centralized user-friendly dashboard.

Real-time Event Monitoring
Advances search and filtering options
Categorized view based on priority, source, rules
Integration Support with third-party services like Slack, etc
Simplifies incident response workflows

Demo of Falco

In this demo, a custom rule of Falco has been configured to detect if a pod is reading the senstive files or not in a Kubernetes cluster. And when a suspicious pod violates the defined rule then the Falco Engine detects the event and forwards it to the Slack by Falcosidekick.
All these events can also be then visualized using the dashboard of falcosidekick-ui.

Please note the recorded demo can be found in this link and the source code in the given GitHub repository.

https://drive.google.com/file/d/1Svv0lSg4_Xgf6cVzdXJeduvUmRqtkQRy/view?pli=1

Limitations and Challenges of Falco

While powerful, Falco has some limitations:

🧠 Kernel dependency: It relies on the Linux kernel.
⛔ Detection, not prevention: Falco only detects anomalies, it doesn’t prevent them.
🔔 Potentially noisy alerts: Rules need careful tuning to avoid excessive alerts.
🪟 Limited Windows support: Primarily focused on Linux.
🛠️ Rule management: Custom rule writing requires expertise.

Reference Links

Falco Documentation: https://falco.org
GitHub Projects: https://github.com/falcosecurity/falco
Falco Rules: https://falco.org/docs/concepts/rules/
Falco Installation on Kubernetes: https://falco.org/docs/setup/kubernetes/
Falcosidekick: https://github.com/falcosecurity/falcosidekick
Falcosidekick UI: https://github.com/falcosecurity/falcosidekick-ui
Demo Recording & Code: https://github.com/vinod827/cloudkit-lab/tree/main/iac/k8s/falco

AI Agents and the Future of App Development: A Beginner's Guide (101)

Vinod Kumar — Sun, 20 Apr 2025 02:55:23 +0000

Software development is evolving—and fast. The latest game-changer? AI agents. These are not just fancy chatbots; they are smart digital coworkers who can think, reason, and act. If you're a developer, tech enthusiast, or just curious about the future of apps, this beginner's guide will walk you through the basics—with no jargon.

1. The Old Way of Building Apps: The 3-Tier Model

Before AI came into play, most apps followed a basic three-part architecture:

Frontend: The user interface (web, mobile).
Backend: The logic or brain (business rules, workflows).
Database: Where all the data is stored.

This model worked well but had a limitation: everything had to be explicitly coded. Apps couldn't "think" or adapt—they just did what they were programmed to do.

2. Meet AI Agents: Smarter, More Flexible Workers

AI agents are like intelligent digital interns. They can:

Understand natural language commands.
Break down vague tasks into concrete steps.
Use tools (like APIs or databases) to take action.
Learn and adapt from feedback.

They don’t just follow strict rules—they figure things out, often in creative ways. Give them the right context and tools, and they can handle everything from customer support to DevOps.

3. Tools: Giving AI Agents Superpowers

To actually perform tasks, AI agents need tools. Think of these as plugins or APIs that allow them to:

Fetch data (from databases or APIs)
Send notifications (via email, Slack, etc.)
Perform transactions (process payments, file reports)

Each tool includes:

A clear purpose (e.g., get today's weather).
Input/output definitions.
Boundaries to keep everything safe and controlled.

You don’t need to rewrite your whole app—just give the agent the tools it needs.

🛠️ Want to try building your own AI agent? Frameworks like LangChain and LangGraph give you building blocks for AI workflows. LangChain connects agents to tools and data. LangGraph lets you design agent-based systems using stateful workflows, memory, retries, and branching logic.

4. What is MCP (Model Context Protocol)?

If you’re assigning 10, 50, or 100 tools to an AI agent, you need a way to manage the connections. That’s where MCP (Model Context Protocol) comes in.

MCP is like a universal translator that sits between your AI agent and all its tools.

It helps you:

Connect agents to tools and data sources easily.
Keep things consistent, secure, and reusable.
Avoid writing custom glue code for every new integration.

It’s like USB for AI apps—plug and play.

5. Real-World Use Cases

Here are just a few examples of how AI agents are already transforming apps:

Customer Support: Answer questions, fetch order status, escalate issues.
Finance Apps: Monitor spending and suggest savings tips.

With the right design, AI agents can truly become part of your team.

6. Why This Matters: The Future of Apps

With AI agents + tools + MCP, we’re moving toward apps that:

Are faster and cheaper to build.
Understand users more naturally.
Adapt to change without full rewrites.
Act like collaborative teammates.

Whether you’re building a chatbot, customer experience flow, or backend automation, this model is more modular, intelligent, and flexible than traditional development.

Final Thoughts

AI agents are no longer a futuristic idea—they’re here and evolving fast. Combined with frameworks like LangChain and LangGraph, and unified by MCP, they are changing how we design, build, and operate applications.

Want to start exploring? Begin by thinking about what tools your app could offer to an agent—and what tasks you'd love to automate or simplify.

The future of app development isn’t just code. It’s collaboration—between humans and intelligent agents.

Secure S3 Access for AWS EKS Pods via IAM Role

Vinod Kumar — Mon, 31 Mar 2025 09:21:23 +0000

The AWS IAM (Identity & Access Management) service allows AWS services to interact with each other based on the policies given to its attached role(s).

We can also use the IAM role with a Kubernetes (k8s) native Service Account (SA) which will allow the Pods running in the Kubernetes cluster or AWS Elastic Kubernetes Service (EKS) to talk to AWS service(s).
In this blog, we will see how we can allow a Pod running in AWS EKS to list the objects in the AWS S3 bucket by using the IAM role with Kubernetes native Service Account.
Go to AWS S3 service and create a bucket and then add few objects to it.

Create an Identity Provider:-
a) Copy the OIDC (OpenID Connect) provider URL from the existing AWS EKS cluster, for instance, in my case this is the URL https://oidc.eks.us-east-1.amazonaws.com/id/8B7D06AD395F38CE1EE8EF0AF2922255

b) Go to IAM -> Identity Providers and click on the ‘Add providers’ button and then select ‘OpenID Connect’.

Paste the copied OIDC URL (from EKS) under the ‘Provider URL’ option and click the ‘Get thumbprint’ button and put sts.amazonaws.com under the ‘Audience’ option as shown below. Give tags as necessary and click the ‘Add provider’ button at the end to add the identity provider.

Create an IAM Policy to read objects in the S3 bucket:-
Go to IAM Policy and create a custom policy with the following JSON to read the objects from the S3 bucket:-

{
 “Version”: “2012–10–17”,
 “Statement”: [
    {
      “Effect”: “Allow”,
      “Action”: [
               “s3:ListBucket”,
               “s3:GetObject”
       ],
       “Resource”: “arn:aws:s3:::k8s-nest”
    }
 ]
}

Create an IAM Role with Trust Relationship with Identity Provider:-
Go to the IAM and create a new role with ‘Web identity’ trust type and select the right Identity Provider (from the dropdown) that we created earlier and Audience as sts.amazonaws.com and click on the ‘Next: Permissions’ button as shown below:-

Attach the previously created custom policy to read the S3 bucket objects:-

Give a name to the role and create it.

Then open the same role again and edit its trust relationship to make sure that only a specific Kubernetes Service Account can assume this Role.

Following policy, will only allow the Kubernetes service account named ‘my-sa’ (in the namespace ‘dev’) to assume the role ‘custom-read-s3-bucket-objects’ using the AWS STS AssumeRoleWithWebIdentity.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Federated": "arn:aws:iam::195725532069:oidc-provider/oidc.eks.us-east-1.amazonaws.com/id/8B7D06AD395F38CE1EE8EF0AF2922255"
      },
      "Action": "sts:AssumeRoleWithWebIdentity",
      "Condition": {
        "StringEquals": {
          "oidc.eks.us-east-1.amazonaws.com/id/8B7D06AD395F38CE1EE8EF0AF2922255:sub": "system:serviceaccount:dev:my-sa"
        }
      }
    }
  ]
}

All configurations at the AWS side are done. Now head towards, your CLI (Command Line Interface) to create a Service Account and Pod in the same AWS EKS Cluster.

Create a namespace ‘dev’ in Kubernetes:-

kubectl create ns dev

Create a Service Account named ‘my-sa’:-
Create a service account annotating the IAM Role ARN created before. as shown below.

apiVersion: v1
kind: ServiceAccount
metadata:
 name: my-sa
 namespace: dev
 annotations:
   eks.amazonaws.com/role-arn:  arn:aws:iam::195725532069:role/custom-read-s3-bucket-objects
kubectl create -f my-sa.yaml

Schedule a new Pod using this Service Account:-
Instead of the default service account, use the one which was created above i.e. my-sa, and attach it to the Pod as shown below.

apiVersion: v1
kind: Pod
metadata:
  labels:
    run: my-pod
  name: my-pod
  namespace: dev
spec:
  serviceAccountName: my-sa
  initContainers:
  - image: amazon/aws-cli
    name: my-aws-cli
    command: ['aws', 's3', 'ls', 's3://k8s-nest/']
  containers:
  - image: nginx
    name: my-pod
    ports:
    - containerPort: 80
  dnsPolicy: ClusterFirst
  restartPolicy: Always
status: {}
kubectl create -f my-pod.yaml

Now, check the logs of the init container, you will notice that the Pod was successfully able to assume the role and communicate with AWS S3 to list all the objects in it securely with the principle of least privilege reducing the blast radius.

kubectl logs my-pod my-aws-cli -n dev

Hope you liked this article :)

Summary

In this blog, we learned that how we can use the AWS IAM Role and create Trust Relationship with the Kubernetes Service Account and use that service account with the Pods running inside the AWS EKS to make calls to AWS S3 with the principle of least privilege.
Behind the scene, the OIDC federation allows the Pod to assume the IAM role via the Kubernetes Service Account with AWS STS to receive a JSON Web Token (JWT). In Kubernetes, we then use the projected service account tokens which are valid OIDC tokens, giving each Pod encrypted signed tokens that can be verified by STS against the OIDC provider for establishing its identity by exchanging the OIDC tokens for IAM Role credentials using AssumeRoleWithWebIdentity of AWS STS.
As usual, you will find the complete source code here at my GitHub repository. Please feel free to fork it and add more IaC (Infrastructure as Code) to it:-
https://github.com/vinod827/cloudkit-lab/tree/main/iac/k8s/iam-roles-with-k8s-service-account

Enforcing Kubernetes Deployments: A Deep Dive into Policy as Code with Kyverno

Vinod Kumar — Wed, 11 Sep 2024 16:22:08 +0000

Introduction

This is an introductory blog on Cloud Native Computing Foundation's or CNCF’s incubating project called Kyverno (greek word which means ‘Govern’). In this blog, we will learn what Kyverno is and how we can leverage it in our Kubernetes cluster to ensure the security and governs of our application deployments.

Authorization in Kubernetes

Before heading to Kyverno, just a quick refreshment on how the authorization (or AuthZ) works internally in Kubernetes.
If you are a Kubernetes administrator then you can decide and customize different authorization modes in Kube API Server configuration. It is AlwaysAllow as shown below ( — authorization-mode=AlwaysAllow) however we can always change it to a different mode like AlwaysDeny, Node Authorizer, ABAC (attribute based), RBAC or WebHook. You can read more about each modes here but we are interested with the mode as WebHook where the authorization works with the help of the third-party popular engines like Kyverno or OPA (Open Policy Agent).

An Admission Controller is the most critical module in Kubernetes that decides how an incoming request from a user/machine is validated before it grants the permission. Not just validation, the Admission Controller is also useful to implement the better security measures to protect the cluster.

The Admission Controller performs ValidatingAdmissionWebHook and MutatingAdmissionWebhook.

Following is the flow when Kyverno is used in the cluster:-

During these(Mutating/Validating Admission) phases of Admission Controller, we can consult the incoming request with the third-party policy enforcement engines like Kyverno to actually perform the mutation or validation based on the rules (or policies) defined.

Kyverno Overview

Kyverno is a Kubernetes-native policy engine designed to validate, mutate, and generate configurations for Kubernetes resources. Unlike the other popular enforcement engines like OPA, you do not need to learn a different language like Rego to write the policies. You can define the policies (as Code) natively just like another Kubernetes resources in YAML format.

Kyverno operates as an admission controller, enforcing custom policies when a resource is created, updated, or deleted. You can use Kyverno to enforce security best practices, manage resource quotas, apply custom validation rules, or even automate Kubernetes operations.

Architecture

Kyverno Policies and Rules

A policy in Kyverno is a set of rule(s) where each rule consists of a match declaration with an optional exclude declaration and only one of validate, mutate, generate or verifyImages as child declarations, as shown below:-

A Kyverno Policy can be applied at cluster level (with kind as ClusterPolicy) or a namespace level (with kind as Policy). Once a policy has been applied then Kyverno enforces the best practices ensuring the governance in the cluster during the application deployments.

Application and Testing of Kyverno

This sample Kyverno policy is a cluster level policy (as the kind is ClusterPolicy) which ensures that all the deployments created in the cluster will be tagged appropriately with the specific label, app. And if not then it throws an appropriate error message because validationFailureAction=Enforce

Note, when validationFailureAction=Audit then Kyverno only warns but not fails the deployment.

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: enforce-app-deployment-label
spec:
  validationFailureAction: Enforce 
  rules:
    - name: check-for-label
      match:
        resources:
          kinds:            
            - Deployment
      validate:
        message: "You must have the label, 'app' for all deployments."
        pattern:
          metadata:
            labels:
              app: "?*"

This is just one simple example however you can create complex policies like image pull from authorized source, minimum number of replica count, no latest tag on images, etc. as required, refer to doc for more details.

Applying the Policy

To apply this policy, you need to:

First install the Kyverno in your Kubernetes cluster. Your best friend here is the official documentation for installation, head towards this link
Apply the policy using kubectl kubectl apply -f enforce-app-deployment-label.yaml
Verify if the Kyverno cluster policy has been applied successfully or not

~ » kubectl get clusterpolicy                                                                                                                                                                                                                     
NAME                           ADMISSION   BACKGROUND   VALIDATE ACTION   READY   AGE    MESSAGE
enforce-app-deployment-label   true        true         Enforce           True    107s   Ready

Testing the Policy

Let us now test the policy by creating a Deployment without the required label:-

apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-app-deploy
spec:
  replicas: 1
  selector:
    matchLabels:
      app: my-app
  template:
    metadata:
      labels:
        app: my-app
    spec:
      containers:
      - image: vinod827/myawesomeapp:3.0
        name: my-simple-app

When you try to creating it, the Kube API server will reject the request with an error message because it is missing the required app label:

~ » kubectl create -f deploy.yaml                                                                                                                                                                                                                 
Error from server: error when creating "deploy.yaml": admission webhook "validate.kyverno.svc-fail" denied the request:

resource Deployment/default/my-app-deploy was blocked due to the following policies

enforce-app-deployment-label:
  check-for-label: 'validation error: You must have the label, ''app'' for all deployments.
    rule check-for-label failed at path /metadata/labels/'

Conclusion

By using Kyverno for Webhook based admission control, the Kubernetes administrators can easily define and enforce custom policies on Kubernetes resources. It offers an intuitive, Kubernetes-native approach to managing policies like label enforcement, resource validation, and security policies, all while simplifying governance in cloud-native environments.

References

https://kubernetes.io/docs/reference/access-authn-authz/authorization/
https://kyverno.io/docs/installation/

Harnessing Apache Kafka on Kubernetes with Strimzi

Vinod Kumar — Thu, 29 Aug 2024 13:40:46 +0000

Strimzi is a CNCF incubating project (moved on Feb 08, 2024) that allows us to run Apache Kafka on the Kubernetes cluster.
The Apache Kafka is a leading tool for building the real-time, event-driven applications. You can also develop and run highly scalable & fault-tolerant data streaming, and data pipelines using Kafka. However, deploying and managing the Kafka infrastructure is always a bit tricky & complicated and this is where Strimzi comes into the picture where it simplifies the whole experience of Kafka on Kubernetes.
In this blog, we will understand what Strimzi is all about and how it is helpful to both the developers & the organization in our Engineering domain.

Why Strimzi to run Kafka on Kubernetes?

Strimzi enables running an Apache Kafka cluster on Kubernetes with various deployment configurations. It uses the operator pattern that makes it easy for running Kafka on Kubernetes.
As a developer, you can leverage Strimzi to run a local Kafka cluster on Minikube on your machine for your end-to-end application development locally without making use of real Kafka servers, say, running on the Cloud or on-premise thus saving the potential cost to the company.
For production, you can customize Strimzi configuration as per your requirements, for instance, rack awareness to distribute Kafka brokers across various AZ (Availability Zones) on Cloud or use Kubernetes' native taints and tolerations to run Kafka on the dedicated nodes. The Kafka can also be exposed outside Kubernetes using NodePort, Load Balancer, or Ingress all of which can be easily secured using TLS certificates.
Strimzi's Kubernetes-native management extends beyond the broker to include Kafka topics, users, and Kafka Connect through Custom Resources. This allows you to manage the complete Kafka applications using your existing Kubernetes processes and tools like kubectl.

Core Principles of Strimzi

High Security: Ensures a secure Kafka cluster with support for TLS, certificate management, and OAuth authentication
Simplicity and Flexibility: Offers a straightforward yet highly configurable setup, enabling Kafka access through Kubernetes-native NodePort, Ingress, and LoadBalancer options
Dedicated Node Deployment: Runs Kafka on dedicated nodes within a Kubernetes cluster
Kubernetes Integration: Allows interaction and management of the cluster using Kubernetes-native tools like kubectl, operator-based approaches, and GitOps
Seamless Integration: Integrates smoothly with other projects like OpenTelemetry, Prometheus, OPA, and more

Architecture - Strimzi Operators

Strimzi provides three operators to manage Kafka on Kubernetes and they are broadly classified as:-

Cluster Operator - This is the main operator, responsible for deploying the Kafka cluster and managing broker configurations. It is also responsible for managing Kafka versions upgrades by rolling out newer version on one broker at a time. It also supports other operands like Kafka Connect, etc.
Topic Operator - Using the KafkaTopic custom resource, this operator is responsible for creating, deleting, and managing the topic(s) on the Kafka cluster created by users.
User Operator - It is responsible for managing the cluster users and related ACL (Access Control List) permissions on the topic using KafkaUser custom resource.

Advantages of using Strimzi

Local Development Support: Developers can use Strimzi with Minikube on their local machines to quickly set up a Kafka cluster for development and testing.
Enhanced Security: Strimzi offers TLS encryption and strong authentication/authorization to safeguard data streams.
Exceptional Performance: Kafka, backed by Strimzi, delivers high throughput and low latency, enabling efficient real-time data processing and analytics.
Robust Data Handling: Strimzi ensures data resilience with features like message ordering, replay capabilities, and message compaction, providing reliable data storage and processing.
Simplified Deployment: Strimzi streamlines Kafka cluster management with custom resources, allowing easy configuration through YAML files. This approach speeds up deployment and minimizes errors.
Scalability and Reliability: Leveraging Kafka's scalability and fault tolerance, Strimzi supports seamless data streaming as your infrastructure grows.

Setting up Kafka using Strimzi on Minikube

As a pre-requisites, you need Docker for Desktop and Minikube configured on your local machine. First start the Minikube using minikube start command on terminal and then follow the steps below to setup Kafka on it.

(1) Create a new Kubernetes namespace for Kafka deployment

kubectl create namespace kafka 
kubectl config set-context --current --namespace=kafka # To default to kafka ns

(2) Deploy the Strimzi operator on this newly created namespace, kafka.

kubectl create -f 'https://strimzi.io/install/latest?namespace=kafka' -n kafka

(3) Now, deploy the Apache Kafka Cluster using the Strimzi CRD (Custom Resource Definition)

kubectl apply -f https://strimzi.io/examples/latest/kafka/kraft/kafka-single-node.yaml -n kafka

To verify the setup of Kafka on Minikube, run the following commands as shown below:-

~ » kubectl get po -n kafka                                                                                                                                                                                       vinod827@Vinods-MacBook-Pro NAME                                          READY   STATUS    RESTARTS   AGE kafka-consumer                                1/1     Running   0          4m33s my-cluster-dual-role-0                        1/1     Running   0          7m22s my-cluster-entity-operator-6b5c9f5764-s5xbc   2/2     Running   0          6m58s strimzi-cluster-operator-865f986d89-tplcs     1/1     Running   0          8m47s                                                                                                                                                                                                              vinod827@Vinods-MacBook-Pro ~ » kubectl get kafka -n kafka                                                                                                                                                                                    vinod827@Vinods-MacBook-Pro NAME         DESIRED KAFKA REPLICAS   DESIRED ZK REPLICAS   READY   METADATA STATE   WARNINGS my-cluster                                                  True    KRaft (base) ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

Voila! it is all done in a very simplified manner.

Let's test the connectivity by sending and receiving the message from this Kafka cluster.

Producer, sending a message:-

~ » kubectl -n kafka run kafka-producer -ti --image=quay.io/strimzi/kafka:0.43.0-kafka-3.8.0 --rm=true --restart=Never -- bin/kafka-console-producer.sh --bootstrap-server my-cluster-kafka-bootstrap:9092 --topic my-topic If you don't see a command prompt, try pressing enter. >hello, Vinod! Demo for Strimzi

Receiver, receiving the message:-

~ » kubectl -n kafka run kafka-consumer -ti --image=quay.io/strimzi/kafka:0.43.0-kafka-3.8.0 --rm=true --restart=Never -- bin/kafka-console-consumer.sh --bootstrap-server my-cluster-kafka-bootstrap:9092 --topic my-topic --from-beginning If you don't see a command prompt, try pressing enter. hello, Vinod! Demo for Strimzi

Conclusion

Strimzi simplifies deploying and managing Apache Kafka on Kubernetes, making it easier for organizations and developers to handle complex data streaming needs. Its Kubernetes-native approach enhances Kafka's flexibility, scalability, and reliability, whether for local development or large-scale production. By embracing Strimzi, you can harness the full power of Kafka on Kubernetes, ensuring your data infrastructure is both robust and future-proof. Start exploring Strimzi today to see how it can transform your Kafka deployments.

References

https://strimzi.io/
https://strimzi.io/docs/operators/latest/overview

Creating custom VPC on AWS using OpenTofu

Vinod Kumar — Mon, 13 May 2024 02:36:58 +0000

The OpenTofu is a Linux Foundation project which is a complete opensource Infrastructure as Code tool, an alternative to the popular Terraform. This essentially means it supports natively Terraform’s HCL (HashiCorp Configuration Language) to write the infrastructure as code.

In this blog, we will see how we can you OpenTofu as Infrastructure as Code (IaC) to provision a custom Virtual Private Cloud (VPC) on Amazon Web Services.

Following is the internal architecture of our custom VPC that we are going to provision on AWS using the OpenTofu:-

However, let us first compare OpenTofu with other popular IaC tools like AWS Cloud Formation or Terraform.

Now let us go ahead and create the VPC using OpenTofu.

Step 1. Install OpenTofu in your system first.

You can execute the following command on MacOS terminal to install the binary for OpenTofu. For other operating systems, refer this documentation link.



brew update
brew install opentofu

Step 2. Setup the AWS provider configuration

Create a file, say 00_provider.tf and copy the following code. Here we have used a variable for AWS region with default as us-east-1 (Northern Virginia) and AWS as the required provider. Also, to connect to our account, we have mapped this to the profile name myaws (which is there in the local path $HOME/.aws/profile).



variable aws_region {
    default = "us-east-1"
    description = "AWS region where the resources will be provisioned"
}


# Configure the AWS Provider
terraform {
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 5.0"
    }
    # helm = {
    #     source = "hashicorp/aws"
    #     version = "~> 2.6"
    # }
  }
}

# Configure region and profile
provider "aws" {
  region = var.aws_region
  profile = "myaws"
}

Step 3. Create a custom VPC configuration and save it in a file, say 01_vpc.tf



resource "aws_vpc" "mycustomvpc" {
    cidr_block = "10.0.0.0/16"
    enable_dns_support = true
    enable_dns_hostnames = true

    tags = {
        "owner" = "vinod"
        "Name" = "my custom VPC"
    }
}

Step 4. Create Internet Gateway and attach it to the VPC



resource "aws_internet_gateway" "igw" { 
    vpc_id = aws_vpc.mycustomvpc.id
    tags = {
        "owner" = "vinod"
        "Name" = "IGW"
    }
}

Step 5. Create Subnets for the VPC



resource "aws_subnet" "private-us-east-1a" {
  vpc_id     = aws_vpc.mycustomvpc.id
  cidr_block = "10.0.1.0/24"
  availability_zone = "us-east-1a"

  tags = {
    "subnet" = "private-us-east-1a"
    "Name" = "Private Subnet"
  }
}

resource "aws_subnet" "private-us-east-1b" {
  vpc_id     = aws_vpc.mycustomvpc.id
  cidr_block = "10.0.2.0/24"
  availability_zone = "us-east-1b"

  tags = {
    "subnet" = "private-us-east-1b"
    "Name" = "Private Subnet"
  }
}

resource "aws_subnet" "public-us-east-1a" {
  vpc_id     = aws_vpc.mycustomvpc.id
  cidr_block = "10.0.3.0/24"
  availability_zone = "us-east-1a"
  map_public_ip_on_launch = true

  tags = {
    "subnet" = "public-us-east-1a"
    "Name" = "Public Subnet"
  }
}

resource "aws_subnet" "public-us-east-1b" {
  vpc_id     = aws_vpc.mycustomvpc.id
  cidr_block = "10.0.4.0/24"
  availability_zone = "us-east-1b"
  map_public_ip_on_launch = true

  tags = {
    "subnet" = "public-us-east-1b"
    "Name" = "Public Subnet"
  }
}

I have created 4 subnets (2 private and 2 public).

Step 6. Create a NAT Gateway and EIP configuration

Create a NAT gateway and attach it to the public subnet. The NAT Gateway allows our instances running within private subnets to access Public Internet for Operating Systems and other software patch updates.



resource "aws_eip" "nat" {
  vpc = true

  tags = {
    "Name" = "EIP"
    "Owner" = "Vinod"
  }

}

resource "aws_nat_gateway" "nat" {
  allocation_id = aws_eip.nat.id
  subnet_id     = aws_subnet.public-us-east-1a.id

  tags = {
    "Name" = "NAT Gateway"
    "Owner" = "Vinod"
  }

  # To ensure proper ordering, it is recommended to add an explicit dependency
  # on the Internet Gateway for the VPC.
  depends_on = [aws_internet_gateway.igw]
}

Step 7. Create route configuration and its association with subnets

Create two route tables (one as private and another as public) with route to NAT Gateway and Internet Gateway respectively. Associate them to their respective private and public subnets.



resource "aws_route_table" "privateroute" {
  vpc_id = aws_vpc.mycustomvpc.id

  route {
    cidr_block = "0.0.0.0/0"
    nat_gateway_id = aws_nat_gateway.nat.id
  }

  tags = {
    Name = "private"
  }
}

resource "aws_route_table" "publicroute" {
  vpc_id = aws_vpc.mycustomvpc.id

  route {
    cidr_block = "0.0.0.0/0"
    gateway_id = aws_internet_gateway.igw.id
  }

  tags = {
    Name = "public"
  }
}

resource "aws_route_table_association" "privateassociation_a" {
  subnet_id      = aws_subnet.private-us-east-1a.id
  route_table_id = aws_route_table.privateroute.id
}
resource "aws_route_table_association" "privateassociation_b" {
  subnet_id      = aws_subnet.private-us-east-1b.id
  route_table_id = aws_route_table.privateroute.id
}
resource "aws_route_table_association" "publicassociation_a" {
  subnet_id      = aws_subnet.public-us-east-1a.id
  route_table_id = aws_route_table.publicroute.id
}
resource "aws_route_table_association" "publicassociation_b" {
  subnet_id      = aws_subnet.public-us-east-1b.id
  route_table_id = aws_route_table.publicroute.id
}

Initialize the tofu project to install all dependencies, modules, etc. by executing on the same directory where all the above .tf files are present



tofu init

To validate our configuration and doing a dry run (without actually provisioning any resources), execute



tofu validate
tofu plan

This will output a summary plan of our change like below for us to review:-



OpenTofu used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
  + create

OpenTofu will perform the following actions:

  # aws_eip.nat will be created
  + resource "aws_eip" "nat" {
      + allocation_id        = (known after apply)
      + arn                  = (known after apply)
      + association_id       = (known after apply)
      + carrier_ip           = (known after apply)
      + customer_owned_ip    = (known after apply)
      + domain               = (known after apply)
      + id                   = (known after apply)
      + instance             = (known after apply)
      + network_border_group = (known after apply)
      + network_interface    = (known after apply)
      + private_dns          = (known after apply)
      + private_ip           = (known after apply)
      + ptr_record           = (known after apply)
      + public_dns           = (known after apply)
      + public_ip            = (known after apply)
      + public_ipv4_pool     = (known after apply)
      + tags                 = {
          + "Name"  = "EIP"
          + "Owner" = "Vinod"
        }
      + tags_all             = {
          + "Name"  = "EIP"
          + "Owner" = "Vinod"
        }
      + vpc                  = true
    }

  # aws_internet_gateway.igw will be created
  + resource "aws_internet_gateway" "igw" {
      + arn      = (known after apply)
      + id       = (known after apply)
      + owner_id = (known after apply)
      + tags     = {
          + "Name"  = "IGW"
          + "owner" = "vinod"
        }
      + tags_all = {
          + "Name"  = "IGW"
          + "owner" = "vinod"
        }
      + vpc_id   = (known after apply)
    }

  # aws_nat_gateway.nat will be created
  + resource "aws_nat_gateway" "nat" {
      + allocation_id                      = (known after apply)
      + association_id                     = (known after apply)
      + connectivity_type                  = "public"
      + id                                 = (known after apply)
      + network_interface_id               = (known after apply)
      + private_ip                         = (known after apply)
      + public_ip                          = (known after apply)
      + secondary_private_ip_address_count = (known after apply)
      + secondary_private_ip_addresses     = (known after apply)
      + subnet_id                          = (known after apply)
      + tags                               = {
          + "Name"  = "NAT Gateway"
          + "Owner" = "Vinod"
        }
      + tags_all                           = {
          + "Name"  = "NAT Gateway"
          + "Owner" = "Vinod"
        }
    }

  # aws_route_table.privateroute will be created
  + resource "aws_route_table" "privateroute" {
      + arn              = (known after apply)
      + id               = (known after apply)
      + owner_id         = (known after apply)
      + propagating_vgws = (known after apply)
      + route            = [
          + {
              + carrier_gateway_id         = ""
              + cidr_block                 = "0.0.0.0/0"
              + core_network_arn           = ""
              + destination_prefix_list_id = ""
              + egress_only_gateway_id     = ""
              + gateway_id                 = ""
              + ipv6_cidr_block            = ""
              + local_gateway_id           = ""
              + nat_gateway_id             = (known after apply)
              + network_interface_id       = ""
              + transit_gateway_id         = ""
              + vpc_endpoint_id            = ""
              + vpc_peering_connection_id  = ""
            },
        ]
      + tags             = {
          + "Name" = "private"
        }
      + tags_all         = {
          + "Name" = "private"
        }
      + vpc_id           = (known after apply)
    }

  # aws_route_table.publicroute will be created
  + resource "aws_route_table" "publicroute" {
      + arn              = (known after apply)
      + id               = (known after apply)
      + owner_id         = (known after apply)
      + propagating_vgws = (known after apply)
      + route            = [
          + {
              + carrier_gateway_id         = ""
              + cidr_block                 = "0.0.0.0/0"
              + core_network_arn           = ""
              + destination_prefix_list_id = ""
              + egress_only_gateway_id     = ""
              + gateway_id                 = (known after apply)
              + ipv6_cidr_block            = ""
              + local_gateway_id           = ""
              + nat_gateway_id             = ""
              + network_interface_id       = ""
              + transit_gateway_id         = ""
              + vpc_endpoint_id            = ""
              + vpc_peering_connection_id  = ""
            },
        ]
      + tags             = {
          + "Name" = "public"
        }
      + tags_all         = {
          + "Name" = "public"
        }
      + vpc_id           = (known after apply)
    }

  # aws_route_table_association.privateassociation_a will be created
  + resource "aws_route_table_association" "privateassociation_a" {
      + id             = (known after apply)
      + route_table_id = (known after apply)
      + subnet_id      = (known after apply)
    }

  # aws_route_table_association.privateassociation_b will be created
  + resource "aws_route_table_association" "privateassociation_b" {
      + id             = (known after apply)
      + route_table_id = (known after apply)
      + subnet_id      = (known after apply)
    }

  # aws_route_table_association.publicassociation_a will be created
  + resource "aws_route_table_association" "publicassociation_a" {
      + id             = (known after apply)
      + route_table_id = (known after apply)
      + subnet_id      = (known after apply)
    }

  # aws_route_table_association.publicassociation_b will be created
  + resource "aws_route_table_association" "publicassociation_b" {
      + id             = (known after apply)
      + route_table_id = (known after apply)
      + subnet_id      = (known after apply)
    }

  # aws_subnet.private-us-east-1a will be created
  + resource "aws_subnet" "private-us-east-1a" {
      + arn                                            = (known after apply)
      + assign_ipv6_address_on_creation                = false
      + availability_zone                              = "us-east-1a"
      + availability_zone_id                           = (known after apply)
      + cidr_block                                     = "10.0.1.0/24"
      + enable_dns64                                   = false
      + enable_resource_name_dns_a_record_on_launch    = false
      + enable_resource_name_dns_aaaa_record_on_launch = false
      + id                                             = (known after apply)
      + ipv6_cidr_block_association_id                 = (known after apply)
      + ipv6_native                                    = false
      + map_public_ip_on_launch                        = false
      + owner_id                                       = (known after apply)
      + private_dns_hostname_type_on_launch            = (known after apply)
      + tags                                           = {
          + "Name"   = "Private Subnet"
          + "subnet" = "private-us-east-1a"
        }
      + tags_all                                       = {
          + "Name"   = "Private Subnet"
          + "subnet" = "private-us-east-1a"
        }
      + vpc_id                                         = (known after apply)
    }

  # aws_subnet.private-us-east-1b will be created
  + resource "aws_subnet" "private-us-east-1b" {
      + arn                                            = (known after apply)
      + assign_ipv6_address_on_creation                = false
      + availability_zone                              = "us-east-1b"
      + availability_zone_id                           = (known after apply)
      + cidr_block                                     = "10.0.2.0/24"
      + enable_dns64                                   = false
      + enable_resource_name_dns_a_record_on_launch    = false
      + enable_resource_name_dns_aaaa_record_on_launch = false
      + id                                             = (known after apply)
      + ipv6_cidr_block_association_id                 = (known after apply)
      + ipv6_native                                    = false
      + map_public_ip_on_launch                        = false
      + owner_id                                       = (known after apply)
      + private_dns_hostname_type_on_launch            = (known after apply)
      + tags                                           = {
          + "Name"   = "Private Subnet"
          + "subnet" = "private-us-east-1b"
        }
      + tags_all                                       = {
          + "Name"   = "Private Subnet"
          + "subnet" = "private-us-east-1b"
        }
      + vpc_id                                         = (known after apply)
    }

  # aws_subnet.public-us-east-1a will be created
  + resource "aws_subnet" "public-us-east-1a" {
      + arn                                            = (known after apply)
      + assign_ipv6_address_on_creation                = false
      + availability_zone                              = "us-east-1a"
      + availability_zone_id                           = (known after apply)
      + cidr_block                                     = "10.0.3.0/24"
      + enable_dns64                                   = false
      + enable_resource_name_dns_a_record_on_launch    = false
      + enable_resource_name_dns_aaaa_record_on_launch = false
      + id                                             = (known after apply)
      + ipv6_cidr_block_association_id                 = (known after apply)
      + ipv6_native                                    = false
      + map_public_ip_on_launch                        = true
      + owner_id                                       = (known after apply)
      + private_dns_hostname_type_on_launch            = (known after apply)
      + tags                                           = {
          + "Name"   = "Public Subnet"
          + "subnet" = "public-us-east-1a"
        }
      + tags_all                                       = {
          + "Name"   = "Public Subnet"
          + "subnet" = "public-us-east-1a"
        }
      + vpc_id                                         = (known after apply)
    }

  # aws_subnet.public-us-east-1b will be created
  + resource "aws_subnet" "public-us-east-1b" {
      + arn                                            = (known after apply)
      + assign_ipv6_address_on_creation                = false
      + availability_zone                              = "us-east-1b"
      + availability_zone_id                           = (known after apply)
      + cidr_block                                     = "10.0.4.0/24"
      + enable_dns64                                   = false
      + enable_resource_name_dns_a_record_on_launch    = false
      + enable_resource_name_dns_aaaa_record_on_launch = false
      + id                                             = (known after apply)
      + ipv6_cidr_block_association_id                 = (known after apply)
      + ipv6_native                                    = false
      + map_public_ip_on_launch                        = true
      + owner_id                                       = (known after apply)
      + private_dns_hostname_type_on_launch            = (known after apply)
      + tags                                           = {
          + "Name"   = "Public Subnet"
          + "subnet" = "public-us-east-1b"
        }
      + tags_all                                       = {
          + "Name"   = "Public Subnet"
          + "subnet" = "public-us-east-1b"
        }
      + vpc_id                                         = (known after apply)
    }

  # aws_vpc.mycustomvpc will be created
  + resource "aws_vpc" "mycustomvpc" {
      + arn                                  = (known after apply)
      + cidr_block                           = "10.0.0.0/16"
      + default_network_acl_id               = (known after apply)
      + default_route_table_id               = (known after apply)
      + default_security_group_id            = (known after apply)
      + dhcp_options_id                      = (known after apply)
      + enable_dns_hostnames                 = true
      + enable_dns_support                   = true
      + enable_network_address_usage_metrics = (known after apply)
      + id                                   = (known after apply)
      + instance_tenancy                     = "default"
      + ipv6_association_id                  = (known after apply)
      + ipv6_cidr_block                      = (known after apply)
      + ipv6_cidr_block_network_border_group = (known after apply)
      + main_route_table_id                  = (known after apply)
      + owner_id                             = (known after apply)
      + tags                                 = {
          + "Name"  = "my custom VPC"
          + "owner" = "vinod"
        }
      + tags_all                             = {
          + "Name"  = "my custom VPC"
          + "owner" = "vinod"
        }
    }

Plan: 14 to add, 0 to change, 0 to destroy.

Finally, execute the following command to create the custom VPC on AWS



tofu apply

You will need to confirm with yes when prompted on the terminal. If you wish to avoid that prompt then use the flag as —-auto-approve like shown below



tofu apple --auto-approve

Voila! its all done :-)

All your custom VPC resources will be created.

Output

If you wish to delete all the resources of the custom VPC, then execute:-



tofu destroy

Summary

In this blog, we have seen what OpenTofu is, how it compares as an open source project with other popular IaC tools and how we can install it in our system to create a custom VPC on Amazon Web Services.

Hope you like the article. Please do share your feedback.

Like always, you will find all the source code used in this blog as a reference at this GitHub project. You can star this GitHub repository to get all updates happening on this active project.

https://github.com/vinod827/k8s-nest/tree/main/iac/aws/terraform/creating-custom-vpc

All k8s manifests lives here. Contribute to vinod827/k8s-nest development by creating an account on GitHub.
github.com

Connect me on LinkedIn and Twitter.

Building a Custom VPC and EKS Cluster on AWS with Terraform

Vinod Kumar — Wed, 01 May 2024 11:54:40 +0000

In today’s rapidly evolving cloud infrastructure landscape, managing resources efficiently and securely is paramount. Infrastructure as Code (IaC) tools like Terraform empower developers and operators to automate the provisioning and management of cloud resources. In this tech blog, we will delve into how Terraform can be utilized to create a custom Virtual Private Cloud (VPC) and an Amazon Elastic Kubernetes Service (EKS) cluster atop it.

Terraform Configuration

You can visit HashiCorp site here to install the Terraform on your machine depending upon your Operating System.

In this blog, we are going to provision a Custom VPC with Security Group and then provision and EKS (Elastic Kubernetes Service) with autoscaling on AWS.

Now, let’s take a closer look at the Terraform script used to orchestrate our infrastructure.

It is always recommended to split these resources into multiple .tf files (terraform files) as they easy to manage and debug later.

Defining variables — Variables like the version of the EKS Cluster (1.29), VPC CIDR range and AWS region can be defined in variables.tf file and can be used across other terraform configuration

## Variables
variable "eks_version" {
  default     = 1.29
  description = "EKS version"
}

variable "vpc_cidr" {
  default     = "10.0.0.0/16"
  description = "CIDR range of the VPC"
}

variable "aws_region" {
  default = "us-east-1"
  description = "AWS Region"
}

# Configure the AWS Provider
provider "aws" {
  region = var.aws_region
  profile = "myaws"
}

Custom VPC and Security Group — The script starts by defining a custom VPC using the terraform-aws-modules/vpc/aws module. It specifies VPC characteristics like CIDR block, availability zones, public and private subnets, and DNS settings. An AWS security group is created to manage inbound and outbound traffic for EKS worker nodes. Ingress and egress rules are defined to control traffic flow. Copy the below script into vpc.tf file.

## VPC setup
data "aws_availability_zones" "available" {}

locals {
  cluster_name = "myekscluster-${random_string.suffix.result}"
}

resource "random_string" "suffix" {
  length  = 9
  special = false
}

module "vpc" {
  source  = "terraform-aws-modules/vpc/aws"
  version = "5.7.0"

  name                 = "myvpc-for-eks"
  cidr                 = var.vpc_cidr
  azs                  = data.aws_availability_zones.available.names
  private_subnets      = ["10.0.1.0/24", "10.0.2.0/24"]
  public_subnets       = ["10.0.4.0/24", "10.0.5.0/24"]
  enable_nat_gateway   = true
  single_nat_gateway   = true
  enable_dns_hostnames = true
  enable_dns_support   = true

  tags = {
    "kubernetes.io/cluster/${local.cluster_name}" = "shared"
    "owner" = "Vinod"
  }

  public_subnet_tags = {
    "kubernetes.io/cluster/${local.cluster_name}" = "shared"
    "kubernetes.io/role/elb"                      = "1"
  }

  private_subnet_tags = {
    "kubernetes.io/cluster/${local.cluster_name}" = "shared"
    "kubernetes.io/role/internal-elb"             = "1"
  }

}

## Security group setup
resource "aws_security_group" "all_worker_mgmt" {
  name_prefix = "all_worker_management"
  vpc_id      = module.vpc.vpc_id
}

resource "aws_security_group_rule" "all_worker_mgmt_ingress" {
  description       = "Allow inbound traffic from eks"
  from_port         = 0
  protocol          = "-1"
  to_port           = 0
  security_group_id = aws_security_group.all_worker_mgmt.id
  type              = "ingress"
  cidr_blocks = [
    "10.0.0.0/8",
    "172.16.0.0/12",
    "192.168.0.0/16",
  ]
}

resource "aws_security_group_rule" "all_worker_mgmt_egress" {
  description       = "Allow outbound traffic to anywhere"
  from_port         = 0
  protocol          = "-1"
  security_group_id = aws_security_group.all_worker_mgmt.id
  to_port           = 0
  type              = "egress"
  cidr_blocks       = ["0.0.0.0/0"]
}

EKS Cluster — The EKS cluster is provisioned using the terraform-aws-modules/eks/aws module. A file eks.tf can be created with below content like cluster name, version, subnet IDs, and other configurations such as managed node groups to provision the resources using terraform.

## EKS setup with autoscaling on worker nodes
module "eks" {
  source          = "terraform-aws-modules/eks/aws"
  cluster_name    = local.cluster_name
  cluster_version = var.eks_version
  subnet_ids      = module.vpc.private_subnets

  enable_irsa = true

  tags = {
    cluster = "demo-cluster"
  }

  vpc_id = module.vpc.vpc_id

  eks_managed_node_group_defaults = {
    ami_type               = "AL2_x86_64"
    instance_types         = ["t3.medium"]
    vpc_security_group_ids = [aws_security_group.all_worker_mgmt.id]
  }

  eks_managed_node_groups = {
    node_group = {
      min_size     = 1
      max_size     = 3
      desired_size = 2
    }
  }
}

Outputs — The outputs are optional though but very useful to refer the resources created through terraform later like the cluster endpoint, cluster id, etc. It can be created with following contents in a file outputs.tf file.

## Outputs
output "cluster_id" {
  description = "EKS cluster id"
  value       = module.eks.cluster_id
}

output "cluster_endpoint" {
  description = "Endpoint for EKS control plane."
  value       = module.eks.cluster_endpoint
}

output "cluster_security_group_id" {
  description = "Security group ids attached to the cluster control plane."
  value       = module.eks.cluster_security_group_id
}

output "region" {
  description = "AWS region"
  value       = var.aws_region
}

output "oidc_provider_arn" {
  description = "ARN of OIDC Provider"
  value = module.eks.oidc_provider_arn
}

And finally, you need versions.tf file with following script:-

terraform {
  required_providers {
    random = {
      source  = "hashicorp/random"
      version = "~> 3.1.0"
    }
    kubernetes = {
      source  = "hashicorp/kubernetes"
      version = ">=2.7.1"
    }
    aws = {
      source  = "hashicorp/aws"
      version = "~> 5.0"
    }
    local = {
      source  = "hashicorp/local"
      version = "~> 2.1.0"
    }
    null = {
      source  = "hashicorp/null"
      version = "~> 3.1.0"
    }
    cloudinit = {
      source  = "hashicorp/cloudinit"
      version = "~> 2.2.0"
    }
  }
}

Once you have all these terraform files ready then you can run following command to initialize the terraform provider plugin (AWS in this example) and modules.

terraform init
Perform a dry run using following command to see all the changes that would be applied:

terraform plan

And then execute following to finally creating the resources on AWS:

terraform apply --auto-approve

Conclusion

By leveraging Terraform, we have automated the creation of a custom VPC and an EKS cluster, simplifying infrastructure management and ensuring consistency across environments. This script serves as a foundation for building scalable and resilient Kubernetes clusters on AWS. Whether you’re deploying a development sandbox or a production-grade environment, Terraform provides the flexibility and control needed to meet your requirements.

Designing a scalable Webhook using AWS Serverless Stack

Vinod Kumar — Sun, 03 Mar 2024 14:08:59 +0000

We have often used Webhook a lot during our interactions with the systems unknowingly but do we know what exactly a Webhook is and how it is better than other system design patterns like short/long polling?

A Webhook is an HTTP/S call that gets triggered between the two systems where the source system triggers a specific event that leads to the notification of the targeted application. A very common example is when you swipe your credit card at the merchant for a Point of Sale (POS) transaction, you receive a notification in your phone about the transaction that you made.

Just think of the same scenario of using a Point of Sale (POS) transaction using the concept of a short/long polling pattern where the mobile application or client (in this above example) has to constantly poll the status of your bank account frequently to see any latest updates for any transaction. This leads to the wastage of computing power costing huge money to the company.

Benefits of using Webhooks

Realtime communication
Loose-coupled architecture
No compute resource wastage unlike in the case of short/long polling
Encryption of the communication in transit over HTTPS
Tokenized, only authorized system to invoke the Webhook

In this blog, I have designed a Webhook system using the AWS serverless services: Highly Scalable, Cost-effective, Resilient, and Highly Available system.

Architecture

Note: You can also use your own Auth system to secure this Webhook Endpoint using a serverless AWS Cognito or SAML-based system (not shown in the architecture) at the API Gateway

How does this serverless Webhook work?

The DNS (Domain Name System) for the Webhook Endpoint is configured with the AWS Route 53 service with an alias record pointing to the endpoint of the API Gateway.

The API Gateway is a regional service from AWS and serves as the entry point for incoming requests efficiently routing them to the AWS SQS (Simple Queue Service). The API Gateway has a default rate limit of 10,000 requests per second (RPS) and a default burst limit of 5,000 requests.

The SQS Queue used here is FIFO (First In First Out) based as the ordering of every request is critical to the Webhook system and not just that the SQS FIFO provides content-based deduplication out of the box, which means if a request from the producer system is triggered twice by mistake within 5-minute span then it will be considered as duplicate and will not reach to the queue. The SQS will act as a reliable buffer, decoupling components and ensuring no data loss during traffic spikes or service failures, allowing the system to retry.

All the messages received in the SQS FIFO will be processed in the same order by the AWS Lambda which can scale out by itself up to 1,000 concurrent executions of Lambda every 10 seconds. The AWS Lambda will be used for the actual processing of the Webhook business logic here. When handling potentially high volumes of requests, Lambda functions further enhance scalability by allowing for event-driven, serverless execution of code, automatically scaling up or down based on demand.

All the logs generated by it will be streamed to the AWS CloudWatch Logs for monitoring and debugging purposes.

If a message is not processed timely or if the SQS queue receives a bad message then it will be moved to the Dead Letter Queue (DLQ) for re-processing of them.

The AWS Certificate Manager will be used to manage the SSL certificates and ensure that they are valid and rotated before their expiration date ensuring that all HTTPS requests are encrypted in transit

This combination of API Gateway, SQS, and Lambda enables robust, cost-effective resilient, and scalable architectures capable of handling varying workloads and maintaining high availability under challenging conditions.

Conclusion

In conclusion, implementing a scalable Webhook system using AWS serverless services offers numerous advantages in terms of real-time communication, loose-coupled architecture, and efficient resource utilization. By leveraging AWS API Gateway, SQS, and Lambda, we can design a robust, cost-effective, resilient, and highly available architecture capable of handling varying workloads and maintaining high availability under challenging conditions. By adopting this architecture, organizations can streamline their workflows, improve system reliability, and reduce operational costs associated with traditional polling-based approaches. Embracing serverless technologies on AWS empowers developers to focus on building and innovating, while AWS handles the underlying infrastructure management, enabling scalable and resilient solutions for diverse use cases.