DEV Community: Adil Ansari

Setup Multi Node Kafka Cluster (KRaft)

Adil Ansari — Tue, 09 Jun 2026 12:27:45 +0000

In this guide I will walk you through running a 3‑node Apache Kafka cluster on AWS EC2 using KRaft (Kafka’s built-in metadata quorum). At the end of this tutorial you’ll end up with:

3 EC2 instances: kafka-01, kafka-02, kafka-03
Each node acts as broker + controller
Ports:
- 9092 for broker/client traffic
- 9093 for controller quorum traffic

This setup uses PLAINTEXT for simplicity. For production, add TLS + SASL, and proper monitoring.

Prerequisites

1) 3 EC2 instances in the same VPC (different AZs for HA)

Recommended: t3a.small or bigger to start (Kafka loves RAM and disk IOPS)
Disk: at least 20–50 GB per node (more if you plan to retain data longer)

2) Each node must be able to reach the other nodes on 9092 and 9093

Now let's move to setup our very own kafka cluster setup from scratch.

Step 1: Set hostnames (run on all 3 nodes)

On each node, set the hostname to match:

On kafka-01:

sudo hostnamectl set-hostname kafka-01

On kafka-02:

sudo hostnamectl set-hostname kafka-02

On kafka-03:

sudo hostnamectl set-hostname kafka-03

Then add the same /etc/hosts block on all nodes:

<KAFKA_01_PRIVATE_IP> kafka-01
<KAFKA_02_PRIVATE_IP> kafka-02
<KAFKA_03_PRIVATE_IP> kafka-03

Quick sanity check (from each node):

getent hosts kafka-01 kafka-02 kafka-03

Step 2: Install Java (run on all 3 nodes)

Kafka runs on the JVM, so Java is non‑negotiable.

On Ubuntu:

sudo apt-get update
sudo apt-get install -y openjdk-17-jre-headless
java -version

I used ubuntu for this tutorial, you can use other linux distributions. Most steps would be similar.

On Amazon Linux 2023:

sudo dnf install -y java-17-amazon-corretto-headless
java -version

Step 3: Download and install Kafka (run on all 3 nodes)

We’ll install Kafka under /opt and run it as a dedicated kafka user.

sudo useradd --system --create-home --home-dir /opt/kafka --shell /usr/sbin/nologin kafka || true
sudo mkdir -p /opt/kafka
sudo chown -R kafka:kafka /opt/kafka

Download Kafka 4.3.0 binary (run in /opt):

If curl isn’t installed:

# Ubuntu
sudo apt-get install -y curl

# Amazon Linux 2023
sudo dnf install -y curl

cd /opt
sudo curl -fL -o kafka_2.13-4.3.0.tgz "https://www.apache.org/dyn/closer.lua/kafka/4.3.0/kafka_2.13-4.3.0.tgz?action=download"
sudo tar -xzf kafka_2.13-4.3.0.tgz
sudo chown -R kafka:kafka /opt/kafka_2.13-4.3.0

Create a log directory (this is where Kafka stores data):

sudo mkdir -p /opt/kafka/logDir
sudo chown -R kafka:kafka /opt/kafka/logDir

Step 4: Create the Kafka config (per node)

We’ll use a custom config file: /opt/kafka_2.13-4.3.0/config/custom-server.properties.

Common config (same on all nodes)

Create the file and paste this base (we’ll change the node-specific lines next):

# ==================== Node Roles & Identity ====================
process.roles=broker,controller
node.id=1

# ==================== Network & Listeners ====================
listeners=PLAINTEXT://kafka-01:9092,CONTROLLER://kafka-01:9093
advertised.listeners=PLAINTEXT://kafka-01:9092

listener.security.protocol.map=CONTROLLER:PLAINTEXT,PLAINTEXT:PLAINTEXT
inter.broker.listener.name=PLAINTEXT
controller.listener.names=CONTROLLER

# ==================== KRaft Quorum Configuration ====================
controller.quorum.voters=1@kafka-01:9093,2@kafka-02:9093,3@kafka-03:9093

# ==================== Storage Layout ====================
log.dirs=/opt/kafka/logDir

# ==================== Topic Defaults & System Replication ====================
num.partitions=6
offsets.topic.replication.factor=2
transaction.state.log.replication.factor=2
transaction.state.log.min.isr=1

# ==================== Log Retention Policies ====================
log.retention.hours=168
log.segment.bytes=1073741824
log.retention.check.interval.ms=300000

Now edit the node-specific values on each node:

On kafka-01
- node.id=1
- listeners=...kafka-01...
- advertised.listeners=...kafka-01...
On kafka-02
- node.id=2
- listeners=...kafka-02...
- advertised.listeners=...kafka-02...
On kafka-03
- node.id=3
- listeners=...kafka-03...
- advertised.listeners=...kafka-03...

Use your editor of choice:

sudo vi /opt/kafka_2.13-4.3.0/config/custom-server.properties

A note on `advertised.listeners`

Whatever you set here is what clients will use to connect. If you plan to connect from your laptop:

advertised.listeners must be reachable from your laptop (security group + routing + correct hostname/IP)

If you only connect from inside the VPC:

keep it on private hostnames/IPs (recommended)

Step 5: Generate a cluster ID (run once, then reuse on all nodes)

On any one node (e.g. kafka-01):

cd /opt/kafka_2.13-4.3.0
sudo -u kafka bin/kafka-storage.sh random-uuid

Example output:

CPRRAdoxRDaL5L1S2nwuJw

Save it somewhere safe. I like keeping it in a file so it’s not lost:

echo "<PASTE_YOUR_CLUSTER_ID_HERE>" | sudo tee /opt/kafka/cluster.id >/dev/null

Now copy that value to all three nodes (same exact cluster ID).

Step 6: Format the log directory (run on all 3 nodes)

This step initializes the KRaft metadata in your configured storage.

On each node:

cd /opt/kafka_2.13-4.3.0
KAFKA_CLUSTER_ID="$(cat /opt/kafka/cluster.id)"
sudo -u kafka bin/kafka-storage.sh format \
  --cluster-id "$KAFKA_CLUSTER_ID" \
  --config config/custom-server.properties

If you re-run the command later and it complains the directory is already formatted, that’s normal (and a good sign).

Step 7: Create `systemd` service on all 3 nodes

Create /etc/systemd/system/kafka.service:

[Unit]
Description=Apache Kafka (KRaft)
After=network.target

[Service]
Type=simple
User=kafka
Group=kafka
WorkingDirectory=/opt/kafka_2.13-4.3.0

Environment=KAFKA_HEAP_OPTS=-Xmx1G -Xms1G
Environment=KAFKA_JVM_PERFORMANCE_OPTS=-XX:+UseG1GC -XX:MaxGCPauseMillis=20 -XX:InitiatingHeapOccupancyPercent=35 -XX:+ExplicitGCInvokesConcurrent

ExecStart=/opt/kafka_2.13-4.3.0/bin/kafka-server-start.sh /opt/kafka_2.13-4.3.0/config/custom-server.properties
Restart=on-failure
RestartSec=3
LimitNOFILE=100000

[Install]
WantedBy=multi-user.target

Start and enable it:

sudo systemctl daemon-reload
sudo systemctl enable --now kafka.service
sudo systemctl status kafka.service

Step 8: Verify the KRaft quorum is healthy

From any node:

cd /opt/kafka_2.13-4.3.0
sudo -u kafka bin/kafka-metadata-quorum.sh \
  --bootstrap-controller kafka-01:9093 \
  describe --status

You should see all voters listed and one node acting as leader.

If this fails, 90% of the time it’s one of these:

9093 blocked between nodes (security group issue)
wrong hostnames (bad /etc/hosts)
node.id mismatch vs controller.quorum.voters

Step 9: Create and test your first topic

Create a topic (replication factor 2 for 3 brokers is OK for a lab):

cd /opt/kafka_2.13-4.3.0
sudo -u kafka ./bin/kafka-topics.sh --create \
  --topic first-topic \
  --bootstrap-server kafka-01:9092 \
  --replication-factor 2 \
  --partitions 3

Describe the topic:

sudo -u kafka ./bin/kafka-topics.sh --describe \
  --bootstrap-server kafka-01:9092 \
  --topic first-topic

Start a producer:

sudo -u kafka ./bin/kafka-console-producer.sh \
  --topic first-topic \
  --bootstrap-server kafka-01:9092

In another terminal, start a consumer:

sudo -u kafka ./bin/kafka-console-consumer.sh \
  --topic first-topic \
  --from-beginning \
  --bootstrap-server kafka-01:9092

Type a few messages in the producer and confirm they show up in the consumer.

Troubleshooting (quick hits)

Clients can’t connect
- Check advertised.listeners (clients connect to that, not listeners)
- Confirm 9092 inbound rules for the client source IP/SG
Nodes don’t form a quorum
- Confirm 9093 is open between the Kafka nodes only
- Verify controller.quorum.voters matches the correct hostnames and IDs
- Ensure each node’s node.id is unique
Permission errors under /opt/kafka/logDir
- Fix ownership: sudo chown -R kafka:kafka /opt/kafka/logDir
Service “starts” but immediately stops
- Use sudo journalctl -u kafka.service -n 200 --no-pager to see the real error

Automation Script

You can automate the Linux-specific steps required to set up Kafka on your Ubuntu or Amazon Linux 2023 VM using the shell script provided in the GitHub repository adilansari488/kafka-multi-node-cluster-setup. Any setup steps outside the VM, such as AWS EC2, Security Group configuration, must still be completed manually.

Summary

If you have followed the above steps correctly, you should now have a Kafka cluster running with three nodes.

Feel free to give your feedback and suggestions.

HAPPY KAFKA 😊

Connect

Get Started with Amazon Transcribe in Easy Steps

Adil Ansari — Sun, 01 Dec 2024 07:48:50 +0000

INTRODUCTION

Amazon Transcribe is a fully managed, automatic speech recognition (ASR) service that makes it easy for developers to add speech to text capabilities to their applications. [AWS]

Key Features of Amazon Transcribe

Batch transcription and Realtime Transcription are Both Available
Supports Multiple Languages
Speaker Identification
Custom Language Model
Custom Vocabulary

There are various other features as well. You can go through their website for more information about Amazon Transcribe features and use cases.

Pre Requisite

The only prerequisite is that you should have an AWS Account with sufficient permissions to access Transcribe and S3. You can use Transcribe from the AWS Console or through AWS SDKs available for multiple languages. For this demo, I'll be utilizing a Lambda function with the Python 3.10 runtime to call the Transcribe API.

Creating Transcribe Job

Step 1: Prepare an S3 Bucket

Before creating the Lambda function, ensure you have an S3 bucket set up to store the media file you want to transcribe.

Step 2: Create a Lambda Function

Create a Lambda function and attach an IAM role with the following permissions:
- AmazonTranscribeFullAccess (AWS Managed Policy)
- S3:GetObject permission (or a fine-grained custom policy if needed).
Refer to this article for guidance on setting up a Lambda function.
Copy the code from this GitHub repository and paste it into the Lambda function editor.

Step 3: Create a Test Event

Create a test event with the following parameters:

{
  "MediaFileUri": "s3://your-bucket-name/media-file.mp4"
}

Step 4: Deploy and Test the Code

Deploy the Lambda function.
Execute the test event. You will receive a response similar to the following:

Step 5: Verify the Transcription

Navigate to the AWS Transcribe service in the AWS Management Console to verify and download the transcription output.

Creating Transcription Downloading Job

Step 1: Create another Lambda Function

Create another Lambda function and attach an IAM role with the following permissions:
- AmazonTranscribeFullAccess (AWS Managed Policy)
- Copy the code from this GitHub repository and paste it into the Lambda function editor.

Step 2: Create a Test Event

Create a test event with the following parameters:

{
  "TranscriptionJobName": "your-jobname-here"
}

Step 3: Deploy and Test the Code

Deploy the Lambda function.
Execute the test event. You will receive a response similar to the following:

Conclusion

We have learned how to use Amazon Transcribe for generating transcription of media files.
We have setup two lambda functions for creating transcription job of a media file stored in S3 and downloading transcription.

Hope you have learned something new.
If you liked this article make sure you give it a heart and comment down your suggestions/feedback.

References

Secure AWS API Gateway with IAM

Adil Ansari — Sun, 15 Sep 2024 05:48:34 +0000

If you're using AWS API Gateway and want to secure it with IAM credentials from your AWS users, you're in the right place.
AWS offers several methods to secure REST APIs, including API keys, Lambda authorizers, and Amazon Cognito, but in this post, we'll focus on using IAM. I'll guide you step-by-step through the process of securing your API Gateway with IAM.

Let’s get started!

API Gateway

Amazon API Gateway is fully managed service by AWS that helps you create and manage APIs for your applications. It acts as a middleman, handling requests and sending them to the right backend services, like AWS Lambda or EC2.

Why IAM

By default, API Gateway APIs are available on a public URL, meaning anyone who has the link can use your API. To protect your data and services, you need to secure it. Using IAM Authorization ensures that only approved users can access your API, keeping your data safe by controlling who can send requests and use it.

Creating an API

For this article I'll be creating a sample REST API in API Gateway and will secure it using IAM.

To create an API, open API Gateway Service in AWS Console and create a new REST API and Deploy it.

In above screenshots, you can check that I have deployed a sample pet store API and invoked it using its invoke URL and its working fine.

Add IAM Authorization

Now let's add IAM authorization to our API method.

Click on the method under the path that you want to secure and then choose Method request and click on Edit button.

In Method request settings choose AWS IAM under Authorization dropdown and click on Save.

Redeploy API and test your API endpoint using invoke URL.

Now your API is secure, and it will return an HTTP 403 (Authentication Error) when accessed without IAM credentials.

Now test your API with IAM credentials using postman. In postman, Go to Authorization tab and choose AWS Signature under Auth Type dropdown and enter your access key, secret key, session token (if STS), and AWS Region. Now try to hit your API endpoint, it will give 200 response code.

Note

Your IAM user should have execute-api:Invoke permission on the API that you want to execute.
AWS Signature authentication method is a specific way to secure APIs provided by AWS. While many client-side libraries may not support it natively, you can use the AWS SDK to call the API Gateway or utilize third-party libraries such as requests-aws4auth in Python, along with similar libraries available for other programming languages.

Conclusion

In this article, We have seen how to secure our AWS API Gateway APIs using IAM credentials and learned how to call them using postman.

If you find this article helpful, do Like, and Follow @adilansari , Adil's Linkedin for more useful content related to AWS, Cloud, DevOps, Linux, and More.

References

Deploying a Web Socket Application on Kubernetes

Adil Ansari — Sun, 09 Jun 2024 17:44:53 +0000

What is a WebSocket

WebSocket is a computer communications protocol, providing a simultaneous two-way communication channel over a single Transmission Control Protocol (TCP) connection. [Wikipedia]

Deploying a WebSocket application on Kubernetes can seem daunting, but this guide will simplify the process for you.

Prerequisites

A running kubernetes cluster.
A WebSocket app to be deployed.
Docker to containerize that application.

You can get a sample websocket app from this github repo adilansari488/websocket-sample-app.

Now let's create kubernetes manifest files.

Manifest files

Deployment.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: ws-app-deployment
  namespace: default
  labels:
    app: ws-app
spec:
  replicas: 1
  selector:
    matchLabels:
      app: ws-app
  template:
    metadata:
      labels:
        app: ws-app
    spec:
      containers:
        - name: ws-app
          image: ws-app:latest
          imagePullPolicy: Always
          securityContext:
            privileged: true
          ports:
            - containerPort: 8819
          resources:
            limits:
              cpu: 1000m
              memory: 1000Mi
            requests:
              cpu: 100m
              memory: 100Mi

Service.yaml

apiVersion: v1
kind: Service
metadata:
  name: ws-app
  namespace: dev
spec:
  selector:
    app: ws-app
  ports:
    - name: ws-app
      port: 8819
      targetPort: 8819

In above manifest files, I have declared port no. 8819 for my websocket app but it can be any port on which your app will run.

Deploying to kubernetes

Now build a docker image of your application or use this repository or directly use this docker image adilansari488/websocket-app to test.
Now apply deployment.yaml and service.yaml to your k8s cluster and get the IP of your service.

Now update the service ip in your client.py and test the connection.

Conclusion

Congratulations! You’ve successfully deployed your first websocket application. If you found this helpful, please like, share, and follow Adil Ansari for more valuable content.

Fetch AWS Secrets in Your Kubernetes Realm

Adil Ansari — Sun, 21 Jan 2024 12:11:19 +0000

INTRODUCTION

It is very crucial to store our credentials securely. And this article will guide you how you can leverage the scaling of aws cloud and power of kubernetes to store and use securely your credentials in your apps deployed on kubernetes environment.
In this guide I'll be using AWS Secrets Manager to store secrets securely and external-secrets in k8s to fetch those secrets securely.

AWS Secrets Manager

AWS secrets manager is a service from AWS that you can use to store your secrets/credentials securely and then use them in your applications to access your secure resources.

Creating Secret in AWS Secret Manager

First let's create our first secret on aws.
Search Secrets manager on your aws console and open it.

You'll have this page, here you can create and store secrets.

Click on store a new secret and select your secret type. For this demo we are choosing Others type of secret. Here you can store your secrets in key value pair, or you can give it json data also by clicking on Plaintext mode.

Then click on next and give this secret a name and description(optional). Then click on next.

If you want to rotate your secrets value then turn on the Automatic rotation else leave it as it is and click on next. For this demo we are keeping optional things as default.

Now on review page review your secret and click on Store to create your secret.
Now you have your secrets stored in aws secrets manager. Its time to fetch it in your k8s pods.

Setup External Secrets in K8s

First you need to install external-secrets helm chart.
Use below commands to install external-secrets:

helm repo add external-secrets https://charts.external-secrets.io
helm install external-secrets \
   external-secrets/external-secrets \
    -n external-secrets \
    --create-namespace

Now you need to create a secret in your kubernetes that will have credentials to access aws secrets manager.
Use below commands to create k8s secret:

echo -n 'KEYID' > ./access-key
echo -n 'SECRETKEY' > ./secret-access-key
kubectl create secret generic awssm-secret --from-file=./access-key --from-file=./secret-access-key -n external-secrets

You can create above secret in any namespace.

Now create a Cluster Secret Store so that our k8s secrets can fetch secrets from aws.
cluster-secret-store.yaml

apiVersion: external-secrets.io/v1beta1
kind: ClusterSecretStore
metadata:
  name: test-secret-store
spec:
  provider:
    aws:
      service: SecretsManager
      region: us-east-2
      auth:
        secretRef:
          accessKeyIDSecretRef:
            name: awssm-secret
            key: access-key
            namespace: external-secrets
          secretAccessKeySecretRef:
            name: awssm-secret
            key: secret-access-key
            namespace: external-secrets

kubectl apply -f cluster-secret-store.yaml

Now we are ready to create our first external secret (es) to fetch secrets from AWS Secret manager.
my-firs-external-secret.yaml

apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: my-first-es # name for your es resource
  namespace: my-ns # your namespace name
spec:
  refreshInterval: 5m
  secretStoreRef:
    name: test-secret-store # clutser secret store name
    kind: ClusterSecretStore
  target:
    name: my-secret # name you want to give to your secret
    creationPolicy: Owner
  dataFrom:
  - extract:
      key: myFirstSecret #name of aws secret

kubectl apply -f my-firs-external-secret.yaml

Let's verify if our resources are deployed or not. use commands as shown in the screenshot below:

Now let's deploy a test application and fetch aws secrets.
test-app-deployment.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: test-app-deployment
  namespace: my-ns
  labels:
    app: test-app
spec:
  replicas: 1
  selector:
    matchLabels:
      app: test-app
  template:
    metadata:
      labels:
        app: test-app
    spec:
      containers:
        - name: test-app
          image: test-app:latest
          imagePullPolicy: Always
          securityContext:
            privileged: true
          envFrom:
            - secretRef:
                name: my-secret # name of the secret
          resources:
            limits:
              cpu: 500m
              memory: 500Mi
            requests:
              cpu: 100m
              memory: 100Mi

test-script.py

from os import environ
# This python script will read the values of aws secrets from os
# environment as we have deployed our app that way.

user = environ.get("myUser")
password = environ.get("myPassword")
otherSecret = environ.get("secret1")

print("my username is: ",user)
print("my password is: ", password)
print("my other secret is: ", otherSecret)

now create a docker image with test-app:latest name and test your app. It will fetch secrets value from AWS secrets manager.

Conclusion

We have learned how to setup external secrets using helm.
We have seen how to create and store secrets in secret manager.
We have deployed our k8s external secret to fetch secrets value from aws.
We have deployed a test app to see our implementaion.

Hope you liked the article and found it useful.
Feel free to ask any questions / give suggestions in the comments.

Thank You for reading the article 😊

Reference - external-secrets

URL Redirection Using API Gateway

Adil Ansari — Sun, 03 Dec 2023 08:14:26 +0000

API gateway is a popular choice of many developers and cloud engineers for multiple use cases. However redirecting to a new URL using API Gateway can be challenging unless you give proper response headers to it.
This article is a quick demo of how to achieve URL redirection using API gateway and Lambda function.

Use Cases of URL Redirection:

URL Shortening Services.
Website Rebranding
Affiliate Links
etc.

What is URL Redirection

URL redirection is the technique to move the user to a new URL.
For example in a URL shortening service, when you click on a short URL it redirects you to the original URL.

Now let's start creating Lambda Function.

Creating Lambda Function

To create a lambda function, follow first part of this article How to Schedule a Lambda Function
In lambda function, return StatusCode:302 and below headers to redirect to redirct_url: "headers": {"Location": redirect_url} For Example:-

def lambda_handler(event, context):
    # URL to which you want to redirect
    redirect_url = "https://www.google.com"
    response = {
        "statusCode": 302,
        "headers": {
            "Location": redirect_url,
        },
    }
    return response

Our Lambda Function is ready. Now let's create API Gateway (You can use your existing one also).

Creating API Gateway

Go to API Gateway Service in AWS and click on Create API.
Select REST API or HTTP API (In this demo we are choosing REST API) and click on Build.
Give your API a name and Create API.
Now create a resource and enable CORS also for testing.

Now create a method and choose integration type as Lambda Function and don't forget to switch on the lambda proxy integration switch. Now select your lambda function and click on Create method.

Now Deploy the API by clicking on Deploy API button and select/create a stage.
Now on the stages section, you will get the Invoke URL for your API. It would be in this format - https://yourApiID.execute-api.AwsRegion.amazonaws.com/stage
In my API, it is https://5m78szju8k.execute-api.ap-south-1.amazonaws.com/dev.

So just copy this URL and add resource name at the last like this:
https://5m78szju8k.execute-api.ap-south-1.amazonaws.com/dev/demo and hit the URL.

Hurrayyy!! Now you are redirected to your your desired page (Our case it is google.com).

Conclusion

You can create GET/POST APIs to redirect to a different URL.
You can also pass the redirect_url in your POST request body and use it later to redirect.
Or you can fetch this URL from somewhere else like Databases or from other services.
Use cases are unlimited it depends on you.

**Hope you liked this article. If you find it useful, Like and share this article with the ones who need it. And follow me and support my contributions.
dev.to LinkedIn

How To Schedule a Lambda Function in AWS

Adil Ansari — Sat, 11 Nov 2023 08:01:32 +0000

AWS Lambda function is a FaaS(Function as a Service) offered by AWS. FaaS is a cloud computing model that falls under the umbrella of serverless computing.

AWS Lambda function is beneficial when we want to execute our code without worrying about the underlying servers and resources. No matter how many users hit your lambda function, it works smoothly as AWS maintains the underlying infrastructure.

Now what if you want your lambda function to get executed only at a particular time, or at some scheduled time daily, hourly, weekly, or whenever you want?

Here Cloudwatch EventBridge (Earlier known as Cloudwatch Events) comes into the picture. You can schedule your aws services to perform at some particular time or if a particular event occurs. In this article, we'll focus on how to automate the lambda function to get executed at some frequency of time with the help of Cloudwatch EventBridge.

Creating a Lambda Function

First login to your AWS console and then search for lambda service

Now open the lambda service and create a lambda function (if you already have a lambda function then you can skip this part).
On the next page, click on Create a function

Now on the Create Function page, enter the name of the lambda function, and choose your programming language and architecture.

Note:- You can also use a blueprint of a pre-created function or container image for your function, but we are continuing from scratch.

Now let other settings be on default and click on Create function.

Now on the next page, write your code, deploy the function, and test it. The function should be running properly.

Now our function is ready, let's schedule it.

Creating EventBridge Rule

First search for EventBridge Service and open it.

After opening the EventBridge page, select EventBridge Rule and then click on Create Rule.

After clicking on Create Rule button, enter some basic details for the event you are going to create and then choose Schedule for a scheduled event rule.

Below you can see two options Continue to create rule and Continue in EventBridge Scheduler. For this demo, we'll go with Continue to create rule.

Now on the next page, you can define your schedule as a cron expression. choose your cron expression as per your need and click on Next.

On the next page, choose your lambda function and again click on Next.

On the next page, you can add tags, that are completely optional. Now click on Next and review and create your rule.

Now your EventBridge rule has been created. You can also verify this in your lambda function, you can see that the EventBridge rule is triggering the lambda function.

Below, Inside the Monitor tab, you can verify that my lambda function is getting executed every minute (Because I set my cron expression for every minute).

So this is how you can schedule your lambda function.

Conclusion

We have seen how you can create your lambda function. and we automated it using the EventBridge rule.

Hope you enjoyed it and found it useful.

DEV Community: Adil Ansari

Setup Multi Node Kafka Cluster (KRaft)

Prerequisites

Step 1: Set hostnames (run on all 3 nodes)

Step 2: Install Java (run on all 3 nodes)

Step 3: Download and install Kafka (run on all 3 nodes)

Step 4: Create the Kafka config (per node)

Common config (same on all nodes)

A note on advertised.listeners

Step 5: Generate a cluster ID (run once, then reuse on all nodes)

Step 6: Format the log directory (run on all 3 nodes)

Step 7: Create systemd service on all 3 nodes

Step 8: Verify the KRaft quorum is healthy

Step 9: Create and test your first topic

Troubleshooting (quick hits)

Automation Script

Summary

Get Started with Amazon Transcribe in Easy Steps

INTRODUCTION

Key Features of Amazon Transcribe

Pre Requisite

Creating Transcribe Job

Step 1: Prepare an S3 Bucket

Step 2: Create a Lambda Function

Step 3: Create a Test Event

Step 4: Deploy and Test the Code

Step 5: Verify the Transcription

Creating Transcription Downloading Job

Step 1: Create another Lambda Function

Step 2: Create a Test Event

Step 3: Deploy and Test the Code

Conclusion

References

Secure AWS API Gateway with IAM

API Gateway

Why IAM

Creating an API

Add IAM Authorization

Note

Conclusion

References

Deploying a Web Socket Application on Kubernetes

What is a WebSocket

Prerequisites

Manifest files

Deploying to kubernetes

Conclusion

Fetch AWS Secrets in Your Kubernetes Realm

INTRODUCTION

AWS Secrets Manager

Creating Secret in AWS Secret Manager

Setup External Secrets in K8s

Conclusion

URL Redirection Using API Gateway

Use Cases of URL Redirection:

What is URL Redirection

Creating Lambda Function

Creating API Gateway

Conclusion

How To Schedule a Lambda Function in AWS

Creating a Lambda Function

Creating EventBridge Rule

Conclusion

A note on `advertised.listeners`

Step 7: Create `systemd` service on all 3 nodes