DEV Community

2

Amazon Kinesis Firehose

Difinition and some features

So in today's lesson we are going to talk about amazon kinesis firehose and this service is used to load real time streams into data lakes warehouses and analytic services.

So amazon kinesis firehose is an extract transform and load (ETL) service and it's used to capture transform and deliver streaming data.

It's also a fully managed service for delivering real time streaming data.
Its destinations could be:

  • Amazon S3
  • Amazon RedShift
  • Elastic Search

So a destination of the amazon kinesis firehose is the output of the service.

It's used to load new data in near real time which is like 60 seconds and you need to know that this is not a real time streaming service, it's a near real time.
Unlike the Amazon Kinesis data streams, which is a real time streaming service.

Also the delivery streams automatically scale up and down in order to handle gigabytes per second or more of input data rate.

So amazon kinesis firehose could be used to transform the data before being delivered to a specific destination.

Sources and Destinations of Amazon Kinesis Firehose.

Sources
So it could be a producer application writing its data directly to the kinesis firehose like:

  • Amazon CloudWatch Logs
  • Amazon CloudWatch Events
  • AWS IOT
  • Kinesis Data Stream
  • Kinesis Agent

Note: Kinesis Agent is a prebuilt java application which monitors certain files and then sends this information to the delivery stream.

Destinations

  • Amazon S3
  • Amazon RedShift
  • Elastic Search
  • Splunk
  • And many more...

GitHub
LinkedIn
Facebook
Medium

Image of Timescale

🚀 pgai Vectorizer: SQLAlchemy and LiteLLM Make Vector Search Simple

We built pgai Vectorizer to simplify embedding management for AI applications—without needing a separate database or complex infrastructure. Since launch, developers have created over 3,000 vectorizers on Timescale Cloud, with many more self-hosted.

Read more

Top comments (0)

Create a simple OTP system with AWS Serverless cover image

Create a simple OTP system with AWS Serverless

Implement a One Time Password (OTP) system with AWS Serverless services including Lambda, API Gateway, DynamoDB, Simple Email Service (SES), and Amplify Web Hosting using VueJS for the frontend.

Read full post

👋 Kindness is contagious

Please leave a ❤️ or a friendly comment on this post if you found it helpful!

Okay