DEV Community

Wakeup Flower
Wakeup Flower

Posted on

Kinesis Data Streams, Data Analytics, Data Firehose comparison

Amazon Kinesis Services Comparison

Feature Kinesis Data Streams (KDS) Kinesis Data Analytics (KDA) Kinesis Data Firehose (KDF)
Purpose Real-time streaming ingestion and buffering Real-time stream processing and analytics Streaming ingestion → automatic delivery to destinations
Data storage Up to 7 days (configurable) Temporary processing only No storage — streams processed and delivered immediately
Input Producers (IoT devices, APIs, applications) KDS or Firehose streams Producers or KDA outputs
Output Multiple consumers (Lambda, KDA, Firehose, etc.) Processed data → storage or streams Amazon S3, Redshift, Elasticsearch, Splunk, custom HTTP endpoints
Processing None built-in (just streaming) SQL queries or Apache Flink for real-time analytics Optional transformation via Lambda
Scalability Very high throughput Limited by analytics job capacity Managed — auto scales
Use cases Real-time ingestion for processing later Real-time analytics and transformation Simplified data delivery to storage or analytics services
Best for Streaming large amounts of raw data that will be processed later Real-time analytics on streaming data Streaming ETL and delivery to analytics/storage without complex setup

How They Work Together

Here’s the flow:

  1. Kinesis Data Streams → collects raw streaming data from sources (e.g., IoT devices, API Gateway).
  2. Kinesis Data Analytics → reads from the stream to process, filter, or aggregate data in real time.
  3. Kinesis Data Firehose → takes either the raw stream (from KDS) or processed output (from KDA) and delivers it to storage destinations like Amazon S3, Redshift, or Elasticsearch.

Example — Bicycle Tracking System

Service Role in System
Kinesis Data Streams Collects raw location pings from bikes in real time during peak hours
Kinesis Data Analytics Processes location data to detect congestion, anomalies, or patterns in real time
Kinesis Data Firehose Delivers processed results to S3 for analytics and to DynamoDB for API access

Key Differences Summarized

  • KDS = raw ingestion & buffering
  • KDA = real-time analytics
  • KDF = streaming ETL & delivery to storage

Example Bicycle Tracking Analogy

Kinesis Data Streams = water pipes carrying live location data from bikes into the city’s processing system.

Kinesis Data Analytics = treatment plant filtering and analyzing location streams — figuring out traffic jams or fastest routes.

Kinesis Data Firehose = distribution system sending the analyzed location data into storage lakes (S3) or dashboards for reporting.

Top comments (0)