Amazon Kinesis Services Comparison
Feature | Kinesis Data Streams (KDS) | Kinesis Data Analytics (KDA) | Kinesis Data Firehose (KDF) |
---|---|---|---|
Purpose | Real-time streaming ingestion and buffering | Real-time stream processing and analytics | Streaming ingestion → automatic delivery to destinations |
Data storage | Up to 7 days (configurable) | Temporary processing only | No storage — streams processed and delivered immediately |
Input | Producers (IoT devices, APIs, applications) | KDS or Firehose streams | Producers or KDA outputs |
Output | Multiple consumers (Lambda, KDA, Firehose, etc.) | Processed data → storage or streams | Amazon S3, Redshift, Elasticsearch, Splunk, custom HTTP endpoints |
Processing | None built-in (just streaming) | SQL queries or Apache Flink for real-time analytics | Optional transformation via Lambda |
Scalability | Very high throughput | Limited by analytics job capacity | Managed — auto scales |
Use cases | Real-time ingestion for processing later | Real-time analytics and transformation | Simplified data delivery to storage or analytics services |
Best for | Streaming large amounts of raw data that will be processed later | Real-time analytics on streaming data | Streaming ETL and delivery to analytics/storage without complex setup |
How They Work Together
Here’s the flow:
- Kinesis Data Streams → collects raw streaming data from sources (e.g., IoT devices, API Gateway).
- Kinesis Data Analytics → reads from the stream to process, filter, or aggregate data in real time.
- Kinesis Data Firehose → takes either the raw stream (from KDS) or processed output (from KDA) and delivers it to storage destinations like Amazon S3, Redshift, or Elasticsearch.
Example — Bicycle Tracking System
Service | Role in System |
---|---|
Kinesis Data Streams | Collects raw location pings from bikes in real time during peak hours |
Kinesis Data Analytics | Processes location data to detect congestion, anomalies, or patterns in real time |
Kinesis Data Firehose | Delivers processed results to S3 for analytics and to DynamoDB for API access |
Key Differences Summarized
- KDS = raw ingestion & buffering
- KDA = real-time analytics
- KDF = streaming ETL & delivery to storage
Example Bicycle Tracking Analogy
Kinesis Data Streams = water pipes carrying live location data from bikes into the city’s processing system.
Kinesis Data Analytics = treatment plant filtering and analyzing location streams — figuring out traffic jams or fastest routes.
Kinesis Data Firehose = distribution system sending the analyzed location data into storage lakes (S3) or dashboards for reporting.
Top comments (0)