Amazon Kinesis Services Comparison
| Feature | Kinesis Data Streams (KDS) | Kinesis Data Analytics (KDA) | Kinesis Data Firehose (KDF) |
|---|---|---|---|
| Purpose | Real-time streaming ingestion and buffering | Real-time stream processing and analytics | Streaming ingestion → automatic delivery to destinations |
| Data storage | Up to 7 days (configurable) | Temporary processing only | No storage — streams processed and delivered immediately |
| Input | Producers (IoT devices, APIs, applications) | KDS or Firehose streams | Producers or KDA outputs |
| Output | Multiple consumers (Lambda, KDA, Firehose, etc.) | Processed data → storage or streams | Amazon S3, Redshift, Elasticsearch, Splunk, custom HTTP endpoints |
| Processing | None built-in (just streaming) | SQL queries or Apache Flink for real-time analytics | Optional transformation via Lambda |
| Scalability | Very high throughput | Limited by analytics job capacity | Managed — auto scales |
| Use cases | Real-time ingestion for processing later | Real-time analytics and transformation | Simplified data delivery to storage or analytics services |
| Best for | Streaming large amounts of raw data that will be processed later | Real-time analytics on streaming data | Streaming ETL and delivery to analytics/storage without complex setup |
How They Work Together
Here’s the flow:
- Kinesis Data Streams → collects raw streaming data from sources (e.g., IoT devices, API Gateway).
- Kinesis Data Analytics → reads from the stream to process, filter, or aggregate data in real time.
- Kinesis Data Firehose → takes either the raw stream (from KDS) or processed output (from KDA) and delivers it to storage destinations like Amazon S3, Redshift, or Elasticsearch.
Example — Bicycle Tracking System
| Service | Role in System |
|---|---|
| Kinesis Data Streams | Collects raw location pings from bikes in real time during peak hours |
| Kinesis Data Analytics | Processes location data to detect congestion, anomalies, or patterns in real time |
| Kinesis Data Firehose | Delivers processed results to S3 for analytics and to DynamoDB for API access |
Key Differences Summarized
- KDS = raw ingestion & buffering
- KDA = real-time analytics
- KDF = streaming ETL & delivery to storage
Example Bicycle Tracking Analogy
Kinesis Data Streams = water pipes carrying live location data from bikes into the city’s processing system.
Kinesis Data Analytics = treatment plant filtering and analyzing location streams — figuring out traffic jams or fastest routes.
Kinesis Data Firehose = distribution system sending the analyzed location data into storage lakes (S3) or dashboards for reporting.
Top comments (0)