Building a Near Real-Time Stock Analytics Pipeline on AWS (and the Silent Bug That Hid My Data)

Building a Near Real-Time Stock Data Pipeline on AWS: What I Learned and Debugged

Tech stack: Python, boto3, Amazon Kinesis Data Streams, AWS Lambda, DynamoDB, Amazon S3, AWS Glue, Amazon Athena, Amazon SNS, IAM, CloudWatch

Repository: amazon-kinesis-stock-analyst-project

Introduction

I wanted to understand how data moves through a cloud pipeline, from an external API to a streaming service, through serverless processing, into storage, and finally into analytics and notifications.

So I built a stock market data pipeline using Python and AWS.

The Python script sends stock price records to Amazon Kinesis Data Streams. AWS Lambda processes the incoming records, storing processed data in DynamoDB and raw data in Amazon S3. AWS Glue and Amazon Athena make the archived data available for SQL queries, while DynamoDB Streams, another Lambda function, and Amazon SNS provide the infrastructure for trend-based email alerts.

This is a near-real-time pipeline, not a real-time trading system. Records are sent every 30 seconds and processed in small batches.

Credit: I built this project while taking a project course by Lucy Wang. The course provided the project brief and starter code, including the Lambda code. This post isn't a walkthrough. It's my field notes on the changes I made, what broke, and how I debugged it. If you want the guided version, I recommend the course.

The Architecture

Here's how the data moves through the system:

Stock API
    |
    v
Python Producer
    |
    v
Kinesis Data Streams
    |
    v
Processing Lambda
    |
    +------> DynamoDB (processed data)
    |
    +------> S3 (raw data)
                 |
                 v
              AWS Glue
                 |
                 v
             Amazon Athena
             (SQL analytics)

DynamoDB Streams
    |
    v
Alert Lambda
    |
    v
Amazon SNS
    |
    v
Email notification

Each service has a specific responsibility:

Service	Purpose
Kinesis Data Streams	Receives incoming stock price records.
AWS Lambda	Processes records, calculates metrics, and identifies potential anomalies.
DynamoDB	Stores processed records for fast lookups.
Amazon S3	Stores raw records for historical analysis.
AWS Glue	Catalogs the data stored in S3.
Amazon Athena	Queries the archived data using SQL.
DynamoDB Streams	Captures changes to DynamoDB items.
Lambda + SNS	Supports trend detection and email notifications.
CloudWatch	Provides logs for monitoring and debugging.

I built and tested the project on Windows using PowerShell and a Python virtual environment, with my AWS resources configured in eu-north-1.

What I Changed Beyond the Starter Code

Since I planned to publish the project publicly, I made several changes to the configuration and security setup.

1. Moved configuration out of the code

The sample script contained hardcoded configuration values, including a placeholder stream name.

I changed it to read settings from environment variables loaded from a local .env file.

For example:

STREAM_NAME = os.getenv(
    "KINESIS_STREAM_NAME",
    "stock-market-stream",
)

AWS_REGION = os.getenv("AWS_REGION")

I also added a .env.example file to document the expected configuration without exposing my local values.

The actual .env file is excluded from version control through .gitignore.

2. Kept AWS credentials out of the repository

The Python script contains no AWS access keys. Instead, boto3 retrieves credentials through the AWS CLI configuration.

I also restricted the producer's IAM permissions to writing records to the intended Kinesis stream rather than granting unnecessary access to other services.

3. Reduced unnecessary logging

The original script printed the full Kinesis response, including request metadata.

I reduced the output to useful information, such as the stock symbol, price, and shard identifier. This made the logs easier to read and safer to share when troubleshooting.

4. Added secret-protection measures

Before publishing the repository, I added:

• A .gitignore entry for .env.
• Secret scanning.
• GitHub push protection.

These safeguards help reduce the risk of accidentally committing credentials. They complement, rather than replace, appropriate IAM permissions and careful review of commits.

The Bug That Cost Me an Hour

This was the most valuable part of the project for me.

My Python script printed a success message every 30 seconds:

INFO Sent AAPL price=330.32 shard=shardId-000000000002

But nothing appeared downstream.

• DynamoDB had no processed items.
• CloudWatch had no expected Lambda log streams.
• Kinesis Data Viewer showed no records in the intended stream.
• The Lambda event source mapping reported no records processed.

Everything looked correct at first, so I started debugging from the middle of the pipeline.

Step 1: Tested the processing Lambda independently

I sent a manually constructed test event shaped like a Kinesis event to the processing Lambda.

The function returned a successful response, but that alone wasn't enough. My handler caught some errors internally, meaning an invocation could appear successful even when processing had failed.

I checked the logs and verified that the test record appeared in DynamoDB and that the expected file was written to S3.

That gave me evidence that the processing and storage path could work independently.

Step 2: Checked the Lambda trigger

Next, I checked the event source mapping connecting Kinesis to Lambda.

It was enabled and attached to the intended stream. The status indicated that no records had been processed.

Combined with the empty Kinesis Data Viewer, this pointed me toward the ingestion side rather than immediately blaming the Lambda function.

My question changed from:

Why isn't Lambda running?

to:

Are my records reaching the Kinesis stream that Lambda is reading?

Step 3: Investigated the Python producer

I went back to the producer and checked its environment.

I found this error:

ModuleNotFoundError: No module named 'dotenv'

The python-dotenv package wasn't installed in my virtual environment.

My dependency list had been generated using pip freeze, but it didn't include the package the script required.

There was another problem: the import was wrapped in an except ImportError block that silently ignored the failure.

As a result, the script continued without loading the .env file.

That meant the intended AWS region wasn't being loaded from the project's configuration. Boto3 could instead use the region configured through the AWS CLI or another applicable AWS configuration source.

The producer could therefore appear healthy while sending records somewhere other than the stream I was monitoring.

Step 4: Fixed the configuration and verified the pipeline

I installed the missing dependency, restarted the producer, and monitored the pipeline in sequence.

My verification checklist was:

1. Confirm records appeared in Kinesis.
2. Confirm the processing Lambda received records.
3. Confirm processed items appeared in DynamoDB.
4. Confirm raw data was written to S3.

This helped me isolate the problem instead of changing multiple AWS resources unnecessarily.

The biggest lesson: A successful message from the producer doesn't prove that the intended consumer received the data.

Technical Decisions I Can Explain

Raw data in S3, processed data in DynamoDB

I used S3 for raw records and DynamoDB for processed records.

S3 provides economical object storage for historical data, while DynamoDB supports fast item-level lookups. Keeping the raw data also provides a useful starting point for historical analysis.

Athena instead of maintaining a query database

AWS Glue and Athena allowed me to query the data stored in S3 using SQL without maintaining a continuously running database.

Athena charges according to the amount of data scanned, so query design and data layout still matter when controlling costs.

Decimal instead of float for DynamoDB

Python's float type isn't supported by boto3's DynamoDB serializer for numeric attributes.

I used Decimal for numeric values written to DynamoDB to avoid serialization errors and handle decimal values appropriately.

Small batches for a low-volume stream

The producer sends records every 30 seconds, and the Lambda event source mapping uses a small batch size.

A batch size of two doesn't guarantee that Lambda runs exactly once every minute. Actual invocation timing also depends on polling, record arrival, and the event source mapping's configuration.

The goal was to keep the pipeline simple while learning how batching affects processing latency and invocation overhead.

Kinesis on-demand mode

I chose on-demand mode so I wouldn't need to determine a fixed shard capacity upfront.

However, on-demand doesn't mean free. Costs depend on the pricing model and usage, so unused resources still need to be cleaned up.

What I Learned About the Data

Repeated prices don't necessarily mean a broken pipeline

During testing, I noticed the same stock price appearing repeatedly.

The script fetched daily stock bars rather than continuously changing market prices. Sending the same daily value every 30 seconds didn't make the underlying data update every 30 seconds.

The US market was also closed during my test, contributing to the repeated values.

This taught me an important distinction: ingestion frequency is not the same as data freshness.

Trend alerts need meaningful price movement

The alert logic compares a five-record moving average with a 20-record moving average.

A crossover can indicate a potential trend change, but repeated or nearly identical prices provide little movement for testing this logic.

A better test would use data containing meaningful price variation and known crossover scenarios, allowing the alert conditions to be validated deliberately.

Timestamp formats matter

I also identified a potential issue with the alert function's time-window logic.

The processing path writes timestamps in a format like this:

2026-10-02T12:00:00Z

The alert function constructs its cutoff using a space between the date and time instead of the T.

When timestamps are compared as strings, inconsistent formats can produce incorrect time-window results. The exact impact depends on the stored values and comparison logic.

A safer implementation would use consistent UTC timestamps and compare normalized values or actual datetime objects.

It's a reminder that a pipeline can successfully move data while still having problems with the logic applied to that data.

Verifying the Pipeline

The following screenshots provide evidence of the main stages of the project.

1. Kinesis Data Streams

This is where I checked whether the producer's records were reaching the intended stream.

2. DynamoDB

The processed records should appear in DynamoDB after the processing Lambda handles incoming events.

3. Amazon S3

The raw records are stored in S3 for historical analysis.

4. Amazon Athena

Athena provides a way to query the archived data using SQL.

These screenshots show different parts of the system. Together, they help readers follow the data from ingestion to storage and analytics.

Cleanup and Cost Awareness

After completing the project, I removed the AWS resources to avoid unnecessary ongoing charges.

My cleanup covered the producer and the AWS resources used by the pipeline, including Kinesis, Lambda, DynamoDB, SNS, S3, Glue, CloudWatch log groups, and associated IAM roles.

The important lesson is that stopping an application doesn't automatically remove the cloud resources it uses.

Before deleting resources, preserve any evidence or data you want to keep and check for dependencies. After cleanup, verify that the intended resources have actually been removed.

What I'd Improve Next

If I extended the project, I'd focus on six areas:

1. Use fresher market data. Introduce a suitable source of more frequent price updates so the trend logic can be tested with meaningful variation.
2. Fix and test the alert time window. Normalize timestamps and create test cases for expected alert and non-alert scenarios.
3. Apply least-privilege IAM to both Lambda functions. Limit each function to the actions and resources it actually needs.
4. Externalize remaining configuration. Move bucket names and SNS topic ARNs into environment-specific configuration.
5. Introduce Terraform. Make the infrastructure reproducible and simplify setup and teardown.
6. Improve failure handling. Configure suitable retries, monitoring, and failure destinations or dead-letter queues where supported.

These improvements would make the project easier to reproduce, maintain, and extend.

What I Learned

This project taught me more than how to connect AWS services. It taught me how to investigate a distributed system when one component appears healthy but the overall pipeline isn't behaving as expected.

My biggest takeaways:

• Trace data end to end. Verify the producer, stream, consumer, and storage independently.
• Make configuration failures visible. Missing dependencies and configuration should never fail silently.
• Test components in isolation. A controlled Lambda test can help rule out entire sections of a pipeline.
• Understand the data, not just the infrastructure. Frequent ingestion doesn't automatically mean fresh data.
• Build with security and cost in mind. Restrict permissions, protect credentials, and clean up unused resources.

I'm continuing to build my practical AWS skills, and this project gave me a clearer understanding of how streaming, serverless processing, storage, analytics, and event-driven notifications fit together.

Explore the Project

• GitHub: amazon-kinesis-stock-analyst-project
• LinkedIn: samobeng14dev

Technologies used: Python, boto3, Amazon Kinesis Data Streams, AWS Lambda, DynamoDB, DynamoDB Streams, Amazon S3, AWS Glue, Amazon Athena, Amazon SNS, IAM, CloudWatch, Git, and GitHub.