Resolving timeout exceptions and zombie processes in a Real-Time data pipeline

This is a submission for DEV's Summer Bug Smash: Clear the Lineup powered by Sentry.

Project Overview

A real-time cryptocurrency exchange dashboard that aggregates trade data from five different exchanges. The application opens web-socket connections, consumes the data in a Kafka cluster, uses spark to process the data, and stores the data in a DynamoDB table. The data can be queried through an API.

Bug Fix

1.Stopping a zombie container

During a docker environment teardown, the local DynamoDB service failed to stop gracefully which resulted in

Error response from daemon: container PID is zombie and can not be killed

2.CPU saturation on a single-threaded process

High-frequency of write batched from Apache spark overloaded DynamoDB leading CPU utilization to spike past 360% and trigger a read timeout exception.

NAME             MEM USAGE / LIMIT   CPU %
dynamodb_local   952.7MiB / 2GiB     367.87%  
 botocore.exceptions.ReadTimeoutError: Read timeout on endpoint URL: "http://dynamodb-local:8000/...

Code

• Docker compose fix: add signal handling and zombie reaping capabilities with init: true
• Optimize the spark pipeline by rate-limiting write batches

# restrict concurrent connection overload
   df_batch.coalesce(2).foreachPartition(
        lambda p: write_predictions_to_dynamo(p, endpoint_bc.value, region_bc.value)
    )

My Improvements

1. Using Docker Init (tini) for process isolation

Python lacks the built-in signal forwarding and orphan process reaping that is handled by standard OS init systems. When sub-processes terminated inside the DynamoDB container, they remained as un-reaped zombies, which locked container resources and prevented docker daemons from cleanly stopping the service.
Setting init: true mounts tini as PID 1 inside the container. This intercepts OS signals, forwards shutdown calls correctly and immediately reaps dead child processes.

1. Storage Mode and Rate Limited Writes
2. Storage Mode & Write Rate Control By default, DynamoDB Local writes operations to disk using SQLite. When Spark streamed continuous high-volume batches, intense disk I/O drove CPU utilization up to 367%, inducing heavy context switching and causing incoming HTTP connections to time out (ReadTimeoutError). The issue was addressed through three coordinated architectural changes:
3. In-Memory Storage: Passed -inMemory flags to DynamoDB Local, removing disk write contention entirely for development testing. - Batch Micro-Throttling: Adjusted the Spark streaming trigger interval from ~15 seconds to 60 seconds and coalesced output partitions down to 1 (df_batch.coalesce(1)), preventing parallel Spark worker tasks from overwhelming the single-threaded local database.
4. Connection Pooling & Pacing: Configured boto3 connection pools to single connections (max_pool_connections=1), chunked item payloads into sets of 5, and inserted micro-pauses (time.sleep(0.1)) inside batch loop iterations to flatten consumption spikes.

Best Use of Sentry