Explaining Transaction Count as Important Constraint for adding additional AWS Kinesis Consumers

USECASE

The aim of this page📝 is to explain the significance of monitoring CloudWatch metrics for Kinesis streams, particularly focusing on GetRecords.Records and SampleCount.
Previously, I was only monitoring the volume limits. However, transaction count limits can also become important when adding additional consumers to the stream. After adding a new consumer, I started noticing latencies and a spike in the ReadProvisionedThroughputExceeded metric.

• GetRecords.Records metric counts the number of records retrieved from a shard in a specified time period.
• SampleCount for GetRecords.Records indicates the number of GetRecords operations (transactions) performed.
• Monitoring SampleCount helps in tracking the frequency of read transactions.
• Kinesis stream limits each shard to five read transactions per second.
• Using SampleCount ensures staying within the transaction quota.
• Adding a new consumer increases read transactions, making SampleCount monitoring crucial.
• Monitoring GetRecords.Bytes gives insight into the volume of data read.
• Balancing GetRecords.Records, GetRecords.Bytes, and SampleCount provides a comprehensive view of stream usage.
• Exceeding transaction limits can lead to throttling or performance issues.
• Regular monitoring helps optimize stream performance and prevent issues.

LINKS

• Quotas and limits - Amazon Kinesis Data Streams
• Monitor the Amazon Kinesis Data Streams service with Amazon CloudWatch - Amazon Kinesis Data Streams