Batch publishing Twitch chatroom messages - with AWS Lambda Durable Multi-Step Executions

#serverless #durable #aws #lambda

Hi all!

AWS have announced yet another AWS Lambda feature release.

In this post I will be covering a use-case for a Lambda that utilises durable multi-step executions. I will be covering overall thoughts, gotchas and summary.

What are Durable AWS Lambda Functions?

AWS Lambda announces durable functions, enabling developers to build reliable multi-step applications and AI workflows within the Lambda developer experience.

Durable functions automatically checkpoint progress, suspend execution for up to one year during long-running tasks, and recover from failures - all without requiring you to manage additional infrastructure or write custom state management and error handling code.

Use-Case

As you may or may not know, I run a Twitch Extension 'Stat-Milestones'. In short the Twitch Extension renders a widget on streamer's profiles, this widget displays a multitude of social goals information to viewers [Followers, Subscribers, Hypetrains, Charities, etc]

Twitch Extensions have the ability [user-permitted] to publish messages to the public chatroom of a Twitch streamer via the Twitch API.

Generally as a viewer you will see messages from Twitch bots/extensions that provide information like reminders to follower the streamer you're viewing or links to external profiles, etc.

For a long while, I have been intending to setup a pipeline to publish daily messages to Twitch streamers chatrooms. With the intention to remind streamers to update their Twitch goals and Stat-Milestones metrics!
Other communication methods are available, but their chatroom will definitely catch their attention!

With the announcement of AWS Lambda Durable Multi-Step Executions, I figure this is the perfect opportunity to have a play with the new feature and hopefully provide some insightful information if you're thinking about adopting Durable executions yourself!

Why not just use Step-Functions?

To note, I use AWS SAM [YAML] to deploy infrastructure for my Twitch Extension.

Personally setting up infrastructure for a step-function takes the above average complexity for IaC, see Create a Step Functions state machine using AWS SAM.

With Durable AWS Lambdas, the need to develop state definitions is now instead done all in-code and the abundant step execution features are found directly within a new tab on the Lambda dashboard.

The Amazon States Language is a JSON-based, structured language used to define your state machine, a collection of states, that can do work (Task states), determine which states to transition to next (Choice states), stop an execution with an error (Fail states), and so on.

Durable Lambda Implementation

In this section I will detail how I created an application to publishing daily reminder messages to Twitch chatrooms.

In order to publish daily reminders to Twitch streamer chatrooms, I thought about how I require to break this down into notable Durable execution steps.

User Discovery
Batch Publishing of reminder messages
Result Aggregatation

Discovery

The goal of this step is to discover users from a Mongo collection who have granted the permissions to allow my Twitch Extension to publish to their chatroom AND still have a valid auth token.

Batch publishing of messages

In this step, the intention is to construct reminder message(s) and publish them to users found in the discovery phase.

I already have an existing pipeline [sqs -> lambda] in the us-east-1 region that processes messages from a SQS queue and constructs API requests to Twitch to publish the messages to the chatrooms directly.

Simply this execution step just requires to construct SQS Messages and publish them to an external Queue (living in us-east-1 not us-east-2!)

Aggregating Results

This execution step is to aggregate the final results of the prior steps... user discovery, message publishing metrics, etc.

Implementation

With the help of AI tools, I set out to create a Python run-time Durable AWS Lambda function that implements the discovery, batch and result aggregations into seperate steps.

Utilising Steps and Map context processing, we can seperate out the discovery and results into singular steps.

For the Batch Publishing of messages, we can do something clever and utilise the context map. This example demonstrates how durable functions combine map() operations with chunking and rate limiting to handle large-scale data processing.

Code

My example Durable lambda code can be found here

The file has to be compiled into a zip with the external packages

requirements.txt
pymongo>=4.10.1
boto3>=1.35.0
aws-durable-execution-sdk-python>=0.1.0

Dashboard Breakdown

The new Durable execution tab found on the Lambda dashboard brings, somewhat familiar widgets to step-functions.

The ability to inspect the inputs/outputs of step executions is a big win for future applications built with Lambda, reduces the need to deep-dive on logs to inspect payloads and results between steps.