DEV Community

Discussion on: Amazon S3 to AWS StepFunctions pattern

Collapse
 
wojciechmatuszewski profile image
Wojciech Matuszewski • Edited

Hey @sheenbrisals .

Regarding the SQS -> Lambda integration (via EventSourceMapping or manual polling) - if the function throws an error, the whole batch of messages will be re-queed to my best knowledge. This might cause a problem where one bad message causes the entire batch to get re-computed repeatedly.

The pattern of manually deleting the SQS messages is also implemented in Lambda Powertools for Python

Collapse
 
sheenbrisals profile image
Sheen Brisals

Thanks! I see your point. Yes, that pattern people use to overcome that issue.

Collapse
 
wojciechmatuszewski profile image
Wojciech Matuszewski

Wow, just as I finished writing this comment, AWS released aws.amazon.com/about-aws/whats-new....

It seems like the approach of manually deleting the SQS messages might no longer be needed. I still need to experiment with the new way ESM handles SQS batches, though.