Good write-up. As commented, CloudTrail is probably an overkill unless there is a specific reason. In most cases, the lambda trigger should fit. If not, SNS or SQS trigger would help.
If you know the S3 object key pattern, then the prefix/suffix based event notification is handy.
In the SQ to Lambda diagram above, not sure if you recommend manual polling for messages from the queue. If so, you need to take care of deleting the messages. Using SQS as the event source for lambda would make that simple and easy. You get to choose the batch size and can adjust the SQS params to make it work the way required.
Regarding the SQS -> Lambda integration (via EventSourceMapping or manual polling) - if the function throws an error, the whole batch of messages will be re-queed to my best knowledge. This might cause a problem where one bad message causes the entire batch to get re-computed repeatedly.
It seems like the approach of manually deleting the SQS messages might no longer be needed. I still need to experiment with the new way ESM handles SQS batches, though.
For further actions, you may consider blocking this person and/or reporting abuse
We're a place where coders share, stay up-to-date and grow their careers.
Good write-up. As commented, CloudTrail is probably an overkill unless there is a specific reason. In most cases, the lambda trigger should fit. If not, SNS or SQS trigger would help.
If you know the S3 object key pattern, then the prefix/suffix based event notification is handy.
In the SQ to Lambda diagram above, not sure if you recommend manual polling for messages from the queue. If so, you need to take care of deleting the messages. Using SQS as the event source for lambda would make that simple and easy. You get to choose the batch size and can adjust the SQS params to make it work the way required.
Hey @sheenbrisals .
Regarding the SQS -> Lambda integration (via EventSourceMapping or manual polling) - if the function throws an error, the whole batch of messages will be re-queed to my best knowledge. This might cause a problem where one bad message causes the entire batch to get re-computed repeatedly.
The pattern of manually deleting the SQS messages is also implemented in Lambda Powertools for Python
Thanks! I see your point. Yes, that pattern people use to overcome that issue.
Wow, just as I finished writing this comment, AWS released aws.amazon.com/about-aws/whats-new....
It seems like the approach of manually deleting the SQS messages might no longer be needed. I still need to experiment with the new way ESM handles SQS batches, though.