I think putting a queue between S3 and Lambda has yet another benefit.
In case, your lambda failed on processing the event from S3 directly connected to it, the event would be lost after second retry. Putting a queue in-between helps to persist the message for longer time.
An alternative would be to add a DLQ to Lambda itself (docs.aws.amazon.com/lambda/latest/...).
Not sure which one is better.
I totally agree with you. I also find it hard to stay on top of all the different limits and behaviours. How many retries you will get depending on the invocation type and resource... Definitely something to think about in advance.
We're a place where coders share, stay up-to-date and grow their careers.
We strive for transparency and don't collect excess data.