Tech Director | Backend, Ops and Technical Communication at North Kingdom | Ex creative coder climbing up (serverless, IAC) and down (operating systems, c) the stack.
Interesting. Up until reading this I would have considered myself one of that lot.
When I started with Lambda I defaulted to SNS because that seemed more naturally event-driven than polling, but then a few times I ended up missing some notifications because I had some edge cases on a Lambda and had to take it down for whatever reason, then changed my heuristic: If I'm integrating with a 3rd party service which provides a reliable SLA I can use SNS, otherwise I'll default to SQS for peace of mind.
I was not aware of retry / "at least once" delivery on SNS, but it makes total sense. I'll keep it in mind next time I have to do it.
The retry is actually in lambda. All SNS + Lambda integrations will be resent until lambda receives the message, since this is an HTTP integration behind the scenes. You have to use caution once it gets to lambda to ensure you don't accidentally drop the message while working on the code.
Make sure you're not actually taking your lambda offline if you want to ensure delivery -- You'll miss any message that is sent before a lambda integration exists. If you're using Javascript, I recommend making your lambda handlers return promises rather than using callbacks and "rejecting" all failures that you want to retry, as they'll enter the retry policy for the lambda. Here's some info on retry: docs.aws.amazon.com/lambda/latest/...
We've found that they seem to retry at a steady pace with some amount of backoff.
You can also set a dead letter queue for messages that exceed the retry limit, which can be processed in a normal queue manner or by some manual process:
We find a lot of reliability in our lambda + SNS + DLQ setups, to the point that SQS almost never gets used. We're not overly cautious about changing lambda code, but you want to ensure that you do reject on failures and you always keep the lambda SNS integration alive between code updates.
Tech Director | Backend, Ops and Technical Communication at North Kingdom | Ex creative coder climbing up (serverless, IAC) and down (operating systems, c) the stack.
Interesting. Up until reading this I would have considered myself one of that lot.
When I started with Lambda I defaulted to SNS because that seemed more naturally event-driven than polling, but then a few times I ended up missing some notifications because I had some edge cases on a Lambda and had to take it down for whatever reason, then changed my heuristic: If I'm integrating with a 3rd party service which provides a reliable SLA I can use SNS, otherwise I'll default to SQS for peace of mind.
I was not aware of retry / "at least once" delivery on SNS, but it makes total sense. I'll keep it in mind next time I have to do it.
The retry is actually in lambda. All SNS + Lambda integrations will be resent until lambda receives the message, since this is an HTTP integration behind the scenes. You have to use caution once it gets to lambda to ensure you don't accidentally drop the message while working on the code.
Make sure you're not actually taking your lambda offline if you want to ensure delivery -- You'll miss any message that is sent before a lambda integration exists. If you're using Javascript, I recommend making your lambda handlers return promises rather than using callbacks and "rejecting" all failures that you want to retry, as they'll enter the retry policy for the lambda. Here's some info on retry: docs.aws.amazon.com/lambda/latest/...
We've found that they seem to retry at a steady pace with some amount of backoff.
You can also set a dead letter queue for messages that exceed the retry limit, which can be processed in a normal queue manner or by some manual process:
docs.aws.amazon.com/lambda/latest/...
We find a lot of reliability in our lambda + SNS + DLQ setups, to the point that SQS almost never gets used. We're not overly cautious about changing lambda code, but you want to ensure that you do reject on failures and you always keep the lambda SNS integration alive between code updates.
ah Lambda! that makes sense. thanks a lot for the detailed answer. Will keep that in mind for next time.