today I’m gonna talk about when to use SNS or SQS. there’s many articles out there that attempt to answer this question but don’t really give you a concrete example of when you should use one or the other so the first thing I’m going to do is go over some technical details and then give you a practical example of when you want to use SNS or SQS so quickly let’s go over the technical comparison between SNS and SQS
· SNS stands for simple notification service while SQS stands for simple queue service
· SNS uses a publisher subscriber system so you may own a topic and you publish to that topic and subscribers get notified of events that are delivered to that topic that’s starkly different to SQS whereas it’s a queuing service for message processing so this example on SQS could be a subscriber to an SNS so whenever someone publishes a message to an SNS your SQS queue can get a message in it that can be processed at a later time so for the SNS kind of as I was alluding to before publishing a message to a topic can deliver too many subscribers so it’s a fan-out approach and those subscribers can be of different types so you can have asked us as a subscriber a lambda function or even an email now moving back to SQS it’s a system that must pull the queue to discover new events so whenever an event is delivered to the queue there’s nothing that’s gonna get invoked there’s no system that automatically can become aware of it you need to have some separate thread that’s pulling the queue to discover when new events get delivered and have a mechanism to actually process it and then delete the message from the queue
· so continuing on that messages in the queue are typically processed by a single consumer or a single service with a very narrow responsibility and different services if they both care about the same events they can have their own separate queues
so you can see that there’s a pretty stark difference from the technical perspective of what these things do if you’re still confused I think there’s a very simple question or two simple questions that you can ask yourself to figure if you want to use SNS or SQS so the first one is do other systems care about an event so if something happens an event occurs in the world does do other people care about it and if the answer is yes you should use SNS because you want to publish a message to your topic and potentially tell other people that that thing happened and to ask us does your system care about an event so do you care specifically when something has happened so you are the receiver of this data and if this is true then SQS is the right choice for you so this is a quick little summary of what these two things do and the key differences between them.
let’s talk about a practical example now of when you would use one or the other so in this example we’re going to be talking about credit card transactions it’s a very common theme in many of my videos so let’s assume that we have a user here and they are making a purchase maybe on some website or through some POS terminal and putting in their credentials to buy some kind of product now they’re making a purchase request to some kind of REST API and we’re just calling this a transaction processing web service and the payload of what gets passed into this web service and maybe it looks something like this so we have a transaction ID some number a customer ID maybe their contact details and the amount that was charged to the customer so from a processing perspective if you get a request containing this kind of event the first thing you need to do is communicate with some kind of credit card authority service practically speaking probably Visa, Mastercard or Amex all these credit card authority services so when an event occurs saying that someone is attempting to purchase something you need to validate that against the authority service so the first thing you do is you probably call that service and say here’s the customer ID maybe they passed in some credentials including their PIN or something like that and that gets validated against the authority service and in this example let’s just assume that they returned 200 ok alright so the transaction is good the credentials are good this thing went.
Now in this ecommerce example this is an event that matters right because someone attempted to purchase something and it went through so it was successful and this is something that is potentially relevant to many different subs systems that exist in your e-commerce ecosystem so maybe we want to publish this event.
so you’re basically telling the world you’re publishing and events to this topic and from this topic there can be multiple different subscribers with all narrow use cases so in this example I have three different ones I have a lambda here and I have two SQS’s and I’m going to fill this out so it’ll make more sense why we have three different ones now.
let’s go through these one by one so in the first case when a message gets delivered maybe we have some kind of customer mailing service and if you’ve ever order anything online maybe on Amazon or any kind of web site you know that when you order something shortly after you get an email that says hey thank you for ordering we really appreciate your business here’s a summary and here’s the address that it’s going to something to this effect so this component is potentially responsible for that very narrow concern so after this event occurs a message gets delivered with this payload to your lambda function and this lambda function maybe it queries some database somewhere for some additional details but in the end it’s gonna send an email to your customer that says thank you for ordering and we appreciate your business okay so that’s one use case right now completely separately.
there’s two other subsystems here that care about this same events but they do very different things so we don’t want to mix or conflict the responsibilities of these different things we want to have a separate system that allows us to process these events in parallel and completely independent of one another so this second system here with a queue it’s some kind of analytics engine so when an event occurs in addition to publishing to this lambda function we’re publishing that same event at some analytics queue okay and this system one of its responsibilities is it cares about how many orders are generated in a single day. you want to show that on some dashboard so when an event occurs this payload gets delivered to this queue and on its own it’s not going to really do anything you need some other system here to pull the queue to actually become aware that something is in there and respond in some sort of way so in this example maybe you have some kind of transaction analytic service that’s hosted on ec2 and part of spinning up your service you have a series of pullers that know to pull messages off of this queue and when they find one maybe they put something in a database somewhere increment some values but at the end of the day their job is to output something that says something like orders today some number and the total amount of revenue is this okay so this is a very different concern than what we’re doing up here this is kind of an analytics perspective and whereas this is a customer use case right now.
for the other queue there’s a separate use case right so it’s a fraud detection service so you’re a bustling e-commerce company and maybe you’re concerned with fraud because certain people have ordered something and the transaction eventually got reversed and you kind of want to have some kind of proactive approach that attempts to detect and mitigate fraud so this is again a very separate use case from your analytics engine and your reminder service this thing is doing something that’s completely different but it also cares about this event and it’s going to use it in its own way so similar to what we had above here we have some fraud detection service that’s also based on ec2 and similar again we’re going to pull the queue to figure out when events occur and once we process the message we delete that message from the queue so no other pullers can receive it and at the end of the day what this system may try to do is determine based on events and based on its own kind of internal processes which transactions are suspicious what should we not deliver immediately and maybe have someone go and look into and just kind of make sure that this thing isn’t a fraudulent case.
so this is an example of a use case where many different consumers care about a singular event and this is where SNS really shines because something is being published and you were distributing that to multiple different consumers so we saw that there’s different consumer types right you can deliver to a lambda function or a queue.
some of you may be asking why would I do one or the other so with a lambda function when you deliver to us on us it’s best effort delivery so if there’s a problem in your logic here you can potentially lose this message whereas from SNS to SQS if a message gets delivered to the queue it’s guaranteed to be present so it’s guaranteed to be processed by some independent service that is pulling and deleting messages from the queue so that’s an example of why you would want to use SQS over lambda understand you multiple different examples here so I’ve shown you know four steps here right so there’s the first stage credit card authority then you do this stuff.
let’s rewind a little bit and try to answer why are we going through all this effort why are we decoupling here and so let’s think about what this world would look like if you did all these four things in sequence maybe in your transaction processing web service and let’s talk about why this is a bad thing so again as I said the first thing you do is you communicate with a credit card authority service and verify that this transaction is good and then if we use this naive approach maybe we want to write some function to send out an email right we attempt to use some email service and send a customer an email and then we continue to the next one we attempt to do some analytics on this event and then finally we go to the fraud detection thing maybe we query some database see if it’s in some kind of blacklist and make some decision based on what to do so why is this a bad thing???
well it’s a bad thing because you have a partial failure scenario so if you’re doing these in sequence you do the first one then the second one then the third one then the fourth one what happens if this one fails if this one fails you need to repeat the whole process and obviously you don’t want to charge the customer again right so by decoupling these applications you have one event that gets distributed to many different consumers and they can all independently in their own way ensure that this message gets processed and this is why we’re leveraging SNS with these different subscriber endpoints.
Latest comments (0)