loading...

Serverless Backends With AWS Cloud: Twitter and Email Campaigns

rob117 profile image Rob Sherling ・5 min read

Let's Begin

Hi! You can read this tutorial series in its original format over at my blog, J-bytes. This is not a short series; it clocks in at over 11,000 words and is a detailed tutorial on how to get a good understanding of several technologies. Many of the articles are long, and require you to actually have an AWS console screen open in front of you. This might be something best to work through when you have a bit of time. Additionally, you can contact me at rob@<blog-domain-name>.com if you have any problems / something cool you wrote / things to chat about.

We are going to make a Twitter and email campaign 'serverless' backend for a site using entirely AWS cloud services (mainly Lambda and DynamoDB) to save a ton of money over classic infrastructure implementations. Again, this is a massive tutorial, so set aside a few hours to complete it. We'll be doing everything from encryption and key management to creating a static site behind AWS Cloudfront.

Super-cool diagram:

cloudcraft diagram of lamda layout

If you don't care about the how and why and just want to get to the project, just head on over to the next post.

If you find yourself in need of a way to register lots of twitter users / emails for a single purpose such as a sign-up campaign, then this is the tutorial for you. You can also make some easy modifications, such as RDS over DynamoDB, if you intend to send emails/tweets more than once.

We will be making two stages (staging and production) in this project to demonstrate stage handling at each level of the project.

This tutorial assumes that the site is static pages hosted in an S3 account, but you can simply adjust if that isn't the case. We will be making a placeholder page to demonstrate proof of concept.

We will allow users to register their email and then send them a confirmation message. Alternatively, we will allow them to register by twitter and decide if they want to follow us. When I asked twitter about their automated follow policy they assured me that we could have users follow us in this way, but only if they explicitly consented (i.e. use a checkbox).

We will encrypt and store their emails and twitter info so that later we can send them emails/send them direct messages/do cool things.

I assume you have an AWS account with sufficient permissions and are not worried about minor charges throughout the project. I assume that you have a domain name of some kind or can easily get one. If you want to send emails you may need it, otherwise don't worry about it. Most of this is free tier, but I cannot guarantee you won't be charged anything. I also assume that you can use Google, and are patient.

Other assumed knowledge:

  • You know how to install packages on your specific machine (this tutorial assumes Mac or Linux, Windows may work but I don't know)
  • Very basic Javascript is nice but not necessary (node.js). You can also use Python or Java for the Lambda code, but you're on your own if you do.
  • Some Ruby is also nice but not necessary (you know what gems are).
  • Git, because always use version control. Always.
  • You are somewhat familiar with the command line (mkdir, cd, touch, etc.)
  • Basic Mysql (optional, but needed for the cool parts at the end where we send tweets)
  • You know what JSON is (even if you can't write it by scratch, you understand the concept)
  • You have at least heard of the following (if not, read a bit on the ones you don't know): AWS Lambda, API Gateway, Data Pipeline, S3, CloudWatch, DynamoDB, CloudFront, CLI, KMS, WAF, IAM, the mail service Mailgun, the tool Postman

This implementation is for you if:

  • You want to remove most hardware bottlenecks from your website and let DB read-write capacity be the main determining factor. Please note that if your request rate rapidly increased over 300 PUT/LIST/DELETE or 800 GET per second, S3 can bottleneck (source). If it goes over 600 combined "lambdas per second" (twitter registrations, callbacks, and email registrations), you will need to request an increase in your lambda limit(source)
  • You don't want to have to deal with waiting for auto-scaling, server crashes, etc.
  • You want to save a truckload of money over using ec2 servers (first 1M lambda requests are free)
  • You want to seem totally cool using a bunch of cloud technologies to make completely fail-safe, robust back ends. The only way it fails is if your AWS region goes out.
  • You are a masochist and can deal with two languages (three if you count SQL), myriad technologies, and the buggy monstrosity that is the AWS Console + DynamoDB's occasionally finicky I/O times all in the same project.

Notes:

If you find this confusing, don't worry. This took me a lot of time to finish and I needed to consult with other people more than once. It took weeks of refinement to get it to a decent place. Also, because of the layout of this guide, some of our decisions aren't going to make sense until later on. For example, why would we build a lambda that always throws errors? (Answer: Only way to redirect with API gateway)(source)

The part with making multiple lambda versions for different stages absolutely sucks. I know. There is nothing that can be done until/if AWS refines their service, unless you skip lambda versioning and staging all together.

Disclaimers:

1) I learned Javascript in 4 hours for this. If you have easy-to-read but clearly better ways to write the lambda code, please let me know and I will publish an update. When I do, you will receive credit at the end of the edited file (by name, email, username, whichever you wish). First-come first-credited.

2) I would love to help you troubleshoot your code, but I unfortunately don't have the time to address issues one-on-one. If you leave me a detailed message about the error you are experiencing with some evidence that it occurs despite directly following the guide I will certainly revisit the post, investigate, and credit you with the changes, but Google + forums are your friend here.

3) This project is all tied together. Cherry picking “how to make an API” or something might leave you disappointed because we make design decisions that are great for our project but are absolutely terrible in others (for example, most API schemes only returning 301 instead of 200).

Why I Built It:

I built this project because my tech lead and I had a brief conversation that went:
Tech Lead: “You built the API in Rails?”
Me: “Yeah, took a few hours because of some weirdness and my juvenile understanding of some keywords, but it's done.”
Lead: “Great. Do it again from scratch in AWS Lambda. I've never done this, but it should be possible. Tell me how it goes.”
I like working with new tech, so I agreed. I looked around and could not find a single how-to guide so I thought that this might actually be useful technology to learn and then share with the world.

Let's begin!

By the way, MIT License. Just check the repo.

Discussion

pic
Editor guide
Collapse
usertwogg profile image
Catherin Cruz

Hi Rob, quick question about the tutorial, how much is the budget need it to run this? I have an AWS account but I guess it's going to be some expenses in there. It's a must have a domain? I have not worked with DynamoDB before, so how much will be that service, any clue?

Collapse
rob117 profile image
Rob Sherling Author

Hey! Thanks for taking the time to look at the article.

You absolutely do not need to have a domain. I had a domain because Mailgun wouldn't send or recognize attempts to send emails after the first three unless I had a domain (which I believe is a bug). You may have much better luck.

The budget is very, very small. The cloudfronts are free because the data we'll send is easily free tier. DynamoDB is free - 25GB capacity and 25 read/writes a month total - and easily covers this tutorial. WAF may cost you about 1 cent. Data pipeline is free for 30 days per pipeline (so create, run, delete after tutorial). The rest should all be free tier, I believe.

Total budget: 10 cents, max? If you actually plan to use this in production, your dynamoDB will be the most expensive part of the equation by far, and is governed entirely by how much throughput you set it to. When you go to set the throughput, it will show you the price for the read/writes that you are requesting.

S3 could theoretically be more expensive if you have a very large site that you are serving statically.

If you have any other questions, please feel free to ask me here or to email me at the email listed in the opening post.

Collapse
usertwogg profile image
Catherin Cruz

Great! It's a green light for testing. When I get some progress done, I will let you know thanks.

Thread Thread
rob117 profile image
Rob Sherling Author

Anytime! Looking forward to it.

Collapse
jervine791 profile image
john ervine

AWS lambda is a fantastic tool but the choice of languages is poor. Ive been using it for the last 8 months for restful microservices and we have had significant problems with startup times from a cold start. Java Lambdas are particulay bad. My advice is benchmark it before you cut over to use it.

Collapse
rob117 profile image
Rob Sherling Author

I haven't used Java lambdas, so I honestly wouldn't know. I have had some strange experiences where (rarely) a particular lambda would take ages to start up (10+ sec), but most ran in just a few seconds.

Collapse
lukaszkuczynski profile image
lukaszkuczynski

Why to use Java if we have python? :)

sincerely,
Past-Java-dev

Collapse
ben profile image
Ben Halpern

What a great series, I'm really excited to work my way through it.

Collapse
rob117 profile image
Rob Sherling Author

Thanks, I appreciate that!