DEV Community: Mikkel

Resources, projects, and lessons from 100 days of (serverless) cloud learning

Mikkel — Sun, 25 Oct 2020 13:19:15 +0000

I recently completed my challenge of spending at least 1 hour a day learning about (serverless) cloud for 100 days almost* in a row.

In this post, I summarise the main topics I covered and the best resources for each, the projects I built, and some lessons for future self-directed learning projects.

*It took me 114 days - with a pandemic and some vacation time thrown in, that's close enough I think!

Topics and resources

The topics that I spent the most time on and found the best resources for were:

Fundamentals and the serverless landscape
Data modelling with DynamoDB
Best practices and design patterns
The business case for 'serviceful serverless'

Fundamentals and the serverless landscape

Despite having an AWS SAA cert and having used AWS at work before the challenge, a key focus for me was gaining a stronger foundation by really understanding the serverless landscape, including tools, best practices, and design patterns.

For this, my favourites were:

Off-by-none: Jeremy Daly's weekly serverless newsletter. More resources than you have time to read, but always some great stuff for all levels in there.
Serverless Stack: an excellent, free resource that gets you going with building a full-stack project using React and Serverless Framework. Having an opinionated guide with links to further best practices and little tricks (like this one) was a life-saver.
Serverless Airline Booking and its accompanying Twitch streams (14+ hours of content). Covers a lot of ground despite not assuming much prior knowledge.
Real-World Serverless: no fluff podcast that goes in-depth on best practices, architectures, and more. I have listened to all the episodes until now.

Data modelling with DynamoDB

I spent a good chunk of days on NoSQL data modelling with DynamoDB using various resources and applying the lessons to my Virtual Watercooler project.

My favourite resources were:

This episode of Serverless Chats (also a great podcast) with Alex DeBrie.
Alex DeBrie's re:Invent talk.
Once I'd watched Alex's talk, I graduated to Rick Houlihan's talk.
After the talks, I read Alex DeBrie's excellent and well-named book, The DynamoDB Book.
The NoSQL Workbench was an excellent thinking tool when applying the lessons.

Best practices and design patterns

Due to the dizzying and constantly increasing number of serverless building blocks, having good examples and understanding best practices to combine them into solid microservices within solid overall architectures is essential.

My favourite resources were:

Jeremy Daly's summary of 19 serverless Microservice Patterns. Excellent reference. Explains e.g. the 'Frugal Consumer' and 'Scalable Webhook' patterns used in my Virtual Watercooler project to cope with a maximum emails-per-second limit for SES.
Serverless Application Lens. Quite readable paper from AWS giving detail on various serverless architectures and considerations.
The intro and linked video in this article by Yan Cui. Gave me the idea to use Algolia as a 'serverless ElasticSearch' and DynamoDB Streams to trigger a Lambda that synchs changes to it in my Sentimental Robot project.
These episodes of the 'Real World Serverless' podcast:
- Episode 15 with Ben Kahoe from iRobot.
- Episode 18 (with Aleksandar Simovic and Slobodan Stojanovic) and 22 (with Gojko Adzic) contain good discussion of hexagonal architecture (Ports & Adapters).

The business case for 'serviceful serverless'

Spending so much time on serverless while working in a very 'servermore' (?) environment at work prompted a lot of thinking about this topic. I've got a ton of notes, so will have to write about it soon.

Some great resources are:

Joe Emison's The Serverless Sea Change. Coining the term 'Serviceful Serverless'.
These episodes of the 'Real World Serverless' podcast:
- Episodes 2 and 3 with Joe Emison.
- Episode 17 with Aleksandar Simovic and Slobodan Stojanovic talking about e.g. 'FinDev' and Wardley Maps.
- Episode 24 with Zack Kanter from Stedi.
Some good discussion about serverless for start-ups between Nader Dabit and Tom McLaughlin in these three threads: one, two, three.
This was a great thread to illustrate both (a) a ridiculously small AWS bill and (b) the internet (Alex DeBrie) helping make it even smaller.
Cloud Programming Simplified: A Berkeley View on Serverless Computing. 35 page paper with the bold claim that serverless 'represents an evolution that parallels the transition from assembly language to high-level programming languages'.

Projects

The majority of my learning was structured around the following projects. Whenever I learnt something new, I would go back and apply it to these.

Virtual Watercooler

Allows you to create an 'organisation', add members to it, and then randomly pair members for coffee chats - either by clicking a button or by choosing a frequency.

Backend tech stack: DynamoDB (single-table design), API Gateway, Cognito, Lambda (Javascript), SQS, SES, Cloudwatch, s3, and CloudFront. All packaged up with Serverless Framework.

Code available here. Wrote this to help me confirm that my approach to letting admins schedule recurring chats made sense.

Sentimental Robot

Scrapes BBC news headlines each morning, runs them through sentiment analysis, and allows the user to search through headlines and sentiments, or just get an overview of the past few days' sentiment. Had already started this but built a lot on top during the challenge.

Backend tech stack: Algolia, DynamoDB and DynamoDB Streams, Comprehend, CloudWatch, API Gateway, Lambda (Javascript), s3, CloudFront. All except Algolia packaged up with Serverless Framework.

Code available here.

ForgotMyGlasses

Tend to forget your glasses? No problem, teach the robot what your friends look like and use it to help you recognise them in the wild.

Backend tech stack: Rekognition, Lambda (Javascript), Cognito Identity Pool (unauthenticated role), and Cloudwatch. Using Serverless Framework as always.

Code available here.

SimpleQuiz (in progress)

Started this right at the end of the challenge. Decided on the data model and got a basic real-time connection working with subscriptions.

Backend tech stack: AppSync (GraphQL), DynamoDB, Lambda (Javascript), Cognito Identity Pool (unauthenticated role), and a few more probably.

Lessons for future learning projects

Open-ended 'benchmark projects'

For my learning, I like to define what Scott Young calls 'benchmark projects', where you pick something that you cannot do right now, but you might be able to do if you improved your skills, e.g. earning a difficult cloud certification.

This strategy worked well for me for the challenge. Had I tried to define specific topics and learning objectives at the start, I'd have failed as I simply did not have enough of a foundation to do so. The learning objectives I'd have set on Day 0, would have hindered rather than helped me on Day 20.

Had I set a more concrete goal or planned my journey in more detail at the start, for example, I'd never have spent the 30 or so hours on NoSQL data modelling that I enjoyed so much.

Instead, I set the goal of building 2 - 3 relatively 'production-ready' fully serverless projects and let that guide my learning. On day one I started building, and then discovered the missing pieces (such as NoSQL data modelling) I needed to fill in.

I wasn't interested in earning another certification (there aren't any fully serverless ones), but I think that would've been an excellent way to guide learning as well.

Learning vs. reporting

At the start, I stuck diligently to reporting progress and learning every single day. Gradually, I ran into two problems, however. Firstly, as topics got more advanced, I did not always have much to report from a day's worth of learning. Additionally, some days I only had 60 minutes available, so reporting would steal away scarce learning time (with the potential to get distracted on Twitter along the way).

In short, I started having more fun and being more effective once I switched to reporting by progress rather than daily. Sometimes that meant grouping 2 - 3 days, sometimes it was daily. But fitting reporting around the learning rather than the other way around is how I'd approach future projects like this.

Becoming 'active on the internet'

Since I stopped playing World of Warcraft and Counter Strike many years ago, I'd become mostly an 'observer' on the internet.

Seeing the 100 Days of Cloud community grow on Twitter and Discord, frequently reporting progress on Twitter, and following other people's learning challenges was super rewarding and got me back into being 'active on the internet'.

'Learning in public' like this is extremely powerful, making for better learning, networking, and accountability all in one. For my future projects, I'd need a very good excuse not to build or learn 'in public'.

Feedback or questions?

I'd love to know if you found this helpful, got any questions I might be able to help with on your cloud journey, or have feedback for me. Twitter or Linkedin are the best places to connect.

How to dynamically create CloudWatch Rules to let users schedule recurring actions

Mikkel — Tue, 28 Jul 2020 08:25:51 +0000

I recently needed to build a feature to let users set their own frequency for a recurring action. Specifically, I wanted to let the user decide how often my app should randomly pair members of their organisation and connect them via email.

I couldn't find many resources on how to easily accomplish this with serverless on AWS, so thought I'd share my learnings and explain the approach I took.

In short, I have an API Gateway / Lambda endpoint which both:

Adds or updates a CloudWatch Rule with the desired frequency. This rule will then trigger a Lambda (when it's time) which pairs and emails organisation members.
Adds or updates a 'RecurrenceRule' DynamoDB item (I store this here to make it easy to display to the user what their current setting is).

Visually, it could look e.g. like this with a simple dropdown:

In this post, I'll explain how I implemented the backend side of this, using The Serverless Framework, API Gateway, Lambda, and CloudWatch.

Note: there may very well be better ways to achieve this. In fact, I'm secretly hoping that by sharing my approach here I will get some feedback to improve my solution, so please reach out if you've got ideas!

Write a 'ManageRecurrenceRules' Lambda

First, write a Lambda function which manages the recurrence rules. This is the one that will be triggered by users clicking in the dropdown example above.

We will need the clients for both DynamoDB and CloudWatch as well as an environmental variable:

const dynamoDb = new AWS.DynamoDB.DocumentClient();
const cloudWatch = new AWS.CloudWatchEvents();
const recurringPairAndEmailFunctionArn = process.env.RECURRING_PAIRANDEMAIL_LAMBDA_ARN;

Ignoring various other things you may want to do (e.g. I retrieve an organisationId from Cognito), we first need to construct a cron expression based on the POST request, e.g.:

const frequency = JSON.parse(event.body);

let scheduleExpression;
switch(frequency) {
    case 'Fridays':
      scheduleExpression = "cron(0 12 ? * FRI *)";
      break;
    case 'Every 5 Minutes':
      scheduleExpression = "cron(0/5 * * * ? *)";
      break;
    case 'Daily':
      scheduleExpression = "cron(0 12 * * ? *)";
      break;
    default:
      // return an 'Invalid frequency' error to the user;

Next, we need to construct a unique rulename. I used an organisationId (which I get from an adminGetUser Cognito request) combined with the stage:

const ruleName = organisationId + "-" + process.env.STAGE;

Use putRule to create or update the rule and then use putTargets to add the desired target Lambda we want the rule to trigger in my case a Lambda function that pairs organisation members and email-intros each pair:

  try {
    await cloudWatch.putRule({
      Name: ruleName,
      ScheduleExpression: scheduleExpression
    }).promise();

    await cloudWatch.putTargets({
      Rule: ruleName,
      Targets: [
        {
          Id: organisationId,
          Arn: recurringPairAndEmailFunctionArn,
          Input: JSON.stringify({
            organisationId: organisationId
          }) // the rule will pass this Input to the Lambda when it triggers it
        }
      ]
    }).promise();

Note that I tell the rule to pass a custom Input to the Lambda target. This means that the target, in my case the 'recurringPairAndEmailFunction', can access the property directly from the event like this:

const orgId = event.organisationId;

Finally, still in the same 'try/catch' block, I add this rule to DynamoDB to easily display the currently selected frequency to the user:

    const params = {
      TableName: process.env.USERS_TABLE,
      Item: {
        PK: organisationId,
        SK: "RecurrenceRule",
        frequency: frequency
      }
    };
    await dynamoDb.put(params).promise();

If all this goes well, we return a lovely 200 OK to the user 🎉

serverless.yml setup

Next, we need to define the API Gateway endpoint and pass the environmental variable to the Lambda (the recurringPairAndEmailFunctionArn used above). In the serverless.yml:

  manageRecurrenceRules:
    handler: manageRecurrenceRules.main
    environment:
      RECURRING_PAIRANDEMAIL_LAMBDA_ARN: arn:aws:lambda:${self:provider.region}:accountidgoeshere:function:${self:service}-${self:custom.stage}-recurringPairAndEmail
    events:
      - http:
          path: rules
          method: post
          cors: true
          authorizer: aws_iam

Note that the target Lambda, recurringPairAndEmail, is defined in the template as simply:

  recurringPairAndEmail:
    handler: recurringPairAndEmail.main

Next, we need to make sure that our Rules have the required permissions to invoke our Lambda. The below allows all rules for my account in this region to invoke the Lambda (which is probably too permissive):

resources:
  - Resources:
      RecurringPairAndEmailInvokePermission:
        Type: AWS::Lambda::Permission
        DependsOn: RecurringPairAndEmailLambdaFunction
        Properties:
          Action: lambda:InvokeFunction
          Principal: events.amazonaws.com
          SourceArn:
            Fn::Sub: "arn:aws:events:${self:provider.region}:accountidgoeshere:rule/*"
          FunctionName: arn:aws:lambda:${self:provider.region}:accountidgoeshere:function:${self:service}-${self:custom.stage}-recurringPairAndEmail

Finally, we need to ensure that our 'ManageRecurrenceRules' Lambda has permissions to add rules and targets, so we add the below in the provider section:

iamRoleStatements:
    - Effect: "Allow"
      Action:
        - events:putRule
        - events:putTargets

Aaand, that's it!

Our users can now choose between Fridays, Daily or Every 5 Minutes as frequencies by making a POST request to the /rules path.

In CloudWatch, a rule will look something like this:

How can the solution be improved? Is this the 'right' way?

Now, back to my secret intention with this post... How can the solution be improved? Is this the 'right' way of implementing this feature?

How can I best work around the constraints on CloudWatch Rules, such as:

The per-region soft limit of 100 CW Rules.
The limit of 5 targets per rule (e.g. so having a rule for each frequency and adding organisations as Input probably isn't the way either).

Thanks for reading and I hope to hear from you either here or on Twitter (@miksimal)!