DEV Community: John Doyle

GitMosaic Hackathon

John Doyle — Tue, 23 May 2023 23:05:55 +0000

What I built

A mosaic showing off your GitHub Avatar based on your git frequency.

Category Submission:

Wacky Wildcards

App Link

https://gitmosaic.com

Screenshots

Description

Compete with others to claim your section of the mosaic with your avatar. Let others commit to your repo to allow your team to contribute together!

Link to Source Code

Action: https://github.com/johncolmdoyle/gitmosaic-action
Application: https://github.com/johncolmdoyle/gitmosaic

Permissive License

Apache License

Background (What made you decide to build this particular app? What inspired you?)

This was inspired by a now archived sub-reddit, /r/place where the community would come together to generate a giant piece of art.

How I built it

I came into the hackathon very late so I kept this very simple. I created a GitHub Action that sent the context of the git commit to an api. The owner of the repo would be identify, and their image would be downloaded and broken into 100 pieces.

Each url was assigned a starting x, y co-ordinate.

Each of the 100 pieces had an off set from those x, y coordinates.

Each commit id would identify a random index in this array. This x, y coordinate was set on the mosaic pointing to this image.

The latest version of each coordinate was returned to the user to display the current mosaic.

Additional Resources/Info

EuroSquares: An AWS Amplify-Powered Game for Eurovision 2023

John Doyle — Sun, 14 May 2023 06:58:20 +0000

Introduction

The Eurovision Song Contest is one of the most watched events on television, with millions of viewers tuning in from around the world to watch the performances of various countries' musical acts.

For Eurovision 2023, I decided to create EuroSquares, a game that allows a group of friends to be assigned random countries to see who will have the most points at the end of the contest. In this post, I'll explain how AWS Amplify was used to power the EuroSquares application, the code is available on GitHub.

Tech Stack

AWS Amplify is a platform that allows developers to quickly and easily build full-stack applications using popular web frameworks and integrate them with AWS services.

EuroSquares was built using React JS and was NodeJS. The application consists of a front-end React application, a REST API, AWS Lambdas, and DynamoDB tables for storing data.

Front End

The front-end of the application was built using React and was designed to be simple, especially with a time crunch on the project. The user interface was designed using the Bootstrap library, which provides a set of pre-built UI components that can be easily customized to fit the needs of the application.

React allowed me to control the content of the calls between the Front End/Mid Tier/Back End, the actions that would trigger the calls, etc.

Mid Tier

The React application communicates with the back-end of the application through a REST API.

With any REST API, it is always good to build out the Postman Workspace to allow easy testing.

There are only 3 resources defined for the project:

Squares
Bands
Score

Amplify gives an option of using a template and linking the API Endpoint with the Lambda and a connected DynamoDB table. I selected the template, Serverless Espress, which provided a lot of code in the template, though the majority of it felt overly bloated for what was effectively CRUD functionality.

Amplify has a strict way of building all the components, and this was the time I ran into some issues where I wanted to go back and grant a function access to multiple DynamoDB Tables. I struggled trying to find the proper way to implement this, and eventually succumbed to granting an over-broad set of permissions:

[
  {
    "Action": [
      "dynamodb:*"
    ],
    "Resource": [
      "*"
    ]
  }
]

Back End

The DynamoDB database was used to store the state of the game, including the current state of the grid, the selected squares, and the points each player had accumulated. The benefit of the database was how highly scalable and resilient it was, with automatic scaling and failover capabilities built in.

There was nothing very complex about this setup, as the project was not data intensive.

Future work, for the next contest, would be to try and hook this into a data source vs manually entering in the data. Furthermore, rather than manually calling the scores endpoint, it would have been better to have a trigger that would automatically recalculate every player's score as a country's points were announced.

Conclusion

In summary, EuroSquares was built very quickly using the AWS Amplify platform, allowing a serverless project to be released in a matter of hours. Managing a lot of the components and hosting helped with the basics.

AWS SSO & GitHub OpenID Connect Setup

John Doyle — Fri, 07 Apr 2023 20:19:56 +0000

After setting up our AWS accounts, we need to ensure that only our user and our CI/CD pipeline have permission to perform updates. AWS has published a lot of resources around best practices.

Identity Center

Since we plan to have multiple AWS accounts, we need to manage access to each of them. The AWS Identity Center enables you to create and manage AWS users, groups, and permissions to grant or deny access to AWS resources across AWS accounts in your organizations.

We do not want to access any of our accounts with the root account. Instead, we will create an IAM user with our primary account's AWS Identity Center that will have access to each of the AWS accounts we need.

Similar to how IAM is generally configured, you have a User, who can be assigned to a Group, and that Group can be granted permissions.

As this is across AWS Accounts, the mapping of permissions is configured account by account. So your development group might have full access to the development account, but maybe only read access to CloudWatch logs etc in the production account.

Single Sign On

Another great feature of AWS Identity Center is that it gives us a personalized URL to sign in with our IAM user and access all the accounts that the user has access to! The portal allows us to either log in to the account or download temporary access credentials for use with the AWS CLI.

Now we login with our user and can get access to which ever account we need:

CI/CD

With our user having access, we start to shift our focus over to our pipeline and CODE! The aim here is to use GitHub Actions to deploy our changes.

It is a bad practice to create an IAM User and use their credentials for our pipeline's permission set. We end up with Access Keys that never get rotated, in an external service. If the Access Key is compromised, it can lead to unauthorized access to your resources, resulting in data breaches, data loss, or other security incidents.

We prevent this by granting GitHub access to assume IAM roles which grants the service temporary credentials to our AWS Account.

Identity Provider

The first step in this configuration is to authorize GitHub as a trusted service. We will perform all of these steps in each AWS account we want GitHub to have access to, so login to the Dev account now.

Once logged in, we can configure create a new Identity Provider. GitHub has provided documentation on setting this up.

Creating a Role

Now that we have a trusted identity provider, select it from the list and you should have the option to Assign Role to it:

Let's create a new role, and it should pre-populate the role with some options. You will need to select the audience from the drop down, but there should only be one.

Assign the role the permissions we will want the CI/CD Pipeline to have. Don't worry too much about this, we will come back to it often to refine the permissions as we go. Finally set a name like GitHub and create the role.

Restricting the Role

Having a role is great, but we don't want anyone using GitHub to be able to assume it.

We will lock this down to our GitHub Repo and to a specific branch.

Open the Trust Relationships for the role.

I will update the Condition statement to only grant access from the dev branch of the github repo:



"Condition": {
   "ForAllValues:StringEquals": {
      "token.actions.githubusercontent.com:ref": "refs/heads/dev",
      "token.actions.githubusercontent.com:aud": "sts.amazonaws.com"
   },
   "StringLike": {
      "token.actions.githubusercontent.com:sub": "repo:johncolmdoyle/holycitypaddle-code:*"
   },
}

Production

Now repeat these same steps for the production account and update the condition statement to reference our main branch:



"Condition": {
   "ForAllValues:StringEquals": {
      "token.actions.githubusercontent.com:ref": "refs/heads/main",
      "token.actions.githubusercontent.com:aud": "sts.amazonaws.com"
   },
   "StringLike": {
      "token.actions.githubusercontent.com:sub": "repo:johncolmdoyle/holycitypaddle-code:*"
   },
}

We are now ready to utilize configure-aws-credentials within our GitHub Actions as we move onto deploying our code!

High Level Architecture

John Doyle — Fri, 31 Mar 2023 14:55:24 +0000

We have our accounts, and we have our requirements - the next step is to sketch out at a high level what services we will use, where they will reside, and a quick point on CI/CD.

Services

There are several AWS services that we will be utilizing in this project:

Amplify

This is a neat little service that allows us to build out web and mobile applications. This service integrates our front end using Flutter or ReactNative, and hooks it straight up to the back end!

Location Services

Launched back in June 2021, this service specializes in maps, geocoding, geofencing, tracking, and routing. Something that will be pretty important when navigating the rivers.

Serverless

API Gatweay, Lambda, and DynamoDB make up the primary serverless triad. By serverless, in this context, it means that we do not manage the underlying infrastructure. Indeed these three services allows our application to scale in and out dynamically while minimizing operational overhead.

File Storage

S3 and CloudFront will be used to manage any media that is uploaded - from logos and images for the application, to images other users upload. They will be stored with an S3 bucket, and then hosted by CloudFront. CloudFront is a content delivery network (CDN), the content in edge locations that are closest to the user, reducing the amount of time it takes to fetch and deliver the content. This improves the user experience by reducing latency and improving performance.

CI/CD

We will be building our application using GitHub and GitHub Actions to deploy the code.

Before we get into deployment, we want to setup our Git Repositories and align them with our AWS Accounts.

Due to the simplicity of this project, I will only setup 2 repos - one for the application that will be deployed to Development and Production, and a second one that manages the Shared Services account.

Next we will setup AWS SSO to allow us to login to our AWS Accounts and also grant our GitHub repos permission to deploy to the AWS Accounts.

Accounts & Organizations

John Doyle — Fri, 31 Mar 2023 13:44:14 +0000

When setting up your AWS account, it's easy to jump right in and start deploying resources, following tutorials, and becoming familiar with your account.

Pause though!

It's important to remember to remove specific resources when you're finished with them to avoid surprise bills later.

Billing Alerts

To avoid this unpleasant surprise, it's a good idea to set up a billing alert for a specific amount each month. You can do this by navigating to AWS Billing Budgets and creating a budget using AWS's preconfigured templates. Choose an amount that you're comfortable with, such as $100 per month.

Navigate to [AWS Billing Budgets](https://console.aws.amazon.com/billing/home?#/budgets/overview) where you can easily create a Budget using AWS configured templates.

Now that you won't be caught off guard by unexpected charges, let's configure your organization.

AWS Organizations

While you have your primary account, it's recommended to create a separate AWS account for each project. This keeps everything related to the project grouped together, both from a budget and resource perspective. When you're finished with a project, you can simply delete the resources in the account and shut it down.

To manage multiple AWS accounts more easily, you can use AWS Organizations, which allows you to consolidate multiple accounts and centrally manage and control them. There is no additional cost to set up AWS Organizations, so it's highly recommended.

Project OUs

Within AWS Organizations you can organize your multiple accounts into Organizational Units, i.e OUs. These are like folders to keep all of your related accounts in.

When kicking off a project, I look at having the following three main AWS accounts:

Production
Development
Shared Services

The Production and Development accounts are straightforward: code is deployed to the Development account, tested, and then promoted to Production. The Shared Services account is more complex, as it hosts resources used by both the Production and Development accounts.

For example, if you're purchasing a domain name, you would register it within the Shared Services account, which would then point the domain to the appropriate accounts so that they can use the primary domain. This keeps the Production account from pointing subdomains back down to Development.

When creating an account, there are a few rules of thumb:

Add tags - I usually add a project and environment tag.
Account Email - I will use email subaddressing, also known as plus sign (+), to create accounts with; but don't login and setup a password with them. We will utilize AWS SSO service for that.

Now after setting up my accounts, I have the following organization unit:

Infrastructure As Code

While setting up accounts I tend to do manually, you can also programmatically do this using Terraform. You could build out scripts that allow you to have a project template which you can run each time you have a new project idea.

provider "aws" {
  region = "us-east-1"
}

resource "aws_organizations_account" "root" {
  name = "my-root-account"
  email = "my-root-account@fake.com"
}

resource "aws_organizations_organizational_unit" "ou" {
  name = "holy city paddle"
  parent_id = aws_organizations_account.root.id
}

resource "aws_organizations_account" "prod" {
  name = "hcp-prod"
  email = "my-root-account+hcp-prod@fake.com"
  parent_id = aws_organizations_organizational_unit.ou.id
}

resource "aws_organizations_account" "dev" {
  name = "hcp-dev"
  email = "my-root-account+hcp-dev@fake.com"
  parent_id = aws_organizations_organizational_unit.ou.id
}

resource "aws_organizations_account" "shared" {
  name = "hcp-shared"
  email = "my-root-account+hcp-shared@fake.com"
  parent_id = aws_organizations_organizational_unit.ou.id
}

Project Requirements

John Doyle — Fri, 31 Mar 2023 04:29:24 +0000

One of the things I love about coding is the ability to start a random side project and try out new things. Over the years, I've played with side projects and participated in hackathons, and I've discovered some best practices for quickly building a solid project.

Recently, I bought a kayak and realized I needed to figure out where to go, when to go, and what are the best things to do. Although there are multiple websites that could help me, I decided to build something that met my exact needs. This side project is called "Holy City Paddlers."

Requirements

Before diving into the code, it's essential to write down some basic goals and key features to minimize the cost of refactoring later.

Requirements typically fall into two primary groups:

Functional
Non-Functional

The easy way to think about these differences are the actions or operations that are needed to take place (functional) vs well.. really everything else - how does it perform, what usability aspects does it need to account for.

Functional Requirements

For this app, there are a few things I want to get from it:

Maps: I want some geolocation based maps, where am I currently, let me plan out my routes between rivers.
Weather Forecasts: Some folks may be all weather kayakers, but I only want to hit the water when its nice and sunny.
River/Tide Information: This is a big one, and it depends on my route. Am I looking to go up-river or down-river. Matching the tides with when I want to go out and where I want to go is critical.
Reviews: While this is personal, it would be great for other people to be able to recommend routes, or even points on the river to launch from.
Wildlife and Landmarks: It would be cool to be able to see historic landmarks along the route.

Non-Functional Requirements

Many things fall under non-functional, but I'm going to kick most of those into the next post around the architecture especially when we thing about performance, reliability etc.

For me, and this project, the number 1 non-functional requirement is around usability.

Usability: I need an easy planning mode coupled with an easy to understand on-water mode.

With these requirements in mind, I'll dive into the architecture in my next post.

Building a Multi-Region app with AWS CDK - Part 1

John Doyle — Sun, 07 Feb 2021 20:35:35 +0000

There are lots of tutorials of deploying basic CRUD applications with AWS CDK using API Gateway, and Lambda connecting to a DynamoDB table. The defacto starter set for a serverless application. You build and deploy, modify the hello-world application to match your needs, you're happy! The API is running and acting perfectly correctly in your default region.. us-east-1...

The Issues Begin

Then November 25th, 2020 strikes. The whole region is down for hours, your API is inaccessible, people are angry at YOU for this inconvenience. You read more about us-east-1 and realize that it tends to have a history of outages... and plan to make the jump to another region.

December rolls around and you've successfully migrated your application to us-west-2. It wasn't too difficult, your CDK app was able to tear down everything in us-east-1, and then deploy everything again in the new region. You let out the breath you'd been holding.

January 7th 2021 started off really well, and as you started looking forward to the end of the day... the alerts start ringing. Your new region, us-west-2, has begun to act up! After two hours of sweating, you are back up.

Multi-Region Solution

After all this pain and stress, you're determined not to be caught flat-footed again. While the goal of a multi-region setup seemed daunting, you roll up your sleeves and dive into it.

For our CRUD application, there are two components we need to deal with:

Ensuring our data is consistent between the regions.
That DNS always resolves to a region that is up.

Thankfully there are solutions to both of these issues in our existing tech stack.

Tech	Single Region Solution	Multi Region Solution
Data	DynamoDB Table	DynamoDB Global Table
DNS	Route53 Simple Routing Policy	Route53 Latency Routing Policy

In part 1, I'll dive into the complexities of implementing a multi-region data configuration.

DynamoDB Global Tables

The DynamoDB Global Tables are a managed solutions from AWS where they keep replicate data changes from one DynamoDB table to all related tables in other regions.

For simplicity of use, there can be a number of "gotchas" from an Infrastructure as Code perspective.

CDK Primary Region

While DynamoDB Global Tables are multi-master, multi-region, from an AWS CDK point of view, we deploy them in only a single region. The resource is configured to replicate to the other regions that you're interested in.

So I would instantiate my stack in us-west-2, and configure it to replicate to us-east-1, us-east-2, and us-west-1.

import * as  dynamodb  from  '@aws-cdk/aws-dynamodb';
// Stack was called in us-west-2
const gloablTable = new dynamodb.Table(this, 'globalTable', {
  partitionKey: {
    name: 'id',
    type: dynamodb.AttributeType.STRING
  },
  billingMode: dynamodb.BillingMode.PROVISIONED,
  replicationRegions: ['us-east-1', 'us-east-2', 'us-west-1'],
  removalPolicy: cdk.RemovalPolicy.DESTROY,
});

Since we want to reference our global table's name later on as environment variables for our Lambdas, we will want to build this out into two stacks.

This will make our app resemble the following:

const appRegions = ['us-east-1', 'us-east-2', 'us-west-1', 'us-west-2'];
const  app  =  new  cdk.App();
const globalstack = new GlobalStack(app,'DynamoDBGlobalStack', {env: {region: 'us-west-2'}});

appRegions.forEach(function  (item,  index)  {
  new AppStack(app, 'AppStack-'.concat(item),  {
    env: {account: process.env.CDK_DEFAULT_ACCOUNT, region: item},
    globalTableName: globalstack.globalTable.tableName
  });
}

Region References - Mocks

You might have noticed that I am passing only the table name to the app stacks, rather than the actual table object which would be best practice. This is because the actual table is specifically referencing our primary table in us-west-2. I'm afraid in our App stack we will need to either mock out the table OR use the AWS SDK to retrieve the full details.

For most use cases, simply mocking out the table should be all that you need.

import * as  cdk  from  '@aws-cdk/core';
import * as  dynamodb  from  '@aws-cdk/aws-dynamodb';

interface  CustomStackProps  extends  cdk.StackProps  {
  readonly  globalTableName: string;
  readonly  env: any;
}

export class AppStack extends cdk.Stack {
  constructor(scope: cdk.Construct, id: string, props: CustomStackProps) {
    super(scope, id, props);
    const globalTable = new dynamodb.Table.fromTableName(this,  'globalTable', props.globalTableName);

We can reference the globalTable to grant IAM permissions etc.

Triggers

I did find that there is ONE instance where there was the need to use the AWS SDK, and that was around implementing triggers off the DynamoDB table. Global Tables are kept in sync by the use of DyanmoDB Streams, and AWS automatically names these streams in the following format:

arn:aws:dynamodb:REGION:AWS-ACCOUNT:table/TABLE-NAME/stream/2021-01-16T19:47:47.531

This timestamp is specific to each region, and so the only way to retrieve this information is to describe the specific region table.

...
  constructor(scope: cdk.Construct, id: string, props: CustomStackProps) {
    super(scope, id, props);

    const  client  =  new  DynamoDB({  region: props.env.region  });

    // Query the regions table
    let  globalTableInfoRequest  =  async  ()  =>  await  client.describeTable({  TableName: props.globalTableName});

    globalTableInfoRequest().then(  globalTableInfoResult  =>  {
      // Mock the table with the specific ARN and Stream ARN
      const globalTable = dynamodb.Table.fromTableAttributes(this, "globalTable", {
        tableArn: globalTableInfoResult?.Table?.TableArn,
        tableStreamArn: globalTableInfoResult?.Table?.LatestStreamArn
      });

      // Lambda
      const triggerLambda = new lambda.Function(this, 'triggerLambda', {
        ...
        environment: {
          TABLE_NAME: props.globalTableName,
          ...
        }
      });

      // Grant read access
      globalTable.grantStreamRead(triggerLambda);

      // Deadletter queue
      const triggerDLQueue = new sqs.Queue(this, 'triggerDLQueue');

      // Trigger Event
      triggerLambda.addEventSource(new DynamoEventSource(globalTable, {
        startingPosition: lambda.StartingPosition.TRIM_HORIZON,
        batchSize: 5,
        bisectBatchOnError: true,
        onFailure: new SqsDlq(triggerDLQueue),
        retryAttempts: 10
      }));
    });
  }
}

Deployment

One specific issue we do run into with this design, is that the multiple app stacks which are deployed in multi regions are dependent on the single global stack that was deployed in one region. Cloudformation does not allow us to create a cross region dependency between stacks. We will want to deploy our GlobalStack first, and then we can deploy all stacks.

We want to synthesize the CDK to produce the Cloudformation template for us:

cdk synth GlobalStack

The we can depoy just the global stack:

cdk deploy GlobalStack

Once that is complete, we can generate all the cloudformation templates - the AWS SDK code is excuted during synthesizing so we need the Global tables deployed beforehand:

cdk synth

Finally we can deploy all stacks:

cdk deploy --all

Since we are performing this ordered deploy, to remember that to tear this down you will need to work in reverse order due to the dependencies that have been created. Tear down all the app stacks first, then tear down the global tables.

Regions Available

A final piece to consider is that DynamoDB Global tables in most, but NOT all regions - specifically the following regions do not support them at the time I write this:

Region Name	Region Code
Africa (Cape Town)	af-south-1
Asia Pacific (Hong Kong)	ap-east-1
Europe (Milan)	eu-south-1
Middle East (Bahrain)	me-south-1

These four regions are disabled by default, and you will need to enable them if you wanted to use them.

Conclusion

DynamoDB Global tables are a great method to quickly implement a multi-region, multi-master data solution that is managed by AWS. In 2019 the pricing model for Global tables was updated that removed the cost to replicate data between regions, which really elevated this into a great solution.

My next post will examine the DNS routing policy that we will want to implement to ensure users are not impacted by any region downtime in the future.

All code for this post can be accessed on GitHub

287 Hours

John Doyle — Tue, 02 Feb 2021 21:52:35 +0000

The race was already on, and I was late. Over a thousand people were ahead of me, with almost 50 teams having submitted completed projects. Yet this is a hackathon, and no one knows what might stand out till the end!

I stumbled onto the Postman API Hackathon a week after it had opened. Back in November 2020, Postman had released a new service called Public Workspaces that was aimed at helping people to collaborate on APIs. With Postman Galaxy, their annual conference coming up on February 2nd-4th, they announced a hackathon with the results announced at it.

The gauntlet was thrown down in the format of a challenge to, “not just create a Postman public workspace with a Collection of APIs, but build something that is creative, has compelling value to developers, addresses a problem, or has community interest.” Alright, I thought, this is doable - I’ve spent late nights banging my head against the table trying to understand why something isn’t acting right.

Inspiration struck with a memory of Boss-Man Paul calling late at night, asking if a service was down. Nope, turned out he just had horrific internet. Yet this did bring me back to wondering if there was a better way to tell. Sure, the API is up for me and maybe for isitdownrightnow.com - but maybe the Dublin data center is experiencing issues that I’m not seeing!

Flexing my knuckles, imagining the crack, I dive into some code - create a GitHub repo, initialize a fresh AWS CDK project, and the hardest part… buy a snazzy domain name. And so, on January 15th api-network.info was acquired!

Instantly I started running into issues as my domain was not resolving no matter how much I refreshed my browser - so the first network took was decided upon, time for NSLoopup to give me insights! This was familiar ground thankfully, hustle some NodeJS code into a Lambda and deploy it.

Yet having a single region tell me its status wasn’t very enlightening - so I need to think broader, multi-region broad. I went down several paths trying to figure a good way to coordinate my requests, both within a region and across them. The final solution I landed on was DynamoDB Global Tables - automatically replicate data across regions and allowing me to set up triggers within each region to fire off on any change to that data!

This sounded perfect - I could add additional multiple regions as I needed to! Back to the true work so, adding more network context than I would have a clue what to do with. Utilizing the newly released Lambda Docker containers I was even able to run some OS-level commands, like traceroute and dig. I assembled my core group of 5 commands and was ready to go global!

At this point, I ran straight into a wall. The perfect solution utilizing Global Tables and Triggers came back to bite in with a vengeance. It turns out that Global Tables are built in a single region, then that region builds the replications in other regions. My original code that simply would duplicate my whole stack in each region was thrown a curveball - now I needed to build my global tables, and then loop over all the regions to add my app stack.

Not too big of a change… until I realized that the DynamoDB triggers that the Lambda’s use are systematically named. With a timestamp. The primary region that built out the Global Tables had no way of telling me what the Stream ARN would be in Hong Kong, the only way to find it would be to describe the table IN Hong Kong. Dr. Frankenstein would be disgusted (at least the CDK team would be) at my abomination as I started calling the AWS SDK within the CDK construct.

This is a hackathon though - please leave your good practices at the door.

Now that I had a multi-region API, I could finally start tackling the actual main hackathon - building out a Public Workspace. Postman had provided links to several example collections to inspire folks, which led me to a big facepalm. Despite having used the application for years, I had never known you could visualize responses. From basic HTML to full-on Bootstrap, to D3JS charts. The quantity of data I was returning was too much to parse manually, this was the perfect solution - to build out charts that highlighted different aspects!

Over the next few days, I added data, massaged it, worked it into several charts that started to give me hope that the project made some sense. From simple bar charts, I stepped up to plotting out the traceroutes on a map, to designing a region reconciliation report on DNS or Requests.

The final hours were drawing to a close, it had been multiple days of late nights, tearing the infrastructure down and rebuilding it. Making decisions that I would go on and completely regret, but there was no time to go back and change them. It was the last day.

Five hours to go, no worries. Code is basically locked in now. Just one last part to do - a video demo. How hard can this be? Open up Photo Booth to record me, and start recording my screen.

Three hours later I just don’t care anymore - stumble over the script, sipping a can of Guinness that is slowly warming. Don’t know when I opened it. Edit the videos together and stare at the upload bar on YouTube. The hackathon site was about to start the 60-minute countdown. If there was an issue at this stage, it might require something stronger than Guinness to power me through to the end.

Yet it succeeded.

The video was uploaded, linked, and submitted with the project.

Time for some much-needed sleep, and fruitlessly thinking that for the next hackathon I’ll be better prepared. Time will tell.

Resources

Follow-up

I'll post up some technical blogs diving into some of the problems I encountered later, from deploying custom Lambda Docker containers to managing multi-region API Keys.

Java Lambda Containers

John Doyle — Sun, 13 Dec 2020 07:11:34 +0000

AWS Re:Invent 2020 held a number of great announcements for AWS Lambda! One of these included the support for Docker Containers - previously the only supported packaging was Zip that was stored in S3 behind the scenes.

At the same time, Lambda also saw the memory configuration expand - from the 3GB limit initially, all the way up to 10GBs! This is especially useful as Lambda will also support Docker images that reach 10GB in size - though this size increase is only for Docker containers, ZIP packages remain at a maximum of 250MB.

AWS provides two ways to support docker lambda

Use an AWS base docker image, all the support run times:

amazon/nodejs12.x-base
amazon/nodejs10.x-base
amazon/python3.8-base
amazon/python3.7-base
amazon/python3.6-base
amazon/python2.7-base
amazon/ruby2.7-base
amazon/ruby2.5-base
amazon/go1.x-base
amazon/java11-base
amazon/java8.al2-base
amazon/java8-base
amazon/dotnetcore3.1-base
amazon/dotnetcore2.1-base

Use your own base docker image and you import the AWS runtime interface client and the lambda talks to the interface client which then talks to your code.

Apparently, Lambda is not actually running the container, rather it will build a function from the container image. I have noticed when running the Java 11 Base Lambda image a pretty significant cold start penalty - up to 20 seconds.

To test out the docker containers, I'll utilize AWS Serverless Application Model to generate, build, and deploy the lambda!

Build & Deploy Docker Lambda

Initialization

To simplify some of these commands, I'll create some environment variables to hold the default region and AWS Account ID:

export AWS_ACCOUNT_ID=$(aws sts get-caller-identity | jq -r '.Account')
export AWS_REGION=$(aws configure get default.region)

Since we will be building out an image for the lambda, we want to store it in an Elastic Container Registry. We can create one with the following command:

aws ecr create-repository --repository-name gizmo-lambda-container | jq '.repository.repositoryUri'

Which should output the container registry url, similar to: 196295636944.dkr.ecr.us-east-1.amazonaws.com/gizmo-lambda-container

We can then authenticate with the AWS ECR repo:

aws ecr get-login-password | docker login --username AWS --password-stdin ${AWS_ACCOUNT_ID}.dkr.ecr.${AWS_REGIO
N}.amazonaws.com

Now we will use the AWS Serverless Application Model to build and deploy our application. In an empty directory we will create our project:

sam init

You will be prompted to select several options:

Which template source would you like to use?
    1 - AWS Quick Start Templates
    2 - Custom Template Location

We want to use AWS Quick Start Templates.

What package type would you like to use?
    1 - Zip (artifact is a zip uploaded to S3)
    2 - Image (artifact is an image uploaded to an ECR image repository)

And here we want to use Image! For this example, I went with the amazon/java11-base base image. Since it is Java we are prompted about the dependency manager, which I recommend gradle.

Build

The skeleton project that is generated has our Dockerfile and application code already setup. The AWS SAM command will build the Docker image for us!

sam build

And we can see the Docker build steps in the output:

Building codeuri: . runtime: None metadata: {'DockerTag': 'java11-gradle-v1', 'DockerContext': './HelloWorldFunction', 'Dockerfile': 'Dockerfile'} functions: ['HelloWorldFunction']
Building image for HelloWorldFunction function
Setting DockerBuildArgs: {} for HelloWorldFunction function
Step 1/16 : FROM public.ecr.aws/lambda/java:11 as build-image
 ---> 9d9f8f6bbeea
Step 2/16 : ARG SCRATCH_DIR=/var/task/build
 ---> Running in 3f077f57a99f
 ---> 1790443364e1
Step 3/16 : COPY src/ src/
 ---> 6357ad3066ed
Step 4/16 : COPY gradle/ gradle/
 ---> 86b36578535b
Step 5/16 : COPY build.gradle gradlew ./
 ---> 7a0e366bbda5
Step 6/16 : RUN mkdir build
 ---> Running in 57ceac09d965
 ---> 6d547fcbb584
Step 7/16 : COPY gradle/lambda-build-init.gradle ./build
 ---> ee1fb2ece820
Step 8/16 : RUN ./gradlew --project-cache-dir $SCRATCH_DIR/gradle-cache -Dsoftware.amazon.aws.lambdabuilders.scratch-dir=$SCRATCH_DIR --init-script $SCRATCH_DIR/lambda-build-init.gradle build
 ---> Running in 06116e2de319
Downloading https://services.gradle.org/distributions/gradle-5.1.1-bin.zip
.................................................................................

Welcome to Gradle 5.1.1!

Here are the highlights of this release:
 - Control which dependencies can be retrieved from which repositories
 - Production-ready configuration avoidance APIs

For more details see https://docs.gradle.org/5.1.1/release-notes.html

Starting a Gradle Daemon (subsequent builds will be faster)
> Task :compileJava
> Task :processResources NO-SOURCE
> Task :classes
> Task :jar
> Task :assemble
> Task :compileTestJava
> Task :processTestResources NO-SOURCE
> Task :testClasses
> Task :test
> Task :check
> Task :build

BUILD SUCCESSFUL in 17s
4 actionable tasks: 4 executed
 ---> f394216fbb70
Step 9/16 : RUN rm -r $SCRATCH_DIR/gradle-cache
 ---> Running in 0b8e94e2670c
 ---> 7f9033989b1a
Step 10/16 : RUN rm -r $SCRATCH_DIR/lambda-build-init.gradle
 ---> Running in 4965d01c3455
 ---> fd5e12377577
Step 11/16 : RUN cp -r $SCRATCH_DIR/*/build/distributions/lambda-build/* .
 ---> Running in 9c29934509fe
 ---> 434e1178004e
Step 12/16 : FROM public.ecr.aws/lambda/java:11
 ---> 9d9f8f6bbeea
Step 13/16 : COPY --from=build-image /var/task/META-INF ./
 ---> af6f91bce6e5
Step 14/16 : COPY --from=build-image /var/task/helloworld ./helloworld
 ---> 9a7cea4f8cbd
Step 15/16 : COPY --from=build-image /var/task/lib/ ./lib
 ---> 09a9974d2dec
Step 16/16 : CMD ["helloworld.App::handleRequest"]
 ---> Running in 11e9683c59dc
 ---> c9fbfd205cd4
Successfully built c9fbfd205cd4
Successfully tagged helloworldfunction:java11-gradle-v1

Build Succeeded

Built Artifacts  : .aws-sam/build
Built Template   : .aws-sam/build/template.yaml

Commands you can use next
=========================
[*] Invoke Function: sam local invoke
[*] Deploy: sam deploy --guided

A very nice aspect of the AWS SAM is the ability to run the container locally!

sam local invoke

With the output matching what we would normally see in the Cloudwatch logs:

Invoking Container created from helloworldfunction:java11-gradle-v1
Image was not found.
Building image..........
Skip pulling image and use local one: helloworldfunction:rapid-1.13.2.

START RequestId: 22cd82ae-e042-4863-9447-4831faf9490a Version: $LATEST
END RequestId: 22cd82ae-e042-4863-9447-4831faf9490a
REPORT RequestId: 22cd82ae-e042-4863-9447-4831faf9490a  Init Duration: 1.02 ms  Duration: 1262.58 ms    Billed Duration: 1300 ms    Memory Size: 128 MB Max Memory Used: 128 MB
{"statusCode":200,"headers":{"X-Custom-Header":"application/json","Content-Type":"application/json"},"body":"{ \"message\": \"hello world\", \"location\": \"71.174.101.210\" }"}%

Deployment

Finally, we can have AWS SAM create our Lambda and connect it to a REST API for us!

sam deploy --guided

And we get the image being pushed to the ECR repository along with the Cloudformation stack deploying the REST API and Lambda.

Configuring SAM deploy
======================

    Looking for config file [samconfig.toml] :  Not found

    Setting default arguments for 'sam deploy'
    =========================================
    Stack Name [sam-app]: gizmo-example
    AWS Region [us-east-1]:
    Image Repository []: 196295636944.dkr.ecr.us-east-1.amazonaws.com/gizmo-lambda-container
    Images that will be pushed:
      helloworldfunction:java11-gradle-v1 to 196295636944.dkr.ecr.us-east-1.amazonaws.com/gizmo-lambda-container:helloworldfunction-c9fbfd205cd4-java11-gradle-v1

    #Shows you resources changes to be deployed and require a 'Y' to initiate deploy
    Confirm changes before deploy [y/N]: y
    #SAM needs permission to be able to create roles to connect to the resources in your template
    Allow SAM CLI IAM role creation [Y/n]: Y
    HelloWorldFunction may not have authorization defined, Is this okay? [y/N]: y
    Save arguments to configuration file [Y/n]: Y
    SAM configuration file [samconfig.toml]:
    SAM configuration environment [default]:

    Looking for resources needed for deployment: Not found.
    Creating the required resources...
        Successfully created!

        Managed S3 bucket: aws-sam-cli-managed-default-samclisourcebucket-1owsb6dv8zl8j
        A different default S3 bucket can be set in samconfig.toml

    Saved arguments to config file
    Running 'sam deploy' for future deployments will use the parameters saved above.
    The above parameters can be changed by modifying samconfig.toml
    Learn more about samconfig.toml syntax at
    https://docs.aws.amazon.com/serverless-application-model/latest/developerguide/serverless-sam-cli-config.html
The push refers to repository [196295636944.dkr.ecr.us-east-1.amazonaws.com/gizmo-lambda-container]
0bd0a3c72bc7: Pushed
11bc4c037913: Pushed
e3d7108f2c0b: Pushed
d1d758bc3380: Pushed
577ed33a1f65: Pushed
d6fa53d6caa6: Pushed
016e6d3f9722: Pushed
898f760e15c4: Pushed
af6d16f2417e: Pushed
helloworldfunction-c9fbfd205cd4-java11-gradle-v1: digest: sha256:507124a3f49f85a91a97508c50acc1b66be3731d1cfeb187a245a44cefb5979e size: 2205


    Deploying with following values
    ===============================
    Stack name                   : gizmo-example
    Region                       : us-east-1
    Confirm changeset            : True
    Deployment image repository  : 196295636944.dkr.ecr.us-east-1.amazonaws.com/gizmo-lambda-container
    Deployment s3 bucket         : aws-sam-cli-managed-default-samclisourcebucket-1owsb6dv8zl8j
    Capabilities                 : ["CAPABILITY_IAM"]
    Parameter overrides          : {}
    Signing Profiles           : {}

Initiating deployment
=====================
HelloWorldFunction may not have authorization defined.
Uploading to gizmo-example/40514ed797d174c979e4a5a29672a62a.template  1199 / 1199.0  (100.00%)

Waiting for changeset to be created..

CloudFormation stack changeset
-------------------------------------------------------------------------------------------------------------------------------------
Operation                         LogicalResourceId                 ResourceType                      Replacement
-------------------------------------------------------------------------------------------------------------------------------------
+ Add                             HelloWorldFunctionHelloWorldPer   AWS::Lambda::Permission           N/A
                                  missionProd
+ Add                             HelloWorldFunctionRole            AWS::IAM::Role                    N/A
+ Add                             HelloWorldFunction                AWS::Lambda::Function             N/A
+ Add                             ServerlessRestApiDeployment47fc   AWS::ApiGateway::Deployment       N/A
                                  2d5f9d
+ Add                             ServerlessRestApiProdStage        AWS::ApiGateway::Stage            N/A
+ Add                             ServerlessRestApi                 AWS::ApiGateway::RestApi          N/A
-------------------------------------------------------------------------------------------------------------------------------------

Changeset created successfully. arn:aws:cloudformation:us-east-1:196295636944:changeSet/samcli-deploy1607463634/9467a1b3-b809-4a57-aecb-8638ec49f4ca


Previewing CloudFormation changeset before deployment
======================================================
Deploy this changeset? [y/N]: y

2020-12-08 16:41:12 - Waiting for stack create/update to complete

CloudFormation events from changeset
-------------------------------------------------------------------------------------------------------------------------------------
ResourceStatus                    ResourceType                      LogicalResourceId                 ResourceStatusReason
-------------------------------------------------------------------------------------------------------------------------------------
CREATE_IN_PROGRESS                AWS::IAM::Role                    HelloWorldFunctionRole            -
CREATE_IN_PROGRESS                AWS::IAM::Role                    HelloWorldFunctionRole            Resource creation Initiated
CREATE_COMPLETE                   AWS::IAM::Role                    HelloWorldFunctionRole            -
CREATE_IN_PROGRESS                AWS::Lambda::Function             HelloWorldFunction                -
CREATE_IN_PROGRESS                AWS::Lambda::Function             HelloWorldFunction                Resource creation Initiated
CREATE_COMPLETE                   AWS::Lambda::Function             HelloWorldFunction                -
CREATE_IN_PROGRESS                AWS::ApiGateway::RestApi          ServerlessRestApi                 Resource creation Initiated
CREATE_IN_PROGRESS                AWS::ApiGateway::RestApi          ServerlessRestApi                 -
CREATE_COMPLETE                   AWS::ApiGateway::RestApi          ServerlessRestApi                 -
CREATE_IN_PROGRESS                AWS::Lambda::Permission           HelloWorldFunctionHelloWorldPer   -
                                                                    missionProd
CREATE_IN_PROGRESS                AWS::ApiGateway::Deployment       ServerlessRestApiDeployment47fc   -
                                                                    2d5f9d
CREATE_IN_PROGRESS                AWS::ApiGateway::Deployment       ServerlessRestApiDeployment47fc   Resource creation Initiated
                                                                    2d5f9d
CREATE_IN_PROGRESS                AWS::Lambda::Permission           HelloWorldFunctionHelloWorldPer   Resource creation Initiated
                                                                    missionProd
CREATE_COMPLETE                   AWS::ApiGateway::Deployment       ServerlessRestApiDeployment47fc   -
                                                                    2d5f9d
CREATE_IN_PROGRESS                AWS::ApiGateway::Stage            ServerlessRestApiProdStage        -
CREATE_IN_PROGRESS                AWS::ApiGateway::Stage            ServerlessRestApiProdStage        Resource creation Initiated
CREATE_COMPLETE                   AWS::ApiGateway::Stage            ServerlessRestApiProdStage        -
CREATE_COMPLETE                   AWS::Lambda::Permission           HelloWorldFunctionHelloWorldPer   -
                                                                    missionProd
CREATE_COMPLETE                   AWS::CloudFormation::Stack        gizmo-example                     -
-------------------------------------------------------------------------------------------------------------------------------------

CloudFormation outputs from deployed stack
----------------------------------------------------------------------------------------------------------------------------------------
Outputs
----------------------------------------------------------------------------------------------------------------------------------------
Key                 HelloWorldFunctionIamRole
Description         Implicit IAM Role created for Hello World function
Value               arn:aws:iam::196295636944:role/gizmo-example-HelloWorldFunctionRole-13AV2P47KN805

Key                 HelloWorldApi
Description         API Gateway endpoint URL for Prod stage for Hello World function
Value               https://uwuy5qpvmg.execute-api.us-east-1.amazonaws.com/Prod/hello/

Key                 HelloWorldFunction
Description         Hello World Lambda Function ARN
Value               arn:aws:lambda:us-east-1:196295636944:function:gizmo-example-HelloWorldFunction-LX0IVO85A1CM
----------------------------------------------------------------------------------------------------------------------------------------

Successfully created/updated stack - gizmo-example in us-east-1

We ended up with a REST API endpoint: https://uwuy5qpvmg.execute-api.us-east-1.amazonaws.com/Prod/hello/:

> curl https://uwuy5qpvmg.execute-api.us-east-1.amazonaws.com/Prod/hello/
{ "message": "hello world", "location": "3.239.84.207" }%

If we curl this, we should get the output - note back to the beginning of the blog post, that I often see a long delay in the first request as the lambda is experiencing the cold start penalty.

Quick Project: KubeconVibes

John Doyle — Thu, 27 Aug 2020 21:16:35 +0000

While attending Kubecon EU 2020, I wanted to maximize my time by watching the most popular sessions. Which hopefully would also be the most useful. When the sessions go up on YouTube, I can look at the view counts to judge, but right now there didn't appear to be anything on the website that provides this information. Time to look behind the curtains!

Turned out they have a nice little API running that you can access utilizing your session cookie. They return the sessions based on the tracks, so I was able to note down the URLs for each track:

101 Track
Application + Development
Case Studies
CI/CD
Community
Customizing + Extended Kubernetes
Experiences
Keynotes
Lightning Talks
Machine Learning + Data
Maintainer Track
Networking
Observability
Operations
Performance
Runtimes
Security + Identity + Policy
Serverless
Service Mesh
Storage
Tutorials

Hitting one of these tracks returned a ton of information on each session including the number of likes and views! BINGO!

The only authentication was utilizing my session cookie, but since it was only a quick project, I decided to pass that parameter in during the CI/CD deploy rather than coding up a request... I didn't want to inadvertently checkin my password during testing...

  let kubeConTracks = [
    {name: '101 Track', url: 'https://onlinexperiences.com/scripts/Server.nxp?LASCmd=AI:1;F:LBSEXPORT!JSON&SQLID=14523&EventPackageKey=55064&RandomValue=1597773745984'},
    ...
  ]

  let options = {
    json: true,
    headers: {
      'Cookie': kubeconCookie
    }
  };

  for(let track of kubeConTracks) {
    request(track.url, options, (error, res, body) => {
      let sessionsArray = body.ResultSet[0];
      for (let session of sessionsArray) {
        console.log(session.NumViews.toString());
      }
    }
  }

Using the AWS CDK I was able to quickly turn this into a Cron Job to pull the information and store it in a DynamoDB table.

    const kubeconCookie = this.node.tryGetContext('kubecon_cookie');

    const consumeFunction = new lambda.Function(this, 'KubeConConsume', {
      runtime: lambda.Runtime.NODEJS_10_X,
      handler: 'index.handler',
      code: lambda.Code.asset('./resources/consumer'),
      description: 'Query KubeCon APIs to get the latest data and insert into DynamoDB',
      timeout: core.Duration.seconds(30),
      environment: {
        DYNAMODB_TABLE: table.tableName,
        DYNAMODB_HISTORY_TABLE: historyTable.tableName,
        KUBECON_COOKIE: kubeconCookie
      },
    });

    const lambdaTarget = new eventstargets.LambdaFunction(consumeFunction);

    new events.Rule(this, 'ScheduleRule', {
      schedule: events.Schedule.cron({ minute: '0' }),
      targets: [lambdaTarget],
    });

I setup two tables, one to contain the latest information and another to group it by time - with an idea of maybe seeing how the popularity of sessions changed over the days.

With the data stored, the only thing remaining was slapping a quick API onto my DynamoDB table and throw up a front end website! I was able to structure everything in the single repo which I liked, seperating the lambdas, the front end and the infrastructure pulling everything together.

The AWS CDK has made it a lot easier to setup an API Gateway, certainly compared to building this up in straight Cloudformation.

    const listFunction = new lambda.Function(this, 'KubeConList', {
      runtime: lambda.Runtime.NODEJS_10_X,
      handler: 'index.list',
      code: lambda.Code.asset('./resources/api'),
      description: 'Used by the API Gateway to return the data.',
      environment: {
        DYNAMODB_TABLE: table.tableName
      },
    });

    table.grantReadData(listFunction);

    const api = new apigateway.RestApi(this, "kubeconvides-api", {
      restApiName: "KubeCon Vibes",
      description: "This service to access the kubecon data.",
      defaultCorsPreflightOptions: {
        allowOrigins: apigateway.Cors.ALL_ORIGINS
      }
    });

    const getIntegration = new apigateway.LambdaIntegration(listFunction, {
      requestTemplates: { "application/json": '{ "statusCode": "200" }' }
    });

    api.root.addMethod("GET", getIntegration);

Actually adding a custom domain to the API Gateway is more complicated than setting up the endpoint itself!

    const domainName = 'kubeconvibes.com'

    const zone = route53.HostedZone.fromLookup(this, 'KubeConZone', {
      domainName: domainName
    });

    const cert = new acm.Certificate(this, 'Certificate', {
      domainName: 'api.' + domainName,
      validation: acm.CertificateValidation.fromDns(zone)
    });

    const customDomain = new apigateway.DomainName(this, 'customDomain', {
      domainName: 'api.' + domainName,
      certificate: cert,
      endpointType: apigateway.EndpointType.EDGE
    });

    new apigateway.BasePathMapping(this, 'CustomBasePathMapping', {
      domainName: customDomain,
      restApi: api
    });

    new route53.CnameRecord(this, 'ApiGatewayRecordSet', {
      zone: zone,
      recordName: 'api',
      domainName: customDomain.domainNameAliasDomainName
    });

In short, I enjoy throwing out quick projects - especially when I want to extend my understanding of some functionality. While none of this had anything to do with Kubernetes, I did learn more about CDK and React!

The code for this project is available here: kubecon-eu-popular-sessions

The website is: KubeconVibes

AWS CDK RDS Backup At Silbo

John Doyle — Fri, 21 Aug 2020 16:51:23 +0000

I was reading Trilochn Parida's article on How to Schedule Backup of MySQL DB and Store it in S3 using Cron Job and it got me thinking that I should show how Silbo handles our own database backups. We utilize AWS RDS which can be configured to automate backups of our database, creating snapshots for the last 35 days.

A rolling 35 days of snapshots is great to quickly restore a short term backup. Yet there are a number of limitations with this system. The 35 days would be the major factor from an audit perspective. The automated backups are also not SQL exports that you can directly access - they are usable only within AWS RDS. When you deal with disaster recovery, you need to plan for regions going down. We want to be able to recovery in another region.

To achieve this, I will walk through some AWS CDK code to demonstrate how to schedule a such a backup policy.

The code for this is up on GitHub.

Overview

With AWS Lambda having predefined quotas: timeout limit of 15 minutes, memory limits etc, we know we will want to perform our backup on an Amazon EC2 instance. To reduce our costs we only want this instance to startup and shutdown just for the period of the backup. So we will have a Lambda create an EC2 instance and configure its commands via it's Instance User Data

EC2 User Data

The most important aspect of this work flow is contained within the EC2 Instance User Data. Here we need to install any dependencies we have, and execute the commands, and shutdown the instance afterwards.

For my example here, I created a PostgreSQL AWS RDS instance with the default version at the time of writing being PostgreSQL 11.6-R1. PostgreSQL can perform an export of a database using pg_dump. Our first few commands will be to get this installed on an Amazon Linux 2:

yum install -y wget tar gcc make
wget https://ftp.postgresql.org/pub/source/v11.6/postgresql-11.6.tar.gz
tar -zxvf postgresql-11.6.tar.gz
cd postgresql-11.6/
./configure --without-readline --without-zlib
make
make install

We want to configure the username and password for pg_dump, and PostgreSQL supports the Password File:

echo "hostname:port:database:username:password" > ~/.pgpass
chmod 600 ~/.pgpass

Finally we want to run pg_dump, and upload the output to S3:

/usr/local/pgsql/bin/pg_dump -h hostname \
           -U username \
           -w \
           -c \
           -f output.pgsql \
           databaseName
S3_KEY=S3BucketName/hostname/$(date "+%Y-%m-%d")-databaseName-backup.tar.gz
tar -cvzf output.tar.gz output.pgsql
aws s3 cp output.tar.gz s3://$S3_KEY --sse AES256

This is all basic logic we need here, you could add more commands in, or change this to use pg_dumpall - whatever is needed!

AWS CDK

With the main logic complete, lets utilize the CDK to stand this all up!

Our first step is to create a bucket where our exports will live:

    const rdsBackupsBucket = new s3.Bucket(this, 'rdsBackupsBucket', {
      blockPublicAccess: s3.BlockPublicAccess.BLOCK_ALL,
      encryption: s3.BucketEncryption.S3_MANAGED
    });

This is just a standalone bucket, but you could create replication logic across regions, archiving logic into Glacier etc.

Now that we have an Amazon S3 bucket, we need to think about how the EC2 instance will be able to access it. We will want to ensure that the EC2 has an Instance Profile with the appropriate permissions. This is broken into 3 parts:

Create an IAM Role that can be used by an EC2 instance
Create an IAM Policy for the role
Create an Instance Profile and attach the role

    const backupInstanceRole = new iam.Role(this, 'backupInstanceRole', {
      assumedBy: new iam.ServicePrincipal('ec2.amazonaws.com')
    });

    backupInstanceRole.addToPolicy(
      new iam.PolicyStatement({
        effect: iam.Effect.ALLOW,
        resources: [rdsBackupsBucket.bucketArn + '/*'],
        actions: [            
          's3:PutObject',
          's3:PutObjectAcl'
        ]
      })
    );

    new iam.CfnInstanceProfile(
      this,
      'backupInstanceProfile',
      {
        roles: [
          backupInstanceRole.roleName,
        ],
      }
    );

Now the User Data we reviewed up above was pretty static, hard coded values etc. We want this stack to be more configurable in regards to the database configuration. I'll configure this as parameters within AWS CDK and pass them as environment variables to the AWS Lambda for the purpose of this simple setup.

<IMPORTANT SEGUE>

When dealing with actual environments, these values should be stored in AWS Secrets Manager. We would then pass the secret name as a parameter via the AWS CDK and AWS Lambda, down into the User Data for the EC2 Instance. Granting the EC2 Instance Profile access to the secrets, it would retrieve them directly and we wouldn't have username/passwords bouncing around.

</IMPORTANT SEGUE>

Back to the CDK! With our variables being passed in via the CDK Context, we will want to retrieve them an set them as environment variables for the lambda. Also, important, we will need to grant the Lambda the required permissions to launch an EC2 instance!

    const ec2_region = this.node.tryGetContext('ec2_region');
    const ec2_type = this.node.tryGetContext('ec2_type');
    const db_host = this.node.tryGetContext('db_host');
    const db_user = this.node.tryGetContext('db_user');
    const db_pass = this.node.tryGetContext('db_pass');
    const db_database = this.node.tryGetContext('db_database');

    const launchingLambda = new lambda.Function(this, 'Lambda', {
      runtime: lambda.Runtime.PYTHON_3_7,
      handler: 'function.lambda_to_ec2',
      code: lambda.Code.asset('./resources'),
      description: 'Backup Database to S3',
      timeout: core.Duration.seconds(30),
      environment: {
        INSTANCE_REGION: ec2_region,
        INSTANCE_TYPE: ec2_type,
        INSTANCE_ROLE: backupInstanceRole.roleName,
        DB_HOST: db_host,
        DB_USER: db_user,
        DB_PASS: db_pass,
        DB_DATABASE: db_database,
        S3_BUCKET: rdsBackupsBucket.bucketName
      }
    });

    launchingLambda.addToRolePolicy(
      new iam.PolicyStatement({
        effect: iam.Effect.ALLOW,
        resources: ['*'],
        actions: ['ec2:*']
      })
    );

    launchingLambda.addToRolePolicy(
      new iam.PolicyStatement({
        effect: iam.Effect.ALLOW,
        resources: [backupInstanceRole.roleArn],
        actions: ['iam:PassRole']
      })
    );

Finally we want to schedule our lambda. In a real environment, a lot of consideration will need to go into the frequency of your updates. What level of reliability do you require? Have you worked out your Disaster Recovery strategy and defined your RTOs (Recovery Point Objective)?

Again, to make this configurable, lets set the cron settings via our CDK Context, pulling out the required values and passing them into the Cloudwatch Event Rule:

    const lambdaTarget = new eventstargets.LambdaFunction(launchingLambda);

    const cron_minute = this.node.tryGetContext('cron_minute');
    const cron_hour = this.node.tryGetContext('cron_hour');
    const cron_day = this.node.tryGetContext('cron_day');
    const cron_month = this.node.tryGetContext('cron_month');
    const cron_year = this.node.tryGetContext('cron_year');

    new events.Rule(this, 'ScheduleRule', {
      schedule: events.Schedule.cron({ 
        minute: cron_minute,
        hour: cron_hour,
        day: cron_day,
        month: cron_month,
        year: cron_year
      }),
      targets: [lambdaTarget],
    });

We can then configure all these settings within cdk.json:

{
  "app": "npx ts-node bin/aws-rds-nightly-backup.ts",
  "context": {
    "@aws-cdk/core:enableStackNameDuplicates": "true",
    "aws-cdk:enableDiffNoFail": "true",
    "@aws-cdk/core:stackRelativeExports": "true",
    "ec2_region": "us-east-1",
    "ec2_type": "t2.large",
    "db_host": "database-1.ct5iuxjgrvl6.us-east-1.rds.amazonaws.com",
    "db_user": "postgres",
    "db_pass": "exampledb",
    "db_database": "testdb",
    "cron_minute": "0",
    "cron_hour": "8",
    "cron_day": "1",
    "cron_month": "*",
    "cron_year": "*"
  }
}

Enjoy some happy backups!

KubeCon EU Day 1

John Doyle — Thu, 20 Aug 2020 01:39:40 +0000

I really enjoyed Day 1 of KubeCon + CloudNativeCon Europe 2020 - Virtual!

Having a convention go fully virtual gives it a very different feel to any convention I'd previously attended - though this is my first KubeCon that I'm attending! That said, KubeCon & CNCF have always been wonderful in putting all their sessions up on YouTube after the convention.

The virtual aspect was very interesting. The sessions themselves had a Q&A feature, which was all fine, though I really liked encouraging folks to chat with the presenters in the CNCF Slack. I would rarely ask questions at a convention, or network with folks, so this felt far more accessible - jump in when you want, lurk the rest of the time! Now if I only could get all the usual swag from the expo hall ;)

Attended Sessions

KubeCon + CloudNativeCon 101: A Beginner's Guide to The Conference

This was the first talk I checked out, presented by Karen Chu and Michelle Noorali. This was far better than I expected as there was a lot of terminology that I wasn't aware of - especially around CNCF as there was sessions aimed at SIGs (Special Interest Group) and TOCs (Technical Oversight Group). Details about the Expo hall, and introducing me to the Hallway Track - which sounded like Lighting Talks but more general? The talk itself also was an interesting combo of pre-recording and then switching over to live for the Q&A!

From Minikube to Production, Never Miss a Step in Getting Your K8s Ready

Heh, this was an interesting talk, but as I've only started on my Kubernetes journey, it became apparent that this was a bit beyond my experience. Though I definitely could relate as Horacio Gonzalez talked about testing Kubernetes locally with Minikube etc, that your laptop will be slow... its going to get hot... but, its going to work :)

It was great to see Horacio and Kevin Georges talk through all the steps to go from MiniKube to Production.

Locally Running
Security Team sets up ingress rules
App breaks, and you figure out a solution
Iterate over ..and over.. and over..

This quickly got DEEP into the networking complexity! Book marked it to return when I will hopefully have a better idea of whats going on...

Tutorial: Hands-On Intro to Cloud-Native CI/CD with Tekton

Welp, double down on Minikube and this time with a CD/CD pipeline with Tekton presented by Joel Lord and Jan Kleinert. I tend to like understanding a pipeline so that I can understand where products come in or patterns can be used. It was pretty straight forward with the pipelines - there are steps, with inputs and outputs. This talk came with actual code, which was great! Let me read the code while the talk is in the background.

I was hoping to learn more about performing the actual deploys, though this really focused on building and pushing the images. The Tekton Hub has a ton of resources with shared tasks and pipelines, which will be really helpful!

Overview

A solid start to the first day :) Will have to check out more of expo hall, review notes some more etc.. but, I'm enjoying myself!