DEV Community: Charlie Thomas

Lifeway's Event Driven Architecture

Charlie Thomas — Tue, 02 Mar 2021 00:00:00 +0000

Back in 2016, Lifeway took on the HUGE challenge of re-architecting it's entire backend eCommerce platform. We went from a monolithic, WebSphere-based system to an event-driven, microservices approach with Apache Kafka and Kubernetes as the backbone of our enterprise. The path to get there wasn't perfect and we made many mistakes in the process, but the end result has set our development teams up for success in the long haul. In this post, we'll take a retrospective look at the architecture we implemented, the challenges we faced during implementation, how it has affected our development teams today, and what Lifeway is prepping for in the future.

The Architecture

At its core, this initiative was about:

Resiliency

By leveraging asynchronous events for integrations between business domains, we projected that the overall resiliency of our platforms would be significantly improved.

Scalability

By embracing the cloud and migrating off of our own data center, we expected to be able to scale to business needs and our customer volume more flexibly.

Elimination of Tight Coupling

By splitting capabilities out into microservices, the expectation was that business domains would no longer be tightly coupled - limiting impact to customers in production incidents and allowing for more graceful fault tolerance.

Evolutionary Architecture

With common interfaces and design paradigms in place, we expected to be able to easily iterate on top of this architecture. Any new large scale feature should be able to be distilled down into its smallest units as Events, and implemented with agility and minimal disruption.

As pictured, we separated previously coupled domains of our business, hosted all on one centralized infrastructure piece, into logical components that are separately managed. Kafka was then introduced as an asynchronous way for these domains to communicate. With the introduction of Kafka, we established an "Enterprise Event Envelope". In essence, this is a contract that all applications had to adhere to when publishing and consuming events to/from other domains.

{
  "id": "4118ca5c-2d28-4ebd-90ca-66eb2ede7314",
  "timestamp": "2021-02-27T16:31:23.885Z",
  "version": 1,
  "payload": {
    "id": "b9b1b8eb-a9a5-4880-a444-fd26a8cda03e",
    "dateCreated": "2021-02-27T16:31:23.885Z",
    "dateUpdated": "2021-02-27T16:31:23.885Z",
    "firstName": "Leeroy",
    "lastName": "Jenkins",
    "gender": "MALE"
  },
  "eventType": "PersonCreated",
  "domain": "Customer"
}

The example event above demonstrates this event envelope. Using this, applications have a standard way to communicate with necessary routing information at the header level and application-specific data at the payload level.

The most important aspect of these "events" is the eventType and the payload. With this, we can adequately communicate when something in our business process has occurred so that other systems can consume and process based on its domain context. In the example above we have a PersonCreated event indicating that a new customer was created either through registration or other means.

Within this paradigm, all of our solutions have a standard way to propagate information out to the Enterprise.

The Struggle

Unsurprisingly, goals as lofty as this came with some pretty serious growing pains. Most notably these included:

Complexity of Microservices

When we broke out our capabilities into individual components, naturally the complexity of our systems and infrastructure grew immensely. In the early stages, we struggled with build pipelines, deployment strategies, and sometimes even struggled to adequately debug issues. As time went on, we used things like K8s, log aggregators, and re-usable pipelines to help simplify this. We've also had to stress the importance of internal development documentation.

Latency

As we introduced more components, redundant API Gateway layers, and features, we noticed that our original architecture introduced unnecessary latency for the end-user. As we learned this, we have slowly been refactoring domains to eliminate these redundant API Gateway layers and reworking features to reduce latency where possible.

Datacenter Dependencies

While we moved the bulk of our enterprise off of a monolithic architecture, we still maintained the fulfillment of our orders, accounting, and general ledger through our legacy Oracle implementation. As a result, we had a networking dependency on our cloud infrastructure being able to connect our CoLo Datacenter via AWS Direct Connect. We mitigated this by isolating the connected components with Kafka, their specific failover routines, and by adding redundancy to the direct connect implementation.

The Gains

As we ironed out these growing pains, these architecture decisions have had a drastic impact on culture and capabilities for our development teams, and not just in how it has improved our systems. We have a homogenous interface for all solutions which enables them to operate independently (not restricting or limiting tooling) and increase speed according to their team's strengths. For example, we have teams using the JVM (Scala, Kotlin, and Java), while other teams are using Node.js, Typescript, and Javascript exclusively. The modularity of this architecture encourages diversity amongst our Dev teams and has resulted in some remarkable innovation.

In addition to this, we have incredible flexibility as our business at Lifeway is looking to evolve to support a more digital church. Some examples include:

The Future

We have laid a great foundation for future development; so it's reasonable to ask: What is Lifeway looking to improve architecturally over the next 2 years? We never want to stop innovating, never want to stop refactoring, and always want to have our finger on the pulse of emerging technologies and architectures at Lifeway.

With this in mind, Lifeway is in the process of evaluating the use of Headless Content Management at Enterprise scale as we have identified more and more large scale e-commerce companies trending in this direction in concordance with the MACH Alliance movement. We have the platform necessary from an e-commerce experience, now we need to shore up the way our content and customer journeys are delivered on the front end.

Spicing Up Your Serverless App Using Curried Functions

Charlie Thomas — Wed, 24 Feb 2021 00:00:00 +0000

I really, really love red curry. You can ask my wife, or inspect my DoorDash order history... it's truly a bit excessive. I also love the functional programming technique called "Currying" too, which isn't quite as expensive 😬 . In this post, we'll specifically be exploring how we can leverage this technique to simplify the way we build Node.js applications with Serverless Framework.

So what exactly is currying?

To get started, let's unpack this technique a little.

Currying is a transformation of functions that translates a function from callable as f(a, b, c) into callable as f(a)(b)(c).

Quote taken from javascript.info

Essentially, it's the practice of "unwinding" the arguments of a function by splitting each argument into a composable higher order function (or a function that returns a function). Let's take a look at a contrived example:

const boringOldAdd = (x, y) => {
  console.log('🥱');
  return x + y;
};

const spicyAdd = (x) => (y) => {
  console.log('🌶️');
  return x + y;
};

Here we have two functions boringOldAdd and spicyAdd that look very similar at first glance, but there is a stark difference in how they are invoked:

const boringResult = boringOldAdd(1, 2);

const spicyResult = spicyAdd(1)(2);

Both functions return the exact same result, but the invocation signature is quite different. Now that the syntax is defined and we've implemented it, it may not be entirely clear how this is actually useful and not just some silly syntax. The key is composability.

It's all about composability!

The driving reason for using this technique in practice is for composition. Building quality software is all about a balance of clean, reusable (or composable) capabilities that can be combined to bootstrap your business processes. In essence, you want to take your application and break it down into small reusable functions that can be used to make more complex functions. For those that may be familiar with Object Oriented Programming, you could draw a correlation between composability and inheritence in that they both strive to abstract capabilities in a way that could be reused in different contexts.

Let's break all this down using our contrived example from earlier. What if we wanted to add 1 to every value in a list using our functions?

const list = [1,2,3,4,5];

// 🥱
const boringList = list.map(n => boringAdd(n, 1));

// 🌶️
const add1 = spicyAdd(1);
const spicyList = list.map(add1);

Here we start to lean into the upside our curried function offers over the normal version. The curried function results in a way to compose together capabilities in a more modular way. Again, this is a very contrived example and you would never use curried functions for something so simple, but with everything defined... let's dig into how to use this somewhere a bit more impactful.

Basic usage with Amazon SQS and Lambda!

When writing functions for your serverless application, there are common tasks that you have to do depending on what vendor you have selected for hosting. For AWS, some of these include:

Serializing SQS bodies and their json messages.
Decoding Kinesis messages from base64 into utf-8.
Extracting path parameters, http headers, or http bodies.

A very basic usage of curried functions could be to extract these vendor specific contracts into a curried function that then passes only the data you need to a business logic function (or your domain). Here is a quick example of doing this for Amazon SQS messages.

const SQSHandler = (businessLogicFn) => async (event) => {
  for (const record of event.Records) {
    const body = JSON.parse(record.body)
    const message = JSON.parse(body.Message)
    await businessLogicFn(message)
  }
}

Now, any time we need write a new SQS handler... we don't have to think about the SQS event contract! We just need to worry about the payload containing data relevant to our system. Generating a function that Serverless Framework could use now looks something like this:

import myBusinessLogic from './my-logic.js';

export const myHandler = SQSHandler(myBusinessLogic); // 🌶️

And the correlating serverless.yml entry...

functions:
  myHandler:
    handler: handlers.myHandler
    events:
      - sqs:
          arn: # SQS Queue

And voila! You now have a composable solution to abstracting the AWS SQS event contract from your business logic that can be used for all of your future handlers. To improve upon this you could:

Add default error handling!
Extract SQS Message Attributes or fields outside of the message body!
Add some debugging utilities like message logging!

But let's not stop here! We can take this a step further and create a framework for middleware around HTTP API handlers with a more advanced application of this technique.

Advanced usage with HTTP API and Lambda!

Expanding on what we did in the SQS Handler example, let's create some reusable functions that can abstract all of our default behaviors away as composable "middleware" functions that adapt the incoming requests and decorate the response from our core business logic.

export const buildMiddleware = (...middleware) => 
  input => middleware.reduce(
    (next, current) => current(next)
  , input)

export const buildHttpHandler = 
  (requestMiddleware, responseMiddleware) => 
    (handler) => async (event) => {
      return Promise
        .resolve(event)
        .then(requestMiddleware)
        .then(handler)
        .then(responseMiddleware)
    }

In the snippet above, we export two different functions. The first of the two, buildMiddleware, takes a list of middleware functions and returns a "reducer" responsible for resolving all potential middleware functions into a final result. This will be used to build both our request middleware layer and response middleware layer. The second export, buildHttpHandler, takes two middleware arguments and return an http handler builder (just like our SQS example above).

import myBusinessLogic from './my-logic.js';
import {
  buildMiddleware,
  buildHttpHandler
} from './http-utils.js';

const requestLogger = (req) => {
  console.log(req);
  return req;
}

const responseLogger = (res) => {
  console.log(res);
  return res;
}

const requestMiddleware = buildMiddleware(requestLogger);
const responseMiddleware = buildMiddleware(responseLogger);

const HttpHandler = buildHttpHandler(
  requestMiddleware,
  responseMiddleware
);

export const myHandler = HttpHandler(myBusinessLogic);

When we combine all of these principals together, we get a composable solution to building our http handlers with utilities that can abstract common system level tasks and enable us to focus more on the business logic. In the example above, we simply use it to add request and response logging to our functions, but you could expand on this to do things like response header decoration or generic request validation.

In Summary

The functional programming technique known as function currying is a very powerful tool for any type of javascript developer to have in their tool belt (backend or frontend).
Using curried functions can, specifically, be used to abstract common code used to build Serverless Applications!
Curry is just plain delicious.

References

Keeping your Node.js projects secure with CI/CD

Charlie Thomas — Thu, 18 Feb 2021 00:00:00 +0000

Lots of times we as developers like to pretend that all the code we are writing is secure by nature. After all, who wants to admit that the product they are crafting is guaranteed to have vulnerabilities given enough time? At the same time, security practices often can work against the developer's overall tooling experience and agility, so we find ourselves in a never ending battle between practicality and crafting ethical and secure software. The bottom line is: security is a discipline that requires us as developers to change our processes at their core so that we can develop quickly and sleep soundly at night.

When in doubt, automate it!

The answer is obvious isn't it? Make the bots do it! Any time a development team is faced with a task that requires significant, recurring work the best approach is to start by asking, "How can I automate this task?".

Unfortunately, a bot isn't going to design secure applications for you (...not yet at least 😬 ), but what can it do for you? To answer this, lets look at what the npm CLI can do for us out of the box.

$ npm audit --audit-level=moderate
$ npm outdated

With these simple npm commands, we have a the basic building block we need to fail our build process when a security vulnerability is found in the dependency list and notify us when our packages are falling behind their recommended versions.

Block the build!

So let's take these CLI tools and put them to practice. For this guide, I will be using GitHub Actions to implement a very basic application of these to help get you bootstrapped.

The general idea is to create an automation job that will run on any pull request to master. First up, lets block the build upon a failing audit.

name: Run Checks

on:
  pull_request:
    branches: [ master ]

jobs:
  audit:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v2
      - name: Setup Node
        uses: actions/setup-node@v1
        with:
          node-version: '14.x'
      - name: CI Install
        run: npm ci
      - name: Audit
        run: npm audit --audit-level=moderate

This workflow will spin up and run the audit command discussed earlier, but to force your team to actually honor this check we need to change some of the configurations in GitHub as seen below:

Try and Keep Up!

Next, lets upgrade our previously defined workflow to include outdated notifications. The goal would be that this does not block incoming pull requests, but rather notifies the team about outdated dependencies via a slack web-hook. Alternatively, you could use something like Dependabot but we'll stick with actions just to demonstrate the basic concept.

name: Run Checks

on:
  pull_request:
    branches: [ master ]
  schedule:
    - cron: '0 17 * * 5' # every friday at 5 PM 😈

jobs:
  check_versions:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v2
      - run: npm ci
      - uses: MeilCli/npm-update-check-action@v4
        id: outdated
      - uses: 8398a7/action-slack@v2
        if: steps.outdated.outputs.has_npm_update != 'false'
        with:
          status: ${{ job.status }}
          text: ${{ steps.outdated.outputs.npm_update_text }}
          author_name: GitHub Actions
        env:
          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
          SLACK_WEBHOOK_URL: ${{ secrets.SLACK_WEBHOOK_URL }}

Rather than rolling our own script that leverages npm outdated, we can use one of the many community maintained actions that are out there! In this example we will use npm-update-check-action. As you can see in the code snippet above, we simply need to pull in the action and pass its output to one of the many slack web-hook actions out there to notify the team if an outdated package is detected.

To improve upon this, some next steps you could experiment with are:

Opening a github issue to track the status of how the team chooses to handle the outdated package.
Implement Dependabot
Have the action comment if this action was triggered via PR.

In Summary

Security is a discipline that should effect the process of application development, packaging artifacts, and deploying.
Developers should strive for automated ways to hold their teams accountable to fixing outdated packages and security vulnerabilities.
With the npm CLI, we have many ways to solve vulnerability and outdated package auditing that we can leverage in our pipelines.

References

Automating your Cloud Infrastructure with Github Actions

Charlie Thomas — Mon, 15 Feb 2021 00:00:00 +0000

Managing cloud resources for a production application can be a major pain. To help teams manage resources, AWS provides the Infrastructure as Code solution Cloudformation, but sometimes automating workflows with cloudformation can be painful in and of itself. Luckily, Github actions and Amazon's out of the box actions kit can make your life much easier!

In this guide we will walk through a series of steps to get a simple pipeline up and running.

Step 1: Define your Cloudformation Stack!

For this example, we will define a simple Cloudformation Stack containing a single dynamo table.

AWSTemplateFormatVersion: '2010-09-09'
Description: My Cloud Resource

Parameters:
  Environment:
    Type: String
    Default: stage

Resources:
  SimpleTable:
    Type: 'AWS::DynamoDB::Table'
    Properties:
      BillingMode: PAY_PER_REQUEST
      TableName: !Sub ${Environment}-tablename
      AttributeDefinitions:
        - AttributeName: id
          AttributeType: S        
      KeySchema:
        - AttributeName: id
          KeyType: HASH

Notice that we have a parameter in the template defined for environment. You can use this if your pipeline needs to deploy to multiple environments.

Step 2: Add permissions keys to your infrastructure github repo!

Next up, you'll want to create some IAM permissions for Github actions to use. As a secure approach, I recommend you create a dedicated user and role for your pipeline to assume when running. In this guide we will define it all in a reusable template to make this a bit future-proof.

Start by defining a user with correlating access keys that we can reference later.

Resources:
  User:
    Type: AWS::IAM::User
    Properties:
      UserName: github-actions-user
  AccessKey:
    Type: AWS::IAM::AccessKey
    Properties:
      UserName: github-actions-user
      Serial: 1
  Credentials:
    Type: AWS::SecretsManager::Secret
    Properties:
      Name: github-actions-user
      SecretString: !Sub |
        {
          "AccessKeyId":"${AccessKey}",
          "SecretAccessKey":"${AccessKey.SecretAccessKey}"
        }

Once you have a the User defined, create a Role that we will allow this User to assume, giving it the permissions it needs to manage the resources in our infrastructure templates.

Resources:
  # ... previous User resources from above
  Role:
    Type: AWS::IAM::Role
    Properties:
      RoleName: github-actions-deployer-role
      AssumeRolePolicyDocument:
        Version: "2012-10-17"
        Statement:
          - Effect: "Allow"
            Principal:
              Service:
                - cloudformation.amazonaws.com
            Action:
              - "sts:AssumeRole"
      ManagedPolicyArns:
        - arn:aws:iam::aws:policy/AmazonDynamoDBFullAccess
        # Any other resources you need to manage should be listed here

Notice that we are giving exact permissions for the deployer to only manage DynamoDB resources in this case. You may even want to restrict it even tighter with your own more specific policies.

Finally, tie the User and Role together with a UserPolicy that allows the User to manage Cloudformation stacks and pass our defined Role to Cloudformation.

Resources:
  # ... previous User and Role resources from above
  UserPolicy:
    Type: AWS::IAM::Policy
    Properties:
      Users:
        - github-actions-user
      PolicyName: github-cloudformation-deploy
      PolicyDocument:
        Version: "2012-10-17"
        Statement:
          - Action:
              - "cloudformation:*"
            Effect: Allow
            Resource: "*"
          - Action: "iam:PassRole"
            Effect: Allow
            Resource: !GetAtt Role.Arn

Once you have all of your IAM written you can manually deploy this with the AWS CLI and subsequently fetch the access credentials.

aws cloudformation deploy \
  --stack-name github-actions-cloudformation-deployer \
  --template-file path/to/your/yaml \
  --capabilities CAPABILITY_NAMED_IAM \
  --region us-east-1

aws secretsmanager get-secret-value \
  --secret-id github-actions-user \
  --region us-east-1 \
  --query SecretString \
  --output text

Then take the output credentials and set up 2 secrets in your github repository under Setting > Secrets > Actions Secrets.

Step 3: Create the pipeline in Github!

Finally, now that you have all the permissions set up and a Cloudformation template ready to deploy, create a Github actions workflow in the .github/workflows directory of your project.

on:
  push:
    branches:
      - master

name: Deploy Cloudformation

jobs:
  deploy-cfn:
    name: Deploy Cloudformation Template
    runs-on: ubuntu-latest
    steps:
    - uses: actions/checkout@v2
    - name: Configure AWS credentials
      uses: aws-actions/configure-aws-credentials@v1
      with:
        aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
        aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
        aws-region: "us-east-1"
    - name: Execute Changeset
      uses: aws-actions/aws-cloudformation-github-deploy@v1
      with:
        name: my-cloudformation-stack
        template: path/to/my/yaml
        role-arn: arn:aws:iam::${{ steps.creds.outputs.aws-account-id }}:role/github-actions-deployer-role
        no-fail-on-empty-changeset: "1"
        parameter-overrides: >-
          Environment=dev

Using the aws-actions actions kit allows us to assume the proper IAM roles and the execute cloudformation changes with ease. Make sure to point to the right yaml file and pass all your parameters that you defined on the stack from Step 1 (in this example we just override the Environment param). Once in place, every commit to the repo will result in your cloudformation template being deployed automatically to the correlating AWS account. From here, you can expand on these workflows to support more advanced features.

Such as:

Multi-stack deployments with Github Actions Matrices
Multi-environment workflows with Github Actions Environments
Change-set linting to get in front of yaml syntax errors and invalid changes.