loading...

A brief summary of our AWS productivity tools

ljacobsson profile image Lars Jacobsson ・5 min read

Table of content

  • Background - What brought us here?
  • cfn-diagram - CloudFormation visualizer
  • evb-cli - EventBridge pattern generator and debugging suite
  • cfn-resource-actions - VS Code extension that lets you interact with deployed resources from the template
  • sam-policies-cli - CLI UI for browsing and injecting SAM Policy Templates
  • iam-policies-cli - CLI UI for building complex IAM policies
  • cwlogs-cli - Tool to quickly launch CloudWatch Logs Insights with multiple log groups pre-selected

A background

When we in January 2017 made the decision to rewrite Sweden's leading online grocery store, MatHem.se, from scratch into a serverless microservices architecture we found ourselves in an extremely promising yet premature tech segment with a lack of tooling due to a yet quite small community.

This led us to adopting a mindset of 'when tasks get repetitive, find a tool for it or write one yourself'.

Much of our tooling solves bespoke in-house procedural bottlenecks, but when we find solutions to tasks that can benefit the community we make sure to publish it in the open.

This post is a summary of the AWS/serverless related productivity tools we have authored.

cfn-diagram

NPM: https://www.npmjs.com/package/@mhlabs/cfn-diagram
GitHub: https://github.com/mhlabs/cfn-diagram

Installation
npm i -g @mhlabs/cfn-diagram

Features
Parses CloudFormation/SAM JSON or YAML and renders a diagram in either draw.io or vis.js network format.

Examples
Cfn-diagram-drawio

Cfn-diagram-html

Reason for existing
Our application is made up of hundreds of microservices, each defined with its own CloudFormation/SAM template. No engineer knows about every stack. Our engineering team has been going through a rapid growth phase which has required onboarding of new developers to be efficient.

To ease the getting to know a stack, we had an initiative that every repository was required to have an up-to-date diagram over the service's resources in the readme file. No matter how good the intentions were, those sort of 'musts' will never be followed and these diagram will be something that 'might be true' and therefore never trusted.

With cfn-diagram any developer new to a stack can quickly pull up a visualisation of how it all hangs together. It's also useful to render in planning meetings instead of drawing on the whiteboard

evb-cli

NPM: https://www.npmjs.com/package/@mhlabs/evb-cli
GitHub: https://github.com/mhlabs/evb-cli

Installation
npm i -g @mhlabs/evb-cli

Features
Pattern generator and debugging tool for EventBridge with features including:

  • Event pattern builder
  • Input transformation builder
  • Event rules browser
  • Generate a diagram over the events flow in a region
  • Local debugging of events where actual events are sent to the console over websockets. These can be forwarded into sam-local for some advanced debugging.

Examples
Demo1
Pattern generation

Demo2
Events flow diagram generator

Demo2

Local debugging

Reason for existing
Historically we have been heavily invested in pub/sub using SNS/SQS, but we were always annoyed with the tight coupling this created between services and teams, so when EventBridge was announced we got very excited.

One of the most powerful features of EventBridge is its content based filtering where you only let through events matching a part of the payload that interests the consumer of the event. Together with EventBridge's Schema Registry an engineer can build event patterns without knowing anything about the producing service.

These event patterns often get quite complex and we found ourselves spending a lot of time first writing, then deploying, then debugging them. Typos were a common mistake both when composing event patterns and input transformations.

Our developers also reported that they find it cumbersome to debug their rules as well as finding it difficult to see the whole picture of how all events hang together, so from that feedback we added local debugging features

cfn-resource-actions

Marketplace: https://marketplace.visualstudio.com/items?itemName=ljacobsson.cfn-resource-actions
GitHub: https://github.com/mhlabs/cfn-resource-actions

Installation
ext install ljacobsson.cfn-resource-actions

Features
Interact with your deployed CloudFormation/SAM templates directly from the template. Turns your template in to an interface into the AWS console

  • Tail lambda log
  • Query DynamoDB tables
  • Copy Physical Id of any resource
  • Consume EventBridge events directly in vs code
  • Send to and Poll SQS
  • Visualise template
  • etc

Examples

Demo1

Invoke lambda + tail its logs

Demo2

CTRL+click from template to Lambda handler

Demo3

Query DynamoDB

Demo4

Visualise stack requires Draw.io Integration extension

Reason for existing
Context switching is a huge cost in productivity loss. Leaving VS Code to go and do stuff in the AWS console is a distraction we want to avoid

sam-policies-cli

NPM: https://www.npmjs.com/package/@mhlabs/sam-policies-cli
GitHub: https://github.com/mhlabs/sam-policies-cli

Installation
npm i -g @mhlabs/sam-policies-cli

Features
CLI UI to quickly inject SAM policy templates into your AWS::Serverless::Function resources

Example

Demo

Reason for existing
We strive to follow the Principle of Least Privilege and finding the correct policy templates requires googling

iam-policies-cli

NPM: https://www.npmjs.com/package/@mhlabs/iam-policies-cli
GitHub: https://github.com/mhlabs/iam-policies-cli

Installation
npm i -g @mhlabs/iam-policies-cli

Features
UI to quickly build simple to complex IAM policies based on resources in a CloudFormation template

Example

Demo

Reason for existing
We strive to follow the Principle of Least Privilege and composing granular IAM policies is time consuming and prone to human errors

cwlogs-cli

NPM: https://www.npmjs.com/package/@mhlabs/cwlogs-cli
GitHub: https://github.com/mhlabs/cwlogs-cli

Installation
npm i -g @mhlabs/cwlogs-cli

Features
Create groups of log groups from resource tags or log group prefixes and launch CloudWatch Logs Insights with those groups predefined

Example

Demo

Reason for existing
CloudWatch Insights Logs is a great way to search your logs. Often you want to search across many log groups to, for example, tracing a correlation id across multiple services. CloudWatch Logs Insights lets you search across up to 20 log groups in one query, but adding log groups from the dropdown is a bit cumbersome.

Discussion

markdown guide