DEV Community: Joseph Sutton

Polyglot Microservices: Federated GraphQL Subscriptions in Golang, Rust, and Nest.js, Pt. 1

Joseph Sutton — Thu, 06 Mar 2025 19:15:02 +0000

I haven't written a post in a long time, but I've been busy. In the past 2 years, I've gotten married and had twin boys. It's been a wild rollercoaster, but I haven't forgotten about learning and sharing what I've learned. I have a lot of drafted articles that I need to get around to finishing, but alas, here we are.

Anyhow, let's get into it. Federated GraphQL Subscriptions. My day-to-day shifts between Node.js, Java, and Rust (although still very new to Rust after 2 years, lol), so my Golang here won't be up to snuff; however, I'm learning Golang more recently and have been taking it semi-seriously in the past week or so.

Moving on... So, we're going to use use Nx because I'm going to do this in one repository. It'll be fine. its-fine.jpg

We're going to have subscriptions in all the microservices. I'm not going to plan or design out anything, just gonna go with it. But the idea is that we're gonna make the Golang service a "spells" subgraph, the Nest.js service a "players" subgraph, and the Rust service will be a "messages" subgraph so players can talk to each other.

We can tie this together on the frontend -- I've never used Svelte before, so let's do that. I want this project to be all about stepping outside of our comfort zone and usual tooling (at least for me).

I have a feeling this will be more work than anticipated.

Let's get into it.

Note: If you're like me and just want to see the code, check it out here: https://github.com/sutt0n/polyglot-fed-gql

Series Navigation

This article will first start off with bootstrapping our monorepo with Nx and building up the first microservice in Golang.

Let's first initialize the monorepo with Nx:

pnpx create-nx-workspace

Spells Subgraph, Schema-first (Golang)

Now, let's create the Golang subgraph.

nx add @nx-go/nx-go
nx g @nx-go/nx-go:application spell-service

We're going to use GQLGen for this. Head to their getting started page, and follow along. We'll create our schema based off of this, run the codegen, and create our necessary resolvers.

We'll need a few things:

Query for a player's spellbook
Mutation for casting spells at a player
Subscription for listening to spells being cast (or cast at us)

Let's write our schema:

extend type Player @key(fields: "id") {
  id: ID! @external
}

enum DamageType {
  FIRE
  ICE
  LIGHTNING
  POISON
  PHYSICAL
}

type CastedSpell {
  spell: String!
  type: DamageType!
  playerId: ID!
  damage: Float!
}

type Mutation {
  castSpell(spell: String!, type: DamageType!, playerId: ID!): Boolean
}

type Subscription {
  spellsCasted(target: String!): CastedSpell
}

type Query {
  spellBook(playerId: ID!): [String]
}

Here's our gqlgen.yml file:

schema:
  - schema/**/*.graphql
exec:
  package: graph
  layout: single-file
  filename: graph/generated.go
federation:
  filename: graph/federation.go
  package: graph
  version: 2
model:
  filename: graph/model/models_gen.go
  package: model
resolver:
  package: graph
  layout: follow-schema
  dir: graph
  filename_template: '{name}.resolvers.go'
call_argument_directives_with_null: true
autobind:
models:
  ID:
    model:
      - github.com/99designs/gqlgen/graphql.ID
      - github.com/99designs/gqlgen/graphql.Int
      - github.com/99designs/gqlgen/graphql.Int64
      - github.com/99designs/gqlgen/graphql.Int32
  UUID:
    model:
      - github.com/99designs/gqlgen/graphql.UUID
  Int:
    model:
      - github.com/99designs/gqlgen/graphql.Int32
  Int64:
    model:
      - github.com/99designs/gqlgen/graphql.Int
      - github.com/99designs/gqlgen/graphql.Int64

Let's run codegen (within the spell-service directory; note that we could add a command for this specific Nx project):

go run github.com/99designs/gqlgen generate

This will generate some files for us. Let's update our graph/resolver.go file to define some subscription-related things. We'll need a channel map for the websocket connections for subscriptions, casted spells, and a mutex for locking:

// graph/resolver.go
type Resolver struct{
  CastedSpells []*model.CastedSpell
  SpellObservers map[string]chan *model.CastedSpell
  mu sync.Mutex
}

var letterRunes = []rune("abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ")

func randString(n int) string {
    b := make([]rune, n)
    for i := range b {
        b[i] = letterRunes[rand.Intn(len(letterRunes))]
    }
    return string(b)
}

Okay, now for our resolvers. We need one for the spellBook query for the player (which we're going to return base spells), a mutation for casting spell at a player, and the subscription for listening to spells casted at a player provided:

// graph/schema.resolvers.go
package graph

import (
    "apps/go-service/graph/model"
    "context"
    "math/rand"
)

// CastSpell is the resolver for the castSpell field.
func (r *mutationResolver) CastSpell(ctx context.Context, spell string, typeArg model.DamageType, playerID string) (*bool, error) {
    // ranodm damage between 1 and 10
    randomDmgFloat := rand.Float64() * 10

    spellToCast := model.CastedSpell{
        Spell:    spell,
        Type:     typeArg,
        PlayerID: playerID,
        Damage:   randomDmgFloat,
    }

    r.CastedSpells = append(r.CastedSpells, &spellToCast)

    r.mu.Lock()

  observer := r.SpellObservers[playerID]

  if observer != nil {
    observer <- &spellToCast
  }

    r.mu.Unlock()

    result := true

    return &result, nil
}

var baseSpells = []string{
    "fireball",
    "ice shard",
    "lightning bolt",
    "earthquake",
    "tornado",
}

// SpellBook is the resolver for the spellBook field.
func (r *queryResolver) SpellBook(ctx context.Context, playerID string) ([]*string, error) {
  spells := make([]*string, len(baseSpells))

  for i, spell := range baseSpells {
    spells[i] = &spell
  }

  return spells, nil
}

// SpellsCasted is the resolver for the spellsCasted field.
func (r *subscriptionResolver) SpellsCasted(ctx context.Context, target string) (<-chan *model.CastedSpell, error) {
    id := target
    spells := make(chan *model.CastedSpell, 1)

    go func() {
        <-ctx.Done()
        r.mu.Lock()
        delete(r.SpellObservers, id)
        r.mu.Unlock()
    }()

    r.mu.Lock()
    r.SpellObservers[id] = spells
    r.mu.Unlock()

    return spells, nil
}

// Mutation returns MutationResolver implementation.
func (r *Resolver) Mutation() MutationResolver { return &mutationResolver{r} }

// Query returns QueryResolver implementation.
func (r *Resolver) Query() QueryResolver { return &queryResolver{r} }

// Subscription returns SubscriptionResolver implementation.
func (r *Resolver) Subscription() SubscriptionResolver { return &subscriptionResolver{r} }

type mutationResolver struct{ *Resolver }
type queryResolver struct{ *Resolver }
type subscriptionResolver struct{ *Resolver }

Ok, now we need to update our server.go file to add the WebSocket transport and define some configured resolvers:

// server.go
package main

import (
    "apps/go-service/graph"
    "apps/go-service/graph/model"
    "github.com/gorilla/websocket"
    "log"
    "net/http"
    "os"
    "time"

    "github.com/99designs/gqlgen/graphql/handler"
    "github.com/99designs/gqlgen/graphql/handler/extension"
    "github.com/99designs/gqlgen/graphql/handler/lru"
    "github.com/99designs/gqlgen/graphql/handler/transport"
    "github.com/99designs/gqlgen/graphql/playground"
    "github.com/vektah/gqlparser/v2/ast"
)

const defaultPort = "8080"

func main() {
    port := os.Getenv("PORT")
    if port == "" {
        port = defaultPort
    }

    srv := handler.New(graph.NewExecutableSchema(graph.Config{Resolvers: &graph.Resolver{
        CastedSpells:   []*model.CastedSpell{},
        SpellObservers: map[string]chan *model.CastedSpell{},
    }}))

    srv.AddTransport(transport.Websocket{
        KeepAlivePingInterval: 10 * time.Second,
        Upgrader: websocket.Upgrader{
            CheckOrigin: func(r *http.Request) bool {
                return true
            },
        },
    })

    srv.AddTransport(transport.Options{})
    srv.AddTransport(transport.GET{})
    srv.AddTransport(transport.POST{})

    srv.SetQueryCache(lru.New[*ast.QueryDocument](1000))

    srv.Use(extension.Introspection{})
    srv.Use(extension.AutomaticPersistedQuery{
        Cache: lru.New[string](100),
    })

    http.Handle("/", playground.Handler("GraphQL playground", "/query"))
    http.Handle("/query", srv)

    log.Printf("connect to http://localhost:%s/ for GraphQL playground", port)
    log.Fatal(http.ListenAndServe(":"+port, nil))
}

And that's it! You can run the following command below and test it out. Note that you'll need to run subscriptions in different tabs than a mutation, otherwise the subscription will cancel out (this is a GraphiQL issue, not a Subscription issue):

nx run spell-service:serve

Here's the GraphQL to run:

subscription {
  spellsCasted(target: "jojo") {
    playerId
    spell
    damage
    type
  }
}

mutation {
  castSpell(spell: "fireball", type: ICE, playerId: "jojo")
}

Ok, let's move onto the Rust microservice in Part 2!

Polyglot Microservices: Federated GraphQL Subscriptions in Golang, Rust, and Nest.js, Pt. 2

Joseph Sutton — Thu, 06 Mar 2025 19:14:04 +0000

Series Navigation

Message Subgraph (Rust)

Now, let's create the Rust subgraph. Let's add a Rust service to our monorepo -- first we'll need the Nx plugin, and then we'll need to generate the project:

nx add @monodon/rust
nx generate @monodon/rust message-service

We'll be using async-graphql for our code-first GraphQL, axum to serve our schema as a websever, and tokio for an async runtime. We'll need some utility libraries for futures (among other things) for a simple broker (since we're not using Redis), as well. Let's add all of the dependencies we'll need for our service to Cargo.toml:

[package]
name = "message-service"
version = "0.1.0"
edition = "2021"

[dependencies]
anyhow = "1.0.90"
async-graphql = { version = "7.0.6", default-features = false, features = ["tracing", "chrono", "playground", "graphiql"] }
async-graphql-axum = "7.0.3"
async-stream = "0.3.5"
axum = { version = "0.7.5", features = ["macros"] }
axum-extra = { version = "0.9.4", default-features = false, features = ["tracing", "typed-header"] }
futures-util = "0.3.30"
futures-channel = "0.3.30"
futures-timer = "3.0.3"
handlebars = "5.1.2"
once_cell = "1.19"
slab = "0.4.9"
tokio = { version = "1.38", features = ["full"] }
tokio-stream = "0.1.17"

I have a specific template I like to use for Rust projects, because I like main.rs to be as small as possible. Let's add some files to bootstrap ourselves:

// main.rs
use rust_service::cli;

#[tokio::main]
async fn main() -> anyhow::Result<()> {
    cli::run().await
}

// lib.rs
pub mod cli;
pub mod graphql;

Let's add some code for our CLI crate:

use anyhow::Context;

pub async fn run() -> anyhow::Result<()> {
  let (send, mut recv) = tokio::sync::mpsc::channel(1);
  let mut handlers = vec![];

  println!("Starting messages graphql subgraph");
  let subgraph_send = send.clone();
  handlers.push(tokio::spawn(async move {
    let _ = subgraph_send.try_send(
      crate::graphql::server::run_server().await.context("Failed to start messages graphql server"),
    );
  }));

  let reason = recv.recv().await.expect("Didn't receive msg");
    for handle in handlers {
        handle.abort();
    }

    reason
}

Ok, now for our GraphQL server. Let's create a Message type:

// src/graphql/types.rs
use async_graphql::SimpleObject;

#[derive(SimpleObject, Clone)]
pub(super) struct Message {
  pub id: String,
  pub text: String,
  pub to_player_id: String,
  pub from_player_id: String,
}

We'll create a broker.rs file in src/graphql as well inspired by this example one from async-graphql: simple_broker.rs. We do need to modify it to add a filter for Stream, however:

    pub fn subscribe_filtered<F>(filter: F) -> impl Stream<Item = T>
    where
        F: Fn(&T) -> future::Ready<bool> + Send + 'static,
        T: Sync + Send + Clone + 'static,
    {
        Self::subscribe().filter(filter)
    }

Ok, now for the schema:

// src/graphql/schema.rs
use async_graphql::{*, futures_util::Stream, futures_util::future};
use super::{types::*, broker::SimpleBroker};

pub struct Query;

#[Object]
impl Query {
  async fn message(&self, id: String) -> async_graphql::Result<Message> {
    // this is here because async-graphql needs one query at least
    Ok(Message {
      id,
      text: "Hello, world!".to_string(),
      to_player_id: "1".to_string(),
      from_player_id: "2".to_string(),
    })
  }
}

pub struct Mutation;

#[derive(SimpleObject)]
pub struct MessagePayload {
  message: Message,
}

#[derive(InputObject)]
struct SendMessageInput {
  text: String,
  to_player_id: String,
  from_player_id: String,
}

#[Object]
impl Mutation {
  async fn send_message(&self, input: SendMessageInput) -> async_graphql::Result<MessagePayload> {
    let message = Message {
      id: "1".to_string(),
      text: input.text,
      to_player_id: input.to_player_id,
      from_player_id: input.from_player_id,
    };

    SimpleBroker::<Message>::publish(message.clone());

    Ok(MessagePayload { message })
  }
}

pub struct Subscription;

#[Subscription]
impl Subscription {
    async fn messages(&self, player_id: String) -> impl Stream<Item = Message> {
        SimpleBroker::<Message>::subscribe_filtered(move |message| future::ready(message.to_player_id == player_id))
    }
}

And finally, the server:

// src/graphql/server/mod.rs

use async_graphql::{http::GraphiQLSource, Schema};
use async_graphql_axum::{GraphQL, GraphQLSubscription};
use axum::{routing::get, Router, response::{IntoResponse, self}};
use tokio::net::TcpListener;

use crate::graphql;

pub async fn run_server() -> anyhow::Result<()> {
  let schema = Schema::build(graphql::Query, graphql::Mutation, graphql::Subscription)
    .finish();

  let app = Router::new()
        .route(
            "/",
            get(graphiql).post_service(GraphQL::new(schema.clone())),
        )
        .route_service("/ws", GraphQLSubscription::new(schema));

    println!("Starting graphql server on port {}", 8081);
    axum::serve(TcpListener::bind("127.0.0.1:8081").await.unwrap(), app)
        .await
        .unwrap();
    Ok(())
}

async fn graphiql() -> impl IntoResponse {
    response::Html(
        GraphiQLSource::build()
            .endpoint("/")
            .subscription_endpoint("/ws")
            .finish(),
    )
}

We can run this with:

nx run message-service:run

And then run the following GraphQL:

subscription {
  messages(playerId: "jojo") {
    text
  }
}

mutation {
  sendMessage(input: { text: "hello", toPlayerId: "jojo", fromPlayerId: "jojo"}) {
    message {
      text
    }
  }
}

As with the Golang service for spells, we can play around with this. We need to use multiple browser tabs again to test the subscription filtering, because this also uses GraphiQL - see this GitHub issue for clarity.

Now, let's move onto the Node.js microservice in Part 3!

Polyglot Microservices: Federated GraphQL Subscriptions in Golang, Rust, and Nest.js, Final

Joseph Sutton — Thu, 06 Mar 2025 19:14:01 +0000

Series Navigation

Hive Gateway (Node.js)

Now, let's tie it all together with the Hive Gateway, made by the awesome team, "TheGuild" over on GitHub.

First, let's create the node project with Nx:

nx g @nx/node:application

Install a few dependencies:

pnpm i @graphql-hive/gateway graphql-ws ws

Add a config in src/gateway.config.ts:

import { defineConfig, LogLevel } from '@graphql-hive/gateway';

export const gatewayConfig = defineConfig({
  port: 8000,
  supergraph: 'apps/gateway/supergraph.graphql',
  logging: LogLevel.debug,
  transportEntries: {
    '*.http': {
      options: {
        subscriptions: {
          kind: 'ws',
        },
      },
    },
  },
});

Let's add a few commands to this project, too:

...
    "hive-gateway": {
      "dependsOn": ["supergraph-compose"],
      "executor": "nx:run-commands",
      "options": {
        "command": "node_modules/.bin/hive-gateway supergraph -c apps/gateway/src/gateway.config.ts"
      }
    },
    "supergraph-compose": {
      "executor": "nx:run-commands",
      "options": {
        "command": "rover supergraph compose --elv2-license=accept --config apps/gateway/supergraph.json --output apps/gateway/supergraph.graphql"
      }
    },
...

Add a supergraph config for supergraph.json:

{
  "federation_version": "=2.9.0",
  "subgraphs": {
    "spells": {
      "schema": {
        "subgraph_url": "http://localhost:8080/query"
      }
    },
    "messages": {
      "schema": {
        "subgraph_url": "http://localhost:8081"
      }
    },
    "players": {
      "schema": {
        "subgraph_url": "http://localhost:8082/graphql"
      }
    }
  }
}

Okay, let's test this out and run the subgraphs first:

nx run-many -t=serve -p player-service message-service spell-service

Now, let's run the gateway:

NOTE: rover CLI is required for this step.

nx run gateway:hive-gateway

You should be able to run and execute Subscriptions through the gateway! Nice! You've now got a polyglot federated supergraph with subscriptions!

If you'd like to see the code, check it out here: https://github.com/sutt0n/polyglot-fed-gql

Polyglot Microservices: Federated GraphQL Subscriptions in Golang, Rust, and Nest.js, Pt. 3

Joseph Sutton — Thu, 06 Mar 2025 19:13:58 +0000

Series Navigation

Player Subgraph (Node.js)

Now, let's create the Nest.js subgraph

nx add @nx/nest
nx g @nx/nest:app apps/player-service

Let's add some dependencies:

pnpm i @nestjs/graphql graphql graphql-yoga @graphql-yoga/nestjs-federation

We're going to be using the GraphQL Yoga server here, because NestJS's federated Apollo configuration doesn't allow for subscriptions -- fun fact (mainly because Apollo charges for federated subscriptions, and we can get that for free if we use Hive Gateway over the Apollo Router).

If you need higher throughput on your router, Wundergraph's Router is free and it's in Golang. I've used it before, but my use-case only required Hive Gateway -- for now.

Anyhow, here's how we set up our src/app.module.ts with the GraphQL module configured for graphql-yoga:

import { Module } from '@nestjs/common';
import { GraphQLModule } from '@nestjs/graphql';
import {
  YogaFederationDriver,
  YogaFederationDriverConfig,
} from '@graphql-yoga/nestjs-federation';
import { PlayerModule } from './player/player.module';

@Module({
  imports: [
    GraphQLModule.forRoot<YogaFederationDriverConfig>({
      driver: YogaFederationDriver,
      autoSchemaFile: {
        federation: {
          version: 2,
          // required for federated subscriptions
          importUrl: 'https://specs.apollo.dev/federation/v2.4',
        },
      },
      subscriptions: true,
    }),
    PlayerModule,
  ],
})
export class AppModule {}

For our player module, we need our resolver in src/player/player.resolver.ts:

import { Args, Mutation, Resolver, Subscription } from '@nestjs/graphql';
import { Player } from './player.model';
import { createPubSub, Repeater } from 'graphql-yoga';

const pubSub = createPubSub();

@Resolver(() => Player)
export class PlayerResolver {
  @Mutation(() => Player)
  enterArea(
    @Args('playerId') playerId: string,
    @Args('area') area: string
  ): Player {
    pubSub.publish(`${area}`, { id: playerId });
    return { id: playerId };
  }

  @Subscription(() => Player, {
    nullable: true,
    resolve: (value: Player) => value,
  })
  nearbyPlayers(@Args('area') area: string): Repeater<Player> {
    return pubSub.subscribe(`${area}`);
  }
}

Now, we need our entity that we're resolving in src/player/player.model.ts:

import { Field, ID, ObjectType } from '@nestjs/graphql';

@ObjectType()
export class Player {
  @Field(() => ID)
  id: string;
}

Let's tie it together in src/player/player.module.ts:

import { Module } from '@nestjs/common';
import { PlayerResolver } from './player.resolver';

@Module({
  providers: [PlayerResolver],
})
export class PlayerModule {}

As with the above services, you'll boot this up and migrate to the proper URL and test it out with multiple browser tabs open:

subscription NearbyPlayers {
  nearbyPlayers(area: "forest") {
    id
  }
}

mutation {
  enterArea(playerId: "1243", area: "forest"){
    id
  }
}

Now, let's move onto the final part: tying it all together with Hive Gateway in part 4!

Scaled Virus Scanner using AWS Fargate, ClamAV, S3, and SQS with Terraform

Joseph Sutton — Fri, 27 May 2022 18:56:14 +0000

Welcome back for more shenanigans!

Some time ago, team I was on ran into a problem of hitting the Lambda deployment (and runtime) size limits: our solo Lambda function + a ClamAV layer with pre-built binaries and virus definitions. If you had a smaller file size requirement to scan, I'm sure that a Lambda layer setup with ClamAV and its binaries and definitions would work great for you; however, it wasn't in our case. We needed to scale our solution to allow for files up to sizes of 512MB.

TL;DR: GitHub repo.

Since we were already using SQS and EC2 for other things, why not use it along with S3 and Fargate? We had a spike to pursue either EC2 or a Fargate consumer client, but Fargate had better maintainability in the long run.

Note that I won't be implementing a cluster policy (yet) in this article, I might save that for another time; however, things should be setup to translate relatively well in that matter.

NOTE: This assumes you have your AWS credentials setup already via aws configure. If you plan on using a different profile, make sure you reflect that in the main.tf file below where profile = "default" is set.

Anyhow, let's get to it. First, let's set up our main configuration in terraform/main.tf:

terraform {
  required_version = ">= 1.0.0"
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 3.29"
    }
  }

  backend "s3" {
    encrypt        = true
    bucket         = "tf-clamav-state"
    dynamodb_table = "tf-dynamodb-lock"
    region         = "us-east-1"
    key            = "terraform.tfstate"
  }
}

# TODO: Make note about aws credentials and different profiles
provider "aws" {
  profile = "default"
  region  = "us-east-1"
}

This has a remote state (didn't I write about that once?), so we need a script to setup the S3 bucket and DynamoDB table for both our state and lock status respectively. We can set that up in a bash script in terraform/tf-setup.sh:

#!/bin/bash

# Create S3 Bucket
MY_ARN=$(aws iam get-user --query User.Arn --output text 2>/dev/null)
aws s3 mb "s3://tf-clamav-state" --region "us-east-1"
sed -e "s/RESOURCE/arn:aws:s3:::tf-clamav-state/g" -e "s/KEY/terraform.tfstate/g" -e "s|ARN|${MY_ARN}|g" "$(dirname "$0")/templates/s3_policy.json" > new-policy.json
aws s3api put-bucket-policy --bucket "tf-clamav-state" --policy file://new-policy.json
aws s3api put-bucket-versioning --bucket "tf-clamav-state" --versioning-configuration Status=Enabled
rm new-policy.json

# Create DynamoDB Table
aws dynamodb create-table \
  --table-name "tf-dynamodb-lock" \
  --attribute-definitions AttributeName=LockID,AttributeType=S \
  --key-schema AttributeName=LockID,KeyType=HASH \
  --provisioned-throughput ReadCapacityUnits=1,WriteCapacityUnits=1 \
  --region "us-east-1"

This does require an S3 policy s3_policy in terraform/templates/s3_policy.json:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": "s3:ListBucket",
      "Resource": "RESOURCE",
      "Principal": {
        "AWS": "ARN"
      }
    },
    {
      "Effect": "Allow",
      "Action": [
        "s3:GetObject",
        "s3:PutObject"
      ],
      "Resource": "RESOURCE/KEY",
      "Principal": {
        "AWS": "ARN"
      }
    }
  ]
}

Now we can run the tf-setup.sh script (don't forget to chmod +x it) via cd terraform && ./tf-setup.sh.

Now that we have our remote state established, let's scaffold out our infrastructure in Terraform. First up, we need our buckets (one for quarantined files and one for clean files), our SQS queue, and an event notification configured on the quarantine bucket for when an object is created. We can set this up via terraform/logistics.tf:

provider "aws" {
  region = "us-east-1"
  alias  = "east"
}

data "aws_caller_identity" "current" {}

resource "aws_s3_bucket" "quarantine_bucket" {
  provider = aws.east
  bucket   = "clamav-quarantine-bucket"
  acl      = "private"

  cors_rule {
    allowed_headers = ["Authorization"]
    allowed_methods = ["GET", "POST"]
    allowed_origins = ["*"]
    max_age_seconds = 3000
  }

  lifecycle_rule {
    enabled = true

    # Anything in the bucket remaining is a virus, so
    # we'll just delete it after a week.
    expiration {
      days = 7
    }
  }
}


resource "aws_s3_bucket" "clean_bucket" {
  provider = aws.east
  bucket   = "clamav-clean-bucket"
  acl      = "private"

  cors_rule {
    allowed_headers = ["Authorization"]
    allowed_methods = ["GET", "POST"]
    allowed_origins = ["*"]
    max_age_seconds = 3000
  }
}


data "template_file" "event_queue_policy" {
  template = file("templates/event_queue_policy.tpl.json")

  vars = {
    bucketArn = aws_s3_bucket.quarantine_bucket.arn
  }
}

resource "aws_sqs_queue" "clamav_event_queue" {
  name = "s3_clamav_event_queue"

  policy = data.template_file.event_queue_policy.rendered
}

resource "aws_s3_bucket_notification" "bucket_notification" {
  bucket = aws_s3_bucket.quarantine_bucket.id

  queue {
    queue_arn = aws_sqs_queue.clamav_event_queue.arn
    events    = ["s3:ObjectCreated:*"]
  }

  depends_on = [
    aws_sqs_queue.clamav_event_queue
  ]
}

resource "aws_cloudwatch_log_group" "clamav_fargate_log_group" {
  name = "/aws/ecs/clamav_fargate"
}

If you read the clamav_event_queue block above, there's an event queue policy - let's not forget that in terraform/templates/event_queue_policy.json:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": "*",
      "Action": [
        "sqs:SendMessage",
        "sqs:ReceiveMessage"
      ],
      "Resource": "arn:aws:sqs:*:*:s3_clamav_event_queue",
      "Condition": {
        "ArnEquals": {
          "aws:SourceArn": "${bucketArn}"
        }
      }
    }
  ]
}

Since this is a "security" thing, we need to make sure it's isolated within its own VPC. I'm no networking guru, so most of the information I obtained was from this Stackoverflow answer. We'll do that in terraform/vpc.tf:

# Networking for Fargate
# Note: 10.0.0.0 and 10.0.2.0 are private IPs
# Required via https://stackoverflow.com/a/66802973/1002357
# """
# > Launch tasks in a private subnet that has a VPC routing table configured to route outbound 
# > traffic via a NAT gateway in a public subnet. This way the NAT gateway can open a connection 
# > to ECR on behalf of the task.
# """
# If this networking configuration isn't here, this error happens in the ECS Task's "Stopped reason":
# ResourceInitializationError: unable to pull secrets or registry auth: pull command failed: : signal: killed
resource "aws_vpc" "clamav_vpc" {
  cidr_block = "10.0.0.0/16"
}

resource "aws_subnet" "private" {
  vpc_id     = aws_vpc.clamav_vpc.id
  cidr_block = "10.0.2.0/24"
}

resource "aws_subnet" "public" {
  vpc_id     = aws_vpc.clamav_vpc.id
  cidr_block = "10.0.1.0/24"
}

resource "aws_route_table" "private" {
  vpc_id = aws_vpc.clamav_vpc.id
}

resource "aws_route_table" "public" {
  vpc_id = aws_vpc.clamav_vpc.id
}

resource "aws_route_table_association" "public_subnet" {
  subnet_id      = aws_subnet.public.id
  route_table_id = aws_route_table.public.id
}

resource "aws_route_table_association" "private_subnet" {
  subnet_id      = aws_subnet.private.id
  route_table_id = aws_route_table.private.id
}

resource "aws_eip" "nat" {
  vpc = true
}

resource "aws_internet_gateway" "igw" {
  vpc_id = aws_vpc.clamav_vpc.id
}

resource "aws_nat_gateway" "ngw" {
  subnet_id     = aws_subnet.public.id
  allocation_id = aws_eip.nat.id

  depends_on = [aws_internet_gateway.igw]
}

resource "aws_route" "public_igw" {
  route_table_id         = aws_route_table.public.id
  destination_cidr_block = "0.0.0.0/0"
  gateway_id             = aws_internet_gateway.igw.id
}

resource "aws_route" "private_ngw" {
  route_table_id         = aws_route_table.private.id
  destination_cidr_block = "0.0.0.0/0"
  nat_gateway_id         = aws_nat_gateway.ngw.id
}


resource "aws_security_group" "egress-all" {
  name        = "egress_all"
  description = "Allow all outbound traffic"
  vpc_id      = aws_vpc.clamav_vpc.id

  egress {
    from_port   = 0
    to_port     = 0
    protocol    = "-1"
    cidr_blocks = ["0.0.0.0/0"]
  }
}

Now that we have the networking configuration done, we can go ahead and implement the ECS / Fargate configuration:

resource "aws_iam_role" "ecs_task_execution_role" {
  name = "clamav_fargate_execution_role"

  assume_role_policy = <<EOF
{
 "Version": "2012-10-17",
 "Statement": [
   {
     "Action": "sts:AssumeRole",
     "Principal": {
       "Service": "ecs-tasks.amazonaws.com"
     },
     "Effect": "Allow",
     "Sid": ""
   }
 ]
}
EOF
}

resource "aws_iam_role" "ecs_task_role" {
  name = "clamav_fargate_task_role"

  assume_role_policy = <<EOF
{
 "Version": "2012-10-17",
 "Statement": [
   {
     "Action": "sts:AssumeRole",
     "Principal": {
       "Service": "ecs-tasks.amazonaws.com"
     },
     "Effect": "Allow",
     "Sid": ""
   }
 ]
}
EOF
}

resource "aws_iam_role_policy_attachment" "ecs_task_execution_policy_attachment" {
  role       = aws_iam_role.ecs_task_execution_role.name
  policy_arn = "arn:aws:iam::aws:policy/service-role/AmazonECSTaskExecutionRolePolicy"
}

resource "aws_iam_role_policy_attachment" "s3_task" {
  role       = aws_iam_role.ecs_task_role.name
  policy_arn = "arn:aws:iam::aws:policy/AmazonS3FullAccess"
}

resource "aws_iam_role_policy_attachment" "sqs_task" {
  role       = aws_iam_role.ecs_task_role.name
  policy_arn = "arn:aws:iam::aws:policy/AmazonSQSFullAccess"
}

resource "aws_ecs_cluster" "cluster" {
  name = "clamav_fargate_cluster"

  capacity_providers = ["FARGATE"]
}

data "template_file" "task_consumer_east" {
  template = file("./templates/clamav_container_definition.json")

  vars = {
    aws_account_id = data.aws_caller_identity.current.account_id
  }
}

resource "aws_ecs_task_definition" "definition" {
  family                   = "clamav_fargate_task_definition"
  task_role_arn            = aws_iam_role.ecs_task_role.arn
  execution_role_arn       = aws_iam_role.ecs_task_execution_role.arn
  network_mode             = "awsvpc"
  cpu                      = "512"
  memory                   = "2048"
  requires_compatibilities = ["FARGATE"]

  container_definitions = data.template_file.task_consumer_east.rendered

  depends_on = [
    aws_iam_role.ecs_task_role,
    aws_iam_role.ecs_task_execution_role
  ]
}

resource "aws_ecs_service" "clamav_service" {
  name            = "clamav_service"
  cluster         = aws_ecs_cluster.cluster.id
  task_definition = aws_ecs_task_definition.definition.arn
  desired_count   = 1
  launch_type     = "FARGATE"

  network_configuration {
    assign_public_ip = false

    subnets = [
      aws_subnet.private.id
    ]

    security_groups = [
      aws_security_group.egress-all.id
    ]
  }
}

The container_definitions from the template_file has the configuration for the log configuration and environment variables. That configuration is found in terraform/templates/clamav_container_definition.json:

[
  {
    "image": "${aws_account_id}.dkr.ecr.us-east-1.amazonaws.com/fargate-images:latest",
    "name": "clamav",
    "logConfiguration": {
      "logDriver": "awslogs",
      "options": {
        "awslogs-region": "us-east-1",
        "awslogs-group": "/aws/ecs/clamav_fargate",
        "awslogs-stream-prefix": "project"
      }
    },
    "environment": [
      {
        "name": "VIRUS_SCAN_QUEUE_URL",
        "value": "https://sqs.us-east-1.amazonaws.com/${aws_account_id}/s3_clamav_event_queue"
      },
      {
        "name": "QUARANTINE_BUCKET",
        "value": "clamav-quarantine-bucket"
      },
      {
        "name": "CLEAN_BUCKET",
        "value": "clamav-clean-bucket"
      }
    ]
  }
]

Since we're using Fargate, we'll need a Dockerfile configuration and an ECR repository (as depicted in the clamav_container_definition.json file above). Let's get the ECR repository configured in terraform/ecr.tf:

resource "aws_ecr_repository" "image_repository" {
  name = "fargate-images"
}

data "template_file" "repo_policy_file" {
  template = file("./templates/ecr_policy.tpl.json")

  vars = {
    numberOfImages = 5
  }
}

# keep the last 5 images
resource "aws_ecr_lifecycle_policy" "repo_policy" {
  repository = aws_ecr_repository.image_repository.name
  policy     = data.template_file.repo_policy_file.rendered
}

You can play around with the number of images, completely up to you. It's just for versioning your Docker image that contains the consumer. Now, for the actual Dockerfile:

FROM ubuntu

WORKDIR /home/clamav

RUN echo "Prepping ClamAV"

RUN apt update -y
RUN apt install curl sudo procps -y

RUN curl -sL https://deb.nodesource.com/setup_14.x | sudo -E bash -
RUN apt install -y nodejs
RUN npm init -y

RUN npm i aws-sdk tmp sqs-consumer --save
RUN DEBIAN_FRONTEND=noninteractive sh -c 'apt install -y awscli'

RUN apt install -y clamav clamav-daemon

RUN mkdir /var/run/clamav && \
  chown clamav:clamav /var/run/clamav && \
  chmod 750 /var/run/clamav

RUN freshclam

COPY ./src/clamd.conf /etc/clamav/clamd.conf
COPY ./src/consumer.js ./consumer.js
RUN npm install
ADD ./src/run.sh ./run.sh

CMD ["bash", "./run.sh"]

This basically installs node, initializes a node project, and installs the bare essentials we need for the consumer: aws-sdk and tmp (for file handling to scan it). The first file, we can create in src/clamd.conf which is the ClamAV configuration (for the daemon that will be listening):

LocalSocket /tmp/clamd.socket
LocalSocketMode 660

Now for the SQS consumer in src/consumer.js:

const { SQS, S3 } = require('aws-sdk');
const { Consumer } = require('sqs-consumer');
const tmp = require('tmp');
const fs = require('fs');
const util = require('util');
const { exec } = require('child_process');

const execPromise = util.promisify(exec);

const s3 = new S3();

const app = Consumer.create({
  queueUrl: process.env.VIRUS_SCAN_QUEUE_URL,
  handleMessage: async (message) => {
    console.log('message', message);
    const parsedBody = JSON.parse(message.Body);
    const documentKey = parsedBody.Records[0].s3.object.key;

    const { Body: fileData } = await s3.getObject({
      Bucket: process.env.QUARANTINE_BUCKET,
      Key: documentKey
    }).promise();

    const inputFile = tmp.fileSync({
      mode: 0o644,
      tmpdir: process.env.TMP_PATH,
    });
    fs.writeSync(inputFile.fd, Buffer.from(fileData));
    fs.closeSync(inputFile.fd);

    try {
      await execPromise(`clamdscan ${inputFile.name}`);

      await s3.putObject({
        Body: fileData,
        Bucket: process.env.CLEAN_BUCKET,
        Key: documentKey,
        Tagging: 'virus-scan=clean',
      }).promise();

      await s3.deleteObject({
        Bucket: process.env.QUARANTINE_BUCKET,
        Key: documentKey,
      }).promise();

    } catch (e) {
      if (e.code === 1) {
        await s3.putObjectTagging({
          Bucket: process.env.QUARANTINE_BUCKET,
          Key: documentKey,
          Tagging: {
            TagSet: [
              {
                Key: 'virus-scan',
                Value: 'dirty',
              },
            ],
          },
        }).promise();
      }
    } finally {
      await sqs.deleteMessage({
        QueueUrl: process.env.VIRUS_SCAN_QUEUE_URL,
        ReceiptHandle: message.ReceiptHandle
      }).promise();
    }
  },
  sqs: new SQS()
});

app.on('error', (err) => {
  console.error('err', err.message);
});

app.on('processing_error', (err) => {
  console.error('processing error', err.message);
});

app.on('timeout_error', (err) => {
 console.error('timeout error', err.message);
});

app.start();

This does the following within a 10 second interval:

1) Pulls the file in through the quarantine bucket via the metadata in the SQS message (message in-flight)
2) Writes it to /tmp
3) Scans it with clamdscan (via the ClamAV daemon, clamd which already has the virus definitions loaded)
4) If it's clean, it puts the file in the clean bucket with a clean tag with the key virus-scan, removes it from the quarantine bucket, and deletes the message
5) If it's dirty, it tags the file as virus-scan = dirty and keeps the virus in the quarantine bucket, and deletes the SQS message

For now, this consumer handles only 1 message at a time; this can be easily configured to handle more messages since the ClamAV daemon scanner is much more efficient.

Now, the last file mentioned in the Dockerfile is the bash script that runs the updater, daemon, and consumer in src/run.sh:

echo "Starting Services"
service clamav-freshclam start
service clamav-daemon start

echo "Services started. Running worker."

node consumer.js

Cool, let's start it up by terraform plan and terraform apply. Once that's finished (you'll have to confirm by typing yes), you should be good to go.

Now, to test it with a script, test-virus.sh:

#!/bin/bash

aws s3 cp fixtures/test-virus.txt s3://clamav-quarantine-bucket
aws s3 cp fixtures/test-file.txt s3://clamav-quarantine-bucket

sleep 30

VIRUS_TEST=$(aws s3api get-object-tagging --key test-virus.txt --bucket clamav-quarantine-bucket --output text)
CLEAN_TEST=$(aws s3api get-object-tagging --key test-file.txt --bucket clamav-clean-bucket --output text)

echo "Dirty tag: ${VIRUS_TEST}"
echo "Clean tag: ${CLEAN_TEST}"

Running that, here's the output we get:

Dirty tag: TAGSET       virus-scan      dirty
Clean tag: TAGSET       virus-scan      clean

There we go. Hopefully y'all learned something! I had a lot of fun with this, and although I felt like I rushed it in a few areas, I look forward to your comments to see what all I missed (or to answer questions).

Y'all take care now.

Achieving a Cleaner State in your React App with Overmind (Basic)

Joseph Sutton — Sat, 05 Feb 2022 17:01:34 +0000

Today's mainly going to be focused on the frontend, because I want to introduce this state management system that I've been really digging lately. It's called Overmind, the same team that made Cerebral. Overmind is somewhat similar to Cerebral, but it supports TypeScript and it's... well, it's not abandoned.

TLDR: GitHub Repository.

Really, Another Daggum State Management System?

Yep. Like all the others say, "bUt tHiS ONe iS DiFfErEnT!" It honestly is - Overmind is a more declarative approach to state management orchestration. You give it a state structure, you tell it how the state is mutated and when the state is mutated, and you'll be a happier developer for it.

Okay, Fine

See? I knew you'd come around! Alright, let's get our boots on with React using TypeScript:

npx create-react-app overmind-shenanigans --template typescript

Now, let's add Overmind to our React project:

npm install overmind overmind-react

Cool, we're done! Just kidding - we need to configure it first in src/presenter/index.ts:

import { createStateHook, createActionsHook } from 'overmind-react';
import { state } from './state';
import * as actions from './actions';
import { IContext } from 'overmind';

export const config = {
  state,
  actions,
};

export type Context = IContext<{
  state: typeof config.state;
  actions: typeof config.actions;
}>;

export const useAppState = createStateHook<Context>();
export const useActions = createActionsHook<Context>();

Note that we're missing a few files, the state and actions files - don't worry, we'll get to those. Since we have our configuration defined, let's go ahead and hook it into our React app in index.tsx:

import React from 'react';
import ReactDOM from 'react-dom';
import './index.css';
import App from './App';
import reportWebVitals from './reportWebVitals';
import { createOvermind } from 'overmind';
import { Provider } from 'overmind-react';
import { config } from './presenter';

const overmind = createOvermind(config);

ReactDOM.render(
  <Provider value={overmind}>
    <App />
  </Provider>,
  document.getElementById('root')
);

Cool. Let's start doing things. First, let's add some good ole' fashioned TODO functionality. We'll use a combination of the component's state (temporary storage for the todo title and description), local storage, and the state managed by Overmind.

State

Let's set up our state structure in src/presenter/state.ts:

export type Todo = {
  title: string,
  description: string,
};

export const state = {
  todos: [] as Todo[],
};

Action

Let's write our action in src/presenter/actions/addTodoAction.ts:

import type { Context } from "../";
import { Todo } from "../state";

export const addTodoAction = (
  { state }: Context,
  { title, description }: Todo
) => {
  state.todos.push({
    title,
    description,
  });
};

For encapsulation's sake (and our config above), let's create our src/presenter/actions.ts file:

import { addTodoAction } from "./actions/addTodoAction";

export { addTodoAction };

Creating our TODO

Nothing special here, pretty simple. This isn't an article about CSS, it's about Overmind. Let's create the components that both add TODOs and list them. First, adding our TODOs with src/components/Todo.tsx:

import React, { useState } from "react";
import { useActions } from "../presenter";

export const Todo = () => {
  const [title, setTitle] = useState<string>('');
  const [description, setDescription] = useState<string>('');

  const actions = useActions();

  return (
    <>
      <div>
        <input
          name="title"
          type="text"
          value={title}
          placeholder="Title"
          onChange={(e) => setTitle(e.target.value)}
        />
        <input
          name="description"
          type="text"
          value={description}
          placeholder="Description"
          onChange={(e) => setDescription(e.target.value)}
        />
      </div>
      <div>
        <button onClick={() => {
          actions.addTodoAction({ title, description })
        }}>Add Todo</button>
      </div>
    </>
  );
};

Notice how we pull in our actions, and call addTodoAction. You can most definitely implement some validation here, too! Now, listing our TODOs with src/components/Todos.tsx:

import React from "react";
import {  useAppState } from "../presenter";

export const Todos = () => {
  const state = useAppState();

  return (
    <>
      {state.todos.map(todo => (
        <ul key={`todo-title-${todo.title}`}>
          <li><b>{todo.title}</b> - {todo.description}</li>
        </ul>
      ))}
    </>
  );
};

Let's put those two components in our src/App.tsx file:

import React from 'react';
import './App.css';
import { Todo } from './components/Todo';
import { Todos } from './components/Todos';

function App() {
  return (
    <div className="App">
      <header className="App-header">
        <Todo />
        <Todos />
      </header>
    </div>
  );
}

export default App;

You'll notice when we refresh the page, things don't persist. If you're normally a React developer, you'll know they won't even before refreshing. Let's talk about persisting our TODOs from the state to local storage with an effect.

Effects

Overmind effects are exactly what you think they are: side effects. You can do anything, from slapping axios to an SQLite library in there. With ours, we're going to just add an effect that accesses local storage.

With that, let's add our setItem effect in src/presenter/effects/setItem.ts:

import { Todo } from "../state";

export const setItem = (key : string, item : Todo) => {
  localStorage.setItem(key, JSON.stringify(item));
}

Now, our src/presenter/effects/getItem.ts:

export const getItem = (key : string) => {
  const item = localStorage.getItem(key);

  if(item) {
    return JSON.parse(item);
  }

  return null;
}

And our encapsulation in src/presenter/effects.ts:

import { getItem } from './effects/getItem';
import { setItem } from './effects/setItem';

export { getItem, setItem };

This will change our config and state context type. Let's go ahead and update that to our config in src/presenter/index.ts:

import { createStateHook, createActionsHook } from 'overmind-react';
import { state } from './state';
import * as actions from './actions';
import { IContext } from 'overmind';
import * as effects from './effects'

export const config = {
  state,
  actions,
  effects,
};

export type Context = IContext<{
  state: typeof config.state;
  actions: typeof config.actions;
  effects: typeof config.effects;
}>;

export const useAppState = createStateHook<Context>();
export const useActions = createActionsHook<Context>();

Now that's updated, we need to do a few things. First, we need to add the effect usage to local storage in our action, src/presenter/actions/addTodoItem.ts:

import type { Context } from "../";
import { Todo } from "../state";

export const addTodoAction = (
  { state, effects }: Context,
  { title, description }: Todo
) => {
  const currentTodos = effects.getItem('todos') || [];

  const newTodo = {
    title, description,
  };

  currentTodos.push(newTodo);

  state.todos = currentTodos;

  effects.setItem('todos', currentTodos);
};

Now, let's try it out. Add some TODOs, and refresh the page. You'll notice that it's still not showing our persisted TODOs in our local storage and that's because we need to initialize the state from local storage with the persisted TODOs. Thankfully, Overmind allows us to do that with an initialization action.

Let's create that initialization action in src/presenter/actions/onInitializationOvermind.ts:

import type { Context } from "../";

export const onInitializeOvermind = (
  { state, effects }: Context
) => {
  const currentTodos = effects.getItem('todos') || [];

  state.todos = currentTodos;
};

Let's add it to our src/presenter/actions.ts:

import { addTodoAction } from "./actions/addTodoAction";
import { onInitializeOvermind } from "./actions/onInitializeOvermind";

export { addTodoAction, onInitializeOvermind };

Now you can refresh the page and it should load any persisted TODOs.

I'll have an article written up on a full-stack application using Overmind with multiple models soon. It'll include the clean architecture that I previously wrote about.

There are some pros and cons to this state management system, as there are with any others. There's a lot of advanced addons / built-in functionality that allows the developer to control the state and how it flows / mutates. For example, Overmind has state machines as well (similar to XState).

However, the best parts I like about Overmind is the encapsulation and testability. If you go over to this article's repository, you'll notice that every effect and action is unit tested.

Thank y'all for reading! My next article will be either a soft skills type of article, or that full-stack clean architecture article that extends on the previous one.

Example App: Clean Architecture with React, Overmind and Local Storage, pt. 1

Joseph Sutton — Sun, 30 Jan 2022 03:56:31 +0000

Disclaimer: This article may seem to bounce around like a squirrel in an acorn field before the winter.

TL;DR: GitHub repository.

So, clean architecture! I'm a huge proponent of it. It's a great way to ensure that your testability of your project is so easy, a monkey could do it. What is clean architecture? It's something that pushes us to completely separate the business logic from the tech stack by allowing us to define clear boundaries by using dependency injection (we'll do this via an applicationContext):

I'm not going to go into detail, because clean architecture is a principle best explained by others; for example, this summary on Gist. Who founded this concept? Uncle Bob Martin. You can check out his Twitter!

How

How are we going to implement this separation? We'll be implementing everything in a monorepo via Lerna. I was going to utilize Terraform with this, but decided that was borderline over-engineering something simple like an Example Application. Maybe in the future!

Structure

How's the packaging going to look? What about the file structure? First off, we'll need a view, ui - that'll be our frontend in which I used create-react-app with a custom template I created.

Secondly, we'll need a place for our business logic, business. This will hold our entities, use-cases, and such. Thirdly, we'll need a place for storage methods, persistence. This is where methods for local storage will live.

Here's what our structure looks like so far:

packages/ui
packages/business
packages/persistence

The View

Let's dive in. So, I stated that I have a create-react-app template. This is essentially my frontend boilerplate for clean architecture that I've made - it's just for frontend and local storage. For TypeScript aficionados, I will be making one shortly after this article. The template has everything wrapped around the frontend, including local storage for persistence; however, I moved things around for this article.

Overmind

I used (https://overmindjs.org/)[Overmind] for state management. It's a more declarative state management system, and allows you to be as complex as you want. It's heavily aimed at allowing the developer to focus on the testability and readability of his/her application.

I'll be writing an article on Overmind as well. 😁

Code

Okay, we're actually diving in now. I promise.

First off, we have our plain ole' index.js which pulls in Overmind to the UI:

import React from 'react';
import ReactDOM from 'react-dom';
import './index.css';
import { createOvermind } from 'overmind';
import { Provider } from 'overmind-react';
import { config } from './presenter/presenter';
import App from './App.jsx';

const overmind = createOvermind(config);

ReactDOM.render(
  <Provider value={overmind}>
    <App />
  </Provider>,
  document.getElementById('root'),
);

That's easy enough. I won't post App.jsx, but it's just going to just reference the Todos component in views/Todos.jsx:

import * as React from 'react';
import { useActions, useState } from '../presenter/presenter';

const Todo = ({ todo }) => {
  useState();
  return (
    <li>
      {todo.title} {todo.description}
    </li>
  );
};

export const Todos = () => {
  const state = useState();
  const {
    addTodoItemAction,
    updateTodoTitleAction,
    updateTodoDescriptionAction,
  } = useActions();

  return (
    <>
      <input
        type="text"
        name="title"
        placeholder="Title"
        onChange={e => updateTodoTitleAction(e.target.value)}
      />
      <input
        type="textarea"
        name="description"
        placeholder="Description"
        onChange={e => updateTodoDescriptionAction(e.target.value)}
      />
      <button onClick={addTodoItemAction}>Add</button>
      <ul>
        {state.todos.map(todo => (
          <Todo key={todo.id} todo={todo} />
        ))}
      </ul>
    </>
  );
};

Since we're diving into Overmind territory, I'll explain some things we have going on here: we have two hooks, useActions and useState which pulls in our current state of the application and Overmind actions. Actions are essentially where state reads and mutations happen, and it's where we inject our applicationContext. I've named the directory where Overmind lives as presenter, because that's where our presentation logic will live.

Let's look at that file, ui/presenter/presenter.js:

import {
  createStateHook,
  createActionsHook,
  createEffectsHook,
  createReactionHook,
} from "overmind-react";
import { state } from "./state";
import { applicationContext } from '../applicationContext';
import { addTodoItemAction } from './actions/addTodoItemAction';
import { updateTodoTitleAction } from './actions/updateTodoTitleAction';
import { updateTodoDescriptionAction } from './actions/updateTodoDescriptionAction';
import { deleteTodoItemAction } from './actions/deleteTodoItemAction';

const actions = {
  addTodoItemAction,
  updateTodoTitleAction,
  updateTodoDescriptionAction,
  deleteTodoItemAction,
};

export const config = {
  state,
  actions,
  effects: applicationContext,
};

export const useState = createStateHook();
export const useActions = createActionsHook();
export const useEffects = createEffectsHook();
export const useReaction = createReactionHook();

After gandering at that, you're probably anxious to see what an action looks like with an applicationContext. Before I show y'all applicationContext, let's gander at the presenter/actions/addTodoItemAction.js:

export const addTodoItemAction = ({ state, effects: { ...applicationContext }}) => {
  const { todoTitle: title, todoDescription: description } = state;

  const todos = applicationContext.getUseCases().addTodoItemInteractor({
    applicationContext,
    title,
    description,
  });

  state.todos = todos;
}

Pretty simple (it gets simpler for those that whom are confused, I promise), really. We grab our use cases from applicationContext. You may be asking, "Why not just include the interactor? Why go through that? Well, let's look at the unit test:

const { createOvermindMock } = require("overmind");
const { config } = require("../presenter");

describe("addTodoItemAction", () => {
  let overmind;
  let addTodoItemInteractorStub;
  let mockTodo = { title: "TODO Title", description: "TODO Description" };

  beforeEach(() => {
    addTodoItemInteractorStub = jest.fn().mockReturnValue([mockTodo]);

    // TODO: refactor
    overmind = createOvermindMock(
      {
        ...config,
        state: { todoTitle: "TODO Title", todoDescription: "TODO Description" },
      },
      {
        getUseCases: () => ({
          addTodoItemInteractor: addTodoItemInteractorStub,
        }),
      }
    );
  });

  it("calls the interactor to add a todo item", async () => {
    await overmind.actions.addTodoItemAction();

    expect(addTodoItemInteractorStub).toHaveBeenCalled();
    expect(addTodoItemInteractorStub).toHaveBeenCalledWith({
      applicationContext: expect.anything(),
      ...mockTodo,
    });
    expect(overmind.state).toEqual(
      expect.objectContaining({
        todos: [mockTodo],
      })
    );
  });
});

I'd much rather mock out applicationContext than use jest.mock for each test. Having a context that unit tests can share for a potentially large codebase will save us a lot of time in writing these tests out. Another reason I believe it's better is for designing/defining your logic via Test Driven Development.

Business

Well, we've covered the actions that call our use-cases, or interactors. Let's dive into our business logic by first taking a look at the interactor being called from our action above, packages/business/useCases/addTodoItemInteractor.js:

import { Todo } from '../entities/Todo';

/**
 * use-case for adding a todo item to persistence
 *
 * @param {object} provider provider object
 */
export const addTodoItemInteractor = ({ applicationContext, title, description }) => {
  const todo = new Todo({ title, description }).validate().toRawObject();

  const todos = [];
  const currentTodos = applicationContext.getPersistence().getItem({
    key: 'todos',
    defaultValue: [],
  });

  if (currentTodos) {
    todos.push(...currentTodos);
  }

  todos.push(todo);

  applicationContext.getPersistence().setItem({ key: 'todos', value: todos });

  return todos;
};

Do you see where we're going with this? This interactor is the use-case surrounding the entity, Todo in the diagram above. It calls two persistence methods, which are just essentially local storage wrappers I've created. Let's take a gander at the unit test for this interactor:

const { addTodoItemInteractor } = require("./addTodoItemInteractor");

describe("addTodoItemInteractor", () => {
  let applicationContext;
  let getItemStub;
  let setItemStub;

  beforeAll(() => {
    getItemStub = jest.fn().mockReturnValue([]);
    setItemStub = jest.fn();

    applicationContext = {
      getPersistence: () => ({
        getItem: getItemStub,
        setItem: setItemStub,
      }),
    };
  });

  it("add a todo item into persistence", () => {
    const result = addTodoItemInteractor({
      applicationContext,
      title: "TODO Title",
      description: "TODO Description",
    });

    expect(getItemStub).toHaveBeenCalled();
    expect(getItemStub).toHaveBeenCalledWith({
      key: "todos",
      defaultValue: [],
    });
    expect(setItemStub).toHaveBeenCalled();
    expect(setItemStub).toHaveBeenCalledWith({
      key: "todos",
      value: [
        {
          title: "TODO Title",
          description: "TODO Description",
        },
      ],
    });
    expect(result).toEqual([
      {
        title: "TODO Title",
        description: "TODO Description",
      },
    ]);
  });
});

Easy, breezy, beautiful. Everything can be stubbed, mocked out, etc. All we care about is the primitive logic within the interactor itself - not what's in local storage, not what persistence we're using whether it's local storage or a remote/local database, and we don't care about the UI or any of the Overmind logic.

All we care about is the business logic. That's all we're testing here, and that's all we care about testing here. Let's take a look at those persistence methods, setItem and getItem.

Persistence

The two methods called above are setItem and getItem. Pretty straight-forward. Honestly, I probably didn't have to wrap them; however, I wanted to show that for persistence to be able to be easily interchangeable no matter what we use, practically nothing has to change inside the interactor.

Let's look at setItem:

module.exports.setItem = ({ key, value }) =>
  localStorage.setItem(key, JSON.stringify(value));

Easy enough. The unit test:

const { setItem } = require('./setItem');

describe('setItem', () => {
  let setItemStub;
  global.localStorage = {};

  beforeEach(() => {
    setItemStub = jest.fn();

    global.localStorage.setItem = setItemStub;
  });

  it('sets the item given the key/value pair', () => {
    setItem({ key: 'todos', value: 'todos value' });

    expect(setItemStub).toHaveBeenCalled();
    expect(setItemStub).toHaveBeenCalledWith('todos', JSON.stringify('todos value'));
  });
});

Simple enough, right? There's a pattern to unit tests and I'm sure with some ideas, one could find a way to reduce boilerplate... or just make a macro since most everything repeated is essential to its respective unit test.

Note: The only reason why we're stringifying with JSON is we're allowing the storage of objects/arrays (if you noticed in the action, the todos are an array).

That's obviously not everything. I didn't want to dive too deep into the specifics. My next article will include us hooking this same setup with a backend (more than likely serverless). What database should we use? DynamoDB or a relational database like PostgreSQL? Maybe both?

Thanks for reading! If I typed something wrong or if you have any questions, comments, concerns, or suggestions then please post them in a comment! Y'all take care now.

Better CircleCI Insights Visualization with Recharts

Joseph Sutton — Thu, 26 Aug 2021 12:24:30 +0000

CircleCI introduced insights some time ago, but I wanted a timeline-like graph of all successful deployments and builds. Since the only way to graph builds is to view each branch, we'll need to do some querying and organizing. I've never used Recharts before, but it was relatively simple to use.

TLDR: Here's the GitHub repository.

Frontend

Since we're just wanting to get up and running, we can create a quick React project with npx create-react-app. After that, we'll have a bare React application and we can start looking into how we can gather metrics on our CircleCI builds and deployments.

Getting the Data

First, we need a script to grab both our deployment and build data. Luckily, CircleCI's API endpoints make that easy enough for us. I'll be using an open source project I'm contributing to currently.

Deployment Data

Let's create our first script for grabbing deployment data. We'll call this script get-deployment-data.js and slap it in our root directory.

const axios = require('axios');
const fs = require('fs').promises;

require('dotenv').config();

(async () => {
  const apiKey = process.argv[2] || process.env.CIRCLE_TOKEN;
  const vcsSlug = process.argv[3] || process.env.VCS_SLUG;
  const workflowName = process.argv[4] || process.env.DEPLOYMENT_WORKFLOW_NAME;

  const { data } = await axios.get(`https://circleci.com/api/v2/insights/${vcsSlug}/workflows/${workflowName}`, {
    headers: { 
      'Circle-Token': `${apiKey}`,
      'Access-Control-Allow-Origin': '*',
    },
  });

  data.items = data.items.map(item => {
    const dtCreated = new Date(item.created_at);

    return { 
      ...item,
      name: `${dtCreated.getMonth() + 1}/${dtCreated.getDate()}`,
    }
  });

  data.items.sort((a,b) => {
    const aDate = new Date(a.created_at);
    const bDate = new Date(b.created_at);

    return aDate > bDate || 0;
  })

  const jsonString = JSON.stringify(data.items);

  await fs.writeFile('./src/data/deployment-data.json', Buffer.from(jsonString, 'utf-8'));
})();

After using this script, it'll create a JSON file of the deployment data in the src/data directory:

node ./get-deployment-data.js abc123-def321-abcdef-123321 gh/sutt0n/test-repo deploy

Build Data

This one is a little tricky, because we can't simply query the workflow name, build for some reason. We have to obtain a select history of builds and grab the branch names returned from that. After we have the branch names, we can query the build workflow for a specific branch. Oh, and CircleCI's API has a rate limiter - so that's fun. We can add axios-retry and that should do the trick:

/**
* get-build-data.js
* Usage: node ./get-build-data.js <circle ci token> <vcs slug> <workflow name>
* Example: node ./get-build-data.js abc123-def321-abc-1234 gh/flexion/ef-cms hourly
* Returns: Creates ./data/build-data.json
**/

const axios = require('axios');
const axiosRetry = require('axios-retry');
const fs = require('fs').promises;

axiosRetry(axios, { retryDelay: axiosRetry.exponentialDelay, retries: 10 });

require('dotenv').config();

(async () => {
  const apiKey = process.argv[2] || process.env.CIRCLE_TOKEN;
  const vcsSlug = process.argv[3] || process.env.VCS_SLUG;

  const results = [];

  let offset = 0;

  for(let i = 0; i <= 3; i++) {
    offset += 100 * i;
    let { data } = await axios.get(`https://circleci.com/api/v1.1/project/${vcsSlug}?shallow=true&limit=100&offset=${offset}`, {
      headers: { 
        'Circle-Token': `${apiKey}`,
        'Access-Control-Allow-Origin': '*',
      },
    });

    results.push(...data);
  }

  const branches = results.map(result => {
    if(result.workflows.workflow_name !== 'hourly' && result.lifecycle === 'finished') {
      return result.branch;
    }

    return null;
  }).filter((x,idx) => x && x.indexOf(x) === idx);

  const allResults = [];

  for(const branch of branches) {
    const {data} = await axios.get(`https://circleci.com/api/v2/insights/${vcsSlug}/workflows/build-and-deploy?branch=${branch}`, {
      headers: { 
        'Circle-Token': `${apiKey}`,
        'Access-Control-Allow-Origin': '*',
      },
    });
    allResults.push(...data.items);
  }

  const jsonString = JSON.stringify(allResults);

  await fs.writeFile('./src/data/build-data.json', Buffer.from(jsonString, 'utf-8'));
})();

Charts

I decided the best approach was to graph out our credits and duration (in seconds) using the SimpleLineCart. Here's what our App.jsx looks like:

import './App.css';
import { LineChart, Line, XAxis, YAxis, CartesianGrid, Tooltip, Legend } from 'recharts';
import deploymentStats from './data/deployment-data.json'
import buildStats from './data/build-data.json'

function App() {
  return (
    <div className="App">
      <header>CircleCI Insights</header>
      <div className="insights">
        <h2 className="padding-top-0">Deployment Stats</h2>
        <LineChart
          width={1500}
          height={500}
          data={deploymentStats}
          margin={{
            top: 5,
            right: 30,
            left: 20,
            bottom: 5,
          }}
        >
          <CartesianGrid strokeDasharray="3 3" />
          <XAxis reversed dataKey="name" dy={5} minTickGap={75}/>
          <YAxis />
          <Tooltip />
          <Legend />
          <Line type="monotone" dataKey="credits_used" stroke="#8884d8" dy={30} />
          <Line type="monotone" dataKey="duration" stroke="#82ca9d" dy={30} activeDot={{ r: 3 }} />
        </LineChart>
      </div>
      <div className="insights">
        <h2 className="padding-top-0">Build Stats</h2>
        <LineChart
          width={1500}
          height={500}
          data={buildStats}
          margin={{
            top: 5,
            right: 30,
            left: 20,
            bottom: 5,
          }}
        >
          <CartesianGrid strokeDasharray="3 3" />
          <XAxis reversed dataKey="name" dy={5} minTickGap={75}/>
          <YAxis />
          <Tooltip />
          <Legend />
          <Line type="monotone" dataKey="credits_used" stroke="#8884d8" dy={30} />
          <Line type="monotone" dataKey="duration" stroke="#82ca9d" dy={30} activeDot={{ r: 3 }} />
        </LineChart>
      </div>
    </div>
  );
}

export default App;

Now let's run npm run start and we should see our deployment chart at the top:

Looks like we had a deployment that hanged (referring to the green spike). Thankfully it wasn't entirely synergistic with the credit usage.

Now, for our build chart. By filtering out the failed builds (including cancelled ones), we can get a better visualization on the relationship for our branch builds:

With the JSON returned, we can definitely get information back to that specific build to see what happened, and Recharts has an API event that allows us to click to get the data returned for a specific data point.

Thanks

I hope this helps you in better visualizing your credit usage with CircleCI! Thank you for reading, and please don't hesitate to ask questions or provide feedback in the comment section.

Using Serverless to Scan Files with ClamAV in a Lambda Container

Joseph Sutton — Thu, 19 Aug 2021 00:41:40 +0000

In my previous post, I had used a lambda function with a designated lambda layer. The layer's binaries were created within a Docker image based from Amazon's amazonlinux:2 image. We can use those binaries in conjunction with AWS's lambda container images feature without much worry of deployment size limitations as we did with the lambda function and its layer.

History

For those that did not read the previous post, this is going to establish an S3 bucket with an event trigger towards a lambda function. This lambda function will be a container with the handler code and ClamAV binaries and virus definitions. It will get the S3 object via the metadata in the trigger, scan it, and mark it as clean or dirty per the results of the ClamAV scan.

TLDR: Here's the GitHub repository.

Infrastructure

This is obviously going to be different - instead of using a lambda layer, we'll be using a Docker image stored on ECR. This is nearly effortless, thanks to Serverless.

Serverless

By default, Server will create an ECR repository for us and the image will live in it. All we have to do is give it the path of the Dockerfile.

service: clambda-av

provider:
  name: aws
  runtime: nodejs14.x
  ecr:
    images:
      clambdaAv:
        path: ./
  iamRoleStatements:
    - Effect: Allow
      Action:
        - s3:GetObject
        - s3:PutObjectTagging
      Resource: "arn:aws:s3:::clambda-av-files/*"

functions:
  virusScan:
    image:
      name: clambdaAv
    memorySize: 2048
    events:
      - s3: 
          bucket: clambda-av-files
          event: s3:ObjectCreated:*
    timeout: 120

package:
  exclude:
    - node_modules/**
    - coverage/**

Dockerfile

Since we're using Javascript, we'll be using the nodejs14 image as the base. Unfortunately, we cannot easily install our ClamAV binaries through this image and thus have to use the amazonlinux:2 image, as stated above. Fortunately, Docker allows us do that with ease via multi-stage Docker builds. I've never done this up until now, but it was a pretty quick and interesting process:

FROM amazonlinux:2 AS layer-image

WORKDIR /home/build

RUN set -e

RUN echo "Prepping ClamAV"

RUN rm -rf bin
RUN rm -rf lib

RUN yum update -y
RUN amazon-linux-extras install epel -y
RUN yum install -y cpio yum-utils tar.x86_64 gzip zip

RUN yumdownloader -x \*i686 --archlist=x86_64 clamav
RUN rpm2cpio clamav-0*.rpm | cpio -vimd

RUN yumdownloader -x \*i686 --archlist=x86_64 clamav-lib
RUN rpm2cpio clamav-lib*.rpm | cpio -vimd

RUN yumdownloader -x \*i686 --archlist=x86_64 clamav-update
RUN rpm2cpio clamav-update*.rpm | cpio -vimd

RUN yumdownloader -x \*i686 --archlist=x86_64 json-c
RUN rpm2cpio json-c*.rpm | cpio -vimd

RUN yumdownloader -x \*i686 --archlist=x86_64 pcre2
RUN rpm2cpio pcre*.rpm | cpio -vimd

RUN yumdownloader -x \*i686 --archlist=x86_64 libtool-ltdl
RUN rpm2cpio libtool-ltdl*.rpm | cpio -vimd

RUN yumdownloader -x \*i686 --archlist=x86_64 libxml2
RUN rpm2cpio libxml2*.rpm | cpio -vimd

RUN yumdownloader -x \*i686 --archlist=x86_64 bzip2-libs
RUN rpm2cpio bzip2-libs*.rpm | cpio -vimd

RUN yumdownloader -x \*i686 --archlist=x86_64 xz-libs
RUN rpm2cpio xz-libs*.rpm | cpio -vimd

RUN yumdownloader -x \*i686 --archlist=x86_64 libprelude
RUN rpm2cpio libprelude*.rpm | cpio -vimd

RUN yumdownloader -x \*i686 --archlist=x86_64 gnutls
RUN rpm2cpio gnutls*.rpm | cpio -vimd

RUN yumdownloader -x \*i686 --archlist=x86_64 nettle
RUN rpm2cpio nettle*.rpm | cpio -vimd

RUN mkdir -p bin
RUN mkdir -p lib
RUN mkdir -p var/lib/clamav
RUN chmod -R 777 var/lib/clamav

COPY ./freshclam.conf .

RUN cp usr/bin/clamscan usr/bin/freshclam bin/.
RUN cp usr/lib64/* lib/.
RUN cp freshclam.conf bin/freshclam.conf

RUN yum install shadow-utils.x86_64 -y

RUN groupadd clamav
RUN useradd -g clamav -s /bin/false -c "Clam Antivirus" clamav
RUN useradd -g clamav -s /bin/false -c "Clam Antivirus" clamupdate

RUN LD_LIBRARY_PATH=./lib ./bin/freshclam --config-file=bin/freshclam.conf

FROM public.ecr.aws/lambda/nodejs:14

COPY --from=layer-image /home/build ./

COPY handler.js ./

CMD ["handler.virusScan"]

This Dockerfile does two things:

Builds the ClamAV binaries into a stage aliased layer-image along with the ClamAV virus definitions
Builds the Lambda image with the handler itself, and then pulls in the ClamAV binaries and virus definitions from the layer-image stage

Handler

This doesn't change the handler much from my previous post:

const { execSync } = require("child_process");
const { writeFileSync, unlinkSync } = require("fs");
const AWS = require("aws-sdk");

const s3 = new AWS.S3();

module.exports.virusScan = async (event, context) => {
  if (!event.Records) {
    console.log("Not an S3 event invocation!");
    return;
  }

  for (const record of event.Records) {
    if (!record.s3) {
      console.log("Not an S3 Record!");
      continue;
    }

    // get the file
    const s3Object = await s3
      .getObject({
        Bucket: record.s3.bucket.name,
        Key: record.s3.object.key
      })
      .promise();

    // write file to disk
    writeFileSync(`/tmp/${record.s3.object.key}`, s3Object.Body);

    try { 
      // scan it
      execSync(`./bin/clamscan --database=./var/lib/clamav /tmp/${record.s3.object.key}`);

      await s3
        .putObjectTagging({
          Bucket: record.s3.bucket.name,
          Key: record.s3.object.key,
          Tagging: {
            TagSet: [
              {
                Key: 'av-status',
                Value: 'clean'
              }
            ]
          }
        })
        .promise();
    } catch(err) {
      if (err.status === 1) {
        // tag as dirty, OR you can delete it
        await s3
          .putObjectTagging({
            Bucket: record.s3.bucket.name,
            Key: record.s3.object.key,
            Tagging: {
              TagSet: [
                {
                  Key: 'av-status',
                  Value: 'dirty'
                }
              ]
            }
          })
          .promise();
      }
    }

    // delete the temp file
    unlinkSync(`/tmp/${record.s3.object.key}`);
  }
};

Summary

From our previous adventure (this is the last time I'm linking it, I swear), this removes the extra step of building the binaries with a bash script. It also removes the need for a lambda layer.

If you'd like to check out the full code, again, it's in the GitHub repository. Please do not hesitate to ask questions or post any comments or issues you may have in this article or by opening an issue on the repository if applicable. Thanks for reading!

Using Serverless to Scan Files with a ClamAV Lambda Layer

Joseph Sutton — Thu, 12 Aug 2021 20:00:46 +0000

Update: I've written how to do this with lambda containers as well!

Let's create an environment that scans a file via an S3 event by utilizing ClamAV binaries on a Lambda layer. You can retrieve the full source code at this GitHub repository.

Note: As of 8/11/2021, the unzipped size of the ClamAV binaries and virus definitions are under the limit. This may change in the future.

Serverless config

So, we'll need a few things: an S3 bucket, a function, and a lambda layer. I've also included logging groups, and permissions in the serverless.yml file:

service: clambda-av

provider:
  name: aws
  runtime: nodejs14.x
  iamRoleStatements:
    - Effect: Allow
      Action:
        - s3:GetObject
        - s3:PutObjectTagging
      Resource: "arn:aws:s3:::clambda-av-files/*"

functions:
  virusScan:
    handler: handler.virusScan
    memorySize: 2048
    events:
      - s3: 
          bucket: clambda-av-files
          event: s3:ObjectCreated:*
    layers:
      - {Ref: ClamavLambdaLayer}
    timeout: 120

package:
  exclude:
    - node_modules/**
    - coverage/**

layers:
  clamav:
    path: layer

Dockerfile

Before we deploy this, we need to get our ClamAV binaries. Amazon has their own Docker base image we can use to build these binaries. With the base image, we can start making our binaries. Through the power of trial and error, I've found the necessary binaries that's required. Here's the full Dockerfile:

FROM amazonlinux:2

WORKDIR /home/build

RUN set -e

RUN echo "Prepping ClamAV"

RUN rm -rf bin
RUN rm -rf lib

RUN yum update -y
RUN amazon-linux-extras install epel -y
RUN yum install -y cpio yum-utils tar.x86_64 gzip zip

RUN yumdownloader -x \*i686 --archlist=x86_64 clamav
RUN rpm2cpio clamav-0*.rpm | cpio -vimd

RUN yumdownloader -x \*i686 --archlist=x86_64 clamav-lib
RUN rpm2cpio clamav-lib*.rpm | cpio -vimd

RUN yumdownloader -x \*i686 --archlist=x86_64 clamav-update
RUN rpm2cpio clamav-update*.rpm | cpio -vimd

RUN yumdownloader -x \*i686 --archlist=x86_64 json-c
RUN rpm2cpio json-c*.rpm | cpio -vimd

RUN yumdownloader -x \*i686 --archlist=x86_64 pcre2
RUN rpm2cpio pcre*.rpm | cpio -vimd

RUN yumdownloader -x \*i686 --archlist=x86_64 libtool-ltdl
RUN rpm2cpio libtool-ltdl*.rpm | cpio -vimd

RUN yumdownloader -x \*i686 --archlist=x86_64 libxml2
RUN rpm2cpio libxml2*.rpm | cpio -vimd

RUN yumdownloader -x \*i686 --archlist=x86_64 bzip2-libs
RUN rpm2cpio bzip2-libs*.rpm | cpio -vimd

RUN yumdownloader -x \*i686 --archlist=x86_64 xz-libs
RUN rpm2cpio xz-libs*.rpm | cpio -vimd

RUN yumdownloader -x \*i686 --archlist=x86_64 libprelude
RUN rpm2cpio libprelude*.rpm | cpio -vimd

RUN yumdownloader -x \*i686 --archlist=x86_64 gnutls
RUN rpm2cpio gnutls*.rpm | cpio -vimd

RUN yumdownloader -x \*i686 --archlist=x86_64 nettle
RUN rpm2cpio nettle*.rpm | cpio -vimd

RUN mkdir -p bin
RUN mkdir -p lib
RUN mkdir -p var/lib/clamav
RUN chmod -R 777 var/lib/clamav

COPY ./freshclam.conf .

RUN cp usr/bin/clamscan usr/bin/freshclam bin/.
RUN cp usr/lib64/* lib/.
RUN cp freshclam.conf bin/freshclam.conf

RUN yum install shadow-utils.x86_64 -y

RUN groupadd clamav
RUN useradd -g clamav -s /bin/false -c "Clam Antivirus" clamav
RUN useradd -g clamav -s /bin/false -c "Clam Antivirus" clamupdate

RUN LD_LIBRARY_PATH=./lib ./bin/freshclam --config-file=bin/freshclam.conf

RUN zip -r9 clamav_lambda_layer.zip bin
RUN zip -r9 clamav_lambda_layer.zip lib
RUN zip -r9 clamav_lambda_layer.zip var
RUN zip -r9 clamav_lambda_layer.zip etc

You'll note that the COPY ./freshclam.conf . line implies that there's another file that we need and you'd be correct:

DatabaseMirror database.clamav.net
CompressLocalDatabase yes
ScriptedUpdates no
DatabaseDirectory /home/build/var/lib/clamav

Building the binaries with Docker

Next is our bash script build.sh to run Docker on the Dockerfile above to build and extract the binaries:

#!/bin/bash

rm -rf ./layer
mkdir layer

docker build -t clamav -f Dockerfile .
docker run --name clamav clamav
docker cp clamav:/home/build/clamav_lambda_layer.zip .
docker rm clamav
mv clamav_lambda_layer.zip ./layer

pushd layer
unzip -n clamav_lambda_layer.zip
rm clamav_lambda_layer.zip
popd

Scanning Handler

Furthermore, we also need our handler to do the scanning and tagging of the file. The tagging is more of a placeholder. You can create a separate quarantine bucket or a separate clean bucket - whichever you prefer.

Now, we need our handler and we should be set:

const { execSync } = require("child_process");
const { writeFileSync, unlinkSync } = require("fs");
const AWS = require("aws-sdk");

const s3 = new AWS.S3();

module.exports.virusScan = async (event, context) => {
  if (!event.Records) {
    console.log("Not an S3 event invocation!");
    return;
  }

  for (const record of event.Records) {
    if (!record.s3) {
      console.log("Not an S3 Record!");
      continue;
    }

    // get the file
    const s3Object = await s3
      .getObject({
        Bucket: record.s3.bucket.name,
        Key: record.s3.object.key
      })
      .promise();

    // write file to disk
    writeFileSync(`/tmp/${record.s3.object.key}`, s3Object.Body);

    try { 
      // scan it
      const scanStatus = execSync(`clamscan --database=/opt/var/lib/clamav /tmp/${record.s3.object.key}`);

      await s3
        .putObjectTagging({
          Bucket: record.s3.bucket.name,
          Key: record.s3.object.key,
          Tagging: {
            TagSet: [
              {
                Key: 'av-status',
                Value: 'clean'
              }
            ]
          }
        })
        .promise();
    } catch(err) {
      if (err.status === 1) {
        // tag as dirty, OR you can delete it
        await s3
          .putObjectTagging({
            Bucket: record.s3.bucket.name,
            Key: record.s3.object.key,
            Tagging: {
              TagSet: [
                {
                  Key: 'av-status',
                  Value: 'dirty'
                }
              ]
            }
          })
          .promise();
      }
    }

    // delete the temp file
    unlinkSync(`/tmp/${record.s3.object.key}`);
  }
};

You may be asking yourself on the --database option above, "Wait... why /opt/?" That's because all of the layer files are placed into that directory for the lambda function to use once it's mounted at runtime.

Here's the algorithm:

If a file is clean, then it is tagged with a key/value pair of av-status = 'clean'
If a file is NOT clean (virus), then it is tagged with a key/value pair of av-status = 'dirty'

Since this is for demonstration purposes, you can obviously customize this flow however you'd like. 😀

Deploying

Now that we have all of our files, we can do the following:

Build the binaries via ./build.sh (make sure it's executable via chmod +x build.sh after creation)
Run sls deploy

joseph@bertha > sls deploy
Serverless: Packaging service...
Serverless: Excluding development dependencies...
Serverless: Excluding development dependencies...
Serverless: Creating Stack...
Serverless: Checking Stack create progress...
........
Serverless: Stack create finished...
Serverless: Uploading CloudFormation file to S3...
Serverless: Uploading artifacts...
Serverless: Uploading service clambda-av.zip file to S3 (41 KB)...
Serverless: Uploading service clamav.zip file to S3 (222.57 MB)...
Serverless: Validating template...
Serverless: Updating Stack...
Serverless: Checking Stack update progress...
........................
Serverless: Stack update finished...
Service Information
service: clambda-av
stage: dev
region: us-east-1
stack: clambda-av-dev
resources: 9
api keys:
  None
endpoints:
  None
functions:
  virusScan: clambda-av-dev-virusScan
layers:
  clamav: arn:aws:lambda:us-east-1:**********:layer:clamav:8

Now, let's test it by uploading a clean file to S3. I happen to have a PDF laying around:

joseph@bertha > aws s3 cp ~/document.pdf s3://clambda-av-files/
upload: ../../document.pdf to s3://clambda-av-files/document.pdf

The downside is that clamscan in the Lambda layer takes ~30 seconds or so boot, load the virus definitions, and scan the file with the results. We can check the tag after some time via:

joseph@bertha > aws s3api get-object-tagging --bucket clambda-av-files --key document.pdf

{
    "TagSet": [
        {
            "Key": "av-status",
            "Value": "clean"
        }
    ]
}

We can test the virus scanner with a test virus signature found at EICAR:

X5O!P%@AP[4\PZX54(P^)7CC)7}$EICAR-STANDARD-ANTIVIRUS-TEST-FILE!$H+H*

After saving that text to a file called test-virus.pdf. Let's upload it and see what happens:

joseph@bertha > aws s3 cp ~/test-virus.pdf s3://clambda-av-files/
upload: ../../test-virus.pdf to s3://clambda-av-files/test-virus.pdf

After waiting another thirty seconds or so, let's check the tag on it:

joseph@bertha > aws s3api get-object-tagging --bucket clambda-av-files --key test-virus.pdf

{
    "TagSet": [
        {
            "Key": "av-status",
            "Value": "dirty"
        }
    ]
}

Drawbacks and potential problems

There are a few drawbacks to this, especially since the size of the Lambda layer is quite large:

File size constraints due to the /tmp means you cannot upload a file larger than 512MB.
ClamAV virus definitions will no doubt get larger, thus potentially interfering with the maximum deployment size (which is 250MB, including layers, as of 8/11/2021).
Lambda code storage limitation may eventually be reached with consecutive deployments, although this can be mitigated with the serverless-prune-plugin.

Thanks

Thank y'all for reading. If you have any questions, feel free to ask in the comments! I also welcome suggestions. 🙂

Speeding Up Your Terraform Projects with S3-backend Remote States

Joseph Sutton — Thu, 05 Aug 2021 17:09:23 +0000

Update 8/6/2021: Abstracted out the Resource property in the s3_policy.json file, since the script below handles replacing that. 🙂

Ever wonder if there's a quicker way to initialize your infrastructure with Terraform on a remote state? My go-to is the S3 backend, but manually setting that up can take more time versus making a script to do the work for us.

Assuming you already have your AWS credentials setup already, we need to create two files. The first file is our S3 policy to let Terraform view the items, and to get and update our Terraform state.

Our S3 policy, s3_policy.json:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": "s3:ListBucket",
      "Resource": "RESOURCE",
      "Principal": {
        "AWS": "ARN"
      }
    },
    {
      "Effect": "Allow",
      "Action": [
        "s3:GetObject",
        "s3:PutObject"
      ],
      "Resource": "RESOURCE/KEY",
      "Principal": {
        "AWS": "ARN"
      }
    }
  ]
}

So, if you're wondering about the ARN, RESOURCE, and KEY, that's okay. Our script will handle those by replacing those via sed magic, so when you change the bucket name in the bash script, you don't need to update the policy file. 😀

The second file is our actual bash script, let's say tf-setup.sh (remember to chmod +x this file so you can execute it!):

#!/bin/bash

# Create S3 Bucket
MY_ARN=$(aws iam get-user --query User.Arn --output text 2>/dev/null)
aws s3 mb "s3://tf-remote-bucket" --region "us-east-1"
sed -e "s/RESOURCE/arn:aws:s3:::tf-remote-bucket/g" -e "s/KEY/terraform.tfstate/g" -e "s|ARN|${MY_ARN}|g" "$(dirname "$0")/s3_policy.json" > new-policy.json
aws s3api put-bucket-policy --bucket "tf-remote-bucket" --policy file://new-policy.json
aws s3api put-bucket-versioning --bucket "tf-remote-bucket" --versioning-configuration Status=Enabled
rm new-policy.json

# Create DynamoDB Table
aws dynamodb create-table \
  --table-name "tf-lock-table" \
  --attribute-definitions AttributeName=LockID,AttributeType=S \
  --key-schema AttributeName=LockID,KeyType=HASH \
  --provisioned-throughput ReadCapacityUnits=1,WriteCapacityUnits=1 \
  --region "us-east-1"

Now, let's configure our main.tf:

terraform {
  required_version = ">= 1.0.0"
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 3.29"
    }
  }

  backend "s3" {
    encrypt        = true
    bucket         = "tf-remote-bucket"
    dynamodb_table = "tf-lock-table"
    region         = "us-east-1"
    key            = "terraform.tfstate"
  }
}

provider "aws" {
  region = "us-east-1"
}

A terraform init outputs:

Terraform has been successfully initialized!

You may now begin working with Terraform. Try running "terraform plan" to see
any changes that are required for your infrastructure. All Terraform commands
should now work.

If you ever set or change modules or backend configuration for Terraform,
rerun this command to reinitialize your working directory. If you forget, other
commands will detect it and remind you to do so if necessary.

You should be all set to run terraform plan and terraform apply! I hope this comes as a useful tool in your toolbox and/or pipeline. If you have any questions or run into any problems, please let me know in the comments!

A Faster Setup for Remote Terraform States

Joseph Sutton — Wed, 04 Aug 2021 15:03:32 +0000

Assuming you already have your AWS credentials setup already, we need to create two files. The first file is our S3 policy to let Terraform view the items, and to get and update our Terraform state.

Our S3 policy, s3_policy.json:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": "s3:ListBucket",
      "Resource": "arn:aws:s3:::my-bucket",
      "Principal": {
        "AWS": "ARN"
      }
    },
    {
      "Effect": "Allow",
      "Action": [
        "s3:GetObject",
        "s3:PutObject"
      ],
      "Resource": "arn:aws:s3:::my-bucket/terraform.tfstate",
      "Principal": {
        "AWS": "ARN"
      }
    }
  ]
}

So, if you're wondering about the ARN, that's okay. Our script will handle that. You can even go as far as abstracting the Resource out for each Statement item as well!

The second file is our actual bash script, let's say tf-setup.sh (remember to chmod +x this file so you can execute it!):

#!/bin/bash

# Create S3 Bucket
MY_ARN=$(aws iam get-user --query User.Arn --output text 2>/dev/null)
aws s3 mb "s3://tf-remote-bucket" --region "us-east-1"
sed -e "s/RESOURCE/arn:aws:s3:::tf-remote-bucket/g" -e "s/KEY/terraform.tfstate/g" -e "s|ARN|${MY_ARN}|g" "$(dirname "$0")/s3_policy.json" > new-policy.json
aws s3api put-bucket-policy --bucket "tf-remote-bucket" --policy file://new-policy.json
aws s3api put-bucket-versioning --bucket "tf-remote-bucket" --versioning-configuration Status=Enabled
rm new-policy.json

# Create DynamoDB Table
aws dynamodb create-table \
  --table-name "tf-lock-table" \
  --attribute-definitions AttributeName=LockID,AttributeType=S \
  --key-schema AttributeName=LockID,KeyType=HASH \
  --provisioned-throughput ReadCapacityUnits=1,WriteCapacityUnits=1 \
  --region "us-east-1"

Now, let's configure our main.tf:

terraform {
  required_version = ">= 1.0.0"
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 3.29"
    }
  }

  backend "s3" {
    encrypt        = true
    bucket         = "tf-remote-bucket"
    dynamodb_table = "tf-lock-table"
    region         = "us-east-1"
    key            = "terraform.tfstate"
  }
}

provider "aws" {
  region = "us-east-1"
}

A terraform init outputs:

Terraform has been successfully initialized!

You may now begin working with Terraform. Try running "terraform plan" to see
any changes that are required for your infrastructure. All Terraform commands
should now work.

If you ever set or change modules or backend configuration for Terraform,
rerun this command to reinitialize your working directory. If you forget, other
commands will detect it and remind you to do so if necessary.

You should be all set to run terraform plan and terraform apply! If you have any questions or run into any problems, please let me know in the comments!