DEV Community: Welcoming Sloth

Quickly create a local postgres database with docker

Welcoming Sloth — Mon, 12 Jun 2023 13:38:37 +0000

Here are some quick package.json scripts to create, remove, start, and stop a local docker postgres database.

{
  "scripts": {
    "db:create": "docker run --name DB_NAME -e POSTGRES_PASSWORD=postgres -p DB_EXTERNAL_PORT:5432 -d postgres",
    "db:remove": "docker rm DB_NAME",
    "db:start": "docker start DB_NAME",
    "db:stop": "docker stop DB_NAME"
  }
}

Used env vars:
DB_NAME: Name for the docker container to create.
DB_EXTERNAL_PORT: What local port the database should be accessible from. Use 5432 for default pg port.

A dependency to deploy Next apps to GitHub Pages ⚡

Welcoming Sloth — Sat, 06 May 2023 17:29:15 +0000

Recently I made a small tool named neght, heavily inspired by storybook-deployer, to help developers deploy their static Next.js apps to GitHub Pages.

Preview

What is Neght 💭

Neght is a tool / dependency you can add to your project to easily deploy it to GitHub Pages. It builds the project in a way that's compatible with GitHub Pages and then pushes it to a designated branch.

How it works 👩‍🔧

First, configure your Next app to be static site, which consists of adding output: 'export' to your next config.

Then, add neght as a dependency (yarn add neght).

Finally, add neght as a script to your package.json.

"scripts": {
  "build": "next build",
  "deploy": "neght --script build --branch gh-pages"
}

That's it! Now when you run the deploy script neght will take care of building your Next project and pushing it to gh-pages branch of your repository.

All that remains for you to do is to make sure that GitHub serves the gh-pages branch through GitHub Pages and you're all done.

Links 🔗

GitHub: https://github.com/Tomburgs/neght
npm: https://www.npmjs.com/package/neght

Generative art, flow fields, and books

Welcoming Sloth — Wed, 31 Aug 2022 10:45:30 +0000

Backstory

A few weeks ago, while working on ReadShape, I was faced with a not very surprising issue. Some books don't have covers. Never did, never will.

How is this possible? Well, when you walk into a book store of course all the books will have covers, even if just to keep all the paper together. But we live in the digital age, where anyone can publish a book without a publisher and in some cases without using proper grammar. In such cases you will often times encounter books without a cover.

This creates a somewhat unique problem. Of course, you can go with a simple solution and just display a cover-not-found.jpeg, but that's not over-engineered enough.

This led me down the colorful & artsy path of generative art.

What is generative art?

Essentially, generative art is art which was generated by a machine using an algorithm.

The cool thing about it is that it can output wildly different results each time it's ran.

Examples

This is some generative art from Tyler Hobbs titled "Fidenza". Each a different invocation of the same algorithm.

Here is a different algorithm from Kjetil Golid titled "Tilework".

Flow fields

As could be seen from the examples, there's a fair bit of variance in output that each algorithm can produce.

In many cases the algorithm needs to produce some amount of overall uniformity. The Fidenza output would not look nearly as good if each line would go in a new direction randomly.

To have this overall uniformity often times something called a "flow field" (also known as "vector field") is used. Essentially what it does is it creates a grid where each vector contains a direction.

It is more easy to understand with the following picture.

Now that each point sets you going in some direction you simply pick a point and follow the arrows for N steps.

Generating a flow field is something I won't go into, but if you're curious about learning more have a look at this post from Tyler Hobbs.

Book covers

Now, back to why we're going down this rabbit hole in the first place — books.

Generative art seems to be just the thing I'm looking for. However, something that still has to be kept in mind is that for my use case, given the same inputs, I always want to return the same output. This is simply so that I could save on my S3 bill and generate these covers on the fly.

This means a few things — no random values, no unnecessary dependencies, and it has to be fast.

What I ended up doing was creating a murmur hash based on the input arguments, because murmur is fast and always returns a uint32 value. Perfect for doing probability-based decisions, creating rgb values, and good enough for creating a flow field.

If you're interested in viewing the source code, have a look here: https://github.com/ReadShape/covers

Here are is an example of the outcome

It's simple, fast, and contains a good amount of variety. You can play around with it here.

Conclusion

I hope you learned something new today, and hopefully you'll be able to see these covers in action soon on ReadShape.

You're also welcome to contribute to the GitHub repository.

— T

Tasks & Advisory Locks

Welcoming Sloth — Mon, 11 Jul 2022 19:01:49 +0000

This post is on how to utilize postgres advisory locks to create introspectable asynchronous job orchestration between microservices

Pretext

Recently I found myself with a task which required creating asynchronous, introspectable, jobs.

Now, this issue is nothing new. In fact, there's plenty of really good solutions out there. This can be easily achieved with tools such as kafka, mqs, there's even tools like cadence which take this to the next level.

However, while these are all very good solutions, they can also be too much. What if all you have is a single postgres instance? Well, that's the situation I was in, and the solution was fairly simple — advisory locks!

Before I go further, let me explain the situation I was in.

Situation

Lets say a user request something from your system which requires some process to be completed in the background.

It could take a minute, or an hour, or 3 months. No matter how long the task takes, the point is the response cannot be returned immediately.

In a simple world

In this example the users ask from the system to generate them an invoice.

Creating a database entry in a tasks table would be simple enough, it's in pending state while being worked on, and deleted or set to done when it's finished.

All we really need to do is be able to resume the task if the system running the task crashes or is interrupted.

However, we're running in a microservice environment, and that makes everything that used to be simple an adventure.

The microservice adventure

So what do we do now? We don't want 20 instances performing the same task at the same time.

In reality the "just make a database entry" approach is still valid, but now we need to know whether a process should resume processing any pending tasks once it restarts.

Of course, if a service crashes it won't have the chance to update the entry to say "available for pickup".

A simple solution, if you're using postgres, is to use advisory locks.

Advisory locks

PostgreSQL provides a means for creating locks that have application-defined meanings. These are called advisory locks, because the system does not enforce their use — it is up to the application to use them correctly.

— From Postgres Documentation

Essentially what advisory locks allow you to do is to create a flag without actually locking any resources.

In the case of transaction level advisory locks the lock will go away when you
A. Commit the transaction
B. Rollback the transaction (which includes the client disconnecting)

This is great for us, because we don't actually want to lock the table, as that would prevent any other tasks from using it. It also means the advisory lock will go away if the process crashes.

Here's how we can utilize it

This looks great, but in practice, there's a few gaps we need to fill.

One of them is that each advisory lock requires a bigint as its identifier.

If your user IDs are already ints, then this is not an issue (you might only want to namespace them).
If they are IDs, such as KSUID or UUID then you need to hash them into an int. I won't go into this as there's plenty of existing sources already.

We can create a transaction-level lock using the below query

SELECT pg_try_advisory_xact_lock(42) -- 42 is the User ID

Query for checking if a lock exists would look something like this

SELECT granted FROM pg_locks WHERE locktype = 'advisory' AND objid = '42' -- 42 is the LockID which is our User ID

Resume on restart

The one last thing which remains is to pick up the existing tasks upon startup.

It could look something like this.

Essentially, upon restart, query the database for any tasks in pending state and see if there is an advisory lock for any of them. If there isn't one, that means that no one is processing them and it should be picked up!

That's it!

That's all it takes to create simple, asynchronous, interopable tasks.

Pulumi EKS with ALB + RDS

Welcoming Sloth — Fri, 03 Jun 2022 16:02:11 +0000

Introduction

Salutations,

I recently needed to set-up aws infrastructure for my startup.

Unfortunately, google lacked good up-to-date resources for what I was trying to do.

It took a lot of mistakes, hair pulling, and a considerable AWS bill to end up here, which is why I will share my findings with you, so that you don't have to repeat all those same mistakes.

Now, this is the app that we're going to deploy:

Microservices
- Users Service
- GraphQL Service (entry point which accepts HTTP calls, will route queries to the appropriate service)
RDS PostgreSQL Instance
Kubernetes Ingress + AWS Load Balancer Controller

This is purely an API setup, but you could quite easily add frontend to this setup if you wish.

We will also add OIDC access control for our cluster, so that the pods could use other AWS services such as S3 & SQS.

Create a VPC & tag the subnets.

First thing's first, we need to create a VPC and tag the subnets.

We will be tagging our subnet groups, so that the alb controller will be able to use them. You can read more about it here.

const vpc = new awsx.ec2.Vpc('friendly-sloth-vpc', {
  numberOfNatGateways: 0, // We won't be needing NAT gateways.
  // Creation args for subnets
  subnets: [
    {
      type: 'public',
      tags: {
        'kubernetes.io/role/elb': '1',
      },
    },
    {
      type: 'private',
      tags: {
        'kubernetes.io/role/internal-elb': '1',
      }
    }
  ],
});

Create a security group

Next, let's create a security group for our cluster.

We will be allowing all inbound & outbound traffic (which is why fromPort & toPort are -1).

You can of course change this to be more strict, just remember to let the HTTP (80 & 443) & DB (5432 for default PGSQL) calls through.

const clusterSecurityGroup = new aws.ec2.SecurityGroup('clustersecgrp', {
  vpcId: vpc.id,
  ingress: [{
    protocol: 'all',
    fromPort: -1,
    toPort: -1,
    cidrBlocks: ['0.0.0.0/0'],
  }],
  egress: [{
    protocol: 'all',
    fromPort: -1,
    toPort: -1,
    cidrBlocks: ['0.0.0.0/0'],
  }]
});

Create RDS instance

Next, let's create our RDS instance.

We will need to create a subnet group for it.
I will be creating it on the private subnets (because I am very secure).

You can also make it use the public subnets to be able to access it from outside the VPC. Just remember to set the database then to publiclyAccessible: true as well (please don't do this for prod).

const dbSubnetGroup = new aws.rds.SubnetGroup('dbsubnet-group', {
  subnetIds: vpc.privateSubnetIds,
});

const rds = new aws.rds.Instance('friendly-sloth-db', {
  instanceClass: 'db.t2.micro',
  engine: 'postgres',
  engineVersion: '14.2',
  dbName: 'sloths-db',
  username: 'sloth',
  password: 'verysecuresloth',
  allocatedStorage: 20,
  deletionProtection: true,
  dbSubnetGroupName: dbSubnetGroup.id, // assign it to subnet
  vpcSecurityGroupIds: [clusterSecurityGroup.id], // add it to cluster's security group
});

Create the cluster + service account

Lets create the cluster

Remember to set createOidcProvider: true so that we can give the cluster permission to use other AWS services.

const cluster = new eks.Cluster('sloth-cluster', {
  vpcId: vpc.id,
  publicSubnetIds: vpc.publicSubnetIds,
  clusterSecurityGroup: clusterSecurityGroup,
  createOidcProvider: true,
});

const clusterOidcProvider = cluster.core.oidcProvider;

Next, lets create a kubernetes namespace

const namespace = new k8s.core.v1.Namespace(
  'sloth-kubes',
  { metadata: { name: 'sloth-kubes' } },
  { provider: cluster.provider },
);

Now, permission time! (service account)

const saName = 'sloth-service-account';

const saAssumeRolePolicy = pulumi
  .all([clusterOidcProvider?.url, clusterOidcProvider?.arn, namespace.metadata.name])
  .apply(([url, arn, namespace]) => (
    aws.iam.getPolicyDocument({
      statements: [
        {
          actions: ['sts:AssumeRoleWithWebIdentity'],
          conditions: [
            {
              test: 'StringEquals',
              variable: `${url.replace('https://', '')}:sub`,
              values: [`system:serviceaccount:${namespace}:${saName}`],
            },
            {
              test: 'StringEquals',
              variable: `${url.replace('https://', '')}:aud`,
              values: ['sts.amazonaws.com']
            },
          ],
          effect: 'Allow',
          principals: [{ identifiers: [arn], type: 'Federated' }],
        },
      ],
    })
  ));

const saRole = new aws.iam.Role(saName, {
  assumeRolePolicy: saAssumeRolePolicy.json,
});

const saPolicy = new aws.iam.Policy('policy', {
    description: 'Allow EKS access to resources',
    policy: JSON.stringify({
        Version: '2012-10-17',
        Statement: [{
            Action: [
              // Configure which services the SA should have access to.
              // Make it more strict / loose however you see fit.
              'ec2:*',
              'sqs:*',
              's3:*',
            ],
            Effect: 'Allow',
            Resource: '*',
        }],
    }),
});

// Attach the policy to the role
new aws.iam.RolePolicyAttachment(saName, {
  policyArn: saPolicy.arn,
  role: saRole,
});

// Create the kubernetes service account itself
const serviceAccount = new k8s.core.v1.ServiceAccount(
  saName,
  {
    metadata: {
      namespace: namespace.metadata.name,
      name: saName,
      annotations: {
        'eks.amazonaws.com/role-arn': saRole.arn,
      },
    },
  },
  { provider: cluster.provider }
);

Create ingress controller

Now for the fun bit — let's create AWS ALB ingress controller.
We will make pulumi use a helm chart from the official aws alb repository.

First, however, more policies!
You can get the iam policy from the alb repository

curl -o iam-policy.json https://raw.githubusercontent.com/kubernetes-sigs/aws-load-balancer-controller/main/docs/install/iam_policy.json

I created a separate util called createIngressControllerPolicy which looks like this

export const createIngressControllerPolicy = () => (
  new aws.iam.Policy(
    'ingress-ctrl-policy',
    {
      policy: {
        // The JSON you got from above.
      }
    }
  );
);

Now, we can attach it to the same service account role

const ingressControllerPolicy = createIngressControllerPolicy();

new aws.iam.RolePolicyAttachment(`${saName}-ingress-ctrl`, {
  policyArn: ingressControllerPolicy.arn,
  role: saRole,
});

Now that we're done with policies, let's create an ingress class which will connect the controller with the ingress.

const albIngressClass = new k8s.networking.v1.IngressClass('alb-ingress-class', {
  metadata: {
    name: 'sloth-alb',
  },
  spec: {
    controller: 'ingress.k8s.aws/alb',
  }
}, { provider: cluster.provider });

Now, the promised controller!
We will create the controller from a helm chart. We will programmatically create it from the repository using Pulumi.

// Important! Use k8s.helm.v3.Release instead of k8s.helm.v3.Chart.
// Otherwise, you will run into all sorts of issues.
const alb = new k8s.helm.v3.Release(
  'alb',
  {
    namespace: namespace.metadata.name,
    repositoryOpts: {
      repo: 'https://aws.github.io/eks-charts',
    },
    chart: 'aws-load-balancer-controller',
    version: '1.4.2',
    values: {
      // @doc https://github.com/kubernetes-sigs/aws-load-balancer-controller/tree/main/helm/aws-load-balancer-controller#configuration
      vpcId: vpc.id,
      region: 'eu-central-1', // Change to the region of the VPC. Since EU is the best region, we're going with that.
      clusterName: cluster.eksCluster.name,
      ingressClass: albIngressClass.metadata.name,
      // Important! Disable ingress class annotations as they are deprecated.
      // We're using a proper ingress class, so we don't need this.
      // @see https://kubernetes-sigs.github.io/aws-load-balancer-controller/v2.4/guide/ingress/ingress_class/
      disableIngressClassAnnotation: true,
      createIngressClassResource: false, // Don't need it, since we already made one.
      serviceAccount: {
        name: serviceAccount.metadata.name,
        create: false, // Don't need it, since we already made one.
      },
    },
  },
  { provider: cluster.provider }
);

Kubernetes services

Now, lets set up kubernetes deployments & services for our Users & GraphQL API.

Since both are fairly similar, we will create a utility for it.

type KubeServiceOpts = {
  image: docker.Image,
  serviceType?: ServiceSpecType;
};

export const createKubeService = (name: string, opts: KubeServiceOpts): {
  deployment: k8s.apps.v1.Deployment,
  service: k8s.core.v1.Service,
} => {
  const { image, serviceType } = opts;

  const appLabels = { app: name };
  const defaultPort = 4200;

  const deployment = new k8s.apps.v1.Deployment(`${name}-dep`, {
    metadata: {
      namespace: namespace.metadata.name,
      labels: appLabels,
    },
    spec: {
      replicas: 1,
      selector: { matchLabels: appLabels },
      template: {
        metadata: { labels: appLabels },
        spec: {
          serviceAccountName: serviceAccount.metadata.name,
          containers: [{
            name,
            image: image.imageName,
            ports: [{
              name: `${name}-port`,
              containerPort: defaultPort,
            }],
            env: [
              { name: 'AWS_DEFAULT_REGION', value: 'eu-central-1' }, // Again, best region there is.
            ]
          }],
        },
      },
    },
  }, { provider: cluster.provider });

  const service = new k8s.core.v1.Service(`${name}-svc`, {
    metadata: {
      name,
      namespace: namespace.metadata.name,
      labels: appLabels,
    },
    spec: {
      ...(serviceType && { type: serviceType }),
      selector: appLabels,
      ports: [
        {
          name: `${name}-http`,
          port: 80,
          targetPort: defaultPort,
        },
        {
          name: `${name}-https`,
          port: 443,
          targetPort: defaultPort,
        }
      ]
    },
  }, { provider: cluster.provider });

  return { deployment, service };
};

Now, lets use it

createKubeService('graphql', {
  serviceType: 'NodePort', // ALB needs the service which will be receiving the traffic to be of type `NodePort`.
  image: myGraphqlServiceImage,
});

createKubeService('users', {
  image: myUsersServiceImage,
});

Finally, the ingress itself!

Now, for the final part, lets create the ingress itself!
Now that we have our ingress controller & services, we can go ahead and create the ingress itself.

We will be using only HTTP in this example, to use HTTPS you will need a certificate which you can specify with the annotation alb.ingress.kubernetes.io/certificate-arn. Then, specify the https ports for the paths.

const apiIngress = new k8s.networking.v1.Ingress('api-ingress', {
  metadata: {
    name: 'api-ingress',
    namespace: namespace.metadata.name,
    labels: {
      app: 'api',
    },
    annotations: {
      // @doc https://kubernetes-sigs.github.io/aws-load-balancer-controller/v2.4/guide/ingress/annotations/#annotations
      'alb.ingress.kubernetes.io/target-type': 'ip',
      'alb.ingress.kubernetes.io/scheme': 'internet-facing',
      'alb.ingress.kubernetes.io/backend-protocol': 'HTTP',
    },
  },
  spec: {
    // Important! Connect it to our Ingress Class.
    ingressClassName: albIngressClass.metadata.name,
    rules: [
      {
        http: {
          paths: [
            {
              pathType: 'Prefix',
              path: '/graphql',
              backend: {
                service: {
                  name: 'graphql',
                  port: {
                    name: 'graphql-http', // We named our service ports in `createKubeService` function.
                  }
                },
              }
            },
          ]
        }
      }
    ],
  }
}, { provider: cluster.provider, });

That's it!

Now we can export the url for our load balancer to see it in the console when we deploy.

export const url = apiIngress.status.loadBalancer.ingress[0].hostname