DEV Community: Kyle J. Kemp

A Better Deployment Process (Monorepos, CapRover, and Netlify)

Kyle J. Kemp — Sat, 12 Dec 2020 19:56:07 +0000

Deployment, one of the first pipelines to set up and one of the most important pipelines to have. I recently set up a really handy deployment process with Github Actions and wanted to share more, for anyone who finds themselves in a similar situation. Here's an overview of my situation:

I have a monorepo, which makes deployment somewhat harder than I had hoped
I wanted to deploy the backend to a private server with CapRover (this post will assume that CapRover is set up already)
I wanted to deploy the frontend to Netlify, without building every time the backend updates
I wanted the ability to deploy independently, together, manually, and automatically

For reference, I'll share a link to my project so it'll be easy to look up the finer points of my implementation.

A few problems I ran into upfront:

CapRover makes deployment somewhat difficult if you have a monorepo, and I personally dislike vendor-specific files
Netlify makes deployment somewhat difficult if you have a monorepo (as well), and I simply could not get their build.ignore settings to work, and the vendor-specific files too
Docker

Client Action

So, first, lets work on the manual process for Netlify w/ GitHub Actions, since it's so much easier. Thankfully, there's a Netlify action that makes this easy:

- name: Deploy to Netlify
  uses: nwtgck/actions-netlify@v1.1
  with:
    publish-dir: './client/dist'
    production-branch: master
    github-token: ${{ secrets.GITHUB_TOKEN }}
    deploy-message: "Deploy from GitHub Actions"
    enable-pull-request-comment: false
    enable-commit-comment: true
    overwrites-pull-request-comment: true
  env:
    NETLIFY_AUTH_TOKEN: ${{ secrets.NETLIFY_AUTH_TOKEN }}
    NETLIFY_SITE_ID: ${{ secrets.NETLIFY_SITE_ID }}

Of course, this assumes the app is built to client/dist first, but from there, it's straightforward - turn off Netlify auto deploys, since this will cover it. Here's my full action for client side manual deployments.

Server Action

The server action, for me, was much more difficult since I'm not well-versed with Docker. First, I had to come up with a Dockerfile (this is the easiest way to deploy to CapRover, IMO). I had some issues trying to find one that would work with my old version of cWS, turns out I had to use node 13.14 specifically. But that's neither here nor there, here's the Dockerfile I came up with (current version here):

FROM node:13.14.0-alpine
RUN apk add --no-cache libc6-compat
RUN ln -s /lib/libc.musl-x86_64.so.1 /lib/ld-linux-x86-64.so.2
RUN mkdir -p /usr/src/app
WORKDIR /usr/src/app
ADD https://www.google.com /time.now
COPY ./package.json /usr/src/app
COPY ./package-lock.json /usr/src/app
COPY ./tsconfig.json /usr/src/app
COPY ./server /usr/src/app/server
COPY ./shared /usr/src/app/shared
RUN npm install
RUN cd server && npm run setup && npm cache clean --force
RUN cd server/content && npm install --unsafe-perm
RUN cd server && npm run build
ENV NODE_ENV production
ENV PORT 80
EXPOSE 80
CMD cd server && npm start

I was running into some weird issues with needing a particular native function from ld-linux-x86-64.so.2, which alpine doesn't have. I searched around and found that I can symlink a different .so file and make this work, so that was one problem sorted out.

After that, I ran into some issues installing a sub-repo that gets cloned, which threw some weird error that ended up being an npm issue, fixed by adding --unsafe-perm to that npm install. Everything else is pretty straightforward for people who use docker, I imagine.

One might notice I structure things a bit differently than most. I think in other Dockerfiles I've seen copying a single package.json file, installing, and going from there. Here, I had to copy server/, shared/, and the root package.json file to get the install process correct.

From there, the action is really straightforward, but note that it must be run on linux. It looks like this:

- name: Deploy Server
  uses: AlexxNB/caprover-action@v1
  with:
    server: 'https://captain.server.rair.land'
    password: '${{ secrets.CAPROVER_PASSWORD }}'
    appname: 'game'

This action is yet a bit more convoluted because it does a build before it starts doing anything in the Dockerfile, then it does it again in the Dockerfile, but the time overhead isn't significant enough for me to want to fix it. Here's my full action for server deployments.

Automatically Deploying

The two actions above handle the manual deployments, if needed. Seeing the full files shows how to set up manual deployments with GitHub Actions, but the tl;dr is on: workflow_dispatch.

Now, that's nice to have for testing, hotfixing, or whatever the case may be, but in general this is something worth automating. Thankfully, GitHub Actions is super flexible! It's really easy to compose the two manual tasks above into something that only deploys on tags (the full action is here):

name: Release New Tags

on:
  push:
    tags:
      - 'v*' # Any pushed tag

jobs:
  build:
    name: Create Release

    runs-on: ${{ matrix.os }}

    strategy:
      matrix:
        # os: [macos-latest, ubuntu-latest, windows-latest]
        os: [ubuntu-latest]
        node-version: [15.x]

    steps:
    - uses: actions/checkout@v2

    - name: Use Node.js ${{ matrix.node-version }}
      uses: actions/setup-node@v1
      with:
        node-version: ${{ matrix.node-version }}

    - run: npm install

    - run: npm run lint

    - run: npm run build

    - name: Deploy Server
      uses: AlexxNB/caprover-action@v1
      with:
        server: 'https://captain.server.rair.land'
        password: '${{ secrets.CAPROVER_PASSWORD }}'
        appname: 'game'

    - name: Deploy to Netlify
      uses: nwtgck/actions-netlify@v1.1
      with:
        publish-dir: './client/dist'
        production-branch: master
        github-token: ${{ secrets.GITHUB_TOKEN }}
        deploy-message: "Deploy from GitHub Actions"
        enable-pull-request-comment: false
        enable-commit-comment: true
        overwrites-pull-request-comment: true
      env:
        NETLIFY_AUTH_TOKEN: ${{ secrets.NETLIFY_AUTH_TOKEN }}
        NETLIFY_SITE_ID: ${{ secrets.NETLIFY_SITE_ID }}

This would automate client and server deployments on tags, but it could easily be changed to on push, or on any event GitHub Actions supports.

Making a Reddit Reply Bot

Kyle J. Kemp — Thu, 25 Apr 2019 15:57:55 +0000

There's a lot of documentation on writing a reddit bot in Python, but I had a lot of trouble finding even basic documentation for Node - even some of the libraries that are listed on reddits official wiki are dead or 5 years old (read: don't support new reddit very well). So, I wanted to write about a simple and common use-case: replying to a user who tags you.

Creating a Reddit Application

First, head on over to https://www.reddit.com/prefs/apps and hit "create app" - you need to do this so reddit isn't using your personal user account. You should also sign up for a new reddit account for your bot (especially if it can be summoned). Make sure you add your main account and bot account as developers on this application.

When creating an app, you need to fill in the field similarly to this:

You don't need to worry about the about uri or redirect uri fields, as you won't be using them, so you can put in whatever you would like.

Once in, you'll see a screen like this:

Take note of this, because you'll need this information in just a second.

Starting the Node Project

For something like this, I found it very easy to use snoostorm (a wrapper around snoowrap). This library makes it exceptionally simple to get comments as they come in.

You first need to make a snoowrap object, then use that to make a CommentStream object. To do that, you'll need your reddit bot username, password, application secret, and application id. You must specify a unique user-agent, so call it something like my-node-js-bot. So, configure it like so (mine is configured based on the picture above):

const Snoowrap = require('snoowrap');
const { CommentStream } = require('snoostorm');

const client = new Snoowrap({
    userAgent: 'my-node-js-bot',
    clientId: 'qR6rJnQ7sEJZDw',
    clientSecret: 'OCoo9pYnlC2K6fxQQxbcIPQ5MA4',
    username: 'myusernamebutactuallybot',
    password: 'mypasswordbutactuallybot'
});

With this client object, you can finally start listening for new comments! Head over to /r/testingground4bots and hop into a thread or make your own. Then, add some code to start watching for comments:

// pollTime is 10000 because reddit is very strict on posting too frequently
// at first, you'll only be able to post once every 10 minutes, so make sure you get it right!
const comments = new CommentStream(client, { 
    subreddit: 'testingground4bots', 
    limit: 10, 
    pollTime: 10000 
});

comments.on('item', (item) => {
    console.log(item);
});

Start the bot up, and you'll see a flood of comments in your terminal. You may be wondering why that is - you haven't even seen any new ones come in yet! Well, the client will always give you the first X entries (in this case, 10) when you start your bot up, and then it will track from there.

We can fix that pretty easily:

// reddits api doesn't use millis
const BOT_START = Date.now() / 1000;

const comments = new CommentStream(client, { 
    subreddit: 'testingground4bots', 
    limit: 10, 
    pollTime: 10000 
});

comments.on('item', (item) => {
    if(item.created_utc < BOT_START) return;

    console.log(item);
});

Great, now you only see the newest comments as they come in. Hopefully you have an established enough reddit account to make some posts on this subreddit. If you do so, you'll see your terminal fill with them fairly quickly after posting.

Making It Interact

So far, you've got a bot and it reads comments - that's a great start! But you want it to interact with your audience, right? So, how about a good ol' hello world? Check it out:

const BOT_START = Date.now() / 1000;

const comments = new CommentStream(client, { 
    subreddit: 'testingground4bots', 
    limit: 10, 
    pollTime: 10000 
});

comments.on('item', (item) => {
    if(item.created_utc < BOT_START) return;

    item.reply('hello world!');
});

There, any time a comment comes in, the bot will reply to it with "hello world!" Wait... that might talk a bit often, won't it? It might get just a little bit annoying. Reddit recommends replying specifically when your bot is mentioned, so, there's a fairly easy way to do that:

const BOT_START = Date.now() / 1000;

const canSummon = (msg) => {
    return msg && msg.toLowerCase().includes('/u/myusernamebutactuallybot');
};

const comments = new CommentStream(client, { 
    subreddit: 'testingground4bots', 
    limit: 10, 
    pollTime: 10000 
});

comments.on('item', (item) => {
    if(item.created_utc < BOT_START) return;
    if(!canSummon(item.body)) return;

    item.reply('hello world!');
});

There! So, what this does is make sure that the comment your bot finds actually refers to the bot itself. This canSummon function will do basic checking to make sure your bot doesn't erroneously spam a bunch of peoples comments. Make a comment now that says /u/myusernamebutactuallybot (rather, you should check for your own bots name), and you should see a reply shortly afterwards that says "hello world!" in reply.

That's all you gotta do! 🎉

Introducing: Inveinta - Another Inventory Management Tool

Kyle J. Kemp — Wed, 31 Jan 2018 16:46:48 +0000

A few weeks ago I was feeling unproductive, and for me the best way to quash that feeling is to hammer out a small project from start to finish. A few people were talking about needing a solution to manage large amounts of inventory + some other features like collaborative editing and checking in and out their stuff. I've wanted to make this for a while but now was the time to hammer out it. So I spun up a quick Ionic project and started to learn how to use the new Firestore tech (a vast improvement over the old Realtime Database IMO) and got to work.

My idea was to make it so you use or create "mixins" to categorize your lists of items, using sets of fields to define your list. So, if you want to have a list of MTG trading cards for sale, you'd pick "Magic the Gathering" "for sale". If you wanted to have a check in/check out system for video games, you'd pick "Video Games" "Library". Of course, you can make your own mixins as well - there's no way to cover every possible need.

You can see the end result here, but since that's just the app (I haven't gotten time to make a "proper" site for it), here's some examples:

As well, here are some pics of the tool in action.

What do you think?