I log, you log, we log for the love of logs

CK Lee — Thu, 04 Jun 2020 10:41:49 +0000

The number of Docker containers deployed in our production boxes have steadily grown over the past 3 years. Last count - 284 containers running. I love logging but I'm shy to say we only just implemented centralised logging for these Docker containers yesterday.

We are a profitable bootstrapped startup, and we don't have VC's money bags 💰 and unicorns 🦄 to chase. At TAGGUN, our top priority is to serve our customers by building receipt and invoice OCR scanning API that doesn't suck. So, the power of scarcity has deemed centralised logging as a YAGNI until a few days ago.

I wished I had a cheatsheet to implement centralised logging for Docker containers. So, I'm sharing it here with you to save you time and money.

How is it done?

Jump to the Docker host
Create or Modify the daemon.json found at /etc/docker
You may need to sudo -i first to access this area as the root user.
Restart the docker daemon (this will take this box down for about 15-20 seconds) sudo systemctl restart docker
Verify the docker logger docker info --format '{{.LoggingDriver}}'
docker ps (This is to validate that docker has restarted without issues)

What configuration for daemon.json?

{
    "log-driver": "syslog",
    "log-opts": {
        "labels": "docker-<machine_name>",
        "tag": "docker-<machine_name>[{{.ImageName}}/{{.Name}}]",
        "mode": "non-blocking"
    }
}

Breakdown

log-driver is built in logging driver for docker; syslog is the one we're using because many server monitoring agents can easily read them from Syslogs.
labels don't get logged to syslog it seems; left in because of docker best practices.
tag does get logged to syslog. This is how you can let your log viewers knows what the application name, and processes are.
mode is set to non-blocking which means a memory buffer is used for logging to syslog. if the log backpressure exceeds a certain point, logs will simply be dropped from this buffer, rather than blocking.

My recommended logging as a service platforms

Site24x7
Loggly
Sematext
~~Elastic Cloud~~ (I love Elasticsearch and ELK stack. But my managed cluster is down too frequently)

[Cover photo by Andrew Ridley on Unsplash]

Making Command-line interface (CLI) for fun and profit

CK Lee — Fri, 24 Jan 2020 09:54:16 +0000

Next time you have an idea 💡 "I know... Why don't I write a script to automate this thing?" Actually, you should not, you should write a CLI instead.

CLI has better structure than scripts. CLI uses flags and help text to avoid mistakes. For example:

$ moa --help
Moa makes it easier to run your local node.js apps.

USAGE
  $ moa

OPTIONS
  -c, --commands=commands  [default: start] commands to run.
  -h, --help               show CLI help
  -r, --root=root          [default: ../] root path to run.

DESCRIPTION
  ...
  Have some fun

Scripts are messy. And can get out of control.

At Taggun, we run a lot of node.js application. We make APIs for receipt OCR scanning, by the way. So, I recommend oclif: The Open CLI Framework by Heroku.

How to create a new CLI

Install node.js first.

Start by scaffolding

$ npx oclif single moa
? npm package name (moa): moa
$ cd moa

Modify your flags in `src\index.js`

MoaCommand.flags = {
  // add --help flag to show CLI version
  help: flags.help({char: 'h'}),
  commands: flags.string({char: 'c', description: 'commands to run. Comma-separated.', default: 'start'}),
  root: flags.string({char: 'r', description: 'root path to run.', default: '../'}),
}

Modify your description in `src\index.js`

MoaCommand.description = `Moa makes it easier to run your local node.js apps.

Modify your command in `src\index.js`

class MoaCommand extends Command {
  async run() {
    const {flags} = this.parse(MoaCommand)
    const commands = flags.commands.split(',')
    const {root} = flags
    find.file(/(?<!node_modules\/.*)\/package\.json$/, root, files => {
      files.forEach(async file => {
        const cwd = require('path').dirname(file)
        commands.forEach(async command => {
          this.log(cwd, `npm run ${command}`)
          const npm = spawn('npm', ['run', command], {cwd})
          for await (const output of npm.stdout) {
            this.log(`${file}: ${output}`)
          }
        })
      })
    })
  }
}

In this example, you will need to run npm i find to install find npm module.

This simple CLI will look for all node.js application in the [root] and run npm run [command]. Great if you need to run multiple node.js application to start coding.

Run it

./bin/run -c start -c ../taggun/

Publish and share your CLI

See https://oclif.io/docs/releasing

DEV Community: CK Lee