Part 3: Developing stock-checker app

This post is part of a series detailing my journey with golang from learning the language to entering the DigitalOcean App Platform Hackathon.

The app I built can be found on GitHub.

Part 3 details:

• golang packages used to build stock-checker
• docker setup for both development and production
• Integrating with Twilio SMS API

Overview

stock-checker is a golang app that spawns a task every hour to check whether any of the following UK retailers have an Xbox Series X console in stock:

• Argos
• Amazon
• Currys
• Game
• John Lewis
• Shopto
• Smyths

This is done using a web automation library that navigates each site in a headless chromium browser.

If the app determines stock is available, a text message is sent using Twilio to the configured mobile number.

golang packages

A number of different packages were used to build this app which are detailed in this section.

These packages were discovered on Awesome Go.

Web Automation - go-rod/rod

rod is a high-level driver for the DevTools Protocol.

This package is used to navigate the various retailer's sites to find the Xbox Series X console listing. Once found, it checks whether the console is in or out of stock.

Notable features include:

• Bypass bot detection
• Custom browser launch
• Concurrent execution using PagePool

Configuration - spf13/viper

Viper is a configuration package that supports reading configuration files in various formats, as well as from environment variables.

This is being used to control:

• The scheduler interval
• Debug settings for rod
• Enables/disables sending notifications and the secrets used to communicate with Twilio.

I used an envfile to define the app's configuration. Originally, I tried using a YAML file but due to a couple of issues, I had to swap the file format.

To help others, here are the issues I ran into when using YAML:

• GitHub Issue: If you want to override a YAML subobject property with an environment variable, you have to name the variable using the following format: PREFIX_SECURE.KEY. DigitalOcean App Platform environment variables do not support . in the name.
• GitHub Issue: If you want to override a YAML subobject property with an environment variable and unmarshal that object to a struct, this isn't currently possible. This is because viper.UnmarshalKey doesn't take account of environment variables.

Scheduler - go-co-op/gocron

goCron is a scheduling package that lets you run functions periodically at a pre-determined interval.

This is being used to run a task once an hour that checks the various retailers and if applicable, sends a notification.

docker support

To simplify running the app on DigitalOcean, I added docker support which this section covers.

Dockerfile

This amazing JetBrains blog post series greatly inspired the Dockerfile created for this project.

Using multi-stage builds it provides both a development image with debugging support and a production image.

# Builder
FROM golang:1.15.6-buster AS builder
COPY . /src
WORKDIR /src
RUN go get github.com/go-delve/delve/cmd/dlv
RUN go build -gcflags="all=-N -l" -o app-dev
RUN go build -o app

# Base runner
FROM debian:10.7 AS base-runner
RUN apt-get update && apt-get install -y \
    ca-certificates \
    chromium \
    && rm -rf /var/lib/apt/lists/*

# Dev runner
FROM base-runner AS dev-runner
WORKDIR /server
COPY config.env /server
COPY --from=builder /src/app-dev /server/app
COPY --from=builder /go/bin/dlv /server
EXPOSE 40000
CMD ["/server/dlv", "--listen=:40000", "--headless=true", "--api-version=2", "--accept-multiclient", "exec", "/server/app"]

# Prod runner
FROM base-runner AS prod-runner
WORKDIR /server
COPY config.env /server
COPY --from=builder /src/app /server
CMD ["./app"]

docker-compose

To assist with development, this docker-compose file starts a development container with debugging enabled and enables a few rod debug options via environment variables.

version: "3.9"

services:
  app:
    build:
      context: ./..
      target: dev-runner 
    security_opt:
      - seccomp:unconfined
    cap_add:
      - SYS_PTRACE
    container_name: stock-checker-$USER

    ports:
      - "40000:40000" # DEBUG

    environment:
      - SC_ROD_TRACE=true
      - SC_ROD_PAGEPOOLSIZE=1

Remote debugging using VS Code

To remote debug the app within docker, we need to create a launch configuration by following this guide.

The resulting config looks like this:

{
    "version": "0.2.0",
    "configurations": [
        {
            "name": "Launch and debug in Docker",
            "type": "go",
            "request": "attach",
            "mode": "remote",
            "cwd": "${workspaceFolder}",
            "port": 40000,
            "host": "127.0.0.1",
            // This corresponds to the directory the app is built in
            "remotePath": "/src"
        },
    ]
}

Integrating with Twilio

Twilio is being used to send a SMS when the app detects a Xbox Series X console is in stock.

Originally, I was going to use saintpete/twilio-go which I discovered here. However I ran into issues with this package due to incompatible package versions.

In the end, I went with a simpler approach and ended up following this blog post to integrate with their SMS API directly.

Next

That's it for this post.

In the final part, I will detail my DigitalOcean App Platform Hackathon submission as well as deploying to DigitalOcean.