DEV Community: jhot

Dear Companies, Stop Mucking With Our Computers

jhot — Fri, 19 May 2023 16:50:00 +0000

This post is partially a rant, partially a cautionary tale, and probably a PSA.

This morning I was just getting into my work day when I lost connection to the work VPN for some reason. This happens on occasion and I didn't think much of it, but upon trying to reconnect I get an error reaching the VPN server. I reach out to a coworker on Slack (whatever websocket or pub/sub it uses was still connected) and they aren't having any issues and my other devices are fine so I just decided to do a reboot and see if that fixes things.

The company I work for uses a service called Mosyle for mobile device management and it enforces two-factor auth on our Macbooks by blocking the entire screen after login and forcing us to login (again) to our SSO provider and complete two-factor auth before the UI is usable. The unfortunate part is that if you have no internet or some sort of internet issue, you lose UI access to your machine and there's nothing you can do about it (there's probably a way to dig in and disable all this and I've tried some things but don't want to fully brick my work laptop).

So PSA #1: if your company uses this software or something similar, don't shut down if you think you won't have internet. Just put it to sleep unless you absolutely have to turn it off. This also defeats most of the purpose of the two-factor software since it doesn't run on login from lock/sleep.

Luckily for me, I had taken a crucial step that allowed me to still do things on my machine, even with this full screen prompt in the way. You may have noticed that I kept mentioning UI access being blocked, that's because SSH access is unrestricted. You see, by the time you see the UI blocker you've already logged in and the OS is running is the background you just can't see it. I had enabled SSH access on the Macbook (Settings > General > Sharing > Remote Login) previously and without this important step I would have been up a creek. So PSA #2 is to enable SSH access on your computer (if it's Windows...someone will have to chime in with a solution) and get familiar with some terminal basics if you aren't already.

Once I was in I had a pretty good idea what had happened. Since I was connected to WiFi and was able to do things on the local network, just not most things on the internet (and Slack was still working before the reboot), I figured it was a DNS issue. So I did a quick cat /etc/resolv.conf and saw that my DNS servers had been hijacked by the work VPN, set to non-public servers, and not restored when I lost connection. So I just had to networksetup -setdnsservers Wi-Fi dns-ip-1 dns-ip-2 and I was back in business. If you're on Linux there are a number of ways to change your DNS and it will depend on your distro, version, and other factors so just look it up.

Rant

So what if I was someone who didn't know about all these acronyms I've used in this post? I would have to find a way to contact my IT staff who may or may not be able to remote access my machine, and otherwise just send in my laptop to HQ for them to troubleshoot and fix. What if I had deadlines? Company policy would also prevent me from using another one of my computers for work things, so I would be out of luck for like a week.

I totally get protecting company data, but putting extra barriers for people with physical access to our machines doesn't do much and mostly just harms the employees. Yes, enforce device encryption. Yes, enforce strong passwords. Yes, mess with people who don't lock their computers when not in use. Yes, enforce two-factor auth on company websites/services. BUT, an attacker using physical access to laptops is one of the least likely attack vectors, and if they have physical access, using something like a screen blocker isn't going to stop them.

Homebrew and Private GitHub Repositories

jhot — Thu, 27 Jan 2022 17:22:24 +0000

I recently developed an internal CLI tool for my organization to use and, of course, wanted to make it easy for the other devs to install. That means I wanted to make it installable from Homebrew, since most, if not all, of my colleagues use it to manage their installed applications. I was able to tap a private repo by setting the HOMEBREW_GITHUB_API_TOKEN environment variable with a GitHub access token, but I was getting 404 errors from curl when installing. The documentation and information I could find by searching was either non-existent or outdated, so I figured I would get what worked for me out there for others to use.

Background

The tool I wrote is a little CLI tool written in go. When I tag a commit on main with a version semver, our CI tool uses GoReleaser to build binaries for various architectures, create a GitHub release, and update the Homebrew formula. GoReleaser is definitely not necessary, but makes things incredibly easy. Here's my .goreleaser.yaml for reference:

before:
  hooks:
    # You may remove this if you don't use go modules.
    - go mod tidy
builds:
  - env:
      - CGO_ENABLED=0
    goos:
      - linux
      - windows
      - darwin
    goarch:
      - amd64
      - arm64
    ignore:
      - goos: windows
        goarch: arm64
archives:
  -
    replacements:
      amd64: x86_64
      darwin: Darwin
      linux: Linux
    format_overrides:
      - goos: windows
        format: zip
brews:
  -
    tap:
      owner: myorg
      name: myrepo
    download_strategy: GitHubPrivateRepositoryReleaseDownloadStrategy
    custom_require: "lib/custom_download_strategy"
    commit_author:
      name: My Name
      email: my.name@my.org
    folder: HomebrewFormula
checksum:
  name_template: 'checksums.txt'
snapshot:
  name_template: "{{ incpatch .Version }}-next"
changelog:
  sort: asc
  filters:
    exclude:
      - '^docs:'
      - '^test:'

I did not create a separate repo for the Homebrew formula and instead just put the formula in the tool's repo in a directory named HomebrewFormula. This makes the tap command a bit longer, but I'm fine with that so I don't have to have an extra repo for each tool I want to deploy with Homebrew.

GoReleaser creates gzips for each platform/arch (zip for Windows) named like ${toolname}_${semver_without_leading_v}_${platform}_${arch}.tar.gz. This is important for our Homebrew download strategy.

The Homebrew Sauce

For whatever reason, you can't curl a release asset from a private repo even with a valid access token. So we need some code to use the GitHub API to get the asset's API URL. This comes in the form of a custom download strategy.

I found some old code that did just what's needed but it used some Homebrew functions that have moved or changed. So after some more digging, trial, and error I was able to get it working with the current version of Homebrew (3.3.12 as of writing this).

HomebrewFormula/lib/custom_download_strategy.rb

require "download_strategy"

# S3DownloadStrategy downloads tarballs from AWS S3.
# To use it, add `:using => :s3` to the URL section of your
# formula.  This download strategy uses AWS access tokens (in the
# environment variables `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY`)
# to sign the request.  This strategy is good in a corporate setting,
# because it lets you use a private S3 bucket as a repo for internal
# distribution.  (It will work for public buckets as well.)
class S3DownloadStrategy < CurlDownloadStrategy
  def initialize(url, name, version, **meta)
    super
  end

  def _fetch(url:, resolved_url:, timeout:)
    if url !~ %r{^https?://([^.].*)\.s3\.amazonaws\.com/(.+)$} &&
       url !~ %r{^s3://([^.].*?)/(.+)$}
      raise "Bad S3 URL: " + url
    end

    bucket = Regexp.last_match(1)
    key = Regexp.last_match(2)

    ENV["AWS_ACCESS_KEY_ID"] = ENV["HOMEBREW_AWS_ACCESS_KEY_ID"]
    ENV["AWS_SECRET_ACCESS_KEY"] = ENV["HOMEBREW_AWS_SECRET_ACCESS_KEY"]

    begin
      signer = Aws::S3::Presigner.new
      s3url = signer.presigned_url :get_object, bucket: bucket, key: key
    rescue Aws::Sigv4::Errors::MissingCredentialsError
      ohai "AWS credentials missing, trying public URL instead."
      s3url = url
    end

    curl_download s3url, to: temporary_path
  end
end

# GitHubPrivateRepositoryDownloadStrategy downloads contents from GitHub
# Private Repository. To use it, add
# `:using => :github_private_repo` to the URL section of
# your formula. This download strategy uses GitHub access tokens (in the
# environment variables `HOMEBREW_GITHUB_API_TOKEN`) to sign the request.  This
# strategy is suitable for corporate use just like S3DownloadStrategy, because
# it lets you use a private GitHub repository for internal distribution.  It
# works with public one, but in that case simply use CurlDownloadStrategy.
class GitHubPrivateRepositoryDownloadStrategy < CurlDownloadStrategy
  require "utils/formatter"
  require "utils/github"

  def initialize(url, name, version, **meta)
    super
    parse_url_pattern
    set_github_token
  end

  def parse_url_pattern
    unless match = url.match(%r{https://github.com/([^/]+)/([^/]+)/(\S+)})
      raise CurlDownloadStrategyError, "Invalid url pattern for GitHub Repository."
    end

    _, @owner, @repo, @filepath = *match
  end

  def download_url
    "https://#{@github_token}@github.com/#{@owner}/#{@repo}/#{@filepath}"
  end

  private

  def _fetch(url:, resolved_url:, timeout:)
    curl_download download_url, to: temporary_path
  end

  def set_github_token
    @github_token = ENV["HOMEBREW_GITHUB_API_TOKEN"]
    unless @github_token
      raise CurlDownloadStrategyError, "Environmental variable HOMEBREW_GITHUB_API_TOKEN is required."
    end

    validate_github_repository_access!
  end

  def validate_github_repository_access!
    # Test access to the repository
    GitHub.repository(@owner, @repo)
  rescue GitHub::HTTPNotFoundError
    # We only handle HTTPNotFoundError here,
    # becase AuthenticationFailedError is handled within util/github.
    message = <<~EOS
      HOMEBREW_GITHUB_API_TOKEN can not access the repository: #{@owner}/#{@repo}
      This token may not have permission to access the repository or the url of formula may be incorrect.
    EOS
    raise CurlDownloadStrategyError, message
  end
end

# GitHubPrivateRepositoryReleaseDownloadStrategy downloads tarballs from GitHub
# Release assets. To use it, add `:using => :github_private_release` to the URL section
# of your formula. This download strategy uses GitHub access tokens (in the
# environment variables HOMEBREW_GITHUB_API_TOKEN) to sign the request.
class GitHubPrivateRepositoryReleaseDownloadStrategy < GitHubPrivateRepositoryDownloadStrategy
  def initialize(url, name, version, **meta)
    super
  end

  def parse_url_pattern
    url_pattern = %r{https://github.com/([^/]+)/([^/]+)/releases/download/([^/]+)/(\S+)}
    unless @url =~ url_pattern
      raise CurlDownloadStrategyError, "Invalid url pattern for GitHub Release."
    end

    _, @owner, @repo, @tag, @filename = *@url.match(url_pattern)
  end

  def download_url
    "https://api.github.com/repos/#{@owner}/#{@repo}/releases/assets/#{asset_id}"
  end

  private

  def _fetch(url:, resolved_url:, timeout:)
    # HTTP request header `Accept: application/octet-stream` is required.
    # Without this, the GitHub API will respond with metadata, not binary.
    curl_download download_url, "--header", "Accept: application/octet-stream", "--header", "Authorization: token #{@github_token}", to: temporary_path
  end

  def asset_id
    @asset_id ||= resolve_asset_id
  end

  def resolve_asset_id
    release_metadata = fetch_release_metadata
    assets = release_metadata["assets"].select { |a| a["name"] == @filename }
    raise CurlDownloadStrategyError, "Asset file not found." if assets.empty?

    assets.first["id"]
  end

  def fetch_release_metadata
    release_url = "https://api.github.com/repos/#{@owner}/#{@repo}/releases/tags/#{@tag}"
    GitHub::API.open_rest(release_url)
  end
end

# ScpDownloadStrategy downloads files using ssh via scp. To use it, add
# `:using => :scp` to the URL section of your formula or
# provide a URL starting with scp://. This strategy uses ssh credentials for
# authentication. If a public/private keypair is configured, it will not
# prompt for a password.
#
# @example
#   class Abc < Formula
#     url "scp://example.com/src/abc.1.0.tar.gz"
#     ...
class ScpDownloadStrategy < AbstractFileDownloadStrategy
  def initialize(url, name, version, **meta)
    super
    parse_url_pattern
  end

  def parse_url_pattern
    url_pattern = %r{scp://([^@]+@)?([^@:/]+)(:\d+)?/(\S+)}
    if @url !~ url_pattern
      raise ScpDownloadStrategyError, "Invalid URL for scp: #{@url}"
    end

    _, @user, @host, @port, @path = *@url.match(url_pattern)
  end

  def fetch
    ohai "Downloading #{@url}"

    if cached_location.exist?
      puts "Already downloaded: #{cached_location}"
    else
      system_command! "scp", args: [scp_source, temporary_path.to_s]
      ignore_interrupts { temporary_path.rename(cached_location) }
    end
  end

  def clear_cache
    super
    rm_rf(temporary_path)
  end

  private

  def scp_source
    path_prefix = "/" unless @path.start_with?("~")
    port_arg = "-P #{@port[1..-1]} " if @port
    "#{port_arg}#{@user}#{@host}:#{path_prefix}#{@path}"
  end
end

class DownloadStrategyDetector
  class << self
    module Compat
      def detect(url, using = nil)
        strategy = super
        require_aws_sdk if strategy == S3DownloadStrategy
        strategy
      end

      def detect_from_url(url)
        case url
        when %r{^s3://}
          S3DownloadStrategy
        when %r{^scp://}
          ScpDownloadStrategy
        else
          super(url)
        end
      end

      def detect_from_symbol(symbol)
        case symbol
        when :github_private_repo
          GitHubPrivateRepositoryDownloadStrategy
        when :github_private_release
          GitHubPrivateRepositoryReleaseDownloadStrategy
        when :s3
          S3DownloadStrategy
        when :scp
          ScpDownloadStrategy
        else
          super(symbol)
        end
      end
    end

    prepend Compat
  end
end

There are some additional download strategies that I have not tested but left in the file just in case I needed them in the future. Then in our Homebrew formula we can just reference this file and tell Homebrew to use our GitHubPrivateRepositoryReleaseDownloadStrategy.

HomebrewFormula/mytool.rb

# typed: false
# frozen_string_literal: true

# This file was generated by GoReleaser. DO NOT EDIT.
require_relative "lib/custom_download_strategy"
class Mytool < Formula
  desc ""
  homepage ""
  version "1.1.5"

  on_macos do
    if Hardware::CPU.arm?
      url "https://github.com/myorg/mytool/releases/download/v1.1.5/mytool_1.1.5_Darwin_arm64.tar.gz", :using => GitHubPrivateRepositoryReleaseDownloadStrategy
      sha256 "abc123..."

      def install
        bin.install "mytool"
      end
    end
    if Hardware::CPU.intel?
      url "https://github.com/myorg/mytool/releases/download/v1.1.5/mytool_1.1.5_Darwin_x86_64.tar.gz", :using => GitHubPrivateRepositoryReleaseDownloadStrategy
      sha256 "qwerty987..."

      def install
        bin.install "mytool"
      end
    end
  end

  on_linux do
    if Hardware::CPU.arm? && Hardware::CPU.is_64_bit?
      url "https://github.com/myorg/mytool/releases/download/v1.1.5/mytool_1.1.5_Linux_arm64.tar.gz", :using => GitHubPrivateRepositoryReleaseDownloadStrategy
      sha256 "f00bar..."

      def install
        bin.install "mytool"
      end
    end
    if Hardware::CPU.intel?
      url "https://github.com/myorg/mytool/releases/download/v1.1.5/mytool_1.1.5_Linux_x86_64.tar.gz", :using => GitHubPrivateRepositoryReleaseDownloadStrategy
      sha256 "xyz543..."

      def install
        bin.install "mytool"
      end
    end
  end
end

Then all that's needed is to run the following commands to install:

HOMEBREW_GITHUB_API_TOKEN=ghp_abc123...
brew tap myorg/mytool https://github.com/myorg/mytool
brew install mytool

Adding Sharing Control To OwnTracks

jhot — Thu, 07 Oct 2021 22:47:36 +0000

In my quest to remove Google from all facets of my life, there are two products that remain near-essential. Google Photos and Google Maps. Google Photos will be hard to replace, it's very polished, handles large amounts of photos with ease, and has great sharing features. Google Maps will also be hard to stop using due to its accuracy and data set, but I am currently using Google Maps to share my location with a few friends and family members which I could definitely replace.

There is an open source product called OwnTracks that has been around for many years and is somewhat easy to setup. But to make a single server work for multiple friend groups and my family, some modifications need to be made. My mom doesn't need to know where my friends are, and separate friend groups don't need to see each other's locations. OwnTracks defaults let everyone on the server see everyone else's location, though. Luckily, I have come up with a fairly simple workaround.

Docker Config

To start, I'm hosting the Mosquitto MQTT broker and Node-Red on a free Oracle Cloud instance using Docker and Caddy as a reverse proxy. My docker-compose.yml file looks like this:

version: "3.8"

services:
  caddy:
    container_name: caddy
    image: lucaslorentz/caddy-docker-proxy:2.3
    restart: always
    ports:
      - 80:80
      - 443:443
    networks:
      - caddy
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
      - caddy_data:/data
    depends_on:
      - ddns-updater
    deploy:
      placement:
        constraints:
          - node.role == manager
      replicas: 1
    labels:
      caddy.email: ${CERTIFICATE_EMAIL_ADDRESS}

  mosquitto:
    image: eclipse-mosquitto:2
    networks:
      - caddy
      - internal
    ports:
      - 1883:1883
      - 9001:9001
    volumes:
      - mosquitto_data:/mosquitto/data
      - mosquitto_logs:/mosquitto/logs
      - ./apps/mosquitto:/mosquitto/config
    restart: unless-stopped
    labels:
      caddy: mqtt.${DOMAIN_NAME}
      caddy.reverse_proxy: "{{upstreams 9001}}"

  nodered:
    image: nodered/node-red:latest-12
    restart: always
    environment:
      - TZ=${TZ}
    ports:
      - "1880:1880"
    networks:
      - caddy
      - internal
    depends_on:
      - mosquitto
    volumes:
      - nodered_data:/data
    labels:
      caddy: nodered.${DOMAIN_NAME}
      caddy.reverse_proxy: "{{upstreams 1880}}"
      caddy.basicauth: "*"
      caddy.basicauth.admin: "${NODERED_ADMIN_PASS}"
      caddy.log.output: "stdout"

networks:
  # caddy network allows the caddy to see any containers you want to
  # expose to the internet
  caddy:
    driver: bridge
    ipam:
      config:
        - subnet: 172.18.1.0/24
          gateway: 172.18.1.1
      driver: default
  # internal network for any future non-public servers
  internal:
    driver: bridge
    ipam:
      config:
        - subnet: 172.18.2.0/24
          gateway: 172.18.2.1
      driver: default

# list data volumes here
volumes:
  caddy_data: {}
  mosquitto_data: {}
  mosquitto_logs: {}
  nodered_data: {}

So I am exposing Mosquitto via websockets with Caddy handling SSL and I'm exposing Node-Red with basic auth and SSL handled by Caddy. This simplifies the configuration of each service greatly.

Mosquitto Config

Here is my mosquitto.conf configuration file with mosquitto running standard mqtt protocol on port 1883 (we'll use this for Node-Red), websockets on 9001 (for external connections), with user authentication and access control.

listener 1883
protocol mqtt

listener 9001
protocol websockets

persistence true
persistence_location /mosquitto/data
log_dest file /mosquitto/logs/mosquitto.log

allow_anonymous false
password_file /mosquitto/config/passwd
acl_file /mosquitto/config/acl

To control user access to only stuff we want them to see, you can use a simple ACL file like following. It allows users to read anything on their username topic and any subtopics, and they can write to owntracks/username. There's also an owntracks user that can read and write to any topic.

pattern read %u/#
pattern write owntracks/%u/#

user owntracks
topic readwrite #

Now that we have configuration out of the way, you'll want to create users using the mosquitto-passwd utility. This can be run inside the mosquitto docker container by doing docker-compose exec mosquitto mosquitto_passwd .... First create the owntracks user by running mosquitto_passwd -c /mosquitto/config/passwd owntracks which will prompt you for a password. All subsequent users can be added by running mosquitto_passwd -b /mosquitto/config/passwd usernamehere passwordhere. Note: mosquitto will need to be restarted for new users to be picked up.

OwnTracks User Config

To make it much easier on your users, there is a way to share OwnTracks configuration via a link. Unfortunately, the link starts with owntracks:// which will be stripped by all messaging platforms that I've tried, but you can use Branch.io to help with this. First create a .otrc file for each user that looks like this:

{
  "_type" : "configuration",
  "autostartOnBoot" : true,
  "cleanSession" : false,
  "clientId" : "{{username}}-phone",
  "cmd" : true,
  "debugLog" : false,
  "deviceId" : "{{username}}phone",
  "fusedRegionDetection" : true,
  "host" : "mqtt.{{mydomain}}.com",
  "ignoreInaccurateLocations" : 0,
  "ignoreStaleLocations" : 0.0,
  "info" : true,
  "keepalive" : 3600,
  "locatorDisplacement" : 500,
  "locatorInterval" : 300,
  "locatorPriority" : 2,
  "mode" : 0,
  "monitoring" : 1,
  "moveModeLocatorInterval" : 10,
  "mqttProtocolLevel" : 4,
  "notificationEvents" : true,
  "notificationGeocoderErrors" : true,
  "notificationHigherPriority" : false,
  "notificationLocation" : true,
  "opencageApiKey" : "",
  "password" : "{{password}}",
  "ping" : 30,
  "port" : 443,
  "pubExtendedData" : true,
  "pubQos" : 1,
  "pubRetain" : true,
  "pubTopicBase" : "owntracks/%u/%d",
  "reverseGeocodeProvider" : "None",
  "sub" : true,
  "subQos" : 2,
  "subTopic" : "{{username}}/+/+",
  "theme" : 0,
  "tid" : "{{initials}}",
  "tls" : true,
  "tlsCaCrt" : "",
  "tlsClientCrt" : "",
  "tlsClientCrtPassword" : "",
  "username" : "{{username}}",
  "ws" : true
}

Make sure to replace the {{variables}} with the appropriate values. Then you can turn this configuration into an owntracks link by running echo "owntracks:///config?inline=$(openssl enc -a -A -in configname.otrc)". This will print the link to the console and you can copy it when configuring Branch.io links.

Once you've created your Branch.io account, you can configure your application like this:

Branch.io will send a user the Play Store or App Store if they don't have OwnTracks installed, otherwise it will open the app link you send. It's really cool. Now, for each user we'll need to create a "Quick Link" with their specific application config. So in Branch select Quick Links from the sidebar, create a new Quick Link and give it a title (I just use the person's username), and then select Link Data. For the key, select $deeplink_path and then paste the link you generated in the terminal into the Value field. You can then save and share that link, and when a user clicks it, OwnTracks will be opened and they can import the configuration and are good to go.

Node-Red Config

The final piece of this puzzle is a layer to re-broadcast messages from owntracks/someuser/somedevice to all the users who they want to share their location with i.e. mom/someuser/somedevice. I decided to use Node-Red for this because it made prototyping easy and allows room for easy future features.

First, drag in a mqtt in node, configure it to point to the local mqtt server (mosquitto:1883) using the owntracks user, and subscribe to owntracks/+/+. Next, connect a function node and put in the following code:

// Customize this
const groups = [
    {
        name: "Group One",
        members: ["user1", "user2", "etc"]
    },
    {
        name: "Group Two",
        members: ["user1", "user3", "user4", "etc"]
    }
];
const users = [
    {
        name: "user1",
        groups: ["Group One", "Group Two"],
        friends: ["user5", "user6"]
    },
    {
        name: "user2",
        groups: ["Group One"],
        friends: []
    }
];

// Don't change this
const originalTopic = msg.topic;
let recipients = [];
const messageUser = msg.topic.match(/owntracks\/(\w+)\//)[1];
const messageUserSettings = users.find(u => u.name === messageUser);
if (messageUserSettings) {
    messageUserSettings.groups.forEach(g => {
        const groupSettings = groups.find(group => group.name === g);
        if (groupSettings) {
            recipients = [...recipients, ...groupSettings.members];
        }
    });
    if (messageUserSettings.friends.length > 0) recipients = [...recipients, ...messageUserSettings.friends];
}
recipients = recipients.filter(r => r !== messageUser);
recipients = recipients.filter((item, pos) => recipients.indexOf(item) === pos);
const messages = recipients.forEach(recipient => {
    node.send({ topic:originalTopic.replace(/^owntracks/, recipient), payload: msg.payload });
});

Let's say user1 posts their location to owntracks/user1/user1phone. This code runs and sees that the messageUser is user1 and then get's all the people in the groups that user1 belongs to (in this case Group One and Group Two) as well as any other friends they want to share with. It then re-broadcasts their original location data message to user2/user1/user1phone, user3/user1/user1phone, etc. Any users not in Group One, Group Two and not user5 or user6 will not see user1 on their map.

Lastly, connect a mqtt out node and select your local MQTT server. Leave the topic blank as it and the message are set by the function node. I also connect a debug node just to make sure all the messages are going out as they should.

I just prototyped this all out so it's a little rough around the edges and a lot of work if you have a bunch of users, but there's lots of room for automation which I will hopefully tackle soon. Let me know if you see any room for improvement!

Caddy Docker Proxy, Like Traefik But Better?

jhot — Fri, 24 Sep 2021 05:36:58 +0000

When I first got serious about self-hosting and spun up a bunch of services, I quickly realized that I needed a reverse proxy (it took me some Ducking to realize what a reverse proxy was) so that I could host multiple services without needing to use weird ports. And at that time I chose Caddy v1 because it had a simple configuration spec and automatic HTTPS. As I was migrating to hosting everything in Docker, I kept seeing Traefik recommended but didn't like that it seemed more complicated than Caddy to me. I did, however, like that Traefik allows for defining rules in Docker Compose right alongside the rest of your configuration, so I began to search around for alternatives.

I quickly stumbled upon Caddy-Docker-Proxy and knew it was just what I was looking for. It makes setting up a basic reverse proxy rule a breeze, but allows for the full power of Caddy for services that require a bit beyond the basics. On top of that, it constantly monitors for changes to docker labels so no restarts are needed to pick up changes. To give you a taste of its power and simplicity, let me give some examples (pretty much straight from my personal docker-compose.yml.

Define the caddy container:

services:

  caddy:
    image: lucaslorentz/caddy-docker-proxy:2.3
    ports:
      - 80:80
      - 443:443
    networks:
      - caddy
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
      - caddy_data:/data
    deploy:
      placement:
        constraints:
          - node.role == manager
      replicas: 1
      restart_policy:
        condition: any
    labels:
      caddy.email: my-email@example.com

A basic reverse proxy:

  miniflux:
    image: miniflux/miniflux:latest
    restart: always
    ports:
      - 8014:8080
    networks:
      - caddy
      - internal
    depends_on:
      - postgres
    environment:
      - DATABASE_URL=postgres://miniflux:${MINIFLUX_DB_PASSWORD}@postgres/miniflux?sslmode=disable
      # ...
    labels:
      caddy: rss.example.com
      caddy.reverse_proxy: "{{upstreams 8080}}"

Equivalent in a Caddyfile:

rss.example.com {
  reverse_proxy 172.XXX.XXX.XXX:8080
}

The only gotcha here is to make sure to use the port that the service is running on in the container, not the port you expose to the host. So in this case, Miniflux is running on port 8080 in the container but is exposed to the host machine on port 8014 (for internal testing purposes), so I tell caddy to point to the container's IP on port 8080. {{upstreams}} is a helper provided by caddy-docker-proxy to get the container's IP within the caddy Docker network.

A more complex example:

  nextcloud:
    image: nextcloud:latest
    restart: always
    networks:
      - caddy
      - internal
    ports:
      - 8001:80
    depends_on:
      - mariadb
      - redis
    volumes:
      - nextcloud_data:/var/www/html/data
      - ./apps/nextcloud/config:/var/www/html/config
      - nextcloud_apps:/var/www/html/apps
    environment:
      - MYSQL_PASSWORD=${NEXTCLOUD_DB_PASSWORD}
      - MYSQL_DATABASE=nextcloud
      - MYSQL_USER=nextcloud
      - MYSQL_HOST=mariadb
      - REDIS_HOST=redis
      - TRUSTED_PROXIES=172.XXX.XXX.XXX/24 # this is the IP range of my caddy network
      - APACHE_DISABLE_REWRITE_IP=1
    labels:
      caddy: nextcloud.example.com
      caddy.reverse_proxy: "{{upstreams 80}}"
      caddy.0_redir: "/.well-known/carddav /remote.php/dav 301"
      caddy.1_redir: "/.well-known/caldav /remote.php/dav 301"
      caddy.header: "Strict-Transport-Security max-age=15552000"

Equivalent in a Caddyfile:

nextcloud.example.com {
  redir /.well-known/carddav /remote.php/dav 301
  redir /.well-known/caldav /remote.php/dav 301
  header Strict-Transport-Security max-age=15552000
  reverse_proxy 172.XXX.XXX.XXX:80
}

Interestingly, the Nextcloud documentation now includes a Caddy v2 example, but when I set up my configuration I based the rules off the nginx example. The only weird thing about this is the number prefixes before the redir directives. This prevents them from grouping together and also orders them (although it doesn't matter in this case).

Local-only access

Lets say you want the benefits of automatic HTTPS, but don't actually want a service to be accessible from outside your network. Caddy allows for only matched requests to be proxied:

  local-only-service:
    image: asdfasdf:latest
    labels:
      caddy: local-only.example.com
      caddy.@local.remote_ip: 192.168.0.0/16 172.16.0.0/12 10.0.0.0/8
      caddy.reverse_proxy: "@local {{upstreams 80}}"

Equivalent in a Caddyfile:

local-only.example.com {
  @local {
    remote_ip 192.168.0.0/16 172.16.0.0/12 10.0.0.0/8
  }
  reverse_proxy @local 172.XXX.XXX.XXX:80
}

These examples should cover many of your potential use cases, and there's always the Caddy community forum for any questions on more complex configurations. If you have any helper containers that you can't live without, I would love to hear about them in the comments.

Browser Automation With Node Red

jhot — Mon, 29 Mar 2021 03:31:55 +0000

Node Red is a fantastic, open source, "low code" automation framework. I have been using it for years as the automation engine for Home Assistant as well as a few other tasks. If you're coding-inclined, you can use Node Red's function nodes to run whatever JavaScript you want. Today I'm going to show you how to use these function nodes to run browser automation tasks using Playwright or Puppeteer.

Note: I'm using Docker to run Node Red and Browserless as a browser instance I can access via API from any machine on my network, but similar steps can be followed if you're running Node Red locally or installed on a server another way.

Step 1: Install the Playwright/Puppeteer module

Navigate to the Node Red data directory. On most systems that defaults to $HOME/.node-red, and in the Docker container it's /data. If you're running Node Red in Docker, you can connect by doing docker exec -it node-red-container-name /bin/bash or if you have NodeJS installed on the host, you can just navigate to wherever the Node Red data directory is mapped.

Now run the appropriate command for the tool(s) you want to install:

Playwright: npm i playwright
Playwright Core: npm i playwright-core (if you're using Browserless)
Puppeteer: npm i puppeteer
Puppeteer Core: npm i puppeteer-core (if you're using Browserless)

Step 2: Add the tools to Node Red's global context

Inside Node Red's data directory should be a file named settings.js. Open that in your editor of choice and find the functionGlobalContext property. Add whatever tool(s) you installed and then restart Node Red so that the changes are picked up.

functionGlobalContext: {
    env: process.env,
    playwright: require("playwright-core")
    // os:require('os'),
    // jfive:require("johnny-five"),
    // j5board:require("johnny-five").Board({repl:false})
}

Note: I have a Browserless instance running in Docker, so I just installed playwright-core since I don't need to install the browsers typically installed with Playwright.

Step 3: Use your tool(s) in Function nodes

const playwright = global.get('playwright');

(async () => {
    node.warn('Started');
    const browser = await playwright.chromium.connect({
        wsEndpoint: "ws://browserless:3000/playwright"
    });
    const context = await browser.newContext();
    const page = await context.newPage();
    await page.goto('https://iknowwhatyoudownload.com/en/peer/');
    node.warn(page.url());
    const downloads = await page.$$('td a');
    msg.payload = downloads.length;
    await browser.close();
    node.send(msg);
})();

Note: if not using Browserless you would just instantiate a browser with const browser = await playwright.chromium.launch(); or use firefox or webkit.

For those that may be new to Async/Await, it allows for cleaner asynchronous programming without callbacks. Javascript doesn't allow the await keyword unless you're inside an async function, so that's why I wrapped my code in:

(async () => {
  //...
})();

Also, since we're using asynchronous code, we have to use node.send(msg); instead of return msg;.

When creating a new browser automation, you're definitely better off testing things on a non-headless machine. When creating a browser instance for testing, you can run it non-headless and in "slowmo" to allow you to watch what is happening. Here's how to do it in Playwright:

const browser = await chromium.launch({
  headless: false,
  slowMo: 100
});

Adding other NPM modules

You can add other npm modules using this same process to make your life a little easier when writing custom functions and is much easier than developing a custom module. Some potentially handy modules might be: date-fns, mathjs, cheerio, execa.

Is this the ultimate self-hosting setup? I think so...

jhot — Wed, 24 Mar 2021 19:57:06 +0000

Many years back, the cheapo DVD player that my wife and I owned kicked the bucket and I wasn't too keen on buying another one because I could already see the writing on the wall for physical media. So I downloaded and setup Plex Media Server on my gaming PC and began ripping my DVDs and serving them up to my Chromecast. Several years after that, my employer was upgrading workstations, so I paid $20 to bring home one of the old machines to take over Plex duties and run a few other things. Little did I know this would send me down a giant rabbit hole of learning, trials, and fun.

The journey to my current setup

When I got my first server, I knew I wanted to go Linux instead of Windows because of its light weight and I really wanted to learn more about Linux. So I installed the latest LTS Ubuntu Server, Plex Media Server, and OwnCloud and called it good for a while. This worked well as I could just copy any ripped media to the server via Owncloud.

But this left the server sitting idle most of the time and there were other services I was wanting to try out, like Home Assistant. So I started installing more things, but this left me with an uneasy feeling. Multiple Python versions we needed, Ubuntu updates were scary (a jump from PHP 5 to PHP 7 left me with lots of troubleshooting to get Owncloud working after an upgrade), and I had no good backup scheme. So I began looking around for better solutions.

/r/homelab seemed to be big fans of virtualization and Proxmox was often recommended since there's a free community edition. So I wiped the OS drive, installed Proxmox and started creating LXC containers for each service and any related add-ons. This solved a lot of my issues. I no longer needed to worry about conflicting dependencies, I could backup a container before performing a major update to an application, and bind mounts allowed for easy sharing of data between services.

While I was happy with this solution and ran things this way for over 2 years, there were still some problems. My main problem was that management of containers and applications was too isolated. I needed to create scripts on each container OS to automate updates and configuration was also spread inconsistently in random folders on each container. This required lots of notes regarding where configuration was stored, what changes were made to the default configuration, and just didn't seem optimal.

The solution

I know Docker has been around for a really long time now and isn't really in vogue in the tech sphere these days because of k8s and other solutions for managing/scaling distributed container ecosystems. But for a single home-server setup it seemed like the best option to me. I was also planning a hardware update, so while I waited for those pieces to come together I began building out a git repo with my docker infrastructure, any scripts I thought might be handy, and as much of my applications' configuration as possible.

When the time came, I spun up containers one by one to make sure I had everything configured correctly and was able to get everything dialed in just a few hours of testing. For most services, I was able to just copy over the configuration files from the old server, bind them to the container and be up and running with little to no changes necessary. I also added some additional services to take advantage of the additional hardware power available.

Repo structure

docker-compose.yml - one compose to rule them all
.env - passwords and stuff I don't want stored in git
.gitignore - files to exclude from source control like .env, auth files, and SQLite databases.
readme.md - notes
apps/
- appname/
  - init.sh - if I need to do any initializing like cloning a repo and copying files
  - build/ - if I need to build the image myself
    - Dockerfile
    - other build files
  - config/ - plain text configuration files that are mounted to the container
scripts/
- init.sh - install docker, docker compose, create folders, docker configuration
- update.sh - update all containers and prune old images
- backup.sh - backup volumes and other data to the backup directory
- etc

Other directories outside the repo

/mnt/datadrive/
- apps/
  - docker/ - docker data here instead of the OS drive https://docs.docker.com/config/daemon/#docker-daemon-directory
  - appname/ - application data if not using a docker volume
- backup/ - synced offsite via Duplicatti
  - appname/ - files separated by application
/mnt/mediadrive/media/
- movies/
- tvshows/
- music/
- etc...

The pieces that really make this system shine

Docker Compose is the main star of the show. Writing long commands to start a container with all volumes, ports, variables and other configuration is ugly. Sure you can script these commands, but compose simplifies all the docker commands by allowing you to refer to services by simple names instead of IDs. It also makes updates really easy. It's as simple as docker-compose pull and then docker-compose up -d. This will pull any new images and restart only containers with configuration changes or new images.

Caddy-Docker-Proxy is a container that also adds a lot of simplicity to configuring a reverse proxy. It allows you to define your Caddy configuration right from labels in your docker-compose.yml. Those in the know will say this sounds a lot like Traefik, and they wouldn't be wrong. I just think it's a little simpler to use and more flexible.

Awesome backup utilities like this for mysql/mariadb, this one for postgres, and this one for Docker volumes. Along with Duplicatti which can also be run via Docker, this all makes for incredibly simple data backups that are also stored encrypted and off-site in case of an emergency. Since my configuration is also stored in Git, I could be back up and running in the event of hardware failure with very little effort.

Other notes

I have a small SSD boot drive. A zfs mirror of 3TB HDDs for application data. This mirror also has a SSD cache drive to improve read speeds, but that's probably overkill for my situation. I also have a zfs mirror of 8TB HDDs for media files. zfs is awesome and I highly recommend it.

Both MariaDB and PostgreSQL support initialization scripts. When running a database instance for a single stack you can simply set some environmental variables for the default user/database and password. Since I'm using a single instance of each database for multiple services, I use the initialization scripts to create databases and users.

If the container supports it, I suggest setting the UID and GID to your non-root user/group IDs. This is crucial for accessing shared files like media so you don't end up with files owned by root. It also keeps configuration read/writable by your user, otherwise you have to run git commands with sudo.

Shoutouts to some awesome projects

In addition to the projects I've already mentioned, these projects are amazing.

Nextcloud - File storage, office document editing, CalDav/CardDav, and more
Node Red - Flow programming, mainly for IOT
Home Assistant Websocket Plugin for Node Red - Use Node Red for Home Assistant automation
Bitwarden RS - Bitwarden server written in Rust
Pihole - Ad-blocking DNS server
Cloudflared - DNS over HTTPS client that's not tied to Cloudflare (route Pihole requests through this)
Archivy - I use it like Pocket to save web pages for later