DEV Community: Ash Walker

Adding Comments To My Site

Ash Walker — Thu, 17 Oct 2024 14:30:25 +0000

I've been wanting to add a comment section to the site but I don't want to use any third-party services, so I hadn't done so until after finding out about Comentario, which is an open-source comment server that I'm now self-hosting an instance of. This would have been pretty simple to set up if I were using Debian/Ubuntu (for which Comentario has already been packaged) or if I were willing to use Docker, but my server's running NixOS, which didn't have a Comentario package.

Until now! I made one myself.

This ended up being much more complicated than I initially assumed because, while building the backend was pretty easy, the frontend is built with Yarn, which isn't supported very well by the build tools available in Nixpkgs. I ended up having to go through a lot of trial-and-error to figure out exactly what build tools the frontend depended on -- for example, Hugo is used, but the docs, as of 2024-10-17, don't mention it.

I initially used mkYarnPackage, because that seemed like the obvious answer, but apparently that's not very useful for building web frontends, so I wasted a lot of time trying to get that to work until I just gave up and wrote a custom build script.

The main issue ended up being the yarn run generate step, which uses OpenAPI Generator, which nixpkgs does provide but which tries to download a specific version of itself at runtime, so I had write a script to patch the build config to use the nixpkgs version.

Its NixOS module is pretty standard for this sort of thing (just a simple systemd service and a Nginx vhost), and adding the web components to the site template was trivial, so at least everything else was easy.

How to Block Scrapers on Every Nginx Virtualhost in NixOS

Ash Walker — Wed, 18 Sep 2024 19:34:00 +0000

A couple months ago I realized that a lot of my home bandwidth was being eaten by AI scrapers constantly refreshing the login screen of the Jellyfin instance I host for my friends on my home server. Regardless of one's opinions about the ethicality of LLMs, the scrapers gathering training data for them are bad for the ecosystem and they're making me pay extra money to Comcast, so: here's how to block them in Nginx (as long as you're using NixOS).

First, here is a robots.txt file containing a list of user agent strings for common scrapers. If you want to add that to a vhost, you can add this to the vhost config:

location =/robots.txt {
  alias /path/to/the/robots.txt/file;
}

Web crawlers are supposed to respect rules set in robots.txt files, but they sometimes ignore them (either through malice or by mistake), so it's also useful to block them entirely.

All you have to do to block a specific user agent in Nginx is to add something like this to the server config, where "GPTBot", "Amazonbot" and "Bytespider" are the user agent strings you want to block:

if ($http_user_agent ~* "(GPTBot|Amazonbot|Bytespider)") {
  return 444;
}

("444" isn't a real HTTP status code; Nginx uses it internally to signal that it should drop the connection without a response.)

Nginx, as far as I know, doesn't let you set common configuration settings shared by all vhosts, so, if you've got more than one vhost, you'll have to do a lot of copy-and-pasting. Nix, however, makes that (relatively) simple.

The naive way to do this in NixOS would be something like this:

services.nginx.virtualHosts = let
  robots = ["GPTBot" "Amazonbot" "Bytespider"];
  rules = lib.concatStringsSep "|" robots;
  robotsTxt = let
    agentsStr = pkgs.lib.concatStringsSep "\n" (map (agent: "User-agent: ${agent}" robots));
  in pkgs.writeText "robots.txt" ''
    ${agentsStr}
    Disallow: /
  '';
in {
  "vhost-A" = {
    # ... other config ...
    locations."=/robots.txt".alias = ${robotsTxt};
    extraConfig = ''
      if ($http_user_agent ~* "(${rules})") {
        return 444;
      }
    '';
  };
  "vhost-B" = {
    # ... other config ...
    locations."=/robots.txt".alias = ${robotsTxt};
    extraConfig = ''
      if ($http_user_agent ~* "(${rules})") {
        return 444;
      }
    '';
  };
  # ... and so on
};

But that gets tedious and it's easy to forget to add the rules to a specific vhost. Instead, you can override the services.nginx.virtualHosts module to automatically apply the rules for you:

let
  robots = ["GPTBot" "Amazonbot" "Bytespider"];
  rules = lib.concatStringsSep "|" robots;
  robotsTxt = let
    agentsStr = pkgs.lib.concatStringsSep "\n" (map (agent: "User-agent: ${agent}" robots));
  in pkgs.writeText "robots.txt" ''
    ${agentsStr}
    Disallow: /
  '';
in {
  options = with lib; {
    services.nginx.virtualHosts = mkOption {
      type = types.attrsOf (types.submodule {
        config = {
          locations."=/robots.txt" = lib.mkDefault {
            alias = robotsTxt;
          };
          extraConfig = ''
            if ($http_user_agent ~* "(${rules})") {
              return 444;
            }
          '';
        };
      });
    };
  };
  config = {
    # normal nginx vhost config goes here
  };
}

Because that overrides the submodule used by virtualHosts.<name>, this configuration will automatically apply to every vhost, including ones defined by external modules.

Addendum, 2024-09-24

I wrote a NixOS module implementing this, including automatically getting the block list from ai-robots-txt.

Apparently, the NixOS manual does actually obliquely reference that you can type-merge submodules, in the documentation for types.deferredModule.

Rebuilding the Website

Ash Walker — Tue, 17 Sep 2024 00:00:00 +0000

Because Cohost is shutting down, I decided to rebuild my whole website so I could use it as a blog. I already have an ActivityPub server, but the software running it (Akkoma) doesn't really work well for long-form posting.

I hadn't been using any sort of site generator for this website -- the whole thing was entirely manual, which involved a lot of copy-and-pasting and would've made something like an RSS feed a Sisyphean effort.

Ideally, I'd have written some sort of bespoke HTTP server in Rust, with on-demand page generation, image processing, and ActivityPub support, but I, unfortunately, have a life outside of yak shaving trivial projects, so, instead, I'm just using Eleventy.

This server runs Nix, though, so I've also set up a relatively simple NixOS module defining a couple systemd services to build the site whenever it detects a change to its source folder. (I'm doing it this way instead of just building it once during nixos-rebuild so that I can post things without having to go in and rebuild the whole system every time.) Nevermind, I had insomnia last night and I don't want to deal anymore with trying to get Eleventy to work in a systemd unit installed through Nix; it's just building in a derivation for now.

(Make sure not to use Git LFS in a Nix flake repo; it's illegal as of Nix 2.20. This has been causing problems for like 2 hours and I only just now discovered why.)

Eleventy is pretty simple to use, so most of the work here has been in wrangling Nix and NodeJS to cooperate with each other and in dealing with weird CSS edge cases (like the tiny gap underneath images in posts that I can't seem to get rid of).

There are a couple extra things I'd like to get to (like CSS for mobile & a dark theme), but this is good enough for now -- I've been at this long enough that I'm typing ; instead of . at the end of my sentences, so I think it's time to work on something else.

Here's the source code, if anyone's interested.