DEV Community: Brandon Rozek

Deploying a Lightweight Git Server with CGit using Docker-Compose

Brandon Rozek — Sat, 21 Jan 2023 02:14:03 +0000

In this post, I’ll talk about how we can setup CGit within a docker-compose setup. We’ll be using the ClearLinux CGit container.

Configuring Apache Webserver

Within the CGit container, an apache webserver is setup to execute the CGit CGI scripts. This configuration is very similar to the default one provided by ClearLinux. However, the default holds your repositories in the /cgit subfolder as I wanted it on the root / folder.

/etc/httpd-cgit.conf

ServerName localhost

# Next two lines changed for new document root
DocumentRoot /var/www/cgit
<Directory "/var/www/cgit">
    AllowOverride None
    Options ExecCGI FollowSymLinks
    Require all granted
</Directory>

# cgid module is required to run the cgit binary
LoadModule cgid_module lib/httpd/modules/mod_cgid.so
<IfModule cgid_module>
        ScriptSock /var/run/cgid.sock
</IfModule>

# Path to cgit stylesheet, graphics
Alias /cgit-data /usr/share/cgit
<Directory "/usr/share/cgit">
    AllowOverride None
    Options None
    Require all granted
</Directory>

# Path to cgit binary
# Next line changed
ScriptAlias / /usr/libexec/cgit/cgi-bin/cgit/
<Directory "/usr/libexec/cgit/cgi-bin">
    AllowOverride None
    Options None
    Require all granted
</Directory>

Configuring CGit

Now to configure cgit itself, we need to create a file called cgitrc. Order matters in the declarations, and from what I can gather you should have your scan-path near the end of the file.

To enable cloning and have it discoverable:

enable-http-clone=1
clone-prefix=https://URL/TO/WEBSITE

To download snapsots of the references

snapshots=tar.gz zip

To enable the git config to override the owner and description fields

enable-git-config=1

Cache up to 1000 output entries

cache-size=1000

Root Page configuration

root-title=Brandon Rozek's Repositories
root-desc=
repository-sort=age

# Start all URLs from the root directory
virtual-root=/

Server the appropriate mime-types of certain files

mimetype.gif=image/gif
mimetype.html=text/html
mimetype.jpg=image/jpeg
mimetype.jpeg=image/jpeg
mimetype.pdf=application/pdf
mimetype.png=image/png
mimetype.svg=image/svg+xml

Styles for the website

css=/cgit-data/cgit.css
logo=/cgit-data/cgit.png
favicon=/cgit-data/favicon.ico
source-filter=/usr/libexec/cgit/filters/syntax-highlighting.sh
about-filter=/usr/libexec/cgit/filters/about-formatting.sh
readme=:README.md
readme=:README

Where to find the repositories

scan-path=/var/www/cgit/

Setting up the container

I prefer using docker-compose to help manage all my containers. The first two volumes map the configuration files we created and the last volume holds our repositories.

cgit:
  image: docker.io/clearlinux/cgit
  container_name: cgit
  hostname: cgit
  volumes:
    - /etc/cgitrc:/etc/cgitrc
    - /etc/httpd-cgit.conf:/etc/httpd/conf.d/httpd-cgit.conf
    - /var/www/cgit:/var/www/cgit
  restart: always

Populating it with Repositories

Within /var/www/cgit, start cloning your repositories:

git clone --bare REPO_URL

If you enabled gitinfo then for each repository you can run git config -e and add the following

[gitweb]
    owner = Name <Email>
    description = Insert your project description here

Aside: Reverse Proxy

In my setup, I have an nginx container that handles all of the traffic. Therefore, I don’t have users enter to the cgit container directly. I handle this by adding the following reverse proxy configuration.

server {
    listen 80;
    listen [::]:80;
    server_name GIT.SERVER.URL

    location / {
        return 301 https://$host$request_uri;
    }
}

server {
    listen 443 ssl http2;
    listen [::]:443 ssl http2;
    server_name GIT.SERVER.URL;

    ssl_certificate /path/to/chain;
    ssl_certificate_key /path/to/private/key;
    include /etc/letsencrypt/options-ssl-nginx.conf;
    ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem;

    location / {
        proxy_pass http://cgit;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Host $server_name;
        # Needed to get around HTTP2 Streaming Errors
        proxy_hide_header Upgrade;
    }
}

Conclusion

After all the configuration, you should be able to pull it up using docker-compose.

docker-compose up cgit

References

These references talked about setting up cgit outside of docker, but they helped me understand the various configuration files needed.

Bmaptool: A simpler way to copy ISOs

Brandon Rozek — Wed, 18 Jan 2023 16:08:20 +0000

Bmaptool is a project created by Intel for creating and copying data using block maps. It’s meant to be a simpler, faster, and more reliable tool than dd.

From their GitHub page:

Faster. Depending on various factors, like write speed, image size, how full is the image, and so on, bmaptool was 5-7 times faster than dd in the Tizen IVI project.

Integrity. bmaptool verifies data integrity while flashing, which means that possible data corruptions will be noticed immediately.

Usability. bmaptool can read images directly from the remote server, so users do not have to download images and save them locally.

Protects user’s data. Unlike dd, if you make a mistake and specify a wrong block device name, bmaptool will less likely destroy your data because it has protection mechanisms which, for example, prevent bmaptool from writing to a mounted block device.

It comes with two commands create and copy. Create generates the block maps which isn’t required to use the application. However, having a bmap will speed up the copying process. The syntax of the copy command is the following

sudo bmaptool copy SRC DST

Let’s say that I want to flash the latest Ubuntu ISO to a USB stick located at /dev/sdX. As the third bullet point claims, I can easily use the URL as SRC and since we don’t have a bmap, we’ll have to specify that with the flag --nobmap.

sudo bmaptool copy --nobmap https://releases.ubuntu.com/22.04.1/ubuntu-22.04.1-desktop-amd64.iso /dev/sdX

Example run on my desktop:

bmaptool: info: no bmap given, copy entire image to '/dev/sdX'
/
bmaptool: info: synchronizing '/dev/sdX'
bmaptool: info: copying time: 3m 7.3s, copying speed 19.5 MiB/sec

Now if we have the ISO downloaded on our computer, we can take the time to create a bmap for it.

sudo bmaptool create ubuntu-22.04.1-desktop-amd64.iso > ubuntu-22.04.1-desktop-amd64.bmap

A bmap file is a human readable XML file that shows the block map and the checksums for each block.

<?xml version="1.0" ?>
<!-- This file contains the block map for an image file, which is basically
     a list of useful (mapped) block numbers in the image file. In other words,
     it lists only those blocks which contain data (boot sector, partition
     table, file-system metadata, files, directories, extents, etc). These
     blocks have to be copied to the target device. The other blocks do not
     contain any useful data and do not have to be copied to the target
     device.

     The block map an optimization which allows to copy or flash the image to
     the image quicker than copying of flashing the entire image. This is
     because with bmap less data is copied: <MappedBlocksCount> blocks instead
     of <BlocksCount> blocks.

     Besides the machine-readable data, this file contains useful commentaries
     which contain human-readable information like image size, percentage of
     mapped data, etc.

     The 'version' attribute is the block map file format version in the
     'major.minor' format. The version major number is increased whenever an
     incompatible block map format change is made. The minor number changes
     in case of minor backward-compatible changes. -->

<bmap version="2.0">
    <!-- Image size in bytes: 3.6 GiB -->
    <ImageSize> 3826831360 </ImageSize>

    <!-- Size of a block in bytes -->
    <BlockSize> 4096 </BlockSize>

    <!-- Count of blocks in the image file -->
    <BlocksCount> 934285 </BlocksCount>

    <!-- Count of mapped blocks: 3.6 GiB or 100.0% -->
    <MappedBlocksCount> 934285 </MappedBlocksCount>

    <!-- Type of checksum used in this file -->
    <ChecksumType> sha256 </ChecksumType>

    <!-- The checksum of this bmap file. When it is calculated, the value of
         the checksum has be zero (all ASCII "0" symbols). -->
    <BmapFileChecksum> e69f56b4cf11a26fba8700bc66a443a20f667d0d0fe1c2d8028715ac060c402d </BmapFileChecksum>

    <!-- The block map which consists of elements which may either be a
         range of blocks or a single block. The 'chksum' attribute
         (if present) is the checksum of this blocks range. -->
    <BlockMap>
        <Range chksum="c396e956a9f52c418397867d1ea5c0cf1a99a49dcf648b086d2fb762330cc88d"> 0-934284 </Range>
    </BlockMap>
</bmap>

Once we have generated the bmap, we can copy the ISO to the device.

sudo bmaptool copy ubuntu-22.04.1-desktop-amd64.iso /dev/sdX

Example run on my desktop:

bmaptool: info: discovered bmap file 'ubuntu-22.04.1-desktop-amd64.bmap'
bmaptool: info: block map format version 2.0
bmaptool: info: 934285 blocks of size 4096 (3.6 GiB), mapped 934285 blocks (3.6 GiB or 100.0%)
bmaptool: info: copying image 'ubuntu-22.04.1-desktop-amd64.iso' to block device '/dev/sdX' using bmap file 'ubuntu-22.04.1-desktop-amd64.bmap'
bmaptool: info: 100% copied
bmaptool: info: synchronizing '/dev/sdX'
bmaptool: info: copying time: 2m 49.2s, copying speed 21.6 MiB/sec

Recall that the bmap generation isn’t necessary, as you can pass in the --nobmap flag.

Decentralized PGP Keys with WKD

Brandon Rozek — Wed, 04 Jan 2023 14:06:38 +0000

After creating a PGP key, it is common to distribute it to various keyservers. However, anyone can upload to these keyservers impersonating someone else. One solution is to use a decentralized identities approach, however, if your email is on your own domain that you tell every soul about, why not have your own website host the key? This is where the Web Key Directory (WKD) protocol comes in.

Setting up WKD

To start we need to create a new folder on our webserver:

mkdir .well-known/openpgpkey/hu

In it, add an empty policy file

touch .well-known/openpgpkey/hu/policy

Now we need to add our key to the folder. The key needs to be stored in the file named after the email’s WKD hash. We can get this hash through the following command:

gpg --with-wkd-hash --fingerprint brozek@brandonrozek.com

Replacing my email with yours. At the current moment, this returns the following for me:

pub ed25519 2022-12-14 [SC] [expires: 2023-12-14]
      5F37 830B FA46 FF78 81F4 7AC7 8DF7 9C3D C5FC 658A
uid [ultimate] Brandon Rozek <brozek@brandonrozek.com>
              o1dbwkdx683fduwgzmrbwa3yip41frdn@brandonrozek.com
uid [ultimate] Brandon Rozek <hello@brandonrozek.com>
              im4cc8qhazwkfsi65a8us1bc5gzk1o4p@brandonrozek.com

The string starting witho1dbwk is the WKD hash for brozek@brandonrozek.com and the string starting with im4cc8qh is the WKD hash for hello@brandonrozek.com.

Let’s store that hash in $WKD

export WKD="o1dbwkdx683fduwgzmrbwa3yip41frdn"

The WKD specification says to upload the non-armored (binary) version of our key.

gpg --no-armor --export brozek@brandonrozek.com > .well-known/openpgpkey/hu/$WKD

After uploading it to our webserver, it needs to be configured with the right content type and access control headers.

In Nginx:

location /.well-known/openpgpkey {
    default_type application/octet-stream;
    add_header Access-Control-Allow-Origin "*";
}

Now we can check our setup using the website:

https://metacode.biz/openpgp/web-key-directory

Using WKD

Many applications currently support WKD, though I’ll show how we can use gpg to search for someone’s key.

gpg --auto-key-locate wkd --locate-key brozek@brandonrozek.com

This will not only locate but import the key into our keystore.

With WKD, we didn’t have to trust anyone but the DNS provider in order to retrieve the key. The biggest downside with this approach, however, is that most people do not have an email on their own domain. Since nowadays, many people use gmail as their primary provider, they will have to fallback to using a different approach for distributing their keys.

References

Decentralized Identities with PGP Annotations and Keyoxide

Brandon Rozek — Wed, 04 Jan 2023 14:00:14 +0000

Under asymmetric encryption, for you to send me a message that only I can read you would need to encrypt the message with my public key. I then would have a corresponding private key that can decrypt the message. Public keys are then usually stored onto keyservers for others to query. When querying for a key, how do you know that the public key actually belongs to me? It turns out, you can’t since anyone can upload a key pretending to be me.

What’s the solution? When PGP first came around, it was built around the Web of Trust concept. The idea is that people would go to key-signing parties and verify in person that they are who they say they are. From a graph would be built out showing who verified who. Sadly this idea didn’t take off. A very small segment of the population attends key signing parties.

Keybase

In 2014, Keybase was created to help solve this issue. The concept behind it is that we have other identities in the web, and by associating a keybase profiles to these identities others can have a strong confidence on who they are speaking to.

For example, I own the website brandonrozek.com and am known as @brozek@fosstodon.org on Mastodon. On those platforms, I would create a post using the private key on keybase which claims that I own the user profile on keybase. Similarly on keybase I would link to my website and mastodon profile to say that I claim those.

For a while, this was working great. Then in 2019 the following article comes out of their blog:

Keybase + Stellar is live for everyone!

Source: https://keybase.io/blog/keybase-stellar-launch

The promotion of cryptocurrency makes you wonder how they are doing financially. Then in 2020, we see:

Keybase joins Zoom

Source: https://keybase.io/blog/keybase-joins-zoom

Upon further reflection, several questions arise:

Why am I depending on Keybase to show which users I’m connected to? Ideally, this should be decentralized.
Keybase holds access to the private key. While this makes the user experience easier since I don’t need to worry about those details. We should be encouraging people to hold their private keys instead.

What’s a great alternative to Keybase then? This is where Keyoxide comes in.

Keyoxide & PGP Notations

Yarmo Mackenbach wanted to create a project that’s decentralized in nature. This means that Keyoxide doesn’t hold the keys. Instead it depends on either:

Web Key Directory (WKD) protocol where the keys are stored on your own server belonging to the domain.
HTTP Keyserver Protocol (HKP) where Keyoxide queries keys.openpgp.org

Within the key you upload, you can add a PGP notation. This allows us to provide additional text on what accounts we own.

For example the notation of:

proof@ariadne.id=dns:brandonrozek.com?type=TXT

claims that I own the domain brandonrozek.com.

To provide the necessary backlink, the Keyoxide documentation says to create a TXT record with my PGP fingerprint.

openpgp4fpr:5F37830BFA46FF7881F47AC78DF79C3DC5FC658A

Notice how nowhere in the process do we reference Keyoxide or their servers. This only depends upon the keys that I upload onto the Internet and the appropriate backlinks. Keyoxide in this case, only serves as a validator, making sure that the links exist.

My Keyoxide profile: https://keyoxide.org/wkd/brozek%40brandonrozek.com

In fact, Keyoxide is open source meaning that anyone can host their own instance to perform the validation checks.

Obtaining Multiple Solutions Z3

Brandon Rozek — Sat, 31 Dec 2022 14:52:00 +0000

Playing around with Diophantine solvers, I wanted to obtain the solutions of the following equation: $$ 5a + 4b - 3c = 0 $$ Let’s encode that using Z3

from z3 import *

# Encode Equation
a, b, c = Ints("a, b, c")
s = Solver()
s.add(5 * a + 4 * b - 3 * c == 0)

# Find solution
result = s.model()
if result == sat:
    print(result)

This code snippet returns

[a = 0, b = 0, c = 0]

Now there are multiple solutions to this Diophantine equation, so how do we get the others? It turns out after searching around StackOverflow (see references) the only way is to add the previous solutions as constraints.

# This encodes the last solution as a constraint
block = []
for var in m:
    block.append(var() != m[var])
s.add(Or(block))

Formulaically, this corresponds to: $$ a \ne 0 \vee b \ne 0 \vee c \ne 0 $$ If you look at the references, it’s hard to encode these constraints generally. This is because Z3 is a powerful SMT solver working with many different theories. Though if we restrict ourselves to the Diophantine equations, we can write a function that acts as a generator for all of the solutions:

import z3

def get_solutions(s: z3.z3.Solver):
    result = s.check()
    # While we still get solutions
    while (result == z3.sat):
      m = s.model()
      yield m
      # Add new solution as a constraint
      block = []
      for var in m:
          block.append(var() != m[var])
      s.add(z3.Or(block))
      # Look for new solution
      result = s.check()

Now for our example, this allows us to do the following:

from z3 import *

a, b, c = Ints("a, b, c")
s = Solver()
s.add(5 * a + 4 * b - 3 * c == 0)

solutions = get_solutions(s)
upper_bound = 10
for solution, _ in zip(solutions, range(upper_bound)):
    print(solution)

The solutions of a linear Diophantine equation can be easily parameterized so I don’t recommend using Z3 in this way. Though I think this exercise is informative for other theories you might be trying to satisfy.

References

Mastodon/Webfinger Alias using HTTP Redirects

Brandon Rozek — Sat, 31 Dec 2022 14:50:00 +0000

Mastodon uses the Webfinger protocol to find users. For example, if you search for @brozek@brandonrozek.com it will access the url:

https://brandonrozek.com/.well-known/webfinger?resource=acct:brozek@brandonrozek.com

As of the time of writing, I do not have a strong interest in running my own Mastodon server. I instead am active on @brozek@fosstodon.org

https://fosstodon.org/.well-known/webfinger?resource=acct:brozek@fosstodon.org

If you try to access the last URL in your web browser, you’ll notice that it doesn’t return anything. This is because the server will only respond if we specify the content type. In curl, we can specify that through the following:

curl -H 'Accept: application/jrd+json' \
    https://fosstodon.org/.well-known/webfinger?resource=acct:brozek@fosstodon.org

Currently, it returns the following

{
  "subject": "acct:brozek@fosstodon.org",
  "aliases": [
    "https://fosstodon.org/@brozek",
    "https://fosstodon.org/users/brozek"
  ],
  "links": [
    {
      "rel": "http://webfinger.net/rel/profile-page",
      "type": "text/html",
      "href": "https://fosstodon.org/@brozek"
    },
    {
      "rel": "self",
      "type": "application/activity+json",
      "href": "https://fosstodon.org/users/brozek"
    },
    {
      "rel": "http://ostatus.org/schema/1.0/subscribe",
      "template": "https://fosstodon.org/authorize_interaction?uri={uri}"
    }
  ]
}

Some people have been copy pasting the contents of this file, onto their webserver. However, this makes it so that we can’t have multiple aliases on this domain. Instead, we can use our webserver (mine is nginx) to redirect the user when the specified handle is requested.

location /.well-known/webfinger {
    default_type application/jrd+json;
    add_header Access-Control-Allow-Origin "*";
    if ($arg_resource = acct:brozek@brandonrozek.com) {
        return 301 $scheme://fosstodon.org/.well-known/webfinger?resource=acct:brozek@fosstodon.org;
    }
}

Now this isn’t a perfect alias, you can’t reference @brozek@brandonrozek.com and have me be notified. Instead, this only works in user discovery.

Personal Web Archive and How I Archive Single Web pages

Brandon Rozek — Sat, 24 Dec 2022 14:01:16 +0000

The Internet Archive is great for providing a centralized database of snapshots of websites throughout time. What happens though when you want to have your own offline copy during times of little to no internet access? Archivebox is one solution to such problem. It behaves similarly to the Internet archive and also allows importing of RSS feeds to save local copies of blog posts. To install it, you can use pipx.

pipx install archivebox

For the rest of this post, however, I want to talk about a simpler tool. A combination of wget and python -m http.server. In the past, I’ve used wget to mirror entire websites. We can adjust the command slightly so that it doesn’t follow links and instead only looks at a single webpage.

wget --convert-links \
     --adjust-extension \
     --no-clobber \
     --page-requisites \
     INSERT_URL_HERE

Now assume we have a folder full of downloaded websites. To view them, we can use any HTTP server. One of the easiest to temporarily setup currently is Python’s built in one.

To serve all the files in the current directory

python -m http.server OPTIONAL_PORT_NUM

If you leave the port number field empty, then this returns

Serving HTTP on 0.0.0.0 port 8000 (http://0.0.0.0:8000/) ...

A few nice side-effects of using wget and python.

Python’s default webserver shows a list of files in the directory. This can make it easier to browse around the web archive.
The wget flags make it so that if you want to archive https://brandonrozek.com/blog/personal-simple-web-archive/ then all you need to access is http://localhost:8000/brandonrozek.com/blog/personal-simple-web-archive. In other words, it preserves URLs.

Now this approach isn’t perfect, if a webpage makes heavy use of javascript or server side features then it’ll be incomplete. Though for the majority of the wiki pages or blog posts I want to save for future reference, this approach works well.

My full script is below:

#!/bin/sh

set -o errexit
set -o nounset
set -o pipefail

ARCHIVE_FOLDER="$HOME/webarchive"

show_usage() {
    echo "Usage: archivesite [URL]"
    exit 1
}

# Check argument count
if ["$#" -ne 1]; then
    show_usage
fi

cd "$ARCHIVE_FOLDER"

wget --convert-links \
     --adjust-extension \
     --page-requisites \
     --no-verbose \
     "$1"

# Keep track of requested URLs
echo "$1" >> saved_urls.txt

Concatenating PDF files in Linux

Brandon Rozek — Sun, 18 Dec 2022 23:38:44 +0000

Every so often I need to combine several images into a single PDF. First, to convert an image to a PDF we can use imagemagick.

convert -quality 100 Image.png Scanned.pdf

To combine or concatenate multiple PDF files, we can use ghostscript.

gs -sDEVICE=pdfwrite \
   -sOUTPUTFILE=output.pdf \
   -dNOPAUSE \
   -dBATCH \
   input0.pdf input1.pdf input2.pdf

Flag	Description
`-sDEVICE`	Device used for processing the output file type. Use `pdfwrite` to write to a PDF file.
`-sOUTPUTFILE`	Path to save the resulting file output.
`-dNOPAUSE`	Disables the prompting and pausing at the end of each page.
`-dBATCH`	Finishes interpreting after processing the inputted files

Alternatively you can use pdftk

pdftk input0.pdf input1.pdf input2.pdf \
    cat output output.pdf

Lastly, you can also use imagemagick. Do note, however, that this program often leads to larger file sizes.

convert input0.pdf input1.pdf input2.pdf output.pdf

Aside: Pixel Densities

One issue I came across is that the pages were of different sizes. This is often because the pages can be of different pixel densities.

To check run pdfimages and look at the 3rd to last and 2nd to last columns:

pdfimages -list filename.pdf 


page num type width height color comp bpc enc interp object ID x-ppi y-ppi size ratio
--------------------------------------------------------------------------------------------
   1 0 image 613 77 rgb 3 8 jpeg no 8 0 1071 1076 9456B 6.7%
   1 1 image 2692 3496 rgb 3 8 jpeg no 9 0 329 329 418K 1.5%
   2 2 image 613 77 rgb 3 8 jpeg no 8 0 915 919 9456B 6.7%
   2 3 image 2300 3016 rgb 3 8 jpeg no 15 0 282 282 322K 1.6%
   3 4 image 613 77 rgb 3 8 jpeg no 8 0 937 942 9456B 6.7%
   3 5 image 2356 3024 rgb 3 8 jpeg no 21 0 288 288 150K 0.7%
   4 6 image 1686 2200 rgb 3 8 jpeg no 27 0 204 204 622K 5.7%
   5 7 image 5100 7016 rgb 3 8 jpeg no 33 0 600 600 1193K 1.1%
   6 8 image 613 77 rgb 3 8 jpeg no 8 0 1104 1110 9456B 6.7%
   6 9 image 2776 3720 rgb 3 8 jpeg no 38 0 339 339 231K 0.8%
   7 10 image 613 77 rgb 3 8 jpeg no 8 0 939 943 9456B 6.7%
   7 11 image 2360 3072 rgb 3 8 jpeg no 44 0 289 289 151K 0.7%

We can then use imagemagick to enforce a certain pixel density. The tradeoff being that the file size might increase.

convert -density 300 input.pdf output.pdf

If you happen to know a different way to enforce a pixel density that doesn’t have a file size increase tradeoff. Please get in touch.

Capturing Quoted Strings in Sed

Brandon Rozek — Sun, 18 Dec 2022 17:55:32 +0000

Disclaimer: This posts assumes some knowledge about regular expressions.

Recently I was trying to capture an HTML attribute in sed. For example, let’s say I want to extract the href attribute in the following example:

<a href="https://brandonrozek.com" rel="me"></a>

Advice you commonly see on the Internet is to use a capture group for anything between the quotes of the href.

In regular expression land, we can represent anything as .* and define a capture group of some regular expression X as $X$.

sed "s/.*href=\"\(.*\)\".*/\1/g"

What does this look like for our input?

echo \<a href=\"https://brandonrozek.com\" rel=\"me\"\>\</a\> |\
sed "s/.*href=\"\(.*\)\".*/\1/g"


https://brandonrozek.com" rel="me

It matches all the way until the second "! What we want, is to not match any character within the quotations, but match any character that is not the quotation itself [^\"]*

sed "s/.*href=\"\([^\"]*\)\".*/\1/g"

This then works for our example:

echo \<a href=\"https://brandonrozek.com\" rel=\"me\"\>\</a\> |\
sed "s/.*href=\"\([^\"]*\)\".*/\1/g"


https://brandonrozek.com

Within a bash script, we can make this a little more readable by using multiple variables.

QUOTED_STR="\"\([^\"]*\)\""
BEFORE_TEXT=".*href=$QUOTED_STR.*"
AFTER_TEXT="\1"
REPLACE_EXPR="s/$BEFORE_TEXT/$AFTER_TEXT/g"

INPUT="\<a href=\"https://brandonrozek.com\" rel=\"me\"\>\</a\>"

echo "$INPUT" | sed "$REPLACE_EXPR"

Drawing Trees in LaTex with Tikz

Brandon Rozek — Tue, 06 Dec 2022 16:01:24 +0000

For the longest time I’ve been avoiding Tikz because I imagined it being too difficult to learn. Usually I create a graphic using a program like Draw.IO and import it as an image. Though this time around, I decided that I’m going to learn how to make trees in Tikz. It turns out, it’s not as bad as I anticipated.

I’m only going to provide a few simple examples in this post. To learn more check out the Tikz documentation on trees.

Remember to have \usepackage{tikz} in the preamble.

To create a Tikz figure, you’ll need to create a tikzpicture environment

\begin{tikzpicture}
 % Tikz Code Here
\end{tikzpicture}

Every tree first begins with a root node.

\begin{tikzpicture}
    \node {Root Node};
\end{tikzpicture}

The semicolon at the end denotes the end of a tikz command.

Now let’s make the root node have one child node.

\begin{tikzpicture}
    \node {Root Node}
        child {node {Child Node}};
\end{tikzpicture}

Notice that the text of the nodes are within the {} after the node command.

To create another child for a node, place it in the same level as the existing child.

\begin{tikzpicture}
    \node {Root Node}
        child {node {Left Child}}
        child {node {Right Child}};
\end{tikzpicture}

The rendered tree may have text overlap as shown in the last screenshot. This is where tikz options come in. We can define the separation distance between siblings. I don’t have many tips for choosing the value other than to play around and see how it looks.

\begin{tikzpicture}
[
    level 1/.style={sibling distance=25mm}
]
    \node {Root Node}
        child {node {Left Child}}
        child {node {Right Child}};
\end{tikzpicture}

To show how the child nesting works, I’ll finish by giving the right child two children nodes.

\begin{tikzpicture}
[
    level 1/.style={sibling distance=25mm},
    level 2/.style={sibling distance=15mm},
]
    \node {Root Node}
        child {node {Left Child}}
        child {
            node {Right Child}
            child {node {Child A}}
            child {node {Child B}}
        };
\end{tikzpicture}

Deploying my Hugo Website through GitHub Actions

Brandon Rozek — Mon, 05 Dec 2022 03:02:08 +0000

For the longest time I’ve held out on deploying my website through GitHub actions. My rationale at the time was:

If I have to execute git push, I might as well run a ./sync script afterwards.

What convinced me otherwise is automated commits. I currently have GitHub actions that sync my Mastodon toots and iNaturalist observations. As part of the sync process, a git commit is made. This commit should then trigger a site rebuild.

How do we create a GitHub action that builds a Hugo website and deploys it via rsync? The rest of this post will go over the components of the GitHub action that triggers when I update my website.

Triggers

I currently have three triggers for my deployment GitHub action:

Manual (workflow_dispatch)
Pushes to the main branch
Daily schedule via cron

on:
  workflow_dispatch:
  push:
    branches: main
  schedule:
    - cron: "21 11 * * *"

Steps

I call my job build_and_publish and have it run on top of the latest Ubuntu image.

jobs:
  build_and_publish:
    runs-on: ubuntu-latest

Step 1: Checkout the Repository

Here we can rely on Github’s checkout action to provide the latest version of the code.

steps:
  - name: Checkout
    uses: actions/checkout@v3
    with:
      submodules: true
      fetch-depth: 0

Since my website relies on submodules, we need to make sure that its included in the checkout. The fetch-depth flag denotes how many commits to retrieve. By default (fetch-depth: 1) it only fetches the latest commit, however setting it to 0 retrieves all commits. This is needed for Hugo’s last modified feature to work.

Step 2: Update the submodules

Even though we checked out the whole repository with its associated submodules, they may be out of date. This step makes sure that we’re using the latest version of the submodule.

- name: Git submodule update
  run: |
    git pull --recurse-submodules
    git submodule update --remote --recursive

Step 3: Setup Hugo

Since Hugo is a static binary, we can pull it straight from their website.

- name: Setup Hugo
  env:
    HUGO_VERSION: 0.105.0
  run: |
    curl -L "https://github.com/gohugoio/hugo/releases/download/v${HUGO_VERSION}/hugo_${HUGO_VERSION}_Linux-64bit.tar.gz" --output hugo.tar.gz
    tar -xvzf hugo.tar.gz
    sudo mv hugo /usr/local/bin

Step 4: Build the website

We can use a separate step to build the website. This along with the deployment are among the few places where this script can fail, so it’s nice to separate it out in case.

- name: Build Hugo Website
  run: hugo

Step 5: Install the SSH key

- name: Install SSH Key
  run: |
    install -m 600 -D /dev/null ~/.ssh/id_rsa
    echo "${{ secrets.BUILD_SSH_KEY }}" > ~/.ssh/id_rsa
    echo "${{ secrets.HOST_KEY }}" > ~/.ssh/known_hosts
    echo "Host brandonrozek.com
        Hostname brandonrozek.com
        user build
        IdentityFile ~/.ssh/id_rsa" > ~/.ssh/config

At this point in our script we need to handle secrets. The post I linked to will likely have the most up to date information, but as of this time of writing, you can add secrets by going to the Settings tab of the repository. A secret is a key-value pair, therefore to access your secret in the GH action, you need to reference the key.

${{ secrets.YOUR_KEY_HERE }}

We need secrets for the SSH key used to deploy the website and the known hosts file so that I don’t have to do host verification. The first line ensures that the permissions of the SSH key is correct, and the last line makes it so that the rsync command within my sync.sh script is simpler. I use my sync.sh not only in the next step of this action but on my own machine which has a different config associated with it.

Step 6: Deployment

- name: Deploy
  run: ./deploy.sh

In my repository there is a deploy.sh with the following contents

#!/usr/bin/env sh
rsync -Pazc --exclude=*.bak --delete public/ build@brandonrozek.com:brandonrozek/

This syncs everything within the public build folder up to my webserver excluding files ended in .bak and removing any files on the webserver that aren’t in the build folder.

Conclusion

Other than the checkout action, each step does not depend on an external library to provide the functionality. I think it’s important to implement each of the steps ourselves, as opposed to relying on a hugo GH action library or a SFTP library. Not only does this safeguard us against supply side attacks, it also makes these actions more portable. I am not counting on GitHub to always allow the usage of their build infrastructure for free.

GitHub action in its entirety:

name: Build and Deploy Hugo Website

on:
  workflow_dispatch:
  push:
    branches: main
  schedule:
    - cron: "21 11 * * *"

jobs:
  build_and_publish:
    runs-on: ubuntu-latest

    steps:
      - name: Checkout
        uses: actions/checkout@v3
        with:
          submodules: true
          fetch-depth: 0

      - name: Git submodule update
        run: |
          git pull --recurse-submodules
          git submodule update --remote --recursive          

      - name: Setup Hugo
        env:
          HUGO_VERSION: 0.105.0
        run: |
          curl -L "https://github.com/gohugoio/hugo/releases/download/v${HUGO_VERSION}/hugo_${HUGO_VERSION}_Linux-64bit.tar.gz" --output hugo.tar.gz
          tar -xvzf hugo.tar.gz
          sudo mv hugo /usr/local/bin          

      - name: Build Hugo Website
        id: build
        run: |
                    hugo

      - name: Install SSH Key
        run: |
          install -m 600 -D /dev/null ~/.ssh/id_rsa
          echo "${{ secrets.BUILD_SSH_KEY }}" > ~/.ssh/id_rsa
          echo "${{ secrets.HOST_KEY }}" > ~/.ssh/known_hosts
          echo "Host brandonrozek.com
              Hostname brandonrozek.com
              user build
              IdentityFile ~/.ssh/id_rsa" > ~/.ssh/config          

      - name: Deploy
        run: ./deploy.sh

Pretty RSS Feeds

Brandon Rozek — Mon, 05 Dec 2022 02:42:36 +0000

Hi, I have a RSS feed. This allows you to subscribe to my words without me even knowing. How does it work? Well you need an RSS link, here’s mine:

https://brandonrozek.com/blog/index.xml

Then you need to input this URL into a newsreader application (Feedly, Feedbin, NetNewsWire, many others) which checks to see if there are any changes in the XML file and presents it in a visually nice way.

In fact, many websites come with a RSS feed. You don’t even need to know the exact URL for it to work, if you type in https://brandonrozek.com into your newsreader, you’ll get options for my various different feeds.

How does that work? Well the website creator needs to include a tag in their website HTML.

<link rel="alternate" 
      type="application/rss+xml" 
      href="https://brandonrozek.com/blog/index.xml" 
      title="Brandon Rozek's Blog" />

Most RSS feeds don’t come styled. Therefore if you click on the RSS link you’ll see a bunch of XML formatted text. That can be confusing for people when they first visit, can we do better?

About Feeds RSS Style

Matt Webb created https://aboutfeeds.com in order to provide a friendly overview into the world of RSS. As part of that he includes a style called pretty-feed.xsl. The file comes with instructions built in, but it’s mostly as simple as adding a style tag in the beginning of the XML

<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet href="https://brandonrozek.com/PATH-TO-YOUR-STYLES/pretty-feed-v3.xsl" type="text/xsl"?>

I hope more bloggers incorporate the style into their feed. I’m waiting for the day that the largest blogging platform Wordpress includes it by default.