DEV Community: Peter Cooper

How to Quickly Play with PostgreSQL 15 Beta using Docker

Peter Cooper — Wed, 25 May 2022 10:32:57 +0000

The first beta of Postgres 15 is out now and the official Docker image for it has also just gone live, making it easy for you to take it for a quick spin without destroying your production systems.

Assuming you have Docker running, here's the quickest way to get going:

docker network create pgnetwork
docker run --name pg15beta1 --network pgnetwork -e POSTGRES_PASSWORD=whatever -d postgres:15beta1

This creates a network called pgnetwork so that other containers can communicate with Postgres easily. A container is then run based on the postgres:15beta1 image, a default password is set for the postgres user (you can change this to whatever you like), and the container is given the name of pg15beta1.

Then to run psql:

docker run -it --rm --network pgnetwork postgres:15beta1 psql -h pg15beta1 -U postgres

This spins up another container (which is deleted after you quit psql thanks to the --rm option) which uses the same Postgres image but to run psql instead. Enter the password you provided in the first step and you're in.

To clean up:

docker stop pg15beta1
docker rm pg15beta1

If you have psql locally and want to experiment with Postgres 15 with apps on your local system, you could expose its port on localhost like so:

docker run --name pg15beta1 -e POSTGRES_PASSWORD=whatever -p 1234:5432 -d postgres:15beta1

This then makes the Postgres 15 server accessible at port 1234 on localhost. Tweak to suit your use case.

How to Create a Bootable USB Stick from the Windows 11 ISO on macOS

Peter Cooper — Mon, 23 May 2022 10:30:39 +0000

An unused Intel NUC has been staring at me from across my living room for months, so I decided to torture it with a Windows 11 install.

Microsoft has a Windows 11 ISO you can download now, so I figured I'd use BalenaEtcher to 'burn' it on to a USB stick and boot the NUC from it. Easy. Wrong.

Creating a USB stick from the Windows 11 ISO that actually boots is tricky on macOS. BalenaEtcher, an otherwise useful and common tool for copying images to USB sticks, doesn't like Microsoft's offering for annoying, complicated reasons.

Luckily, there is a way.

Make sure Homebrew is installed. If you're a dev, it probably is.
Mount the Windows 11 ISO in macOS so it appears on the desktop and is available at /Volumes/CCCOMA_X64FRE_EN-GB_DV9 or similar.
Put the USB stick in.
Use diskutil list at the terminal to find your USB stick. It'll be something like /dev/disk5 – I'll pretend it is for the rest of these instructions.
diskutil eraseDisk MS-DOS "w11" MBR disk5
rsync -avh -progress -exclude=sources/install.wim /Volumes/CCCOMA_X64FRE_EN-GB_DV9/ /Volumes/w11
brew install wimlib
wimlib-imagex split /Volumes/CCCOMA_X64FRE_EN-GB_DV9/sources/install.wim /Volumes/w11/sources/install.swm 4000

Ta-da.

(Please note that certain names and numbers above will vary depending on locale or configuration. Make sure to update those as you follow the instructions.)

How to Install the `nano` Text Editor on AWS CloudShell

Peter Cooper — Tue, 15 Dec 2020 21:58:05 +0000

I know the basics of vim and I even know how to exit it(!) but I'd much rather use GNU Nano as my editor at the terminal. Yet, the Amazon Linux 2 distribution with AWS CloudShell doesn't include it!

Here's what to do to install it into your home folder so that it remains in place long-term:

# Install basic developer tools and dependencies
sudo yum -y groupinstall "Development Tools"
sudo yum -y install git-core zlib zlib-devel gcc-c++ patch readline readline-devel libffi-devel openssl-devel make bzip2 autoconf automake libtool bison curl sqlite-devel xz

# Download and compile nano
cd ~
mkdir ~/bin
wget https://www.nano-editor.org/dist/v5/nano-5.4.tar.xz
tar xf nano-5.4.tar.xz
cd nano-5.4
./configure --prefix=$HOME
make && make install

# Clean up and add ~/bin to PATH
cd ..
rm -rf nano-5.4
rm nano-5.4.tar.xz
echo 'export PATH="$HOME/bin:$PATH"' >> ~/.bash_profile
source ~/.bash_profile

Now you can run nano:

nano

How to Install Ruby on AWS CloudShell

Peter Cooper — Tue, 15 Dec 2020 21:32:58 +0000

If you use AWS and like to interact with AWS services from the shell/terminal, there's a new option from today: AWS CloudShell. CloudShell provides a terminal emulator in the browser complete with the tools and authentication credentials needed to work with your AWS resources.

Python, Node, and numerous other tools are preinstalled on the Amazon Linux 2 based instance out of the box, but.. there's no Ruby! Boo!

While installing Ruby on Amazon Linux 2 is no big deal, only the /home folder is persisted between AWS CloudShell runs, so you'll need to take extra care to install it under your home directory.

Without further ado, here's how to get Ruby 2.7.2 running on AWS CloudShell using rbenv and ruby-build:

# Install basic developer tools and dependencies
sudo yum -y groupinstall "Development Tools"
sudo yum -y install git-core zlib zlib-devel gcc-c++ patch readline readline-devel libffi-devel openssl-devel make bzip2 autoconf automake libtool bison curl sqlite-devel

# Install libyaml to $HOME
wget https://github.com/yaml/libyaml/releases/download/0.2.5/yaml-0.2.5.tar.gz
tar xzf yaml-0.2.5.tar.gz 
cd yaml-0.2.5
./configure --prefix=$HOME
make && make install
cd ..
rm -rf yaml-0.2.5
rm yaml-0.2.5.tar.gz

# Install rbenv and ruby-build
git clone https://github.com/rbenv/rbenv.git ~/.rbenv
cd ~/.rbenv && src/configure && make -C src
echo 'export PATH="$HOME/.rbenv/bin:$PATH"' >> ~/.bash_profile
~/.rbenv/bin/rbenv init
echo 'eval "$(rbenv init -)"' >> ~/.bash_profile
source ~/.bash_profile
mkdir -p "$(rbenv root)"/plugins
git clone https://github.com/rbenv/ruby-build.git "$(rbenv root)"/plugins/ruby-build

# Install Ruby itself
RUBY_CONFIGURE_OPTS=--with-libyaml-dir=$HOME rbenv install 2.7.2
rbenv global 2.7.2
gem env home

There's a LOT to digest here, but here's basically what happens:

Install basic development dependencies (gcc, etc.)
Install libyaml manually to get it into /home
Install rbenv and ruby-build
Get ruby-build to build Ruby 2.7.2
Set Ruby 2.7.2 as our default, global Ruby
Run gem env home just to see everything is working

Note that due to only the /home directory sticking around long term, you might find that some gems with native dependencies fail to install. If the dependencies are binary packages, you'll need to install them prior to installing or upgrading those gems.

To install GCC and a big pile of common native dependencies, you can do this:

sudo yum groupinstall "Development Tools"
sudo yum install git-core zlib zlib-devel gcc-c++ patch readline readline-devel libyaml-devel libffi-devel openssl-devel make bzip2 autoconf automake libtool bison curl sqlite-devel libyaml

Notes on an earlier `libyaml` problem..

In an earlier version of this article, I used the normal libyaml packages and Ruby installed fine.. but once the instance had been deleted due to inactivity, gem list failed to work due to libyaml being dynamically loaded from the system.

The error looked like this:

in 'require': libyaml-0.so.2: cannot open shared object file: No such file or directory - /home/cloudshell-user/.rbenv/versions/2.7.2/lib/ruby/2.7.0/x86_64-linux/psych.so (LoadError)

If you really want to use the system libyaml, this can be resolved by reinstalling libyaml each time using sudo yum -y install libyaml and this could be added to ~/.bash_profile but if you install libyaml to $HOME as in the new instructions up top, this problem goes away.

How to Create a AWS Lambda Layer of Your Gemfile / Ruby Gem Dependencies

Peter Cooper — Sun, 06 Dec 2020 00:06:19 +0000

If you use rooling like AWS SAM or Serverless Framework to build and deploy your serverless Ruby functions, they'll take care of building your dependencies on a function by function basis and package them up for deployment. Great! Use them!

But if you prefer to have your own hands on the controls at all times, using gems with native dependencies (e.g. Nokogiri) can be a nightmare! If you prefer to travel light (or, maybe, like me you like coding adhoc functions in the AWS console), you can instead build your dependencies into a Lambda Layer and then use that layer across whatever Lambda functions you like without worrying about dependencies again.

But how? After an hour of trial and error plus a few tips from this neat article (though that one still used SAM), I figured out the magic spell required to locally build Ruby gems and their dependencies in a local Amazon Linux container and turn them into a layer.

Note: This post assumes you're familiar with the aws client and have the credentials to use Lambda with it. If not, this is too advanced for now. Google the aws CLI client and how to create an IAM user with the right permissions.

The code

Without further ado, here's the magic:

LAYER_NAME="my-ruby-layer"
mkdir $LAYER_NAME && cd $_

bundle init
bundle add http --skip-install
bundle add nokogiri --skip-install
rm Gemfile.lock

docker run --rm -v $PWD:/var/layer \
           -w /var/layer \
           amazon/aws-sam-cli-build-image-ruby2.7 \
           bundle install --path=ruby

mv ruby/ruby ruby/gems
zip -r layer.zip ruby

aws lambda publish-layer-version \
           --layer-name $LAYER_NAME \
           --region eu-west-1 \
           --compatible-runtimes ruby2.7 \
           --zip-file fileb://layer.zip

This results in a LayerVersionArn you can use with your Lambda functions. Once you've done this, loading the gems you need in the usual way (e.g. require 'nokogiri') will Just Work™.

If you use the AWS console, it'll let you pick this from a drop down menu which is how I like to do it:

How the code works

Here's a quick break down of what the code above does. First, we create a folder with the name of our layer:

LAYER_NAME="my-ruby-layer"
mkdir $LAYER_NAME && cd $_

Next, we create a Gemfile and add some gems to it. If you already have a Gemfile, copy it in at this point instead of doing this:

bundle init
bundle add http --skip-install
bundle add nokogiri --skip-install
rm Gemfile.lock

Note: I delete Gemfile.lock because the version of Bundler in the Lambda container clashes with mine.

Let's now instruct Docker to run the image Amazon provides for building Ruby 2.7 dependencies and do the bundle install there (while saving it to our normal filesystem):

docker run --rm -v $PWD:/var/layer \
           -w /var/layer \
           amazon/aws-sam-cli-build-image-ruby2.7 \
           bundle install --path=ruby

A quick directory name tweak is necessary to match the structure that Lambda expects. We can then zip up the ruby folder:

mv ruby/ruby ruby/gems
zip -r layer.zip ruby

Finally, publish the layer to AWS Lambda:

aws lambda publish-layer-version \
           --layer-name $LAYER_NAME \
           --region eu-west-1 \
           --compatible-runtimes ruby2.7 \
           --zip-file fileb://layer.zip

Make sure to set the region to the one you actually want to use the layer from. Every region has its own layers.

Note: If you want to deploy your layer to the public or use it more broadly across all regions, you'll want to search up the AWS Serverless Application Repository.

How to Look at Your GitHub API Rate Limits

Peter Cooper — Sat, 05 Dec 2020 18:14:23 +0000

If you're using any GitHub client library worth its salt, it'll have a way to look at your current rate limits and expiry times. I have lots of tools using my GitHub account, however, and wanted an independent way to look at this data.

Luckily, if you have a Personal Access Token (PAT) it's easy to do so.

Place your personal access token in ~/.githubtoken (or any file of your choice) and then you can do this:

curl https://api.github.com/rate_limit -H "Authorization: token $(cat ~/.githubtoken)"

However, I wanted to be able to use the information in a Ruby script and to show the number of seconds until a quota were to reset, so this Ruby code fits the bill:

require 'http'
require 'json'

TOKEN = File.read(ENV['HOME'] + "/.githubtoken").strip

res = HTTP["Authorization" => "token #{TOKEN}"].get("https://api.github.com/rate_limit")
rates = JSON.parse(res.to_s)

def convert_epoch_to_seconds_remaining(r)
  r["resets_in_seconds"] = (Time.at(r["reset"].to_i) - Time.now).ceil if r["reset"]
  r.values.each { |rr| convert_epoch_to_seconds_remaining(rr) if rr.is_a?(Hash) }
end

convert_epoch_to_seconds_remaining(rates)

puts rates.to_json

The only dependency is the http gem, but you should have this anyway as it's fantastic. If you'd rather have no dependencies, you could use open-uri like so:

require 'json'
require 'open-uri'

TOKEN = File.read(ENV['HOME'] + "/.githubtoken").strip

res = open("https://api.github.com/rate_limit", "Authorization" => "token #{TOKEN}").read
rates = JSON.parse(res.to_s)

def convert_epoch_to_seconds_remaining(r)
  r["resets_in_seconds"] = (Time.at(r["reset"].to_i) - Time.now).ceil if r["reset"]
  r.values.each { |rr| convert_epoch_to_seconds_remaining(rr) if rr.is_a?(Hash) }
end

convert_epoch_to_seconds_remaining(rates)

puts rates.to_json

How to Make curl Request a Site from a Different IP Than in DNS

Peter Cooper — Wed, 11 Nov 2020 19:58:36 +0000

One issue I've encountered over the years has been when moving a Web site between systems and the site is set up on two different servers simultaneously (this could also happen if using a CDN). Let's say example.com is set up on 1.1.1.1 and 2.2.2.2 but DNS is pointing solely to 1.1.1.1

What do we do if we want to request http://example.com/ but from 2.2.2.2?

A system wide approach would be adding an entry to the /etc/hosts file:

2.2.2.2 example.com

This usually works but can require you to flush your local DNS cache and/or force refresh in your browser. You have to remember to remove the line as well, it's not very configurable, and so on.

What about if we want to use curl? You could use /etc/hosts but for debugging purposes it'd be nice if we could override the IP address on a single request basis as part of the command. This is possible!

The way I used to do it was to override the Host HTTP header (the way a HTTP client tells a server which host it's looking for) and request from the new IP address, like so:

curl -H "Host: example.com" http://2.2.2.2/

Fine in the old days, but in the modern HTTPS world various problems are introduced. Luckily curl has a solution in the form of --resolve!

--resolve <host:port:address[,address]...>
       Provide a custom address for a  specific  host  and  port  pair.
       Using  this,  you  can make the curl requests(s) use a specified
       address and prevent the otherwise normally resolved  address  to
       be  used.

So if we want to request example.com from 2.2.2.2, we could do this:

curl http://example.com/ --resolve example.com:80:2.2.2.2 
curl https://example.com/ --resolve example.com:443:2.2.2.2

Multiple entries can be added if needed for more complex situations like chains of redirects, say.

It's also possible to use * as a wildcard host so that any host involved in the request goes to the address you specify (and it could also save you a bit of redundancy and typing!):

curl http://example.com/ --resolve *:80:2.2.2.2

Print Files But Without the First Line at the Terminal

Peter Cooper — Sun, 08 Nov 2020 22:53:10 +0000

I had a bunch of CSV files I wanted to join together and they all had the same 'header' line at the start. I didn't want the header line repeating in my concatenation of the files so I wanted to get all except the first line for my concatenation process (using cp).

Turns out there are several ways to do this. Let's say you have a file a.txt containing such:

And your goal is to get:

You can't use tail in the standard way as you might not know how long the file is. But you can use it with a special + number to specify that tail is to begin on a certain line, like so:

tail -n +2 a.txt

You can also use awk to get the job done by specifying that it can return any lines where the line number count is larger than 1:

awk 'NR>1' a.txt

Or you could use sed to delete the first line before displaying the file:

sed '1d' a.txt

Each approach has various pros and cons mostly around which one you can remember in the moment or if you want to use awk or sed in other ways to make other adjustments, but it's handy to have options here.

To conclude, I took my CSV files and joined them like so:

cp 1.csv all.csv
tail -n +2 2.csv >> all.csv
tail -n +2 3.csv >> all.csv
tail -n +2 4.csv >> all.csv

There are smarter ways to do this all in one go, but I only had a few files to do anyway! :-)

How to install the Kristall Gemini client on Raspberry Pi OS

Peter Cooper — Fri, 06 Nov 2020 21:08:37 +0000

Kristall is an open source client for Gemini, a protocol and set of technologies for lo-fi, high performance hypertext.

There are no binary builds for the ARM architecture of the Pi and you need to install some Qt5 related packages. This post documents how I got things running on standard Raspberry Pi OS on a Raspberry Pi 400.

git clone https://github.com/MasterQ32/kristall.git
cd kristall
sudo apt-get install qt5-default qtmultimedia5-dev libqt5svg5-dev
make
./kristall

Beyond this point you could copy kristall to /usr/local/bin or wherever you want to run it from, as well as install the desktop file to get things into the application menu:

sudo mv kristall /usr/local/bin/
cp Kristall.desktop ~/.local/share/applications/kristall.desktop
update-desktop-database ~/.local/share/applications

List unique paths requested by popularity in NGINX logs (access.log)

Peter Cooper — Mon, 19 Oct 2020 12:57:47 +0000

We had a big pile of NGINX access.log files for our site and wanted to quickly know all of the unique paths that had been requested.

If your access.log file(s) follow a reasonably standard format that looks like this:

127.0.154.222 - - [19/Oct/2020:06:26:59 +0000] "GET / HTTP/1.1" 301 178 "-" "-"

.. then you can use this solution:

awk -F\" '{print $2}' access.log | awk '{print $2}' | sort | uniq -c | sort -g

The output will look like this:

[lots of stuff here]
    104 /xmlrpc.php
    114 /wp-includes/wlwmanifest.xml
    121 /robots.txt
    161 /feed/
    336 /
   3056 //xmlrpc.php
  53786 /wp-login.php

So what's going on?

awk -F\" '{print $2}' access.log splits each line on the first quotation mark and returns the second part.

awk '{print $2}' then skips the HTTP verb (GET/POST/PUT/etc.) and prints out the path (which follows the space after the HTTP verb).

sort sorts the output into groups of the same thing which..

uniq -c then turns into a list of the unique paths only. The -c prefixes the output with the number of non-unique lines.

sort -g then sorts the lines in numeric order.

Want the result in descending numeric order? Use sort -gr instead.

How to manually import files into Middleman site builds

Peter Cooper — Tue, 13 Oct 2020 16:14:33 +0000

Middleman is a neat Ruby-based static site generator that provides a good alternative to Jekyll if you want something simpler and not focused on blogging. As such, we're using it for our new company site.

We ran into an annoying bug with it, however.

We're using Netlify to host the new, Middleman-built site and Netlify lets you use a _redirects file to specify all of the redirected URLs you want on your site. Great, except that if you place a _redirects file in your Middleman site, it doesn't get copied across into the final build of the site!

It turns out the reason for this is that Middleman treats files beginning with an underscore as special (since they could be partials) and you need to manually specify the inclusion of such static files in the build.

Add this to config.rb:

import_file File.expand_path("_redirects", config[:source]), "/_redirects"

import_file manually brings a file from one location to another in your final build, so this brings the _redirects file in the site's 'source' directory to live at /_redirects in the final site and Netlify is happy!

How to Create Joined Bulletpoint Lists with CSS, BBC News-style

Peter Cooper — Tue, 07 May 2019 11:56:30 +0000

When there are live events, BBC News often runs a 'timeline' style list against stories. For example:

I was intrigued to work out how they'd got the bullet points of the list items to be dynamically joined by a line, and set off with my trusty Chrome DevTools to figure it out.

And, boy, their markup is a trip!

Long story short, it's a standard HTML list (the BBC uses <ol> but I went with <ul>) where each list item (<li>) has a :before pseudo-element that's empty content-wise but is marked as being 2 pixels wide with a red background color. This creates the 'line' before each <li>. Further styling then positions this pseudo-element/line.

The key part of the CSS is for the pseudo-element:



li:before {
  background-color: #c00;
  width: 2px;
  content: '';
  position: absolute;
  top: 0px;
  bottom: 0px;
  left: 5px;
}

You also need to make sure that the list item itself uses relative positioning so that the position: absolute on the pseudo-element doesn't base itself on the page but on the list item itself:



li {
  /* You need to turn on relative positioning so the line is placed relative to the item rather than absolutely on the page */
  position: relative;

  /* Use padding to space things out rather than margins as the line would get broken up otherwise */
  margin: 0;
  padding-bottom: 1em;
  padding-left: 20px;
}

(Also note that if you want to have gaps between your line items now, you need to use padding, otherwise the line would get broken up by the margins!)

The bullet-points aren't the standard HTML bullet-points but are just circles drawn using inline SVG. You could in theory use any shape (stars, squares, whatever) or maybe even something like an emoji, but I just copied the circle approach that the BBC uses.

Rather than bore you to tears with a technical explanation of how it works, I've boiled it down to the simplest code example I could come up with, added plentiful comments, and put it on CodePen for you to fiddle around with.

Play with it here:

LATE ADDITION:

Saadat on Twitter has extended the concept above a little more by baking the SVG for the bullets into the CSS by using the :after pseudo-element! This makes the code even cleaner. Their code is in this pen but the crux of it is this:



/* Small bullets for normal list items */
li::after {
  content: '';
  position: absolute;
  background-image: url("data:image/svg+xml,%3Csvg xmlns='http://www.w3.org/2000/svg' aria-hidden='true' viewBox='0 0 32 32' focusable='false'%3E%3Ccircle stroke='none' fill='%23c00' cx='16' cy='16' r='10'%3E%3C/circle%3E%3C/svg%3E");
  background-repeat: no-repeat;
  background-size: contain;
  left: 0;
  top: 2px;
  width: 12px;
  height: 12px;
}