DEV Community: Richard Halford

Setting up virtual machines on Arch Linux

Richard Halford — Fri, 27 Aug 2021 15:21:33 +0000

Virtualisation is an incredibly useful tool, even for personal use. Personally, I use virtual machines for different purposes: testing out operating systems or Linux distributions without having to install them to bare metal; install my own software to test compatibility; try out the latest hyped about desktop environment or window manager. And as time goes on, I find more things to add to the list of uses.

Whilst typing this up, I read about how I can use a virtual Debian install to install (not live load) a Debian distribution onto a USB drive. So I can carry around an operating system with persistent storage, and plug it into any PC.

Note: This article will be focussing on how to run virtual machines on Arch Linux.

KVM

Kernel-based Virtual Machine (KVM), is a hypervisor - an emulator or sorts - that comes baked into the Linux kernel. So if you run Linux, you will have this module included. And what this means to you the user is that you can quickly setup up your system for managing VMs.

But even if KVM is included in the Linux kernel installed on your computer, you first need to check whether your CPU has virtualisation support.

$ LC_ALL=C lscpu | grep Virtualization

Depending on which team you opted to support, the output will vary, but either output will confirm you support KVM.

AMD

Virtualization:                  AMD-V

Intel

Virtualization:                  VT-x

With the confirmation that virtualisation is supported, you will probably want to install some packages to make installing and managing VMs a lot more user friendly.

Virtualisation made easy

$ sudo pacman -S qemu libvirt iptables-nft dnsmasq virt-manager

All the below installations in one.

QEMU

QEMU is an emulator and virtualiser that can use KVM for virtualisation, by using Hardware-assisted virtualisation with your CPU. But with the qemu package alone, you will not be able to use a GUI to manage your VMs, or make their sessions have persistent settings. Meaning that to make things run, you will need to use the command-line every time.

$ sudo pacman -S qemu

Libvirt

To overcome these hurdles, it is recommended to install libvirt, a meta-package that contains the tools for managing your VMs in a convenient way. To do this package justice in knowing what it provides, I'd recommend viewing it's Arch Wiki page.

$ sudo pacman -S libvirt

Libvirt daemon

Next you will need to enable and start the libvirt daemon. This will create some necessary symlinks and also enable the virtlogd.service.

$ sudo systemctl enable libvirtd
Created symlink /etc/systemd/system/multi-user.target.wants/libvirtd.service → /usr/lib/systemd/system/libvirtd.service.
Created symlink /etc/systemd/system/sockets.target.wants/virtlockd.socket → /usr/lib/systemd/system/virtlockd.socket.
Created symlink /etc/systemd/system/sockets.target.wants/virtlogd.socket → /usr/lib/systemd/system/virtlogd.socket.
Created symlink /etc/systemd/system/sockets.target.wants/libvirtd.socket → /usr/lib/systemd/system/libvirtd.socket.
Created symlink /etc/systemd/system/sockets.target.wants/libvirtd-ro.socket → /usr/lib/systemd/system/libvirtd-ro.socket.

Then just start the service.

$ sudo systemctl start libvirtd

Libvirt group

To allow non-sudo user access to the newly enabled libvirt daemon, you should now add the necessary users to the libvirt user group.

$ sudo usermod -aG libvirt richard

Internet

You'll probably want to add internet connectivity to your future VM instances. Allowing you to browse the web or download and install software on another operating system.

Typically for personal use, you would want to create a virtual network that uses on the host system's network connectivity. This means installing two more packages. iptables-nft to replace iptables - that comes with a base install of Arch Linux - and dnsmasq, a DNS forwarder and DHCP server.

$ sudo pacman -S iptables-nft dnsmasq

Virtual Machine Manager

The GUI client I'd recommend using to manage your VMs is Virtual Machine Manager, or simply virt-manager. A brilliant bit of free software developed by Red Hat (they also help develop libvirt).

This program allows the user to; create, start, edit, pause and stop VMs. Whilst providing performance metrics for individual VMs.

$ sudo pacman -S virt-manager

Creating a virtual machine

Once all the above steps have been completed, you may need to reboot your system. Then to create a VM, find and download the operating system you want to try out. And start the virt-manager application.

At first you'll be greeted with an empty list of VMs. To create a new VM, click on the monitor with the star and play button in the top left.

File > New Virtual Machine

Then you will be greeted with a VM creation wizard. For this quick run-through I'm going to create a VM for Debian 11 "Bullseye", from a local install media.

Select the downloaded ISO and find the operating system, if virt-manager does not automatically find it for you.

As Debian 11 is new, I had to choose Debian testing manually.

Then choose how much system memory and the number of CPUs you want to give whilst the VM runs.

Then do the same with how much storage you want to provide. This won't automatically take up the given amount. But, sets a cap on the total amount you have made available to it.

Finally, name your VM and select the network solution for it. This tutorial assumed that you would want to provide a virtual network, so that's what has been selected here.

Now the new VM will be listed.

To start it, select the VM and click on the play button to power it on. Then select Open to view it.

If you've made it this far, thanks for reading!

I hope this article has been helpful in understanding some of the basics of virtualisation on Linux. How to set up your system and manage VMs with KVM, QEMU, libvirt and Virtual Machine Manager. And the steps needed to create a new VM using virt-manager.

If you've got any questions about this process, or think I might have missed something, just ask below.

Originally published on xhalford.com

How to remove Windows from Dual-Boot

Richard Halford — Mon, 23 Aug 2021 15:36:07 +0000

Whenever anyone decides to take a dip into the world of Linux, coming from Windows. It's not normally as a straight swap from operating system to operating system. And the distribution you land on using is normally set up in dual-boot.

That's how I got started. Windows paired up with Linux Mint.

"Because I need Windows to play games."

But, with the progress Valve has made with their Proton compatibility layer. I soon realised that I'm never planning on booting Windows up again, and want the storage on my other drive back. So how do we go about removing Windows from a dual-boot system?

Finding Windows

To find out where Windows is installed on your computer, we can use the fdisk or lsblk command-line utilities.

List partitions using `sudo fdisk -l`

$ sudo fdisk -l
Disk /dev/nvme0n1: 232.89 GiB, 250059350016 bytes, 488397168 sectors
Disk model: KINGSTON SA2000M8250G
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: 098B466C-F305-8C4E-8229-C0882B81A9A6

Device Start End Sectors Size Type
/dev/nvme0n1p1 2048 1128447 1126400 550M EFI System
/dev/nvme0n1p2 1128448 9517055 8388608 4G Linux swap
/dev/nvme0n1p3 9517056 488397134 478880079 228.3G Linux filesystem

Disk /dev/sda: 465.76 GiB, 500107862016 bytes, 976773168 sectors
Disk model: Samsung SSD 850
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: 8BF1E112-9DF8-42A4-B571-E81529CE4C89

Device Start End Sectors Size Type
/dev/sda1 2048 206847 204800 100M EFI System
/dev/sda2 206848 239615 32768 16M Microsoft reserved
/dev/sda3 239616 975736384 975496769 465.2G Microsoft basic data
/dev/sda4 975736832 976771071 1034240 505M Windows recovery environment

List partitions using `lsblk -p`

$ lsblk -p
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS
/dev/sda 8:0 0 465.8G 0 disk
├─/dev/sda1 8:1 0 100M 0 part
├─/dev/sda2 8:2 0 16M 0 part
├─/dev/sda3 8:3 0 465.2G 0 part
└─/dev/sda4 8:4 0 505M 0 part
/dev/nvme0n1 259:0 0 232.9G 0 disk
├─/dev/nvme0n1p1 259:1 0 550M 0 part
├─/dev/nvme0n1p2 259:2 0 4G 0 part [SWAP]
└─/dev/nvme0n1p3 259:3 0 228.3G 0 part /

fdisk is a more detailed than the simple output from lsblk. Letting us identify the drive that the different Windows partitions are installed on.

So now we know that for this system, Windows is installed on the /dev/sda drive and has four partitions; sda1, sda2, sda3, sda4. We also know from fdisk that the drive has a GPT partition table: Disklabel type: gpt.

Now we can just delete these partitions and replace it with a single Linux Filesystem partition. And format it to match the ext4 type that the rest of this system uses.

IMPORTANT: If you have split a single drive into partitions for both Linux and Windows. As in, partitions for Linux and Windows are both under /dev/sda or /dev/nvme0n1 and not on either drive, like in this tutorial. Make sure to note down the partition numbers for the; Microsoft reserved, Microsoft basic data, and Windows recovery environment partitions. As you will need to keep the EFI System for Linux to use when booting.

Deleting Windows

To delete the partitions on the /dev/sda drive, just pass the path to fdisk. I recommend using fdisk as it's readily available, and allows for ease of mind if you think you might have messed up. As everything you do can be undone by hitting q, as long as you never write the changes with w.

To view all options available, enter m and hit return. For deleting partitions, we're just going to use the d command. And go through the process of deleting each partition.

If you want to check the partitions on the drive first, type p to print out the partition table.

$ sudo fdisk /dev/sda

Command (m for help): d
Partition number (1-4, default 4) 4

Partition 4 has been deleted.

Command (m for help): d
Partition number (1-3, default 3) 3

Partition 3 has been deleted.

Command (m for help): d
Partition number (1,2, default 2) 2

Partition 2 has been deleted.

Command (m for help): d
Selected partition 1
Partition 1 has been deleted.

Creating a partition

Whilst inside of the fdisk utility, we can now create a new partition, using n.

Command (m for help): n
Partition number (1-128, default 1):
First sector (34-976773134, default 2048):
Last sector, +/-sectors or +/-size{K,M,G,T,P} (2048-976773134, default 976773134):

Created a new partition 1 of type 'Linux filesystem' and of size 465.8 GiB.
Partition #1 contains a vfat signature.

Do you want to remove the signature? [Y]es/[N]o: Y

The signature will be removed by a write command.

Here I just hit enter to input all the default values, to get a partition that was as big as my drive could allow.

Also notice how fdisk automatically selected the Linux Filesystem partition type. If you want to change this to another, use l to view all partition type options, and t to change to one.

Just in case you're curious about why the partition defaults to starting at sector 2048 (i.e. 1 MB) I found a good explanation over on StackOverflow

As this drive will be used to store all of my games, I will only keep this single partition. But the same process with n can be repeated to create more if needed.

If in the future I decide that my games library isn't that big or I want to create a video library. I can resize partition /dev/sda1 and create new one along side it.

Writing table changes

After deleting the Windows partitions and creating a new Linux Filesystem partition. This is what the drive will look like when written.

Command (m for help): p
Disk /dev/sda: 465.76 GiB, 500107862016 bytes, 976773168 sectors
Disk model: Samsung SSD 850
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: 8BF1E112-9DF8-42A4-B571-E81529CE4C89

Device Start End Sectors Size Type
/dev/sda1 2048 976773134 976771087 465.8G Linux filesystem

Filesystem/RAID signature on partition 1 will be wiped.

To commit these changes and write the table to disk and exit, type in w and hit enter.

Command (m for help): w
The partition table has been altered.
Calling ioctl() to re-read partition table.
Syncing disks.

Formatting the partition

To activate this new partition to be used on my system. I need to format this new partition to be of the same type, ext4, by entering the command sudo mkfs.ext4 /dev/sda1.

$ sudo mkfs.ext4 /dev/sda1
mke2fs 1.46.4 (18-Aug-2021)
Discarding device blocks: done
Creating filesystem with 122096385 4k blocks and 30531584 inodes
Filesystem UUID: 4149ff1e-bd86-4d4f-940f-c79cc910ded0
Superblock backups stored on blocks:
        32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208,
        4096000, 7962624, 11239424, 20480000, 23887872, 71663616, 78675968,
        102400000

Allocating group tables: done
Writing inode tables: done
Creating journal (262144 blocks): done
Writing superblocks and filesystem accounting information: done

Mounting the new partition

Once formatted, the drive partition needs to be mounted, and stay mounted every time the system is booted. To do that, we need to decide where to permanently mount this partition.

Where to mount the partition

A common directory mentioned is in a /media directory. But this is usually associated with removable media, so would be more suitable for my USB devices and CDs.

Typically, the /mnt directory is given as such a place. However, this is typically only meant for temporary mounts of storage devices. And if there are any drives already mounted in sub-directories of /mnt, then they will be hidden whilst a drive is mounted to /mnt.

The most practical option I use as there is no standard place set out for permanent secondary drives, is in a sub-directory of /mnt. For example, /mnt/games. So lets create a new directory there.

$ sudo mkdir -p /mnt/games/

The -p flag options refers to --parents, which will make sure to create all parent directories to games/. Although /mnt should be there anyway.

Automatically mount on boot

To make sure the system mounts the drive partition /dev/sda1 in /mnt/games/ on every boot. Edit the /etc/fstab file, which will already contain your current partitions. Using the UUID of the partition, found with the lsblk -f command.

$ sudo /etc/fstab

# Static information about the filesystems.
# See fstab(5) for details.

# <file system> <dir> <type> <options> <dump> <pass>
# /dev/nvme0n1p3
UUID=05ed8f02-f092-40e5-b72b-38cacce9b370 / ext4 rw,relatime 0 1

# /dev/nvme0n1p2
UUID=9a54cd9a-c6af-403a-a32a-cc08c0adc60b none swap defaults 0 0

# /dev/sda1
UUID=4149ff1e-bd86-4d4f-940f-c79cc910ded0 /mnt/games ext4 defaults 0 2

Only the bottom two lines were added to this file.

What we did: - Used the default of using the file system UUIDs to denote the partition identity. - Assigned the mount point directory /mnt/games. - Given the default options for the partition. - Kept dump as 0 to stop automatically backing up the entire drive if you use dump (not always pre-installed). - Given a pass priority of 2, as the root directory should be checked first at boot time.

Mount without rebooting

Once the changes have been saved, we can run sudo mount /mnt/sda1 without having to reboot the computer yet. This will check the /etc/fstab file and use it to know to mount /dev/sda1/ at /mnt/games.

Now if we run lsblk command, under MOUNTPOINTS the file system will be mounted to the correct directory. Meaning that whenever the system is next booted, it will automatically mount to this point.

$ lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS
sda 8:0 0 465.8G 0 disk
└─sda1 8:1 0 465.8G 0 part /mnt/games
nvme0n1 259:0 0 232.9G 0 disk
├─nvme0n1p1 259:1 0 550M 0 part
├─nvme0n1p2 259:2 0 4G 0 part [SWAP]
└─nvme0n1p3 259:3 0 228.3G 0 part /

Giving user privileges

This new file system is still only accessible to the root user, being in the root / directory. To change this, run the following sudo chown -R richard:users /mnt/games.

Updating GRUB

At this point we have now removed Windows from the system. The Boot manager, Grub, will now need to be updated. To display the currently installed operating systems on your computer.

$ sudo grub-mkconfig -o /boot/grub/grub.cfg
Generating grub configuration file ...
Found linux image: /boot/vmlinuz-linux
Found initrd image: /boot/initramfs-linux.img
Found fallback initrd image(s) in /boot: initramfs-linux-fallback.img
Warning: os-prober will not be executed to detect other bootable partitions.
Systems on them will not be added to the GRUB boot configuration.
Check GRUB_DISABLE_OS_PROBER documentation entry.
Adding boot menu entry for UEFI Firmware Settings ...
done

Now when you next restart your system, Windows will no longer be an option in the boot menu.

If you've made it this far, thanks for reading!

I hope this article has been helpful in understanding some of the utilities found on Linux, such as fdisk and lsblk. And that you now understand the file system of your own machine bit better too.

If you've got any questions about this process, or think I might have missed something, just ask below.

Originally published on xhalford.com

Bing Wallpaper: Without Bing

Richard Halford — Wed, 18 Aug 2021 13:00:14 +0000

Whilst Microsoft's Bing might not be many people's first choice of search engine. It has always had one thing going for it: daily homepage wallpapers.

But what about if you don't want to use Bing just for their wallpaper's? Because if we're honest with ourselves, that's an odd reason to stick with a search engine. Let alone decide to make it your browser's homepage.

Or maybe, like me, you're just too lazy and unimaginative to curate your own wallpaper collection. Don't want to keep storage of all those photos. And like the small spice in life that comes from not knowing what's today's wallpaper?

The reason I wanted to find a way of getting these wallpapers, was to create my own simple browser homepage.

Either way, keep reading on as I'll be going through a small script that will find and download the latest Bing daily wallpaper. Then you can do whatever you'd like with it. And hopefully, you'll learn a bit more about some of the cool things you can achieve from shell scripting!

The script

#!/bin/sh

bing="http://www.bing.com"
xmlURL="http://www.bing.com/HPImageArchive.aspx?format=xml&idx=0&n=1&mkt=en-WW"

saveDir=$XDG_CACHE_HOME'/bing-wp/'
mkdir -p $saveDir
picName="bing.jpg"

picURL=$bing$(echo $(curl -s $xmlURL) | grep -oP "<url>(.*)</url>" | cut -d │ ">" -f 2 | cut -d "<" -f 1)

curl -s -o $saveDir$picName $picURL

exit

Requirements: curl

Breaking it down

This shebang simply tell the parent shell which interpreter to use when running the script.

#!/bin/sh

Tells the interpreter to use your system's shell. Typically this will be either dash or bash.

Here we're setting the variables to be used for creating the URL of the latest Bing wallpaper.

bing="http://www.bing.com"
xmlURL="http://www.bing.com/HPImageArchive.aspx?format=xml&idx=0&n=1&mkt=en-WW"

Then we define the location where the image will be saved to, once downloaded. By using the mkdir utility, to create the folder at the location defined as $saveDir. The -p option flag here, is used to make sure all the necessary parent directories up the branch of bing-wp/ are also created. And finally, the variable picName is assigned to create the image file.

saveDir=$XDG_CACHE_HOME'/bing-wp/'
mkdir -p $saveDir
picName="bing.jpg"

As every time this script is ran, it will write over the current image. $XDG_CACHE_HOME is used to denote the appropriate location for a temporary file, using the XDG Base Directory Specification. It merely represents in most *nix systems, the ~/.cache/ directory.

We can break down this nested and piped variable assignment into more understandable parts.

picURL=$bing$(echo $(curl -s $xmlURL) | grep -oP "<url>(.*)</url>" | cut -d ">" -f 2 | cut -d "<" -f 1)

Like before, this is simply reassigning Bing's base URL to a variable.

picURL=$bing
# picURL=http://www.bing.com

The curl command silently, -s - meaning without progress or error reporting - goes to the URL address assigned as xmlURL, and outputs it's contents.

$(curl -s $xmlURL)
# <?xml version="1.0" encoding="utf-8" ?><images><image><startdate>20210817</startdate><fullstartdate>202108170000</fullstartdate><enddate>20210818</enddate><url>/th?id=OHR.PochuckValley_ROW7250263541_1920x1080.jpg&amp;rf=LaDigue_1920x1080.jpg&amp;pid=hp</url><urlBase>/th?id=OHR.PochuckValley_ROW7250263541</urlBase><copyright>Stairway to Heaven trail, Wawayanda State Park, New Jersey, USA (© Leembe/Getty Images)</copyright><copyrightlink>https://www.bing.com/search?q=wawayanda+state+park+new+jersey&amp;form=hpcapt</copyrightlink><headline>Info</headline><drk>1</drk><top>1</top><bot>1</bot><hotspots></hotspots></image><tooltips><loadMessage><message>Loading...</message></loadMessage><previousImage><text>Previous image</text></previousImage><nextImage><text>Next image</text></nextImage><play><text>Play video</text></play><pause><text>Pause video</text></pause></tooltips></images>

As an example, this is what that resulting XML looks like, as of writing this article. We'll use it as an example to explain the rest of the script.

Within this long line of XML are the <url> and </url> tags. Containing the day's Bing wallpaper URL parameters within. To separate this information from the curl commands output. We need to pipe that resulting string into grep.

grep -oP "<url>(.*)</url>"
# <url>/th?id=OHR.PochuckValley_ROW7250263541_1920x1080.jpg&amp;rf=LaDigue_1920x1080.jpg&amp;pid=hp</url>

Using the -P flag to perform the pattern matching with Perl-compatible regular expressions. Combined with the -o option to only return the matching pattern, and not just highlight it.

(.*) just creates a capturing group of everything between the tags.

Now we just need to remove the <url> and </url> tags surrounding the resource name and parameters of the image address. This will be done by piping this text through the cut command, twice.

cut -d ">" -f 2
# /th?id=OHR.PochuckValley_ROW7250263541_1920x1080.jpg&amp;rf=LaDigue_1920x1080.jpg&amp;pid=hp</url

cut -d "<" -f 1
# /th?id=OHR.PochuckValley_ROW7250263541_1920x1080.jpg&amp;rf=LaDigue_1920x1080.jpg&amp;pid=hp

Using the delimiters, -d, ">" and "<" to separate the tags from the URL segment. And then selecting the delimiter separated fields, -f, to capture the URL.

This snippet of the image URL is then concatenated with the $bing variable using echo.

picURL=$bing$(echo ...)

# picURL=http://www.bing.com/th?id=OHR.PochuckValley_ROW7250263541_1920x1080.jpg&amp;rf=LaDigue_1920x1080.jpg&amp;pid=hp

Finally, putting this altogether, we can call curl to silently -s get the image with $picURL, now pointing towards the day's image. And direct the output -o to the be saved as bing.jpg, inside the $XDG_CACHE_HOME/bing-wp/ directory.

curl -s -o $saveDir$picName $picURL

Remembering to exit the script at the bottom of the file.

Options for Running the Script

I won't get into how to do it here, but just as a pointer on how you can now take this script to run and get the latest Bing wallpaper. Here are a few options on how you might want to go about running the script everyday.

.xinitrc
systemd Timer
CronJob

Hopefully, after reading this tutorial on how to get Bing's latest daily wallpaper, you have learnt something new about shell scripting along the way. Thanks for reading through to the end, and if you've got any questions, ask below!

Originally published on xhalford.com

How to improve the Arch Linux package manager

Richard Halford — Fri, 13 Aug 2021 17:01:53 +0000

Pacman is the package manager for Arch Linux. And out of the box, it's just black and white, and single threaded all over. But, even if you're one to tout "I use Arch btw." and what it's like living on the bleeding edge with it's rolling release updates. You'll likely still want to make those updates take as little time as possible, with important information being colour highlighted.

Luckily pacman comes with a way to make both of these the case, with no more than an edit to it's configuration file.

Colour Output

To make sure pacman produces a colourful output in the terminal. Open up /etc/pacman.conf in your editor of choice, with sudo/doas privileges. Then simply delete the # before Color, under the # Misc options sub-heading.

  # Misc options
  #UseSyslog
- #Color
+ Color
  #TotalDownload
  CheckSpace
  #VerbosePkgLists

Now save and exit, and the next time you update your system packages, pacman will use the same colour scheme as your terminal/tty.

Multi-threaded Performance

With the release of Pacman v6.0.0, comes the ability to have parallel package downloads. Meaning all of your small system packages can be downloaded, all whilst getting the latest Linux kernel.

Because of reasons beyond the scope of this article (read: I don't feel like writing about it), this shouldn't throttle the speed of each download stream. As the Arch repository mirrors you're getting your package updates from, probably aren't sending packets of data to you as fast as your network connection can handle. Making parallel downloads only seem like a good thing. With resulting average install times that are less than just downloading package updates sequentially.

To add this latest feature, you will have to make the following edit, just like with the Color option mentioned above.

  # Misc options
  #UseSyslog
  Color
  #TotalDownload
  CheckSpace
  #VerbosePkgLists
+ ParallelDownloads = 5

If your install of Arch Linux comes after the release of Pacman v6.0.0, you should find that ParallelDownloads option is already there. And just needs to be 'un-commented'.

The number you assign to ParallelDownloads denotes how many download streams you want to enable. By default, this value is set equal to 5, but can be changed depending on your system's hardware. A rule of thumb being adopted seems to be to match the number of threads on your CPU. But, you may find that testing above this may be beneficial, without resulting in performance drops.

Originally published on xhalford.com

Using hook scripts with Certbot

Richard Halford — Fri, 13 Aug 2021 11:28:00 +0000

Today there are a number of ways to setup your email server. Regardless of how, you will have to create an SSL certificate for the domain you plan to host your email on. And a common and convenient way of doing this, is through the instruction generator Certbot. A free and open-source (FOSS) tool created by the Electronic Frontier Foundation (EFF).

"Certbot is a free, open source software tool for automatically using Let’s Encrypt certificates on manually-administered websites to enable HTTPS. Certbot offers domain owners and website administrators a convenient way to move to HTTPS with easy-to-follow, interactive instructions based on your webserver and operating system." - EFF

One of my favourite features of Certbot has come from the convenience of automatically checking and renewing my certificates for all the websites I host. Without having to worry whether or not I should log on to my server and manually renew them. Allowing the ability to fetch my email's to read and reply with, from desktop or mobile. And not be told that my email server isn't trusted without an SSL certificate.

A problem I have found regarding the certificate renewals, however, comes from the way the IMAP server (Dovecot in my case) doesn't reload in order to use the updated certificate. Leaving me email-less until I reload the service.

Online you might find the solution to be as simple as issuing $sudo systemctl reload dovecot, after Certbot has finished doing it's business. But, when I'm trying to check email on my phone, away from a command-line to ssh into. This isn't exactly practical. And I don't want to set a 60 day reminder to just run this command.

Luckily, Certbot comes with the ability to run pre-, post- and deploy-hooks. Simply by adding a shell script to the appropriate folder, found under the /etc/letsencrypt/renewal-hooks/ directory.

Don't forget to make the file executable with the sudo chmod +x /path/to/file command.

Just add the following script to a file - named whatever you want - and put it in /etc/letsencrypt/renewal-hooks/deploy/.

#!/bin/sh
do
        if [ "$domain" = mail.example.com ]
        then
                systemctl reload dovecot
        fi
done

Now every time Certbot runs and successfully issues a certificate to the email server's mail.example.com domain. This little convenience script will run, and subsequently reload Dovecot, the IMAP server.

Originally published on xhalford.com

GoDo: A command line todo list application

Richard Halford — Thu, 12 Aug 2021 18:48:53 +0000

About
Getting Started
- Requirements
- Installation
- Recommendations
- Configuration
GoAPI
- Authentication
- Server
- Daemonize
Licence

About

GoDo aims to help you get organised with your tasks. Designed to be simple and accessible.

Usage:
  godo [command]

Available Commands:
  add         add a new todo
  done        toggle todo as done
  edit        edit a todo
  find        search for a given string
  help        help about any command
  list        list your todos
  priority    label a todo as a priority
  remove      remove a todo
  version     print godo's version

Flags:
      --datafile string   data file to store todos
  -h, --help              help for godo

Use "godo [command] --help" for more information about a command.

Go get things done and checked off the list.

Getting Started

Requirements

The best way to ensure GoDo will work on your machine, is to compile it yourself.

Go (to compile applications)

To do this, all you need is to have Go - the programming language - installed on your computer.

Installation

To install GoDo, all you have to do is run the go get command.

$ go get -u github.com/rsHalford/godo

Recommendations

If you want to have your GoDo list available online, and accessible to other computers. There is GoAPI. A REST API which can handle the JSON created by GoDo.

To edit the necessary variables to hook up GoAPI with a database. You will need to clone the GoAPI repository.

$ git clone github.com/rsHalford/goapi

Then after making the necessary changes to the source code. Build the GoAPI binary, for the operating server it will be ran on.

$ env GOOS=linux GOARCH=amd64 CGO_ENABLED=0 go build github.com/rsHalford/goapi

This example command will build GoAPI to be executable on Debian 10.

Then you would need to have a web server installed and a way to run GoAPI as a daemon.

Apache2 (ProxyPass to your web address)
Systemd (daemonize GoAPI to run in the background)

These recommendations are based on what I found to work best for my setup.

Configuration

The config.yaml can be edited to set-up a connection to GoAPI. As well as, select your preferred text editor.

This file will read from ${XDG_CONFIG_HOME:-$HOME/.config}/godo/config.yaml.

username: "admin"
password: "secret"
api: "https://example.com/api/v1/todo"
editor: "vim"

If you prefer to use a local JSON file to store your todo list, leave the api address blank - "".

By default, GoDo will use whatever text editor you have set as your system's default - $EDITOR.

GoAPI

GoAPI was built to support the GoDo application, in providing a RESTful API.

To setup the server application, there a three changes that need to be made.

Authentication

To secure your todo list online, you will need to change the api_username and api_password variables, required to access the API endpoints. These variables are found in the config.yaml file.

These values will be what you send to the API with each request, using Basic Authentication.

An example of the config.yaml can be found in the project's repository

api_username: "username"
api_password: "password"

Currently GoAPI only supports PostgreSQL. To link up the server to a database, make sure to edit the config.yaml. Providing the db_username, db_password, name and port - relating to your database address.

db_username: "username"
db_password: "password"
name: "goapi"
port: "5432"

Server

GoAPI by default serves on port :8080. As an example, below is a basic Apache configuration file to relay GoAPI to your domain.

Apache Configuration Example:

<VirtualHost *:80>
        ServerName goapi.example.com
        ServerAdmin webmaster@example.com
        ProxyRequests Off
        <Proxy *>
                Require all granted
        </Proxy>
        ProxyPass / http://127.0.0.1:8080/
        ProxyPassReverse / http:127.0.0.1:8080/
        ErrorLog ${APACHE_LOG_DIR}/goapi-error.log
        CustomLog ${APACHE_LOG_DIR}/goapi-access.log combined
</VirtualHost>

Daemonize

To have GoAPI run at all times in the background. You will need to make it run as a daemon. This is possible by creating one as a service with your init system - such as with systemd.

Systemd Service Example:

[Unit]
Description=GoAPI RESTful API

[Service]
ExecStart=/path/to/goapi/excutable
WorkingDirectory=/path/to/goapi
StandardOutput=journal
StandardError=inherit
SyslogIdentifier=goapi
User=user
Group=group
Type=simple
Restart=on-failure

[Install]
WantedBy=multi-user.target

Licence

GoDo and GoAPI are both released under the GNU General Public License v3.0.

Part 5: Arch Linux Installation Guide

Richard Halford — Thu, 12 Aug 2021 15:20:53 +0000

Originally published on xhalford.com

In part 5 of this series, we will be creating a user, adding access to more programs and setting up a graphical environment. Then we can finally boot up into out fresh install of Arch Linux. This follows on from installing Arch, configuring it's setup and boot process in previous part of this guide series.

Notes: There are a few presumptions being made for this guide. Regarding yourself and your current setup.

You will be moving from another *nix system.

This will help with the initial steps, with the programs being used to download the image and write it to a USB drive.

If you are coming from Windows, I will try to link to articles explaining this fairly trivial difference.

Have access to this guide on another device.

As once we get to installing Arch Linux, we'll be stuck in a terminal for some major steps.

Can handle looking deep into the empty void of the tty.

Make sure not to make any typos - as I try to do the same throughout these guides.

What you'll need:

The patience to re-read every command you type, and output given.
It's not the end of the world if you "mess up" at any stage here. You'll just save yourself the time of redoing anything, if you do.

Creating a User

When using the new installation, we don't want to be the root user. Therefore, we should now create a user, and give it a password.

$ useradd -m richard
$ passwd richard

When logged in as this new user, we probably want to be able to have access to some necessary functions. Here we will use the usermod command to do this. Selecting a few appropriate groups to add the user to.

$ usermod -aG wheel,audio,video,input,storage richard

This command will output and error if you use spaces between commas.

Sudo Privileges

As we want to give this user root privileges, by adding it to the wheel group. Without having to rely on entering root with su, and exiting out of the interactive shell every time. We can install the package, sudo, for the ability to perform one-off root commands.

$ pacman -S sudo

To make sure this user has the privileges of root, thanks to being added to the wheel group. We need to comment out the following line in the sudoers file.

 $ EDITOR=nano visudo

%wheel ALL=(ALL) ALL

So that you don't need to enter a password to use sudo, as a wheel group member. You can also uncomment the next line in the file, %wheel ALL=(ALL) NOPASSWD: ALL

Network Manager

For future's sake, and making life easier every time you boot up your computer. You'll want to automatically connect to the internet. For this, a very useful package is NetworkManager. Which comes with both a CLI (nmcli) and TUI (nmtui) program, to view, edit and activate network connections.

$ pacman -S networkmanager

Now to make sure the init system, systemd, has NetworkManager enabled and will run when the computer starts up.

$ systemctl enable NetworkManager

Extending Program Availability

At the moment, the only way to download and install, comes from Arch's official stable repositories; core, extra and community.

If you want to have access to the Arch User Repository, AUR. Which will provide you with an endless amount of programs. We will first need to install the base-devel metapackage, to provide the tools necessary to for compiling software. Including git, a version control system, which will allow you to clone programs from online repositories.

$ pacman -S --needed base-devel git

To make life easier when installing packages from the AUR, it's recommended to install an AUR helper. My recommendation here is to use Yay, as it works in a similar way to pacman.

We will move into the /opt/ directory to install yay, as it seems like an appropriate directory. Given it's meant for installing add-on application software packages. Then we will clone yay, give our new user access to it and then build it.

$ cd /opt
$ git clone https://aur.archlinux.org/yay.git
$ chown -R richard:wheel ./yay
$ cd yay/
$ makepkg -si

And that's it for extending software availability on Arch Linux. Now we can focus on making the operating system user friendly, and add a graphical user interface.

Rebooting Arch

With Arch Linux finally installed, added to the bootloader menu, and with a user to log in with. We can now after all this time, reboot the computer. First we need to exit the chroot environment, unmount all partitions, and shutdown the computer.

$ exit
$ umount -R /mnt
$ shutdown now

With this done all you have to do is remove the USB installation media and hit the power button.

With that done. You should now be prompted to enter the desktop (hostname) login and password for your user.

🥳 You're done! 🎉

But... things still look the same.

Graphical User Interface

So far, we've spent our lives in tty purgatory. Staring at a black screen with harsh white text. We need to get out of this. We need to add a display server and graphical environment.

Display Server & Graphics Drivers

On Linux there a two major display servers available, Xorg and Wayland. In this tutorial I am going to go over installing and setting up with Xorg. It's got more support, and a greater user-base. If you have problems in the future, it'll be easier to solve with Xorg.

To make sure you get all necessary utilities that go along with Xorg. We will install the xorg meta package.

$ pacman -S xorg

Followed by installing the hardware drivers relating to your current system.

If you have an NVIDIA GPU, I would actually suggest using their proprietary drivers 😱

For further explanations for your specific graphics driver.

Desktop Environment or Window Manager

Traditionally, operating systems and Linux distributions come bundled with a desktop environment and programs. You might already be used to using one, but here are a few that you can check out. Including what packages and configurations you may need to get it working.

These desktop environments come with a; file manager, terminal emulator, text editor, icons, browser, etc.

If you want to have a more custom and light-weight graphical user interface. You might want to opt for a window manager like dwm, i3, or Openbox. Then you can decide what extra programs you want to have without the added "bloat" of what comes with desktop environments.

Display Manager

If you chose to use a desktop environment, you will most likely have a display manager. But, if you installed a window manager, you will either have to install one yourself or forgo having one. Without a display manager, you will have to start your Xorg server with the startx command. To make this command run automatically when you log in, edit the systems shell initialisation file.

if [[ "$(tty)" = "/dev/tty1" ]]; then
  exec startx
fi

Remove exec and just have startx if you want to kill the server but remain logged in.

This if statement will run startx if you log in on tty1. But will keep you in the tty if you need to, when using the other five. Which can be helpful if you need to debug a graphical problem. Accessing these other tty's with, Ctrl+Alt+F2 up to F6 on Arch Linux.

User Directories

To create all the standard directories within you user's home directory (Documents, Downloads, etc.). It is useful to install the xdg-user-dirs packager. Which helps keep your /home/ directories organised, by using the XDG Base Directory specification.

$ pacman -S xdg-user-dirs
$ xdg-user-dirs-update

The generated ~/.config/user-dirs.dirs file can be edited to fit your use case. This is what mine looks like given I don't have a desktop with my window manager.

XDG_DESKTOP_DIR="$HOME/"
XDG_DOWNLOAD_DIR="$HOME/Downloads"
XDG_DOCUMENTS_DIR="$HOME/Documents"
XDG_MUSIC_DIR="$HOME/Media/Music"
XDG_PICTURES_DIR="$HOME/Media/Pictures"
XDG_VIDEOS_DIR="$HOME/Media/Videos"
XDG_TEMPLATES_DIR="$HOME/Templates"
XDG_PUBLICSHARE_DIR="$HOME/Public"

Notice how I have also consolidated my pictures, videos and music directories within a Media/ directory. To remove even more /home/user/ clutter.

Summary

At the end of this final part in the Arch Linux Installation Guide, we have covered a lot.

Creating a user account and given it root priveleges.
Installed a way to manage our internet connection.
Provided ourselves with a way of downloading programs from almost anywhere.
Finally rebooted our system, and into our new home.
Added a graphical server and looked at potential desktop environments and window mangers.
Gone through two methods to automatically login and keep things clean.

If after all five parts of this guide you have questions. I will try my best to answer them in the comments below. The best resource really is the Arch Wiki. And would be my recommended first stop before installing, configuring or debugging a problem.

Thank you for taking the time to read through this guide, and I hope you have managed to get your Arch Linux installed and operational.

Follow me on Twitter!

Part 4: Arch Linux Installation Guide

Richard Halford — Thu, 12 Aug 2021 15:17:54 +0000

Originally published on xhalford.com

In part 4 of this series, we will be installing essential packages. Configuring the system. And adding a boot loader to the system. This follows on from the drive partitioning, formatting and mounting in the previous part of this guide series.

Notes: There are a few presumptions being made for this guide. Regarding yourself and your current setup.

You will be moving from another *nix system.

This will help with the initial steps, with the programs being used to download the image and write it to a USB drive.

If you are coming from Windows, I will try to link to articles explaining this fairly trivial difference.

Have access to this guide on another device.

As once we get to installing Arch Linux, we'll be stuck in a terminal for some major steps.

Can handle looking deep into the empty void of the tty.

Make sure not to make any typos - as I try to do the same throughout these guides.

What you'll need:

Maintain the internet connection you setup in part 2.
The patience to re-read every command you type, and output given.
It's not the end of the world if you "mess up" at any stage here. You'll just save yourself the time of redoing anything, if you do.

Installing Arch Linux

Arch Linux uses the package manager, pacman. The Debian equivalent to apt or Fedora's dnf. Before we can do any of that, we need to install three essential packages. That provide everything necessary for you to tell other people that you "use Arch Linux by the way."

In the future I may write about pacman's repository system, and the different ways to use it. But carrying out the commands later on in this tutorial are simple enough.

For now we'll be using the pacstrap script to install packages to the new root directory. Installing the base package, the Linux kernel, and the Linux firmware package to help support common hardware on your device.

$ pacstrap /mnt base linux linux-firmware

The linux kernel package can be changed for the linux-hardened, linux-lts or linux-zen kernels.

Leave that to do it's work for a few minutes...

Then, once done, we'll go through some configurations for the system.

Generating the fstab File

As the Arch Wiki states;

"The fstab file can be used to define how disk partitions, various other block devices, or remote filesystems should be mounted into the filesystem."

To generate this fstab file, enter the below command.

genfstab -U /mnt >> /mnt/etc/fstab

Change Root Into Your New System

Chroot command to change root into this new installation.

arch-chroot /mnt

Now you can use the ls command to view what the file structure of your new home will look like. Look's nice, huh?

Set Time Zone

As we are now in a new installation, we need to set the time zone. To do this, we need to know which region we are from.

$ ls /usr/share/zoneinfo/

And what city is the nearest one to your location.

$ ls /usr/share/zoneinfo/region/

Remembering to replace /region/ with one that is appropriate to yourself.

Now we know the closest location. We'll use the following command to force create a symbolic link - using the -sf arguments.

$ ln -sf /usr/share/zoneinfo/REGION/CITY /etc/localtime

Entering the following, will generate the adjtime file.

$ hwclock --systohc

Localisation

With the system's time zone set, we need to select and then generate the locales. Which are required for certain programs and libraries to function as expected.

We do this by editing the /etc/locale.gen file. Uncommenting locales that relate to you. For example, in the US, this will likely be en_US.UTF-8 UTF-8 - found on line 177.

Using an Editor

As Arch doesn't come with a text editor installed with the base package. We need to install one to make the edit to the locale.gen file.

My recommendations here are to use the beginner-friendly command-line interface editor nano. As it shows the appropriate Ctrl, ^, commands to save and exit the program, once done uncommenting the locales.

Otherwise, install and use vim. And let your mind expand.

To do this, enter the following command - with your chosen editor as the argument.

$ pacman -S nano

Now you can edit that locale.gen file. Once the correct lines have been uncommented, generate the locales.

$ locale-gen

Set the LANG Variable

Next up is to create and then set the LANG variable.

$ nano /etc/locale.conf

LANG=en_GB.UTF-8

Entering the same information for the variable, as you uncommented from the locale.gen file.

Keyboard Layout

Like we did in part 2 of this series, we need to set the keyboard layout for our region. Refer to that article section, to find your KEYMAP value.

$ nano /etc/vconsole.conf

KEYMAP=uk

Network Configuration

Create and edit the hostname file. Here you will enter the name you want your device to be known as on your network.

$ nano /etc/hostname

desktop

Then we need to add our hosts to the /etc/hosts file. Using our hostname (e.g. desktop) where appropriate.

$ nano /etc/hosts

127.0.0.1   localhost
::1         localhost
127.0.1.1   desktop.localdomain desktop

Setting the Root Password

So we don't have to chroot when we load back up. It's best to set the root password now. Using the passwd script, create your password and confirm it.

$ passwd
New password:
Retype new password:
passwd: password updated successfully

Bootloader

Before we're done with setting up the base install and configuring it. We need to to install a few packages to make sure the UEFI firmware can find and start the bootloader, which is in the EFI system partition.

Basically, when you start your computer, Arch Linux will be listed and can be started.

The almost "standard" bootloader these days is GRUB. As it can support our chosen firmware (UEFI), partition table (GPT), can multi-boot, and supports our Linux file system partition (ext4).

To setup GRUB, we also need to install a few other packages that will support it's installation.

$ pacman -S grub efibootmgr os-prober dosfstools mtools

grub is the bootloader.
efibootmgr writes the boot entries, making Arch Linux discoverable.
os-prober is used to discover other installed operating systems on your computer.
dosfstools allows the system to create, check and label FAT32 file systems.
- e.g. format a USB to FAT32.
mtools allows you to access FAT file systems.

Now we need to create an EFI directory in /boot/.

$ mkdir /boot/efi

We can now mount the EFI system partition to that new directory.

$ mount /dev/nvme0n1p1 /boot/efi

Then we need to run the GRUB installation script, grub-install. Which will install the GRUB EFI application and install it's modules.

$ grub-install --target=x86_64-efi --efi-directory=/boot/efi --bootloader-id=grub_uefi --recheck

When the computer starts up, it loads the GRUB configuration file, /boot/grub/grub.cfg. So we need to generate this file.

$ grub-mkconfig -o /boot/grub/grub.cfg

We're Done! (mostly)

At this point you can exit the chroot environment, umount -R /mnt and enter reboot. Before doing that, however, this is a good place to your user account and install a few more packages. Making your new home a bit nicer to look and easier to use, than this black void.

Follow me on Twitter!

Part 3: Arch Linux Installation Guide

Richard Halford — Thu, 12 Aug 2021 15:10:50 +0000

Originally published on xhalford.com

In part 3 of this series, we will be partitioning, formatting and mounting a drive to install Arch on. This follows on from the live environment setup completed in the previous part of this guide series.

Notes: There are a few presumptions being made for this guide. Regarding yourself and your current setup.

You will be moving from another *nix system.

This will help with the initial steps, with the programs being used to download the image and write it to a USB drive.

If you are coming from Windows, I will try to link to articles explaining this fairly trivial difference.

Have access to this guide on another device.

As once we get to installing Arch Linux, we'll be stuck in a terminal for some major steps.

Can handle looking deep into the empty void of the tty.

Make sure not to make any typos - as I try to do the same throughout these guides.

What you'll need:

The patience to re-read every command you type, and output given.
It's not the end of the world if you "mess up" at any stage here. You'll just save yourself the time of redoing anything, if you do.

Partitioning

First, we need to partition our hard drive. Creating a space for the boot, file system and swap partitions. For this, we'll use fdisk, a common Linux utility.

To get started, use the below command to list view the hard drive and it's current partitions.

$ fdisk -l

This should give you a similar view as the lsblk command. Used when preparing the USB drive in part 1.

Now you should be able to identify which drive you have to partition. Next, you will input this drive path as an argument to the fdisk command.

$ fdisk /dev/nvme0n1

Remembering to replace the path given above, with one that corresponds with your system (e.g. fdisk /dev/sda or fdisk /dev/nvme0n1).

This will then prompt you with a series of commands to enter. To save some time I have provided you with some pretty standard options. You can tailor these to what you require - if you know what you're doing.

$ fdisk /dev/nvme0n1
$ m
$ g
$ n
$ 1
$ return
$ +550M

m = shows the help message
g = creates a gpt partition table
n = creates a new partition
1 = partition number using the default
return = first sector starts at 2048 as default
+550M = gives the partition a size of 550M

This should now have created the first partition, that will be used for the EFI system.

You can list this new partition by entering p. This will show device start and end sectors (in mebibits, Mi) and type as "Linux filesystem".

At this point you haven't actually committed any changes yet to your drive. So don't worry if you think you've messed up. As you can fdisk /dev/nvme0n1 and select d to delete a partition, or q to quit without saving all changes.

Swap Partition

To some, the creation of a swap partition is an optional step. But, given how affordable storage space is, the impact of having one is almost negligible. Especially when compared to losing all unsaved work, when your RAM fills up. And your computer needs to be restarted. Also, if you're still one to use hibernation, you will need at least the same amount of swap space as you do RAM, to store your system's state.

So creating this partition is much like the previous step. Hit n, and we'll create the swap partition. Enter 2 for the partition number. Go with the default start position. And finally select how much swap space you want to assign.

$ n
$ 2
$ return
$ +4G

I'm choosing a swap space partition size of 4GB as I have 16GB of RAM. Some might see this as too much and some as too little. Here's an easy to use table by people who know more about this than I do.

File System Partition

Now to create the partition for the system file system. Where your root directory will be created. Again this means following similar steps as before.

$ n
$ 3
$ return
$ return

Selecting the default partition size, by entering return for the deault value. You will be assigning all remaining drive space for the Linux file system.

Assigning Partition Types

Before assigning a type to each partition. We will view what we have done so far, by entering p, to show the current partition table.

Device          Start   End       Sectors   Size    Type
/dev/nvme0n1p1  2048    1128447   1126400   550M    Linux filesystem
/dev/nvme0n1p2  1128448 9517055   8388608   4G      Linux filesystem
/dev/nvme0n1p3  9517056 488397134 478880079 228.3G  Linux filesystem

This is what my table looks like with a 256GB drive.

So far, all partitions are of the type "Linux filesystem". But we need to change the first two, to create both the EFI System and Linux swap partitions.

First up will be the 550M partition (1). Entering the below key presses to;

change partition type, t
select the correct partition, 1
list available partition types, L
choose the EFI System partition type, 1

$ t
$ 1
$ L
$ 1

Then we will do the same to the sencodn partition, creating the swap partition.

$ t
$ 2
$ L
$ 19

As the third partition is already of the right type to hold the root directory we can leave it as is. We can then view the partition table, p.

Device          Start   End       Sectors   Size    Type
/dev/nvme0n1p1  2048    1128447   1126400   550M    EFI System
/dev/nvme0n1p2  1128448 9517055   8388608   4G      Linux swap
/dev/nvme0n1p3  9517056 488397134 478880079 228.3G  Linux filesystem

If you're happy with how everything looks. Hit w, and this new table will be saved.

Formatting Partitions

Now the drive partitions have been created and assigned a type. They will now need to be formatted.

EFI System Partition

As the EFI system partition is independent of the operating system being run. This means you can add a Windows operating system to your computer. Either as a separate drive or another set of future partitions.

With the EFI system partition, we will choose to go with a FAT32 file system. Instead of the standard Linux format, ext4. The Arch Wiki has a handy page all about the EFI system partition, if you have any further questions.

To format the partition, enter the following.

$ mkfs.fat -F32 /dev/nvme0n1p1

Notice the additional partition number, p1, added to the end of the drive path.

Linux Swap Partition

Then we will initialise the swap partition, p2.

$ mkswap /dev/nvme0n1p2

Linux File System Partition

Finally before mounting, we will format the Linux filesystem as ext4.

$ mkfs.ext4 /dev/nvme0n1p3

When it starts to create the journal it may take a few seconds.

Mounting the Formatted Partitions

Now we can mount the formatted partitions and enable the swap space.

$ swapon /dev/nvme0n1p2
$ mount /dev/nvme0n1p3 /mnt

Done. 🎉

We're almost there!

Follow me on Twitter!

Django: Creating An App

Richard Halford — Thu, 12 Aug 2021 14:48:20 +0000

Projects vs Apps?

Django Apps

Django apps are essentially individual feature sets for a website. A few examples of Django apps could be;

Blog app that helps you create and edit your own articles online.
User authentication app, where you can create a login system for your website.
Chat app where you can connect users who can then message each other.

The possibilities are endless, but the idea is to create apps that serve one specific purpose. Then you can use that app in other Django projects. For example, your login system app can be plugged into any future Django projects that need user account creation.

Django Projects

Projects are a collection of apps, and they provide the configurations necessary for the apps to work with each other. As the official Django documentation puts it, "A project can contain multiple apps. An app can be in multiple projects."

There is also a Django apps website djangopackages.org. Where you can find apps that other developers have created, that you can add to your project.

Creating the app

First, make sure to be in the correct directory and activate your python virtual environment.

$ cd django_project
django_project $ source project_venv/bin/activate
django_project (project_venv) $

Now you should be in the same directory as manage.py. To create the app, that we'll call "app".

django_project (project_venv) $ python manage.py startapp app

This should have created the app directory, and filled it with the following files.

django_project/
    project_venv/
    project/
    app/
        migrations/
           __init__.py
        __init__.py
        admin.py
        apps.py
        models.py
        tests.py
        views.py

Before you can use this new app in your project, you must register it in project/settings.py.

 # Application definition

 INSTALLED_APPS = [
     'django.contrib.admin',
     'django.contrib.auth',
     'django.contrib.contenttypes',
     'django.contrib.sessions',
     'django.contrib.messages',
     'django.contrib.staticfiles',
+    'app'
 ]

Hello, World!

Getting started with this new app, we have to start with the "Hello, World!" test.

To do this we need to edit the app/views.py file, to define the index function.

 from django.shortcuts import render
+from django.http import HttpResponse

+def index(request):
+    return HttpResponse("Hello, World!")

Then, we have to create an app specific app/urls.py file, that will define the location of this index function.

django_project (project_venv) $ cd app/
app (project_venv) $ touch urls.py

The reason for creating this file, is to keep this a modular app, as mentioned earlier. Making it easier to edit the app when more code gets added, and plug in to future projects.

Add the following to this file.

from django.urls import path
from . import views

urlpatterns = [
    path('', views.index, name='index'),
]

Finally, we connect the this app view to the project's root URL. By adding this code to the project/urls.py file.

 . . .

 from django.contrib import admin
-from django.urls import path
+from django.urls import include, path

 urlpatterns = [
+    path('', include('app.urls')),
     path('admin/', admin.site.urls),
 ]

Now you can start the server, by running the following command.

$ python manage.py runserver

And you should see a rather plain but working index page like the one below.

Before we're done

Before summarising what was covered in this article. You might have noticed there is already an "admin" app added to the project/urls.py urlpatterns.

This was created when you ran the command django-admin startproject project in the previous Django tutorial. And is technically the first user app you installed for your project.

At the moment when you visit the /admin page you'll be greeted with a login screen. But we'll go over Django super-users and project administration in the future.

Summary

After going through this post, you should now;

Know the difference between Django projects and apps.
Create your first app, and add it to the project's settings.py file.
Define an index page for your project, by creating a simple "Hello, World!" function.
Add your new web page to the project urlpatterns, and be able to view it on your browser.

Hopefully, you have been able to follow these first few steps for creating and setting up your first Django app. If you have any questions, recommendations or found any errors with this post. Please let me know, in the comments below.

Thank you! ✌

Django: Getting Started

Richard Halford — Thu, 12 Aug 2021 14:40:20 +0000

First, create a directory to keep your new Django project nice and organised.

$ mkdir django_project

Then move into that created folder.

$ cd django_project/
django_project $

Installing Django

Before installing Django and starting a project, it is good practice to create a virtual environment (venv) first. That way all package installations are locally installed, and you can then create a list of all the dependencies for the project.

Since Python 3.5 the module venv is the default way to make virtual environments, replacing pyvenv.

django_project $ python -m venv project_venv

Then you can run the following command to activate your created virtual environment. And install the latest version of Django onto it.

django_project $ source project_venv/bin/activate
django_project (project_venv) $ python -m pip install Django

This will also install the required packages; sqlparse, asgiref and pytz.

You can then verify which version of Django you installed. Which can be helpful in the future when debugging your code. As the official documentation is viewable for specific major releases.

django_project (project_venv) $ python -m django --version
3.2

Creating the base project

In comparison to other Python web frameworks, such as Flask. Django comes with some very useful setup files, after you initialise the project.

django_project (project_venv) $ django-admin startproject project

This creates the following directories and files in your root directory (django_project/):

django_project/
    project_venv/
    project/
        manage.py
        project/
            __init__.py
            asgi.py
            settings.py
            urls.py
            wsgi.py

manage.py

This Python file is used to run several important project specific commands.

wsgi.py and asgi.py

WSGI = Web Server Gateway Interface

ASGI = Asynchronous Server Gateway Interface

When deploying your website in the future, it's required to have either one of these files. So that you can direct an application server towards your project (e.g. gunicorn, mod_wsgi, daphne).

settings.py

The settings.py file is used as a configuration file for your Django project.

init.py

With all Python packages, there needs to be an __init__.py file, to let Python know that this directory should be viewed as a package.

Running the development server

Now the basic initial setup of your Django project is complete. You can open up a browser and verify if your Django project works.

django_project (project_venv) $ cd project/
project (project_venv) $ python manage.py runserver

Watching for file changes with StatReloader
Performing system checks...

System check identified no issues (0 silenced).

You have 18 unapplied migration(s). Your project may not work properly until you apply the migrations for app(s): admin,
 auth, contenttypes, sessions.
Run 'python manage.py migrate' to apply them.
April 25, 2021 - 15:57:20
Django version 3.2, using settings 'project.settings'
Starting development server at http://127.0.0.1:0000/

You should then be greeted with the following web page.

Then, when you get the success message. You can quit the server by entering Ctrl-C.

Quit the server with CONTROL-C.

As you may have noticed, there was an error on "unapplied migrations" for certain Django apps. For now this can be ignored. As applying database migrations will be something you will become used to when building out your Django project.

You will also find a created "db.sqlite3" file created alongside your manage.py file and project/ directory. This will be project database, until you change it within project/settings.py.

Selecting an alternative port

You can change which port the development server runs on. By adding a port number argument at the end of the runserver command.

 $ python manage.py runserver 1234

Here, you can also change the ip address if you want to run a development server on different servers on your network.

$ python manage.py runserver 0:1234

Summary

After going through this post, you should now be able to;

Setup your Django project root directory.
Install Django and it's dependencies onto a virtual environment.
Create the basic Django project file structure.
Run a development server in you browser, and know how to change which ip address and/or port to run it on.

The next advised steps to take now, would be to start creating your first Django app! And I'd always recommend to combine this guide with the many others found online (i.e. documentation, youtube, other articles). That way any knowledge gaps can be filled and preferred development techniques can be worked out.

Hopefully, you have been able to follow these first few steps for getting started with a Django web project. If you have any questions, recommendations or found any errors with this post. Please let me know, in the comments below.

Thank you! ✌

Part 2: Arch Linux Installation Guide

Richard Halford — Tue, 10 Aug 2021 16:18:30 +0000

Originally published on xhalford.com

In part 2 of this series, we will be booting into a live Arch Linux environment. With the use of the USB drive we setup in the previous article.

Notes: There are a few presumptions being made for this guide. Regarding yourself and your current setup.

You will be moving from another *nix system.

This will help with the initial steps, with the programs being used to download the image and write it to a USB drive.

If you are coming from Windows, I will try to link to articles explaining this fairly trivial difference.

Have access to this guide on another device.

As once we get to installing Arch Linux, we'll be stuck in a terminal for some major steps.

Can handle looking deep into the empty void of the tty.

Make sure not to make any typos - as I try to do the same throughout these guides.

What you'll need:

USB drive which will have our installation medium loaded to it.
Disable Secure Boot for the installation image to be supported. This can be re-enabled once the install has finished.
Internet connection available.
- Preferably via Ethernet.
- If you will be using WiFi there are a few steps that I will cover. Every install I've done has been this way and it's really no bother.

Loading up the live environment

Before booting the USB, make sure to turn off secure boot in the UEFI.

Plug in your USB and turn on you computer. Making sure to enter the boot menu and select the installation medium on your USB.

The key to press to enter the boot menu will vary depending on your motherboard. If you never see this boot menu option, and you load straight into your original operating system (likely Windows). This will mean you also need to turn off Fast Startup in you UEFI settings.

Now you should be greeted with the virtual console, and see that you are logged in as the root user.

Setting up the live environment for installation

Boot mode

Before anything, we need to verify that we are booting into UEFI mode.

$ ls /sys/firmware/efi/efivars

This should hopefully output the directory. If it didn't, you may be using a different boot mode than UEFI (e.g. BIOS).

Next, we will make sure your keyboard, clock and internet are connected. Before we move onto partitioning the hard drive to make space for our Arch install to live.

Keyboard layout

To get a list of all available key-maps available:

$ ls /usr/share/kbd/keymaps/**/*.map.gz

To narrow down your search, you can be little more advanced and pipe this list into grep, to make the listing easier to read. A simple example to get all Dvorak key-maps:

$ ls /usr/share/kbd/keymaps/**/*.map.gz | grep dvorak

Then, once you've found your key-map of choice, load it up with the following command. Remembering to leave off the .map.gz file extension:

$ loadkeys uk

For the "/usr/share/kbd/keymaps/i386/qwerty/uk.map.gz" keymap.

Now everything should be inserted correctly when you type out the remaining commands.

Make sure we're online

We first need to make sure our computer's network devices are recognised. As an example, my computer's Ethernet (eth13s0) and WLAN (wlp14s0) interfaces are listed with the following command:

$ ip link
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: enp13s0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq state DOWN mode DEFAULT group default qlen 1000
    link/ether 40:8d:5c:50:76:63 brd ff:ff:ff:ff:ff:ff
3: wlp14s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DORMANT group default qlen 1000
    link/ether a4:34:d9:09:77:57 brd ff:ff:ff:ff:ff:ff

If you're connected with an Ethernet cable, you should be okay with connecting to the internet. We can test this by pinging a popular domain we expect to be running. Once we can see that we can connect to the outside world, we can move on to the next step.

$ ping wikipedia.com
PING wikipedia.com (91.198.174.194) 56(84) bytes of data.
64 bytes from ncredir-lb.esams.wikimedia.org (91.198.174.194): icmp_seq=1 ttl=55 time=27.0 ms
64 bytes from ncredir-lb.esams.wikimedia.org (91.198.174.194): icmp_seq=2 ttl=55 time=28.6 ms
64 bytes from ncredir-lb.esams.wikimedia.org (91.198.174.194): icmp_seq=3 ttl=55 time=154 ms
^C
--- wikipedia.com ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2002ms
rtt min/avg/max/mdev = 26.984/69.917/154.174/59.581 ms

Entering Ctrl-C to stop this call (^C shown above)

WiFi connection setup

For wireless connectivity, we will use the preinstalled iwd command line tool, with the following steps:

To enter the tool's interactive prompt.

$ iwctl

Getting the name of the WiFi device. Mine was labelled as wlan0, so remember to replace the following commands with the name of your device.

$ device list

Start a scan for available networks.

$ station wlan0 scan

Bring up a list of the available networks.

$ station wlan0 get-networks

Connect to the network. Replace SSID with the name of the network you want to connect to. You will then be prompted to enter and submit the password for this network.

$ station wlan0 connect SSID

Hopefully show that you are now connected to your network.

$ station wlan0 show

Exiting the interactive prompt with Ctrl-d.

To make sure things are running as they should.

$ ping wikipedia.com

Updating the system clock

This is a simple step.

$ timedatectl set-ntp true

Follow me on Twitter!

DEV Community: Richard Halford

Setting up virtual machines on Arch Linux

KVM

Virtualisation made easy

QEMU

Libvirt

Libvirt daemon

Libvirt group

Internet

Virtual Machine Manager

Creating a virtual machine

How to remove Windows from Dual-Boot

Finding Windows

List partitions using sudo fdisk -l

List partitions using lsblk -p

Deleting Windows

Creating a partition

Writing table changes

Formatting the partition

Mounting the new partition

Where to mount the partition

Automatically mount on boot

Mount without rebooting

Giving user privileges

Updating GRUB

Bing Wallpaper: Without Bing

The script

Breaking it down

Options for Running the Script

How to improve the Arch Linux package manager

Colour Output

Multi-threaded Performance

Using hook scripts with Certbot

GoDo: A command line todo list application

Table of Contents

About

Go get things done and checked off the list.

Getting Started

Requirements

Installation

Recommendations

Configuration

GoAPI

Authentication

Server

Daemonize

Licence

Part 5: Arch Linux Installation Guide

What you'll need:

Creating a User

Sudo Privileges

Network Manager

Extending Program Availability

Rebooting Arch

Graphical User Interface

Display Server & Graphics Drivers

Desktop Environment or Window Manager

Display Manager

User Directories

Summary

Part 4: Arch Linux Installation Guide

What you'll need:

Installing Arch Linux

Generating the fstab File

Change Root Into Your New System

Set Time Zone

Localisation

Using an Editor

Set the LANG Variable

Keyboard Layout

Network Configuration

Setting the Root Password

Bootloader

We're Done! (mostly)

Part 3: Arch Linux Installation Guide

What you'll need:

Partitioning

Swap Partition

File System Partition

Assigning Partition Types

List partitions using `sudo fdisk -l`

List partitions using `lsblk -p`

init.py