DEV Community: Rick Carlino

Tools and Practices I Use In Every Real-World Software Project

Rick Carlino — Tue, 23 Mar 2021 13:34:00 +0000

There are very few books written for junior-to-mid level developers that answer the question "How do I run a real-world software project?". Industry best practices often arise as the result of cross-pollination and institutional/tacit knowledge rather than explicitly prescribed rules that you can read about in a book.

For developers working alone or in small organizations such as startups, these norms may not be obvious. Knowing which tools one needs to deploy a production-scale application is crucial knowledge.

In 2019 I published the article "Software Tools for Hobby-Scale Projects." It is still one of my most popular blog entries. This post will explore the same idea within a professional context and hopefully help new or solo developers get guidance on tools and practices for new projects at small to mid-scale organizations.

Objective: Provide a list of tools and practices that apply to a majority of real-world software projects.

Intended audience: Developers familiar with software authorship wishing to learn about real-world software deployments and practices.

The 12 Factor Methodology

If you run an application with real customers, you should use the 12-Factor Methodology. The 12-Factor Methodology is a set of practices that improve commercial software development and will make your software easier to manage over the long term. Many recommendations are simple yet overlooked (example: "Store config in the environment"). If your application runs on Docker or Heroku, you may already practice many of the 12 factors. If you have not yet read the entire document, I recommend giving it a look. It's a straightforward read and offers a great set of practices for structuring new and existing projects.

A Reliable Version Control System

Software development is costly. The result of that process is computer source code. As the organization and its related software projects grow in complexity, you will need a reliable way to:

Create backups of the project source code.
Revert problematic changes to the project.
Figure out who last modified a particular part of the codebase.
View historical versions of the project.
Merge changes from two developers who created parallel versions of the project.
Figure out what changed between different versions of a project.

Most (but unfortunately not all) developers in the software industry have adopted version control systems (VCS) to solve the issues noted. For all of my software projects, I use Git as my VCS, which is the current industry leader. Alternatives such as Pijuul, Fossil, and Subversion are also available.

For some, choosing to use version control on a project may seem obvious that it does not even require mention. Unfortunately, there are still companies out there that like to host zip files on FTP servers as a form of pseudo-version control. Often this situation is the result of a developer's unwillingness to learn a version control system. Opting out of VCS is inappropriate in all cases.

Choosing to build an in-house version control system (or opting out of version control) will cost you and your organization time and money. You will not create a solution that is more robust than Git. You will lose parts of your source code and historical information about the project. A good VCS can fix severe problems like data loss in seconds. Conversely, without a good VCS in place, certain classes of issues may take days or weeks to resolve if they are fixable at all.

You should use Git on every professional software project. You can find alternatives to Git if you genuinely don't like it, but you must use some turn-key version control system and not build your own. Failing to use a real VCS will result in a loss of time and data.

Although I've yet to find a version control system that is easy (or enjoyable) to grasp at first, you still need to eat your vegetables. The benefits of having a version control system cannot be understated, and that's why it appears in the top half of this list. You need to invest the time to learn it, reap the benefits, and move on.

You can find a good starting point here.

Source Code Hosting

Even if you are the only developer on a project, it is still a good idea to host redundant copies of source code somewhere other than your local machine. In the early days of the internet, it was not uncommon to hear horror stories of shareware projects that ceased development for no reason other than the lead developer had lost the source code. The advent of free and cheap source code hosting has made this type of disaster less common.

Additionally, if you are working on the project with other developers, you will want to discuss and review changes to the Git repository as they happen. Many source code hosting tools offer extras such as features to review and discuss incoming changes easier.

The most popular options are:

The vast majority of developers will choose Github as the current industry leader. Github offers a broad feature set and decent free plans. The market for hosting solutions widened in recent years, with several alternatives gaining traction among developers that wish to avoid Microsoft products.

TL;DR: You will need shared source code backups no matter your project or team size. Many companies that offer source code hosting offer additional features alongside the hosting such as webhooks and pull requests that make the product offerings extremely useful.

Automated Database Backup System

This one is obvious but overlooked. Your backup system needs to be automated- it's too easy to forget, and the consequences are disastrous (see the previous section about version control backups). Once you have a backup system that works, perform periodic "fire drills" to ensure that you can handle backups properly in an actual database failure. If your backup system is cloud-based, be sure to create redundant hard copies periodically. There is no guarantee that your data center will not burn to the ground.

I currently manage my database backups with Heroku. There are too many options to name in this space. Your main concern when shopping for a backup system should always be reliability.

TL;DR: Database backups prevent disasters and are easy to set up, but you need to regularly make sure they will work in an emergency.

A Testing Framework

During a project's lifetime, the scope and importance of features will only increase, and so will the need for stability. Project maintainers ensure the reliability of a codebase by testing it. In the early days of a software project, it is easy to ensure things work. You can manually click buttons and pass in data. Project maturity brings an inevitable challenge- there is no way your team can manually test every change before a release. The number of ways data can be passed and combined makes it mathematically impossible to cover every possible use case within a single release cycle. How can you be sure that a recent update will not break other parts of the application?

For decades, software developers have solved this challenge by writing a second software project to go alongside the main project. This second codebase is known as a "test suite," and it will automate away the process of testing the main application. Sometimes, the test suite will click actual buttons on the screen. These sorts of tests are known as "integration tests." A more common approach is to only test functions and modules in isolation. This second approach is known as "unit testing." Unit testing is more common in the wild because it produces consistent test output, is fast to run, and is easy to setup. Although unit testing is easier to manage, its main criticism is that it does not simulate real-world conditions effectively as integration testing.

Philosophies abound on the most effective way to approach automated testing. There is also extensive jargon and countless books written on the subject. If you add a test suite to a project for the first time, do not be discouraged by philosophical debates. The most important aspect of a test suite is that it exists. Testing is an art that takes time to master, and the best way to increase your project's reliability is to begin writing a large volume of automated tests as soon as possible. For a delightful satire article on the subject, see: "Expert Excuses for not Writing Unit Tests."

Once you are committed to the idea of automated testing, you must pick a library to automate the boring parts. The options are language-specific, and every language ecosystem has a good set of options available. The ubiquity of high-quality testing frameworks means you should not attempt to create an in-house testing framework.

These days, I use Jest for Javascript, RSpec for Ruby, and ExUnit for Elixir.

TL;DR: Manually testing a product before each deployment is mathematically impossible for small teams. Startups and solo projects need a way to test the product for defects in an automated manner.

A Test Coverage Reporter

Once you have a large volume of tests, you will find that they detect many problems that humans could not have noticed. Getting a project to that point takes considerable effort. A codebase will receive new features and remove old ones as it matures. If left unchecked, a test suite's usefulness will deteriorate with time until it no longer is a helpful tool for defect prevention. Developers need a way to ensure that every line of code added to a project has accompanying tests.

The term "test coverage" is used in the software industry to determine what percent of application code is touched by test code. Under ideal circumstances, your application test coverage numbers should only ever go up. For real-world applications with thousands of lines of code, enforcing test coverage is difficult, even in teams where everyone is motivated to write tests. Luckily, there are tools available to record test coverage trends. Running these tools before merging new code is effective in maintaining a test suite's usefulness.

I use CodeCov and Coveralls to manage code coverage metrics. Both tools are excellent industry leaders, though I find that each product has strengths and weaknesses depending on the project's language. I recommend trying both and deciding for yourself.

TL;DR: Even with the best intentions, a test suite will lose effectiveness over time due to neglect. Measuring test coverage will alert you to these situations and inform you of which tests need updates.

Containerized Environments

"Works on My Machine" is a horrible state of affairs in software development and the punchline of many jokes. Running software consistently, reproducibly, and reliably in any environment is a mostly solved problem for small to mid-sized projects. Many of the issues of environment inconsistency (e.g., accidentally running the wrong version of a library) are avoidable by running applications in "containers."

Below is a list of situations that containerization improves:

Running a test-suite in a clean environment, free of state from previous test runs.
Setting up a development environment for new team members
Keeping multiple versions of the same application running in parallel (example: test, staging, and production environments).
Creating temporary sandboxes, such as those required to run tests against proposed changes to a codebase.
Re-creating an environment from backups after hardware failures

In 2021, Docker and Docker Compose are the most used solutions. They solve the problems by hosting software in a "container" with a clean slate on every run. When the application restarts, the container is destroyed. No state is saved for the next run unless you explicitly allow it through the use of "storage volumes."

Currently, I use Docker Compose to manage my containers. Although Docker-based tools still reign as the defacto industry standard for containerization, a cohort of competitors have taken root in recent years. I have not investigated these alternatives because I am still satisfied with Docker and Docker Compose. I welcome comments on this matter.

TL;DR: Running software in containers will prevent most (but not all) environment-specific software problems.

Automated Deployment System

A simple way to reduce the severity of bugs is to deploy as often as possible. Anyone who has worked on a large-scale enterprise project with 8-month release cycles will know what I am talking about- massive releases create huge error backlogs and upset customers. The need to batch significant changes into a single release is a relic of a bygone era when software lived on CD-ROMs and floppy disks. Publishing minor incremental releases regularly should be the norm for most projects. There is rarely anything to gain by batching multiple changes into a single release.

Once you have a reliable test suite that runs in a container with good version control practices, you should automate software releases as much as possible. There are exceptions to this rule, but they are just that- exceptions.

In the last decade, the industry has seen the rising use of "push to deploy" offerings. The premise of such tools is that you can release your software more frequently (multiple deployments per day) by pushing changes to a version control repository managed by a specialized tool. These tools handle the most common use cases, such as pushing changes, reverting changes, restarting applications, and managing configuration. All actions are completed in a single step and with minimal developer intervention. These tools take time to learn and set up, but it truly is a better way to ship software. High adoption rates are a testament to their effectiveness.

The ecosystem for push-to-deploy tools is diverse. In the past, I've automated the deployment of micro-projects with Dokku. For projects where reliability is a more significant concern than cost, I use Heroku. Many language and platform-specific tools are out there. Research the tools offered by your ecosystem.

An alternative title for this section was "Continuous Delivery and Integration Systems." I decided against this title because those terms come with extra baggage and may distract from my main point- you need a sound system to publish new versions quickly. CI/CD practices propose many things beyond increasing deployment time. Readers who are unfamiliar with CI/CD are encouraged to research the topic.

TL;DR: You should automate most aspects of your deployment process and deploy as frequently as possible to increase reliability.

Real-time Error Reporting

What's valid with Pokemon is also true with production error reports: You gotta catch them all. When faced with a software defect, very few users will take the time to report it to you. When they do, they are already upset. Unreported errors aren't good for business since these bugs can translate to lower usage and overall satisfaction.

Real-time error reporting tools help keep track of errors as they happen. They also give you a head start on fixing the problem. Once captured, the tool immediately reports the error to the appropriate communication channels, such as a team Slack or staff email.

A good error reporting tool will capture the error and additional information, such as the user that triggered the error and any relevant stack trace(s). In my experience, error reporting tools are universally easy to install and come in a wide range of price points (with free plans available).

Currently, I use Rollbar for production-grade projects and exception_notifier for small-scale Rails projects.

(Sometimes) Performance Monitoring and Profiling

Identifying performance issues in production is hard. It is especially hard for mature, real-world applications servicing production workloads. In addition to being a complicated process, it is also essential to an application's reliability. The solution is never as simple as npm install performance. Performance monitoring tools help you untangle the situation by recording statistics about your application's runtime characteristics.

Typically, you install a library into your application code and deploy the app to production. Once deployed, the performance monitoring solution will store and upload runtime metrics. You can view graphs of the data in a dashboard, hopefully giving you a better picture of application hot spots. These tools are often language and platform-specific. Many Ruby on Rails performance monitoring tools can help identify numerous anti-patterns and mistakes, such as N+1 queries in ActiveRecord. Although the tools do an excellent job of isolating bottlenecks, it is ultimately the developer's responsibility to devise solutions for each hot spot.

I can't say I have a favorite performance monitoring tool at the moment. The landscape for performance profiling and monitoring tools appears to have two main kinds. On the one hand, you have high-quality tools that are significantly overpriced, with pricing tiers intended for large enterprises. In my experience, performance monitoring tools are one of the highest-priced services that a software startup will pay for, aside from server hosting. On the other hand, there are cheap or free tools that are only marginally useful and do not mirror their competitors' offerings. I've yet to find a performance monitoring solution that offers a good feature set and affordable pricing plans for startups and small businesses.

Depending on your project's scale and maturity level, it may be possible to delay the adoption of a performance monitoring tool. In the case of specialized software used by a small userbase, commercial performance monitoring tools might offer more cost than benefit.

Something Else?

The tools and practices above build a basis for most real-world applications. Did I miss anything? Feel free to post a comment if you would like to share real-world experiences about tools and practices that make your life easier as a developer.

Special thanks to Daniel Legut for proof-reading earlier versions of this article.

What Were Server-Side Image Maps?

Rick Carlino — Tue, 02 Mar 2021 13:46:00 +0000

Author’s note: This article explores server-side image maps, a historic web technology. In 2021, the use of server-side image maps is considered a worst practice. Hopefully, no one thinks I am encouraging the use of server-side image maps. Although they provide hours of fun for weekend hobby projects, they are an accessibility nightmare. Please don’t proliferate server-side image maps in real-world projects in 2021! OK, with that out of the way, let’s get started…

Setting the Stage

I realize that this is another unplanned change that I’m making without getting a consensus or having it added to the spec or whatever.

– Marc Andreeson, discussing the addition of server-side image maps to the Mosaic web browser, May 18th, 1993.

I recently found myself reading over a very old HTML spec and encountered something I had almost completely forgotten about- the ISMAP attribute for IMG tags.

The ISMAP attribute allowed prehistoric web developers to convert images to a “server-side image map”. What follows is a brief overview of my re-discovery of this amusing attribute.

In the year 1993, the web was young. There were no javascript onClick events, and in fact, there was no Javascript at all. CSS would not show up for another three years. Despite a narrow feature set, web developers were building an exciting vision of what the web could become.

In the case of server-side image maps, web developers of the era needed a way to create dynamic clickable graphics, and they didn’t have the luxury of Javascript or CSS. Creating an image hyperlink was already possible, but this only allowed one hyperlink destination per image. What if the image was a map of the United States, and you wanted to direct the user to one of 50 pages based on a single click of that image?

Another use case was the creation of intricate menus for web pages. Since CSS was still in its early days, web developers that wanted a way to build flashy graphics were limited in their options. One workaround that many web designers would deploy was not using HTML for the UI’s graphical elements. Instead, they would design the layout entirely as an image and write a server-side CGI script that could render a subsequent page based on the X/Y coordinates of the last point clicked on the image. This practice is now frowned upon.

Taking this example further, I decided to do some historical re-enactment. What would an interactive drawing app look like in the early 90s? I created this hilariously underpowered drawing app in one sitting last weekend to illustrate the point. The app does not use Javascript or CSS and allows multi-user interaction with a shared monochrome canvas (barely).

Digging Deeper

Advanced amateur Google research has led me to conclude that the ismap attribute was first added to the Mosaic browser in the early 90s, circa 1993. At that time, it was added as a non-standard feature.

Later, in November of 1995, the ismap attribute was referenced in RFC 1866, the HTML 2.0 specification. RFC 1866 is the first official standard I can find that references server-side image maps. If anyone can find earlier documents (official or otherwise), please leave a comment on Reddit or Lobste.rs.

Problems Presented by Server-Side Image Maps

Server-side image maps disappeared from use for a variety of (good) reasons. The most notable reasons are that server-side image maps are inaccessible to users who cannot use pointing devices or view images. Developers of the era were encouraged to provide clickable hypertext links with computer-readable descriptions alongside the image, though this was rarely practiced. The result was an inaccessible website.

Another argument against server-side image maps was that they are not offline-first, though I would argue that the days of being able to keep offline backups of a website are far, far behind us.

Later Refinements: Client-Side Image Maps

The accessibility challenges of server-side image maps eventually led to the introduction of client-side image maps in HTML 3.2.

Like their server-side counterparts, client-side image maps allowed developers to create clickable, interactive graphics on HTML documents. Unlike server-side image maps, the map information is entirely contained within the HTML document. This means that screen readers and text-based browsers can parse the data locally and provide an accessible alternative to the graphical image map. This assumes that the developer followed best practices, such as giving descriptive ALT text for each map region.

Client-side image maps are nested directly within an img tag using various tags that describe shapes for regions within an image. Since an image map tutorial is beyond this article’s scope, I will provide a link to an MDN tutorial here.

Below is a screenshot of a website hosted by Penn State that explains the accessibility aspects of client-side image maps. I’ve provided screenshots below showing how the image map renders in a graphical browser (Dillo) and a text-based one (Lynx). Because the image map was implemented correctly, I was able to extract relevant information from the image map, even in a text-based browser.

Rendering the page in Lynx, a text-based browser:

The same web page, rendered in a graphical web browser:

Image Maps Today

As stated previously, server-side image maps are discouraged in modern websites. On the other hand, client-side image maps are not a deprecated technology, but they are rarely seen on modern websites. I attribute this to the fact that Javascript has matured to the point that it can replace image maps with custom client-side code that is tailored specifically to the needs of the application.

Build a Raspberry Pi Linux System the Hard Way

Rick Carlino — Sat, 23 Jan 2021 20:13:00 +0000

INTENDED AUDIENCE: Raspberry Pi hobbyists. Software developers with no embedded systems experience.

READ TIME: About 10 minutes, excluding exercises.

GOAL: The instructions below will explain how to build a Linux environment for a Raspberry Pi 3B from scratch, focusing on extreme minimalism. I will build most components from source code and use BusyBox as the only user application on the target. The focus is on minimalism for the sake of learning. Optimizations like network boot, secondary bootloaders, compressed filesystems, etc.. will not be covered.

Hardware Requirements
Partition the SD Card
Download Firmware Blobs
Create A Toolchain on Host Machine
Build the Kernel
Copy Kernel and DTBs to Root Partition
Configure Kernel and Bootloader
Create Local Directory for Root Filesystem
Install Kernel Modules
Download BusyBox
Configure BusyBox
Build BusyBox
Root Filesystem, Part II
Power On and Explore the Shell Prompt
Next Steps
Key Terms

Key Terms

BOOT LOADER : A platform specific program that runs before the operating system is loaded. It is responsible for hardware initialization and loading of the OS kernel into memory.

BOOT PARTITION : A disk partition that is used by the bootloader.

BUSYBOX : A single Linux application that emulates many common system utilities, such as cp, ls, mv, etc.. Often used in embedded systems.

DEFAULT KERNEL CONFIGURATION : A set of default options that are applied to the Linux kernel when it is compiled from source code by a developer.

DEVICE TREE BLOBS : A binary format used to describe a machine’s device layout. This information is used by the Linux kernel (and its device drivers) to support platform-specific hardware.

DEVICE TREE OVERLAY : Fragments of a device tree that modify the DEVICE TREE BLOB. They are typically used to make small changes or additions to a DEVICE TREE BLOB.

HOST MACHINE : In embedded systems, the HOST MACHINE is a computer used for software development that is not the end product of the software development activity.

KERNEL MODULES : Code that can be loaded and unloaded from the kernel at runtime. They add functionality to the kernel (often for hardware management) without the need to reboot the system or modify the kernel itself.

LINUX KERNEL : Responsible for managing system hardware and resources as well as providing services to userspace applications. Loaded by the BOOT LOADER at system start.

INIT : The first program loaded at start time by the Linux kernel.

LINUX KERNEL COMMAND LINE : A set of options that are passed to the Linux kernel when it is loaded. It often contains information about how to load the root filesystem.

ROOT FILESYSTEM : A series of directories that contain essential applications (like init and a shell) as well as system configuration.

TARGET MACHINE : The embedded device that is the final product of embedded systems development. See also: HOST MACHINE.

USB TTL CABLE : A cable that allows you to access the Raspberry Pi’s system shell without needing to attach a keyboard and monitor. It is required hardware for this tutorial. They are very inexpensive.

Hardware Requirements

You will need:

A micro SD Card larger than 2 Gigabytes
A micro SD Card reader
A Raspberry Pi 3b (The “target machine”)
An x86-based Ubuntu desktop machine (the “host machine”)
A USB TTL Serial cable

Partition the SD Card

This section will require the creation of two partitions on a blank SD Card:

A FAT16 boot partition of at least 32 MB
An EXT4 root filesystem partition of at least 1 GB

The boot partition will eventually contain proprietary firmware blobs, device tree blobs, the Linux kernel, and two configuration files.

The root filesystem will contain kernel modules and userspace programs (such as ls, cat, etc..).

I recommend using gparted for partitioning. Here’s a short screencast explaining how to perform this action in GParted:

Download Firmware Blobs

Our goal is to build an entire Linux system. Before we can load the kernel, the hardware must be initialized. Hardware initialization is the responsibility of the “boot loader”- platform-specific code that runs before the Linux Kernel is loaded into memory. Every platform will handle the boot loading process differently. In the Raspberry Pi 3b, we must download three files and place them in the boot partition (FAT16) of the SD card. These files are not Open Source, so we cannot compile them ourselves.

All files are available in the Raspberry Pi firmware repository. I will not discuss the topic of boot loaders in depth. You can read more about the boot configuration in the official documentation.

Before proceeding, add the following files to the FAT16 (boot) partition of your SD card:

Create A Toolchain on Host Machine

At this point, you should have an SD card with an EXT4 partition and a FAT16 boot partition. The boot partition should contain the three proprietary blobs listed in the previous section.

With the boot loader installed, we’re ready to compile a Linux kernel from source code. Compiling a Linux kernel requires GCC, a C language compiler. It also requires binary utilities (“binutils”) for handling assembly language files.

Your host machine may already have GCC and Binutils installed, but there’s a problem- the compiler on the host machine emits x86 binaries. A Raspberry Pi uses an entirely different ARM instruction set. To produce a binary that the RPi can execute, we require a compiler that emits ARM instructions, not x86. We could undoubtedly install Raspberry Pi OS (https://www.raspberrypi.org/software/) and perform kernel compilation on the target device. This would be a prolonged process, however. For comparison, I have a 10th gen Core i7 Laptop with 8 physical cores, 8 GB of RAM, and a solid-state drive. Compiling the Linux kernel took approximately 20 minutes on this machine, which is still a considerable amount of time. Compilation on a Raspberry Pi would take even longer.

The better way to compile a Raspberry Pi Linux kernel is via “cross-compilation” through a “cross compiler.” A cross compiler is a compiler that runs on one instruction set but outputs source code in a completely different instruction set than the target. In our case, we need a compiler that runs on an x86 machine but which emits ARM instructions that the RPi can understand.

# Installs `arm-linux-gnueabihf-*` compiler, linker, etc..
sudo apt install bc bison crossbuild-essential-armhf flex git libc6-dev libncurses5-dev libssl-dev

After running the command above, our machine will have a new set of tools available:

What?	Native tool	Cross compiler equivalent
Compiler	gcc	arm-Linux-gnueabihf-gcc
Linker	ld	arm-Linux-gnueabihf-ld
Assembler	as	arm-Linux-gnueabihf-as

Do you see the pattern? In the next step, we will set a CROSS_COMPILE environment variable so that applications will compile via arm-Linux-gnueabihf-gcc rather than the native gcc. Typically, a cross compiler will offer all the native compiler and Binutils’ executables, such as gcc and ld. It will do so, however, with a prefix that denotes the target architecture. In our case, the prefix is arm-Linux-gnueabihf-.

If you want to build a custom toolchain, or build a toolchain from source, look at Crosstool-NG. It is a more advanced way to generate cross compilers.

Build the Kernel

Now that we have a working toolchain, we need to find the latest Linux kernel source and compile it.

Before proceeding, we need to set some environment variables. You must export these variables when performing all steps.

# ==== BASH USERS:

# Set the architecture to ARM:
export ARCH=arm

# We want to use ARM v7:
export KERNEL=kernel7

# Set the cross-compiler prefix (explained in previous section):
export CROSS_COMPILE=arm-linux-gnueabihf-

# ==== FISH USERS:
# set -x KERNEL kernel7
# set -x ARCH arm
# set -x CROSS_COMPILE arm-linux-gnueabihf-

The Raspberry Pi foundation manages its own version of the kernel. We will clone the source code using Git:

# Clone Raspberry Pi Linux kernel
git clone --depth=1 https://github.com/raspberrypi/linux
cd linux

We must then apply the default configuration for the build system based on the target device (An RPi3b, which uses the BCM2709 chipset):

# Apply default config for RPi3
make bcm2709_defconfig

Some users will want to modify the default kernel configuration. You can accomplish this via make menuconfig, though it is not required in our use case.

We are ready to compile the kernel, but we need to determine how many CPU cores our host machine has before we do that. We require this information to enable parallel compilation, which can decrease the compilation time substantially. Perform the following steps:

Determine the number of CPU cores your machine has vs. lscpu.
Multiply the number from step 1 by 1.5.
Pass this number to the -j flag in the next step.

Please note that I have used -j12 in the example below. Your machine may require a different -j value!

# Compile actual kernel, modules, DTBs
# Takes ~20 minutes on modern i7 with SSD
make -j12 zImage modules dtbs

Now is an excellent time to take a break. Kernel compilation takes a while.

Copy Kernel and DTBs to Root Partition

The previous step created several required files:

zImage- a compressed binary Linux kernel.
Device Tree Blobs (DTBs)- metadata that Linux needs to find physical hardware
Device Tree Overlays- modifications and additions to the core DTBs

We need to move these types of files to the FAT16 boot partition previously created in a specific manner.

First, find the location of the boot partition (FAT16) created earlier. On my machine, the path is /media/rick/rpi_boot. It will be different on your device. It will live in the same location as the bootcode.bin, fixup.dat, start.elf files we downloaded earlier.

Within your boot partition, create a new folder called overlays/ at the root level:

mkdir /media/rick/rpi_boot/overlays

Next, run the following within the linux/ source directory of the previous section:

cp arch/arm/boot/zImage /media/rick/rpi_boot
cp arch/arm/boot/dts/bcm2710-rpi-3-b.dtb /media/rick/rpi_boot
cp arch/arm/boot/dts/overlays/disable-bt.dtbo /media/rick/rpi_boot/overlays

Configure Kernel and Bootloader

The Raspberry Pi uses a config.txt file to configure the bootloader.

Create a config.txt file in the boot (FAT16) partition as follows:

# Use the Linux Kernel we compiled earlier.
kernel=zImage

# Enable UART so we can use a TTL cable.
enable_uart=1

# Use the appropriate DTB for our device.
device_tree=bcm2710-rpi-3-b.dtb

# Disable Bluetooth via device tree overlay.
# It's a complicated explanation that you
# can read about here: https://youtu.be/68jbiuf27AY?t=431
# IF YOU SKIP THIS STEP, your serial connection will not
# work correctly.
dtoverlay=disable-bt

Next, we need to edit the Linux command line.

The Raspberry Pi will grab the kernel command line from cmdline.txt. Like config.txt, it lives in the boot partition (FAT16):

Create a cmdline.txt file as follows and save it to the boot partition:

console=tty1 console=serial0,115200 root=/dev/mmcblk0p2 rootfstype=ext4 rootwait

Create Local Directory for Root Filesystem

Now that the Linux kernel is built and the boot loader is configured, we need to shift our focus to building the root filesystem. The root filesystem is where Linux expects to find executables, kernel modules, system files, and many others. The layout is specified by the Linux Filesystem Hierarchy Standard

We will create a root filesystem in a directory on our host machine. Later, we will copy the final directory structure to the EXT4 partition we made at the beginning of the tutorial.

I will name this directory root_fs:

mkdir -p root_fs

Use pwd to determine the absolute path to this directory. Keep this information handy; we will need it in several steps.

Install Kernel Modules

The first thing we will put onto the root filesystem is kernel modules, which were compiled alongside the Linux kernel in a previous step.

cd back into the directory where the Linux source code was, but keep the absolute path to your root_fs directory handy.

We need to set the INSTALL_MOD_PATH environment variable to the absolute path of the root_fs:

export INSTALL_MOD_PATH=/home/rick/linux_root
# Fish shell users:
# set -x INSTALL_MOD_PATH /home/rick/linux_root

After that, cd into the Linux source dir and run:

# May require `sudo`:
make modules_install

After this step, you will find a multitude of *.ko files installed in your root_fs/ directory.

Download BusyBox

The modules are now installed in the root filesystem. Modules are not enough to run a Linux system, however. We need all of the usual tools found on a Linux system, like ls, cd, cp and friends.

Rather than compile each tool individually, we will statically cross-compile BusyBox. BusyBox touts itself as “The Swiss Army Knife of Embedded Linux”. It is a single executable that serves multiple functions, depending on how the executable is loaded. For instance, we can create a symbolic link from BusyBox to /bin/echo. When we execute /bin/echo, BusyBox will note the symbolic link name and run as expected for the echo command. BusyBox implements all of the essential Linux command-line utilities like cd and ls. It also comes with added extras like a lightweight HTTP server and text editor.

Let’s clone a local copy of BusyBox and checkout version 1.33.0:

git clone git://busybox.net/busybox.git --branch=1_33_0 --depth=1
cd BusyBox

Configure BusyBox

Once we have the source on our host machine, we need to tweak some settings from within menuconfig:

make menuconfig

Set the following options:

Setting Location	Setting Value
Settings -> Build static binary (no shared libraries)	Enable
Settings -> Cross compiler prefix	`arm-Linux-gnueabihf-`
Settings -> Destination path for ‘make install’	Same as `INSTALL_MOD_PATH` from kernel modules step

Build BusyBox

We’re ready to compile BusyBox.

# Your -j value will be different.
# Typically, I use 1.5x the number of physical CPU cores.
make -j12
make install

BusyBox is now installed in ../root_fs. You should see many symbolic links within the bin/, sbin/, and usr/ directory of your root filesystem directory ($INSTALL_MOD_PATH).

Root Filesystem, Part II

BusyBox is not quite enough to boot a Linux system. We still need to add:

A place to mount procfs
A place to mount sysfc
A place to mount devfs
The /etc directory
The /etc/init.d configuration directory
The /etc/init.d/rcS file

Run the following commands within /root_fs:

# Create directories to mount stuff:
mkdir proc
mkdir sys
mkdir dev
mkdir etc

# Create config directory:
mkdir etc/init.d

# Create an empty `rcS` file.
touch etc/init.d/rcS

Make rcS executable (Linux will execute this script at boot time):

chmod +x etc/init.d/rcS

Add the following entries to etc/init.d/rcS:

#!/bin/sh
mount -t proc none /proc
mount -t sysfs none /sys

echo /sbin/mdev > /proc/sys/kernel/hotplug
mdev -s

The entries above will mount sysfs and procfs at boot time.

Power On and Explore the Shell Prompt

We’re ready to boot the system.

Do not use a monitor- a USB TTL cable is required! If you do not own such a cable, you can buy one for a few dollars on Amazon. It will change your life as a Raspberry Pi enthusiast, trust me.

Once you power on, you should see some kernel messages scroll by and, eventually, a shell prompt.

# ... Shortened for clarity ...
[4.536592] usb 1-1.1: new high-speed USB device number 3 using dwc_otg
[4.666906] usb 1-1.1: New USB device found, idVendor=0424, idProduct=ec00, bcdDevice= 2.00
[4.680733] usb 1-1.1: New USB device strings: Mfr=0, Product=0, SerialNumber=0
[4.693829] smsc95xx v1.0.6
[4.791032] smsc95xx 1-1.1:1.0 eth0: register 'smsc95xx' at usb-3f980000.usb-1.1, smsc95xx USB 2.0 Ethernet, b8:27:eb:54:61:ef

Please press Enter to activate this console.

/ # echo "Hello, Linux world."
Hello, Linux world.

We now have a working Linux root partition and boot partition. Copy the root_fs/ directory to the SD card and attempt to boot the device.

Next Steps

The Linux system presented in this article is not very useful, but hopefully, it gives you a better picture of how Linux distributions and embedded Linux systems are built. Feel free to reach out to me on Reddit or Lobste.rs with feedback.

Practices that Make Raspberry Pi Work Easier

Rick Carlino — Thu, 21 Jan 2021 23:34:00 +0000

I work on Raspberry Pi hardware daily as part of my job and for fun at my local makerspace. Below I’ve outlined some practices that have streamlined my Raspberry Pi workflow both at work and on hobby projects.

1. Use a Serial TTL Cable When Possible

A traditional setup in new RPi projects is to attach a monitor, keyboard, mouse, and associated cables to configure the device and application. This is an adequate starting point for novices but can be a hassle for medium and large scale projects. It can also add a considerable amount of setup and teardown time to a weekend project, particularly in a shared environment like an office or makerspace.

There’s a much better way to go about this, assuming you have already learned RPi and Linux basics. Instead of attaching your Pi project to a keyboard and monitor, you can use a compact serial TTL cable directly attached to your main laptop. You can then access the Linux shell prompt as a serial USB device.

Once the cable is connected, you can operate from the device’s command line as usual. With a serial TTL cable, it’s simple to perform tasks such as viewing logs and updating code, all without the need for a monitor and with a setup that can fit into your bag.

Although some developers prefer the convenience of SSH-over-WiFi, I still prefer a USB cable. It allows me to investigate problems before WiFi connectivity is available (for example, kernel panics). It also allows me to work on the device outside of the WiFi router’s range.

Using a serial TTL cable instead of a graphical monitor is the single best workflow improvement you can make. Still, it might not seem so obvious on the surface. Every graphical tool on a Raspberry Pi has a command-line based alternative. It will take time to learn, but much less time in the long run than lugging around a monitor. You will also find that command-line tools are much more convenient after the initial learning investment has been made.

Adafruit has written an in-depth tutorial here.

2. Replace `screen` with `tio`

When accessing a Pi via USB, the common practice in Linux is to use screen or minicom on the host machine. Many of these options offer an OK experience. Nevertheless, I’ve found that the interfaces are not always obvious, and often the tools show their age.

Recently I discovered Tio, a modern alternative to Screen. It is much easier to use than any other command-line terminal application while still having a handy command set.

By default, Tio will auto-reconnect to a USB device, even if the device is unavailable. This is great if you move your project around a lot and need to constantly reconnect to the board. As soon as you re-attach the RPi, Tio will start printing logs to the screen by default.

Tio is also very easy to learn, as all commands are viewable via ctrl + t, ?.

3. Use `micro` Text Editor

When editing text files from the command line, vi offers a powerful interface for advanced users. Personally, I’d rather just use an editor that offers inferrable shortcuts. The common suggestion in these cases is to use Nano, and indeed, Nano is easy to learn and easy to install. However, there is a newer project available that reflects more modern usage.

Micro is a text editor built in the last 5 years. It is actively maintained and has a very ergonomic set of default keybindings. Configuration files are stored in plain JSON, though it requires almost no customization and works exceptionally well out of the box. The default key bindings will make sense to anyone that has used a computer in the last decade. You copy test with ctrl+c, move the cursor via arrow keys, and highlight text using the shift key. If you are working in a graphical environment, mouse support works out of the box.

For these reasons, Micro is now my go-to text editor for command line work.

4. Use an HDMI Capture Device Instead of a Monitor

If you’re building a project that requires a graphical display by design, such as a kiosk or media player, a USB TTL serial cable might not be an appropriate solution. That’s not to say you are required to park another computer monitor at your desk. In these situations, it is still possible to avoid hooking a second monitor up through an HDMI capture device. An HDMI capture device is an adapter that takes any HDMI source and turns it into a USB webcam on the PC side.

Once your HDMI capture device is attached, you can then view the stream as any other USB webcam. On Linux, I use Cheese for viewing the display. One additional advantage to this approach is that you can very easily take screenshots of your work.

Although some professional units used for video editing are expensive, most can be purchased cheaply, especially since the video quality is not a huge concern when debugging. At the time of writing, I was able to find several options on Amazon for under USD 20.

5. Use Ethernet-enabled USB Hubs

Modern laptops with integrated Ethernet become harder to find every year. The integrated Ethernet port may have finally gone the way of the floppy drive.

The Raspberry Pi 0 does not have integrated Ethernet, either. In addition to the missing Ethernet port, the Raspberry Pi 0 only has one USB port available. The simple solution to this problem is to purchase a USB hub and attach a USB Ethernet adapter to an available slot.

I’ve recently seen more USB hubs on sites like Amazon that contain an integrated Ethernet port. This is great because it frees up a USB slot and gives you Ethernet access for free. They are also extremely inexpensive, so I like to keep a few in my toolbox.

6. Something Else?

Did I forget something? If you have a trick that has made life easier when developing on the Raspberry Pi, feel free to send me a message on Reddit- I will keep this list as up-to-date as possible.

Video Resources for Embedded Linux Learners

Rick Carlino — Sat, 12 Dec 2020 17:58:00 +0000

Core Concepts and Utilities

Hardware

Filesystems, Data Storage

C, C++, Compilers, Development tools

Booting, Boot loaders, Kernels

Init Systems

System Administration, Services, Tools

RetroTS- Porting the RetroForth VM to Typescript

Rick Carlino — Sun, 11 Oct 2020 22:53:00 +0000

This weekend I ported the Javascript version of RetroForth to Typescript, while also adding support for ES modules. The source code is available here. A live demo can be seen here.

If time allows, I will update the editor to use Monaco and add more HTML5 canvas support.

Keeping an Offline Copy of StackOverflow with Kiwix

Rick Carlino — Sun, 09 Aug 2020 20:03:00 +0000

For developers working in an offline environment such as public transit or rural dead zones, looking up information on sites like Wikipedia or StackOverflow might seem impossible.

Lately, I’ve been using Kiwix to keep offline copies of popular sites like Wikipedia and StackOverflow, aided by the availability of cheap USB mass storage devices. Kiwix is an offline web browser that uses a specialized archive file format to store copies of reference material. It was built to view Wikipedia articles but later expanded for other data sources like StackOverflow and Open Street Map.

Keeping an offline copy is simple:

Download Kiwix Reader
Find relevant ZIM packages, downloading them via BitTorrent. The files are large, often several hundred GB in size. A new USB drive may be helpful.
Open the files using Kiwix.

The experience is like the web version but with a blazing fast load time and the ability to look things up when the internet cuts out.

Software Tools for Hobby-Scale Projects

Rick Carlino — Sat, 02 Nov 2019 22:01:00 +0000

You can do three things with a computer. You can try to make money, and that is unlikely. You can try to become famous, and that never happens. And you can have fun, and that always works.

– Chuck Moore, creator of the Forth programming language

In a low-stress hobby setting, computers offer a vehicle for personal enrichment rather than strictly a tool for business or research. Some noteworthy Open Source projects started as hobby projects, the most notable example being the original Linux Kernel written by Linus Torvalds. Many other well-known programmers have learned their skills as hobbyists pursuing an intellectual curiosity rather than explicitly seeking a career or a serious objective. A well-publicized example is the often quoted newsgroup post written by Linux Torvalds in the early 90s:

I’m doing a free operating system, just a hobby, won’t be big and professional like gnu for 386(486) AT clones.

You can find examples of hobby software on sites like Hackaday or at your local Hackerspace.

Below is a list of useful tools I’ve come in contact with over the years. I share them with the hope that they bring you enjoyment in your next hobby project. If you would like anything added to this list or would like to share something you’ve built using the tools below, please contact me on Lobste.rs or Reddit.

I used the following criteria to select the tools:

They cost less than a coffee or are free.
They are quickly learned.
They allow you to accomplish a single task in a short amount of time (such as a Sunday afternoon)
They are less focused on the needs of long-term projects (scalability, speed, etc.) and more focused on ease of use and prototyping speed.

MyJSON.com - Free Cloud Storage for JSON

Static sites are easy and fun to build, but they can’t easily share persistent data between clients. Usually, this requires you to write a backend API or pay for a service like Firebase. That’s a lot of work, though, and the point of a hobby project is to have fun.

When I need to store very small amounts of unstructured JSON for a toy app, I use the MyJSON API. You can download and edit JSON documents via their CORS enabled REST API so that you don’t need to be bothered with provisioning a backend or paying hosting fees.

The owner of MyJSON seems to be a hobbyist as well. If you find value in MyJSON, consider thanking them on Twitter for their generosity.

REFL.ME - Simple Mobile Alerts

REFL.ME is an Android app that performs HTTP polling on a JSON URL and sends push notifications to your phone when the data changes. When paired with a Cron job and a JSON URL (see MyJSON section above), it offers a great way of creating mobile alerts without the hassle of authoring a full-blown Android app.

Vultr - Dirt Cheap VPS Hosting

Since hobby projects rarely see high-volumes of traffic, the main concern is often cost rather than performance.

Vultr is my go-to solution for cheap hosting. As of this writing, the cheapest plan they offer is $2.50 a month. Although their cheapest plan definitely won’t be able to fuel your next high-powered computing project, it is a great solution when you need to host small-scale tasks such as Cron jobs.

Dokku - A Self-hosted PaaS

Dokku offers many of the features seen in “Platform-as-a-Service” offerings such as Heroku. Unlike many PaaS offerings, you can host it on a cheap VPS service (like Vultr).

Although it is well suited for hobby projects, it’s important to note that Dokku also works wonderfully in production settings.

With Dokku, you can self host a push-to-deploy mechanism for your hobby project and not waste time with server configs. There are lots of community plugins that handle the most common use-cases.

Free MQTT servers

MQTT is a simple, easy-to-learn protocol for real-time communication. It’s most commonly seen in “Internet of Things” projects (Arduino, Raspberry Pi) but can also be used in web projects via WebSocket tunneling.

Numerous freely available servers are listed on MQTT Github Wiki.

These servers are great for simple hobby apps that run on low-end devices like Arduinos, RaspberryPi, or the ESP16 WiFi device. Some servers offer WebSocket support, which allows a browser to talk directly to the device without the need for a backend.

NGrok - Free Firewall Tunnel

If you are hosting a server on a home network with a dynamic IP address, it can be hard or impossible to accept incoming web requests. NGrok provides local workstations with a *.ngrok.io web address for exposing local servers to the Internet. It also provides HTTPS/TLS support, which is pretty cool!

Ngrok is especially useful when you want to experiment with third-party APIs that use WebHooks.

Bookmarklets

A Bookmarklet is a specially formatted bookmark toolbar entry that allows your browser to execute arbitrary code on any webpage. Bookmarklets have unfortunately gone unnoticed in recent years. They are a great alternative to browser extensions when you need to perform simple personal tasks such as scraping a webpage. There are some great tutorials and examples out there.

Cron jobs

Cron jobs are a classic Unix automation tool and a powerful helper for many hobby projects. Although many in the Linux community are well aware of Cron, they deserve mention as a tool in any hobbyist’s toolkit.

Cron jobs allow a Unix-like operating system to execute a script on a repeating timer. Cron jobs are useful for personal tools such as weather and stock market monitoring scripts.

CGI-scripting

I’ve blogged about the History of CGI Scripts in the past. Although it is almost unheard of to see new production systems deployed with CGI scripts in a professional setting, CGI scripts can provide a hobbyist with a cheap and simple solution for dynamic backend scripting. Many shared hosting providers (such as the one I host this blog on) provide CGI-script support at no additional cost.

(New) PatchBay.Pub - No-Signup Pub/Sub over HTTP

I have yet to try PatchBay, though I thought it deserved an honorable mention.

From PatchBay.Pub:

patchbay.pub is a free web service you can use to implement things like static site hosting, file sharing, cross-platform notifications, webhooks handling, smart home event routing, IoT reporting, job queues, chat systems, bots, etc, all completely serverless and requiring no account creation or authentication. Most implementations need nothing but CURL and simple bash snippets.

PatchBay appears to offer a solution similar to MQTT publish/subscribe topics, but entirely over HTTP. It allows developers the ability to create MQTT-like streaming applications using simple tools like CURL.

Check out the PatchBay website for more information

(New) EasyDB.io - DB hosting, no registration

EasyDB was recently featured on Lobste.rs. Like PatchBay, it looks like a great tool for hobby projects. I have not yet tried it for myself.

EasyDB is a one-click, hosted database provider. We give you a database for the programming language of your choice, you build something cool. It’s that simple.

It’s worth noting that it is an “ephemeral” database provider:

EasyDB is an ephemeral database provider. That means that after some time, your database will be removed. By default, databases will stick around for 24 hours. If you make an account with us, the database will last for 72 hours. If you really want to keep it longer, you can upgrade your database to a persistent database for $5/month.

Check the EasyDB website for more information.

What were CGI scripts?

Rick Carlino — Sat, 20 Jul 2019 18:07:00 +0000

A Hypothetical Scenario

It’s 1996. Ruby on Rails does not come out for another 10 years. Javascript is barely an idea yet and not all browsers support it.

You run a personal website from the public_www subdirectory of a shell account provided to you by your local dialup internet service provider.

Having a shell account affords you the ability to serve static HTML files, but during that era, if you wanted to generate dynamic content, like a hit counter that records website visits or a guest book (the comments section of websites in worlds past), things were not so simple.

In this article, we will take a look at CGI scripts, a technology of the early web that enabled software tinkerers to build simple (but useful) dynamic web apps.

CGI Scripts

CGI stands for “Common Gateway Interface”. During the 90s, CGI scripts provided the most viable option for individuals wishing to host dynamically rendered server-side content. If you’ve never seen one in action before, take a look at this example script I wrote. It dynamically displays the current time and user agent to the visitor. The example script is written in Bash, although historically, most CGI scripts were written in Perl (examples).

The source code to my example script is below. If you’ve never written an HTTP request by hand, check out my article about HTTP request anatomy:

#!/usr/bin/bash
echo "Content-type: text/html"
echo ""
echo '<html>'
echo ' <body>'
echo ' <p>The time is: '
# Prints current date/time to STDOUT:
date
echo ' </p>'
echo ' <p>Your HTTP user agent is:'
# A CGI-enabled server will set this ENV var at runtime:
echo $HTTP_USER_AGENT
echo ' </p>'
echo ' </body>'
echo '</html>'

exit 0

As you can see in the example above, all writes to STDOUT were translated to an HTTP response. Some information (such as $HTTP_USER_AGENT or $QUERY_STRING) are provided by the server at runtime. By providing a simple means of input (ENV) and output (STDOUT), CGI scripts were a very powerful tool for early web developers, particularly hobbyists and small scale developers. All you needed was a programming language and a CGI-enabled web server.

CGI scripts were often placed in a cgi-bin directory. When the server received a request to a particular CGI script, it would execute the script rather than serve its content.

The practice was eventually formalized in RFC 3875. Many of the RFC’s collaborators were themselves authors of HTTP servers.

Like the Year 1996, All Things Come to Pass

CGI scripting was undoubtedly useful and continues to be useful for small scale web applications, such as developer utilities, simple form data collection and local intranet tools.

CGI scripts were not the right tool for every job and their obsolescence is the result of better alternatives. The “one process per request” model offered by CGI scripting suffered from a number of problems, most notably an I/O performance loss. A CGI application that uses a database must open a connection to the database once per request, which incurs connection overhead. Furthermore, a CGI script with 100 users will inevitably create 100 OS processes (possibly more if it is executing sub-commands).

Another cause of CGI scripting’s demise was an inability to handle the complexity of large scale applications. Such problems were better handled by web development frameworks which arrived shortly after the CGI script.

Additionally, since the server was directly executing the script, security issues could easily creep in (the script shares the permissions of the HTTP server).

The inefficiencies of CGI scripts eventually led to the creation of more robust solutions such as FCGI, SCGI, numerous web frameworks and, more recently, “Function as a Service” platforms, which addressed many of the concerns listed while also adding additional features.

Links and Resources

Raspberry Pi Weather Station with Secure Scuttlebutt

Rick Carlino — Sat, 06 Apr 2019 14:33:00 +0000

A Secure Scuttlebutt Weather Station

Secure Scuttlebutt is a replicating, peer-to-peer log database.

It is unique because it is theoretically possible to create a mesh of replicating nodes that do not have internet access. Information is “gossiped” over local WiFi. When users move between wifi networks, the “gossip” moves with them, enabling friends (and friends-of-friends) to pass data along the social mesh. (See: SSB Protocol Guide)

Currently, SSB is used for social apps that revolve around chat, microblogging and file sharing. The most popular apps are Patchwork (desktop) and Manyverse (mobile), which can be thought of as peer-to-peer Twitter-analogs. SSB’s gossip makes it well suited for these types of apps.

I have not yet seen SSB used for remote sensor networks, and I think it is a good use case for it, given the protocol’s delay tolerance and ability to store immutable log data.

This month’s project creates a weather station that uses SSB as a replacement for internet connectivity. Weather logs are transmitted via data mule until they eventually find their way to a peer with internet connectivity, at which point the weather data reaches the global mesh of SSB users on the internet.

To put that in more practical terms:

An always-offline box in a remote location collects data.
A data mule connects to the weather station’s internal wifi network while Manyverse (mobile SSB client) is open on the data mule’s mobile phone. The network is only a LAN- there is no internet access at the remote site.
Within a few seconds, the weather station perform UDP peer discovery and exchange gossip.
The data mule goes home.
When the data mule connects to an internet-enabled wifi network at home, the data is gossiped to the wider SSB community (known as the “Scuttleverse”).

See It in Action

If you want to see the weather logs for yourself, you can follow its identity on the Scuttlverse:

@siAx0bQUNVIS3IH2d++o44atOzn8h7BuoULySDiKrHc=.ed25519

How We Built It

It’s a custom NodeJS application with a few dependencies:

node-dht-sensor: For reading an attached temperature sensor.
ssb-client: For writing to an append-only SSB feed.

The application reads the temperature once per day as well as at startup time. The reading is written to the machine’s SSB log so that it can be gossiped and replicated later.

Parts List

Wooden case (custom built)
DHT-22
Realtime Clock
Raspberry Pi 3 (the pi0 was too slow)
RaspAP-GUI (to create a WiFi LAN for UDP peer discovery)
USB battery pack

Operation

At startup, systemd initiates two services:

ssb-server, which provides an HTTP-based API for reading and writing to SSB feeds.
The custom Node app detailed in the section above.

The services run in the background and poll for temperature updates at startup and once per 24 hour period.

The full source code can be found on Github.

Improvements

This was a hobby project. Like all hobby projects, it is neither perfect nor optimized for real world conditions. If we were to spend more time on this project (we probably won’t), I would:

Use deep sleep to conserve battery
Add a solar cell for better off-grid support (we placed it in a spot that had grid power)
Automate the build process (currently, setup is a manual process)
Build a viewer app instead of publishing messages as "type": "post". Some users view the weather reports as “noise” on Patchwork.

Vintage Computer Restoration, Part II

Rick Carlino — Fri, 01 Mar 2019 14:00:00 +0000

In my last article, I set up serial file transfer between an ancient 386 desktop (running Windows 3.1) and a modern laptop running Xubuntu 18. Part I is available here.

Once my 386 was accepting files from a host laptop I was ready to move forward with my goal of accessing the web.

After installing the correct software packages (thanks, Internet Archive) I ran some network-based experiments on the machine. I ultimately met my goal of connecting to a public-facing server (HTTP and Telnet) while also learning a lot in the process. This was my first deep dive into the world of retro computing, and I had lots of fun!

Below I will documents how I connected my 386 to a network.

Experiment I: Telnet/TCP

When I started the project, I assumed that TCP network setup on older machines would be a challenge. To try things out quickly, I decided to pursue a less ambitious goal of piping a single TCP socket over a serial line. This would essentially turn my machine into a terminal emulator and avoid the issue of Windows 3.x networking altogether.

My setup was simple: on one end of a serial-to-USB cable, I had a modern Linux laptop. My 386 machine would sit on the other end of the cable and use the modern laptop as a proxy (mostly to avoid native TCP setup in Windows 3.x).

To facilitate data transfer between the two machines, I used the popular TCPSer application, which acts as a bridge between a TCP socket (Telnet) and a serial port.

The process was even simpler than I had anticipated. In the course of one afternoon, I was able to access several BBS systems and even the Telnet edition of Wikipedia. A thriving community of retro enthusiasts keep the BBS tradition alive and well, even in 2019.

I won’t say too much more about the setup process, since there are plenty of good tutorials that already cover this topic.

Experiment II: A Serial-to-WiFi Adapter

Another experiment I tried was connecting the serial port of my 386 to an ESP-8266 microcontroller. This is essentially the same result as using TCPSer, but with a compact microcontroller instead of a full Linux desktop.

The factory default ESP firmware has a number of Hayes compatible AT commands that allow users to open arbitrary TCP sockets and then read them over a TTL pin. By soldering a MAX-3232 board to an ESP, I was able to convert TTL signals to RS-232 and plug the ESP directly into the 386’s serial port, thereby eliminating the need for the host laptop mentioned in experiment I.

The module quickly connected to my home WiFi and I was able to use the default ESP commands to open connections. I thought this process would be more involved, but it really was as simple as soldering a MAX-3232 to an ESP.

Fun fact: I am fairly certain that the modem has better specs in terms of CPU, storage and memory than the desktop it is connected to.

Experiment III: A Full TCP Stack

After connecting to BBSes and Telnet services over two different channels, I was ready for a more challenging task. My goal was to get a full-blown TCP stack running on Windows 3.1 so that I could do more than just BBSing. To do this, I made various attempts to “trick” my 386 into thinking the modern laptop on the other end of the serial cable was a modem.

I found an interesting article within the Sega Dreamcast community and even tinkered with the idea of writing a custom script that could emulate a modem and dialup ISP. This turned out to be as difficult as anticipated. Due to a meager 4 megabyte memory limit, many of the browsers of the era would simply not load on the machine, citing a lack of memory. After failed attempts to load Netscape and Opera, I was ultimately able to install Internet Explorer 5. This came as a surprise to me, given the history of the three browsers.

After consulting with the Internet, I installed Trumpet Winsock on the target and began the configuration process on the host machine (a modern laptop running Xubuntu 18). Trumpet Winsock is a popular software package that is used on Windows 3.1/Windows 95 machines.

Configuring the Host: SLIP and Serial

As mentioned in part I, the physical layer between the two machines was a USB-to-serial adapter. Nothing new there.

For the data link layer, I chose SLIP, because I did not have the patience to deal with the more advanced PPP protocol. Although not as elegant as PPP, it proved to be easier to manage and was well supported by Trumpet.

To create a link between the two machines, I used slattach on the Linux side of the link. slattach is a tool that puts a standard serial line into “network mode” (SLIP, in my case):

slattach -s 19200 -p slip -d /dev/ttyUSB0

The flags were as follows:

Flag	Meaning
`-s 19200`	Speed / baud rate (my machine could not handle data faster than 19200 baud)
`-p slip`	Sets the line protocol to `slip`, as opposed to `cslip` or `ppp`.
`-d`	Enable debug output.

With my serial line configured for SLIP, I was ready to move up the network stack and begin IP configuration.

Configuring the Host: IP Tables

Once slattach is called, you can verify connectivity by running ifconfig sl0:


sl0: flags=4240<POINTOPOINT,NOARP,MULTICAST> mtu 296
        slip txqueuelen 10 (Serial Line IP)
        RX packets 0 bytes 0 (0.0 B)
        RX errors 0 dropped 0 overruns 0 frame 0
        TX packets 0 bytes 0 (0.0 B)
        TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

slattach has created the appropriate network interface, but we still need to configure the network interface and bring it up:

sudo ifconfig sl0 192.168.5.1 netmask 255.255.255.0 up

In my case, 192.168.5.1 was not in use. Your local network may differ.

To verify that everything worked, I ran ifconfig sl0 again:

sl0: flags=4305<UP,POINTOPOINT,RUNNING,NOARP,MULTICAST> mtu 296
        inet 192.168.5.1 netmask 255.255.255.0 destination 192.168.5.1
        slip txqueuelen 10 (Serial Line IP)
        RX packets 0 bytes 0 (0.0 B)
        RX errors 0 dropped 0 overruns 0 frame 0
        TX packets 0 bytes 0 (0.0 B)
        TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

My Linux laptop could now transfer IP packets over the serial line.

Configuring Trumpet

Now that the host laptop has SLIP and network interfaces configured, I needed to setup Trumpet.

Trumpet is a popular shareware TCP stack for Windows in the mid 90s. Be advised that if you plan on using this software for more than thirty days, you must purchase a license from the author.

I used the following configuration values:

Setting	Value
IP Address	I used `192.168.5.2`, but your IP address availability may differ
Netmask	`255.255.255.0`
Nameserver	You can use `8.8.8.8` if you are unsure. I noticed increased latency compared to my ISPs nameserver.

It is also important to ensure you are not using PPP.

Once properly configured, you should be able to run ping on either end of the network and get a response. Just for fun, I ran a small HTTP server to see if IE5 would load it (it did):

At this point, I am able to send TCP packets between the two machines, but I do not have access to the internet. To contact the outside world, I need to set up network address translation, which is how the 386 packets will travel to the outside world with my laptop acting as an intermediary.

Configuring the Host (Again)

Before proceeding, be sure to enable kernel level IP forwarding on the host:

sudo bash -c "echo 1 > /proc/sys/net/ipv4/ip_forward"

You will need to run the command above on every boot.

After that, you will need to run iptables to route outbound traffic. In my case, the outbound network interface was wlp1s0, but yours may be different (especially if you use ethernet instead of WiFi).

You can determine your outbound network interface by running ifconfig.

# Change the value of `wlp1s0` to the correct interface!!
sudo iptables --table nat --append POSTROUTING --out-interface wlp1s0 -j MASQUERADE
sudo iptables --append FORWARD --in-interface sl0 -j ACCEPT

I could now surf the web!

What I Learned

Fun fact: Although the SLIP protocol does not have a formal IETF standard, it does have a formal nonstandard that is comprised of a few lines of C code.

Although I did not have time to meet my stretch goal of running a web server on the target device, I did have fun revisiting the technology of my childhood. I was able to refresh my memory on many topics that I don’t see day-to-day (such as SLIP, iptables, xmodem and layers 1 and 2 of the OSI model). I also saw an exaggerated perspective of bloat on the modern web. Popular websites such as Google were nearly unusable. Uncompressed images that load quickly on modern devices could take up to 10 minutes to load on a constrained device.

On the flip side of the spectrum, there are still some websites out there (such as these) that performed surprisingly well. TLDP.org loaded quickly and rendered in a nearly identical fashion to the modern host machine.

Vintage Computer Restoration, Part I

Rick Carlino — Tue, 01 Jan 2019 14:00:00 +0000

I attended Vintage Computer Festival Midwest late last year. I purchased a functioning 386-based workstation running Windows 3.1 for $80 USD. The blog articles that follow will document my attempts to get it back on the internet.

Windows 3.1 has a special place in my heart. It was my first operating system. As a child in the mid 90s, it was also my first means of accessing the internet (via Compuserve and later, AOL).

Judging by filesystem dates, the computer operated from 1994 to 1997.

Specs and Hardware

Intel i386 @ 25 Mhz
4 Megabytes of SIPP Memory
120 MB hard drive
VGA Monitor complete with gigantic video card
Serial Mouse (predates PS/2)
AT Keyboard
3.5 Inch Floppy

The device is constrained. In 2019, even some microcontrollers are able to out perform this machine.

Project Goals

My intent for this project is to “make a 386 do things” and document my progress in a way that helps other retrocomputing enthusiasts. Here are some of the things I hope to accomplish:

Goal 1 (DONE): Get file transfer with my (modern) laptop via any means. This probably means via USB Floppy drive, or serial cable.

Goal 2 (DONE): Get it talking to a public-facing network.

Goal 3 (IN PROGRESS): Get it online and browse the text-based Web.

Stretch Goal: Run a public facing server on it for comedic effect. HTTP would be fun, although I may settle for an easier target (such as telnet) if the task turns out to be too ambitious.

Why?

Why not?

Data Transfer Attempt I: Sneakernet

Despite the excitement of Windows 3.1 classics like Solitaire and Minesweeper, I can’t accomplish much on a clean install of Windows 3.1. I need to get software on it somehow. I had a USB floppy drive and a case of floppies I found from a local thrift store.

This seemed like the easiest way to make progress. I could download classic Windows software from The Internet Archive Software Collection, store them to disk via USB floppy drive (on my laptop) and then manually load them on to the Windows 3.1 machine.

The sneakernet approach presented challenges, however:

Many titles required more than 1.44 MB of storage space and the machine does not have a CD-ROM drive (yet).
My floppy disk collection had data integrity issues. Some files appeared to write to disk fine, but were unreadable afterward.
Linux support for USB floppy drives in 2019 is not great. I would hit strange issues, such as drives refusing to mount, endlessly spinning discs and botched system shutdowns whenever the drive was attached. Such errors happened sporadically without warning.

I was able to get a few notable titles (like SkiFree) loaded but it proved to be inconvenient. Although I probably could have split files on my laptop and then reassemble them on the host via pkzip or similar tools, I decided to search for more convenient file transfer methods.

Data Transfer Attempt II: Serial Line Transfer

In the bad old days of home computing, when wide area networks were less ubiquitous and <font/> tags roamed free, users could transfer data between computers via RS-232 lines. Simple transfer protocols such as XMODEM made such transfers possible. The process was as simple as plugging two computers together with a serial cable. Despite their simplicity, the protocols and transport layers were incredibly slow.

The first problem to tackle was purchasing an RS-232-to-USB adapter, since my laptop (and pretty much every computer sold during this millennium) does not come with an RS-232 port.

After purchasing an adapter on Amazon, I realized that many of the models sold require a male-to-female USB extender, which required a second set of online purchases.

Another inconvenience was that the machine had a single serial port. This meant I could use a mouse or perform data transfers, but not simultaneously. Since serial mice are not hot swappable on Windows 3.1, switching from “mouse mode” to “file transfer mode” required a full reboot of my machine. Not to be deterred, I spent some time re-learning Windows 3.1 keyboard shortcuts and was ready to configure a software toolchain for data transfers.

File Transfer Incantation

Now that I had all the required cables and adapters, I was ready to configure the software. The main tools I used were:

GNU screen: Opens a serial port on the Linux side.
sx: “Send XModem”. A *nix tool for doing XModem transfers
terminal.exe: Used for opening serial (COM) ports on Windows 3.1 (screenshot).

To open a serial port and begin a serial transfer, I performed the following incantation:

screen /dev/ttyUSB0 9600
Type ctrl + a
Type :
Enter sx -b FILE_RELATIVE_PATH_GOES_HERE
Click Transfers => Begin Binary Tranfer in Windows 3.1 terminal.exe

This quickly became a tedious task. I ultimately wrote a custom Ruby script to automate the process and reduce the number of steps from 5 to 1.

What’s Next?

My Windows 3.1 machine is healthy, happy and transferring files from the Internet Archive to disk.

In my next post, I will talk about opening TCP sockets for fun things like viewing the Wikipedia Telnet Gateway and my ongoing struggles to get HTTP access on Internet Explorer 5.

If you were a computer user during this era and wish to share advice with me, please contact me via Reddit or Lobste.rs.