DEV Community: Lee Hambley

SeedStudio TRMNL, BYOD, and BYOS

Lee Hambley — Mon, 08 Sep 2025 12:15:27 +0000

I recently picked up 3 of the "TRMNL 7.5" (OG) DIY Kit" for $45 bucks each, it's an ESP32-S3, a 7.5" E-Paper/E-Ink display, and a 3.7V 2000mAh battery in a pretty price effective kit.

TRMNL seems to be a little overselling this collaboration with SeedStudio, making it seem easier than it is, and overall this is I would say good hardware with some questionable communications and positioning over what also, appears to be good software.

Unfortunately you basically cannot easily use this "OG DIY Kit" with TRMNLs hosted offering without paying an additional $40 per device, a specific BYOD licence, which grants "lifetime" access to the TRMNL hosted services.

TRMNL seems to leverage Platform.IO, which is another one of those things which on paper seems like a good idea, but is just vendor lock-in.

That all being said, it turns out to be fairly easy (after a little discovery) to leverage the device in your own way, mostly using the mostly avaialble, mostly documented software they provide with only really minor headaches along the way. Are the headaches worth 40 bucks? Probably not. If I had one of these devices and planned to use it with their portal, I'd probably pay, but I have three, and the math works out in favor of spending 20 hours.

In the end, the changes are small, in the trmnl-firmware repository, which seems to be excellently maintained:

make this diff:

diff --git a/include/config.h b/include/config.h
index e666613..1012a02 100644
--- a/include/config.h
+++ b/include/config.h
@@ -104,6 +104,6 @@ enum WIFI_CONNECT_RETRY_TIME // Time to sleep before trying to connect to the Wi

 #define SERVER_MAX_RETRIES 3

-#define API_BASE_URL "https://trmnl.app"
+#define API_BASE_URL "http://192.168.2.105:2300"

 #endif
diff --git a/platformio.ini b/platformio.ini
index dbcc5e1..0ddea5f 100644
--- a/platformio.ini
+++ b/platformio.ini
@@ -1,5 +1,5 @@
 [platformio]
-default_envs = trmnl
+default_envs = seeed_xiao_esp32s3

 ;       ========================
 ;      Third-party Dependencies

I run VSCode, where there's a Platform.IO extension, and there's a "Build" command which does the heavy lifting. It even noticed the edit to platformio.ini and rebuild when I changed the target.

In the byos-hanami directory, simply run docker-compose up (or podman, etc) after setting appropriate variables.

With that, then I ran:

 esptool.py -p /dev/cu.usbmodem2211401 -b 460800 write_flash 0x10000 .pio/build/seeed_xiao_esp32s3/firmware.bin

to Flash it, using https://github.com/espressif/esptool which is provided by Expressif for interfacing with their modules on MacOS (and other platforms).

The other firmwares in teh ./build/ directory seem to be for the -C3 variant of the ESP32, which I assume is used on other boards.

The Importance of Cadence, and the trade-offs you make.

Lee Hambley — Wed, 22 May 2024 11:22:10 +0000

My personal philosophy on work is shaped strongly by two books in particular:

The Principles of Product Development Flow (978-1935401001)
Joy Inc: How We Built a Workplace People Love (978-1591847120)

They both arrive at sustainable, productive, predictable practices for working on project delivery (mostly software delivery) using very diverse methods.

Joy Inc puts the focus on psychological safety, inventing structures that ultimately minimize risk and leave people feeling safe.

The Principles of Product Development flow takes a radically different approach, speaking of systems as a series of pipes and junctions, and using very dispassionate language to talk about size of work, flowing between nodes in a graph, and talking about "building healthy systems".

Despite the wildly different approaches, both basically arrive at what you may already better know as "agile principles". Short cycles, bite-size chunks of work, frequent contact with stakeholders and predictable cadence.

Short cycles minimize the risk of investing in the wrong direction; bite-size chunks are less likely to cause a "log-jam" when someone loses half a day reading long docs or code reviews; frequent contact with stakeholders ensures nobody is ever surprised, and a predictable cadence is a two-way contract between stakeholders and makers that goalposts won't move, and scope won't creep (..until the next cycle)

The cadence trade-off is one I see most frequently misunderstood in contemporary business practices.

When a company settles on three month cadence for planning (as many do); if you get new information a month into the quarter in theory you should stick to your plan. You planned for three months, presumably that planning investment was not small; and you should not break trust with teams by approaching them in the middle of a cycle with new information.

For many companies, a quarterly cadence, if taken strictly is utterly unthinkable.

If your planning in March for Q2 can be invalidated in early April information about a new potential customer tender, then you cannot run your business on quarterly plannings.

If you allow external stakeholders to dictate your timelines, for example, in the middle of a quarter, you learn on the 19th of the month that you have time until the 30th to scramble to provide something to appease external customer tenders you will undermine all trust from your teams in your processes and deadlines.

Mid quarter, 11 days, a weekend and a public holiday in the middle? What kind of cadence is this? You're sewing disruption into your teams and undermining your credibility as a leadership group.

Conversely, when you do this, you signal desperation to your stakeholders, an acknowledgement that you are inexpert in your field, and you bow to their whims.

Are diodes required for x27 type motors?

Lee Hambley — Thu, 02 May 2024 13:54:24 +0000

tl;dr NO (see below for oscilloscope traces)

I'm a bit of a noob when it comes to electronics, but I have a project using the X27 motor which is surprisingly difficult to find information about.

You should both absolutely use a stepper driver board, and it's completely unnecessary.
It's both completely safe, and unsafe to use them direct on PWM GPIO pins on a classic microcontroller with and without diodes.
Break-out boards are on the market with and without place for flyback (EMF, etc) diodes.

As a rule you should plan with 2x the driver voltage for a flyback diode for any inductive load;

I planned to use the X27 motor on a 3.3v power supply from an STM32 micro controller on a custom PCB, however coarse reading in the internet left me with questions if that's safe; forums (Arduino forums, naturally) recommend using an Arduino as they are "more tolerant".

I found in the STM32F103 datasheet that it's absolute max tolerance for a GPIO pin is 9v, and I'll be powering that with a 5v supply, even though it's a 3.3v logic level.

For the ATMega32U4 on the Arduino Leonardo, the max tolerated voltage is VCC+0.5 (see "§29 Electrical characteristics")

See https://cdn.sparkfun.com/datasheets/Dev/Arduino/Boards/ATMega32U4.pdf

These two traces show that at worst there's about 5v on the bus, which seems to be outside the Arduino's tolerance, but it has ran perfectly OK for more than a week unattended on my desk, and it's well within spec for my STM32 project goals.

I assume this is something to do with driving AC loads with PWM but I can't say for sure. I may also be measuring this wrongly as I grounded the probes to the arduino ground, and not to the "ac" ground.

In the end, I'm just happy that it looks like driving these directly with an STM32 without any diodes or custom driver board should work just fine.

Lead Screw Drawings Optimum F40E Mill

Lee Hambley — Mon, 08 Apr 2024 22:36:52 +0000

I am lucky enough to own an Optimum F40E mill which is a pretty OK piece of hardware for manual milling.

I've been working on a CNC conversion (actually CMC, but close enough) and wanted to change out the poorly made lead screws for hardened ball screws.

One other project online has a walkthrough how to order the 25mm balls crews, however that required modifying the mill casting, and some heavy manufacture of adapters. Also, the person from that post had what appeared to be the standard sized table, I have the long table.

Here are Fusion360 drawings of the X and Y lead screws. I didn't bother to measure which acme profile they were, also I didn't bother to check if they are left or right threads on the retainers on the both ends of Y and the front end of X.

Nominally bearing surfaces are 15mm which should make it possible to buy decent angular contact bearings which will take both the rotational and thrust loads.

Optimum calls this a Bohr-Fräsmaschine (German for "Drill-Millmachine"). If you don't want to deal with Optimum, the handbook PDF is hosted here on imgur as PNGs because it was the easiest way to put this online https://imgur.com/a/ACnzYzE

(also here in higher resolution https://imgur.com/a/CzrLucX)

Nominally SFU2005 should work fine here. If I order ballscrews, I will update this post with the drawings I use to order with in case you want to undertake your own mill conversion.

I've had it with "String"

Lee Hambley — Wed, 05 Apr 2023 17:10:38 +0000

After 17 years working with computers, some of the most fundamental building blocks we reach for every day are still not universal.

General purpose programming languages came into existence in the 1960s with IBM's System/360, with Fortran, COBOL and early Lisps. The use of "String" in computer science goes back even a decade earlier, with a long-since obsolete and esoteirc language COMIT.

I mention this just to highlight that we've had long enough to do better at this, as an industry.

First, let's digress a little into serialization, just to separate the concepteptual part (hello category theory 👋), and the reality that much of what software engineers do day-by-day is serialization, and deserialization of data over the wire, be that from browser to server over the web, or from server to data store such as into a database.

Serialization, and deserialization are such a massive part of what we do, it's no wonder that the "lowest common denominator" is the state of the art we are left with. If we had a better way to share "types" across systems, we'd effectively need everything from browsers, to toasters (thanks IoT folks), to servers, to middlewares, etc to support the new standards. Everyone can "fall back" to a string, so that's unfortunately what we get left with.

Eric S. Raymond, lamented once upon a time that Unix's greatest triumph, and biggest mistake were that "everything is a bag of bytes". The metaphor underpinning most of Linux and Unix's operating model for most of the last 35 years has been "everything is a file", and a file is a sequence of bytes.

"String" utility functions are broadly available, and it's a common paradigm.

Serialization aside for a moment, looking into programming languages themselves, by and large, the String type (or something akin to it, even in ducktyped languages such as Ruby) is ubiquitous.

Strings, character arrays, suchlike are great, except all the times they aren't.

A typical string handling library will expose functions to upercase, lowercase, trim, reverse, transpose, camel-case, title-case, and more, and more, and more. If we're extremely fortunate, we might have a library at hand which can reliably normalize Unicode diacritical characters, where an ¨ and a u can be combined three ways to make a ü (¨u, u¨, or ü), which, if we're folding could be expanded to "ue" if we're in German, most of the time, except place names, but not so for Swedish or Danish.

Strings, in most languages, we are thankful to say are encoded in UTF-8, a variable byte size representation which can represent up to 2,097,152 "code points" (The Unicode Consortium has identified about 1,112,00 so far). The variable size encoding means that an a in UTF-8 is the same as an a in ISO 8859-2, helping us leave the rocky late-1990s behind, where it was different flavours of ISO 8859-2 which left us with documents from Greek computers scrambled when viewed on a French machine, because the values from 128-256 in the extended ASCII character set values were assigned differently based on the locale.

Speaking of ASCII, the lowest of lowest common denominators, a formerly 7 bit encoding, expanded to 8 bits somewhere close to the dawn of time, with 7 bits, a whole 128 codepoints could be encoded, from which most English speakers would recognize around about ~90; the other ~50 are so-called "control characters", printer control, terminal control, a virtual bell, a backspace character, etc.

Every "String" in every programming language in the world will accept valid ASCII.. and that somewhat long introduction lets me finally make two points:

Strings are too liberal, a correctly implemented stack of software would happily accept a string of serialized backspace characters as a valid "name" in a form. It's all character data, after all.
Most of the functions and methods for working with a String in most programming languages actually render our strings useless. Reversing a human name, URL, or or SSN is semantically meaningless and mangles the data; upercasing an email address (the definition of which spans two separate RFCs) is poorly defined.

Strings, are, once the data is deserialized, almost never the correct data type to use, but we don't have either a robust ubiquitous library (let's not even talk about portable calling conventions across languages, before our grandfather's C enters the chat), nor a robust serialization format we can ever dream of getting widespread attention.

There are prescious few specifications talking about

Here's some examples:

ASCII control codes in a Name

Easily enough, someone could submit a form with their name, but mistakenly copy a 0x01 bytes ("START OF HEADING"), that byte is invisible, so if it's ever interpolated into a document (e.g into a PDF) the result is incorrect. Instead of a "String" we should use a type for human names which is defined as only comprising visible alpha-numeric (sorry X Æ A-12) characters, some ligatures, some dashes (unicode defines 25, from which most people would assume 2-3 can be a valid part of a name). It becomes more complicated with non-English names, or names transliterated into their original alphabets, but the rules are the same.

A "Noun" must serialize cleanly to text, and back in a way that would survive printing and scanning (for compatibility with 2000 years of human government beurocracy)
Two nouns might compare equally even if they are in different alphabets ("Munich" and "München" are the same city in English and German, and "München" and "Muenchen" are both valid German spellings, similarly "Cologne" and "Köln" are the same, and "Hamburg" (English, German) and "Hamborg" (French) are all the same.)

Unicode Characters in a restaurant name

Unicode is amazing, but the taxonomy leaves a little to be desired

Phone Number as a "String"

⚠️ This has been sat as a draft for a while, but I'm publishing it incomplete because my point stands, and collecting extra examples helps noone. Strings suck.

GnuPG (PGP) SmartCard over SSH to a VM with a Yubikey

Lee Hambley — Thu, 07 Apr 2022 09:14:10 +0000

Agent Forwarding (GNU Privacy Guard (GPG) & SSH) over SSH.

Most all the blog posts I found for this topic were five or more years old, and were referencing a time prior to GnuPG 2.1 which was where some real changes landed, which made this process possible, and safe. These days even LTS distributions such as Ubuntu 20.04 include GnuPG 2.2 which is yet simplier, and requires even fewer hoops be jumped through.

My objective seemed simple:

From my laptop (macOS Monterey)..
Forward my EC-DSA key with an SSH agent...
..and a GPG Key (actually a smartcard, but that doesn't matter) ..
into a Linux VM over SSH so that I can work there, sign commits, and clone git+ssh:// repositories

It's possible to use a GPG key (and smartcard) as an SSH authentication token, but I'm not interested in that, I have separate SSH and GPG keys, and I'm happy with that set-up.

Check you have new enough software

# Workstation:
$ gpg --version
gpg (GnuPG) 2.3.4    # anything over 2.1 is fine

# VM (ssh target)
$ gpg --version
gpg (GnuPG) 2.2.19

Generate or configure relevant keys:

Smart-card

Out of scope, but it's pretty doable. Follow this guide:

https://support.yubico.com/hc/en-us/articles/360013790259-Using-Your-YubiKey-with-OpenPGP

Generate a GPG Key

$ gpg --gen-key

It is an interactive program which will ask for your real name, and your user email address, complete those, and then show you some output, the important part is the long key ID:

Real name: Example User
Email address: user@example.com
You selected this USER-ID:
    "Example User <user@example.com>"

..... snip .....

pub   ed25519 2022-04-06 [SC] [expires: 2024-04-05]
      7B5CB440DA3A316537466897128986B90599B1B1
uid                      Example User <user@example.com>
sub   cv25519 2022-04-06 [E] [expires: 2024-04-05]

In this case 7B5CB440DA3A3... is the key ID, copy it to the clipboard, or export it to an environment variable, we'll need this a lot.

Generate an SSH key

Run this and follow the prompt...

$ ssh-keygen

This will generate something like an ~/.ssh/id_rsa or ~/.ssh/id_ecdsa or something depending what you configure.

Getting Relevant Socket Addresses

# On your local machine:

$ gpgconf --list-dirs agent-ssh-socket
/Users/<your username>/.gnupg/S.gpg-agent.ssh
$ gpgconf --list-dir agent-socket
/Users/leehambley/.gnupg/S.gpg-agent
$ gpgconf --list-dirs agent-extra-socket
/Users/<your username>/.gnupg/S.gpg-agent.extra


% On the remote machine:
$ gpgconf --list-dirs agent-ssh-socket
/run/user/<your numeric user id, probably>/gnupg/S.gpg-agent.ssh
$ gpgconf --list-dirs agent-socket
/run/user/<your numeric user id, probably>/gnupg/S.gpg-agent

The GPG Agent and SSH Agent sockets should be self-explanatory enough, however the "extra" socket is peculiar, see this from the docs:

Also listen on native gpg-agent connections on the given socket. The intended use for this extra socket is to setup a Unix domain socket forwarding from a remote machine to this socket on the local machine. A gpg running on the remote machine may then connect to the local gpg-agent and use its private keys. This enables decrypting or signing data on a remote machine without exposing the private keys to the remote machine.

The extra socket then is a slightly less privilidged socket which safely allows forwarding to a remote machine without giving that remote machine full control over your local GPG agent (as a the normal socket would have)

Local Configuration

# ~/.ssh/config
Host thevmweworkin
  # this is standard SSH config, mostly
  HostName 192.168.64.11
  User vagrant
  Port 22
  UserKnownHostsFile /dev/null
  StrictHostKeyChecking no
  PasswordAuthentication no
  IdentitiesOnly yes
  LogLevel FATAL

  # This is the GPG/SSH forwarding 
  RemoteForward /run/user/<your numeric user id, probably>/gnupg/S.gpg-agent /Users/<your username>/.gnupg/S.gpg-agent.extra
  RemoteForward /run/user/<your numeric user id, probably>/gnupg/S.gpg-agent.ssh /Users/<your username>/.gnupg/S.gpg-agent.ssh
  ForwardAgent yes
  ExitOnForwardFailure yes

# ~/.gnupg/agent-config.conf
cat ~/.gnupg/gpg-agent.conf
default-cache-ttl 600
max-cache-ttl 7200
pinentry-program /opt/homebrew/bin/pinentry-mac            # brew install gnupg for this, or don't specify pin entry
extra-socket /Users/<your username>/.gnupg/S.gpg-agent.extra  
enable-ssh-support
keep-display
default-cache-ttl 600
max-cache-ttl 7200
keep-tty
keep-display

# Your ~/.zshrc or ~/.bash_profile, etc
eval $(gpg-agent --daemon)

Run this to make sure your agent is running/restarted with the correct config:

$ gpg-connect-agent reloadagent /bye   # will start an agent if you didn't have one running
OK

Remote Configuration

Configure Git to require signing commits:

$ git config [--global] commit.gpgsign true
$ git config --global user.signingkey 34EC1A4D011E7FDFFD6E3722A4F823DC30FA9DA7!  # the exclamation mark makes Git use this key, and not try and detect a subkey to use

Configure SSH to remove local sockets of already running daemons, and allow you to overbind them:

/etc/ssh/sshd_conf

# add the following:
# https://superuser.com/questions/161973/how-can-i-forward-a-gpg-key-via-ssh-agent
StreamLocalBindUnlink yes

Upload the key someplace (Github, Gitlab), and into the VM public keychain

# From the host machine
gpg --output public.pgp --armor --export 34EC1A4D011E7FDFFD6E3722A4F823DC30FA9DA7
gpg --armor --export 34EC1A4D011E7FDFFD6E3722A4F823DC30FA9DA7 | pbcopy

scp public.pgp thevm:~/public.gpg

Then go to your profile and paste the new GPG key into your profile.

Check Everything Works

Enabling support for Python scripts for use with perf uprobes on Ubuntu 20.04

Lee Hambley — Wed, 26 Jan 2022 08:48:16 +0000

Linux's perf subsystem, about which you can read a lot more here https://www.brendangregg.com/perf.html is an invaluable subsystem for tracing the performance of the kernel and user-space code.

It's a bit of the wild-west, it's a fast moving space with lots of moving pieces, and lots of unusual kernel- and user-space interop which makes it tricky to use.

What makes it trickier unfortunately is that Canonical saw fit to remove scripting support from the perf tool in the linux-tools-generic (and linux-tools-$(name -r)) package rendering it practically useless.

Confirm you have this problem with the following:

sudo perf script -s lang

Scripting language extensions (used in perf script -s [spec:]script.[spec]):

  Perl                                       [Perl]
  pl                                         [Perl]
  Python                                     [Python]
  py                                         [Python]

Great, seems like perf script has Python and Perl support, right? ... not quite:

The steps here can be also used to enable Perl scripting support, but I'm sticking with Python, since what little documentation there is covers this specifically.

sudo perf script -g py
Python scripting not supported.  Install libpython and rebuild perf to enable it.
For example:
  # apt-get install python-dev (ubuntu)
  # yum install python-devel (Fedora)
  etc.

Unfortunately the instructions won't help you and there's no way to rebuild the linux-tools-* packages easily even after adding the {python,perl}-dev{,el} packages as suggested.

Rather one is required to get the kernel sources, and patch them and then re-build the entire kernel package tree and then and only then can a version of linux-tools-* with scripting support be installed:

The instructions ultimately are the normal instructions for building a new kernel; but I'll drop a block of shell commands below to save having to read a bunch of convolouted docs with multiple options:

Follow to about half way: https://wiki.ubuntu.com/Kernel/BuildYourOwnKernel
Modify a file to remove NO_LIBPYTHON=1 (note it must be REMOVED, not set to 0)
Compile the kernel from that source tree (resume the build your own kernel guide)
Install the linux-tools package with dpkg -i .. from the newly built packages.

So:

# 1. enable deb sources in /etc/sources.list
sed -i '/deb-src/s/^# //' /etc/apt/sources.list && apt update

# 2. install kernel sources for current kernel (installs to $PWD) e.g ./linux-5.4.0
apt-get source linux-image-unsigned-$(uname -r)

# 3. install python-dev package & general kernel build deps
apt install python-dev libncurses-dev gawk flex bison openssl libssl-dev dkms libelf-dev libudev-dev libpci-dev libiberty-dev autoconf default-jdk


# 4. change into the directory
cd ./linux-generic-5.4.0

# 5. apply this patch:
patch -p1 << EOPATCH
--- ./debian/rules.d/2-binary-arch.mkbak    2022-01-26 09:40:31.791079823 +0100
+++ ./debian/rules.d/2-binary-arch.mk   2022-01-26 09:40:46.671072005 +0100
@@ -702,7 +702,7 @@
    cd $(builddirpa) && $(kmake) syncconfig
    cd $(builddirpa) && $(kmake) prepare
    cd $(builddirpa)/tools/perf && \
-       $(kmake) prefix=/usr HAVE_NO_LIBBFD=1 HAVE_CPLUS_DEMANGLE_SUPPORT=1 CROSS_COMPILE=$(CROSS_COMPILE) NO_LIBPYTHON=1 NO_LIBPERL=1
+       $(kmake) prefix=/usr HAVE_NO_LIBBFD=1 HAVE_CPLUS_DEMANGLE_SUPPORT=1 CROSS_COMPILE=$(CROSS_COMPILE) NO_LIBPERL=1
 endif
 ifeq ($(do_tools_bpftool),true)
    $(kmake) CROSS_COMPILE=$(CROSS_COMPILE) -C $(builddirpa)/tools/bpf/bpftool
EOPATCH

# 6. Build the targets
#    if this fails and you need to install new deps, then
#    reset things with `fakeroot debian/rules clean`
fakeroot debian/rules binary-arch

# 7. Wait (about 15-20 minutes on a decent laptop, probably, YMMV with virtualization) 

# 8. Install the new package with dpkg -i 
#    note: filename here varies a lot by version and arch 
sudo dpkg -i linux-tools-*.deb

After all this then the command sudo perf script -g py should generate a pyton script with hooks for any perf recorders you might have enabled already.

On professional standards

Lee Hambley — Wed, 17 Nov 2021 20:15:51 +0000

Building software occupies an unusual space, it's certainly more of a vocation along side plumbers, electricians and technicians, however there remains an enormous scope for creativity and problem solving because unlike those aforementioned vocations computers more-or-less lack any physical or chemical laws governing our application of techniques or which can constrain our fantastically complicated creations.

I'd go as far to say that the debate about whether or not people need to have computer science degrees is wholly misguided. Scientists ... let's say people with an analytical mind, a fundamental training in the "laws of nature" of computers, these people may make excellent engineers, but science is not engineering, and a computer sciences degree is absolutely not a guarantee of being a capable engineer, just as a physics degree doesn't qualify someone to engineer buildings, bridges or skyscrapers, even if it might equip them with the right analytical skills.

I am generally opposed to certifications and standards, as I believe that easy access to technology-focused knowledge work is such an enormous enabler for individuals, that the floodgates for new boot-camp graduates and interns should be wide, wide open to encourage as many people as possible into the field, as the skills base widens, and as accessible low- and no-code tools proliferate the volume of software in the world continues to grow, and more and more opportunities continue to exist where profitable businesses need extremely highly skilled engineers to assist them, and those engineers can command a premium salary or rate.

Technology enjoyed this way can line the pockets of everyone concerned, the fresh-faced rookies, the 200 year old jaded witches and wizards (architect years are like dog years.), and the company who is profiting all the time (maybe with a smaller gross margin than they wished..) from the software we build.

That all said, in the absence of professional standards such as those which exist in the form of building, HVAC, electrical and plumbing codes, it continues to be exhausting and draining to constantly be held back by sub-standard work, and for that definition of sub-standard to be entirely subjective.

For all the progress a software architect (or someone in a Principal or Distinguished role) can make in fostering a culture of collaboration, design, deliberate decision making, and modelling in collaboration with stakeholders from diverse disciplines (towards business or IT operations), the environmental pressure to make changes at any cost are everpresent, and in the absense of objective measures (including standards & certifications) things can rapidly devolve into the famed "ball of mud" non-architecture which has more in common with the Shanty towns than anything you might recognise as deliberate architecture.

Just as Shanty towns are often shaped by the environmental constraints, lack of resources and rapid growth, software projects can suffer the same. Shanty towns can gentrify info middle-class neighbourhoods, but more often than not, at least in software the growth on the fringes out-paces the gentrification and investment in the core.

The architect(s) must focus their energy, and choose their battles wisely.

Ultimately the architect(s) require the support of the tradespeople working hands-on on the code every day. Often programmers lament that their business and project counterparts don't grant them time for re-factoring, writing tests, or conducting load tests, however their reasoning is likely flawed, nobody negotiates (at least not over the table) those things with their HVAC, plumbing, or electrical technicians.

Those tradespeople have stacks of standards, international and regional to adhere to. An electrician will not add a new circuit to a pre-1950s house (which in many parts of the world lack the 3rd wire in the wall sockets which carries a protective earth to protect against electrocution) without performing the remedial work to re-fit the house with modern wiring, no matter the cost. The liability, insurance, loss of license and with it, their livelihood is a cost so high that with sub-standard wiring in your outdated home, professionals cannot, and will not take shortcuts.

The rise of the open office may indeed correlate with the cost of installing adequate HVAC systems in closed-office environments, and tenants would likely violate their lease and put their employees' health in danger by taking shortcuts, thus HVAC professionals will also not take the risk of assuming the liability of contributing to sub-standard work.

Technology however doesn't directly have these kinds of standards in place, sure there's security standards such as ISO 27001, the GDPR (article 15) mandates that access to data is logged to assist in preserving the data subject's rights, financial and telecommunications software is required to keep data for a long time for auditing, and certain fields such as the payment card industry specifically mandate against storing data.

With all those "non-functional" requirements, however enforcement and policing is extremely lax. Article 15 of the GDPR should render it illegal for any software to show any data without leaving a trace, however most Django, Rails, Express.JS or similar applications will happily serve a GET request and log practically nothing in the logs, which may or may not even be persisted. I make no judgement on whether or not the global invalidation of practically every piece of software on the planet is a bug or feature of the GDPR, but our industry is steadfast in it's refusal to even address the topic.

Digressing somewhat to professional standards, most seasoned architects and engineers would begin to raise questions about whether using some or all of a dynamically typed language, without CI/CD, with a privileged database user account, with access to an SSH console would begin to constitute malpractice, if we drew parallels to tradespeople in the physical world if would be akin to working on a live electrical panel, or sourcing fresh air for HVAC From next to a busy highway, or recycling water from a bathroom into a kitchen, etc, etc.

Less clear, again is perhaps the use of certain paradigms such as stateless- or functional-programming, event sourcing, certain IPC strategies, and various combinations of the above, one may not forget that just like a building is built once, and lived in forever, code too is built once, and lived in for a long time, perhaps it is worth investing.

A passing slight at Agile, before I wrap up, which is that often Agile is used as a tool to keep things moving, no matter the total cost, as long as the immediate costs are bearable, in a strongly opinionated ecosystem where technology choices are made for you, this may be appropriate. I might be tempted here to draw parallels between "renting" space someone else designed, living out of a small studio apartment, to how it is to "rent" space in an opinionated inflexible framework, there may come a time when really investing into "where you live" as an engineer in a business context makes more sense, but having rented for the first few years of your life, you may lack the maturity to do sufficient, just-in-time planning; a delicate skill which takes years and years of vocational training to master.

Of course, just as hand-building a shed in your garden under a certain size is not subject to building regulation, nor should all software be subject to regulation, we must, at all costs preserve the accessibility of our industry at large, and preserve it as a creative practice in which people can innovate, but we also urgently need to develop a philosophy of our own, and begin to identify objective measures of good and bad software, I might start with something like:

Some application of schema definition tools Any polymorphic type system, json schema, AVRO schema, etc, etc. Polymorphism (i.e interface types, traits, generics, etc) is important to avoid hard static typing, but dynamic typing leaves too much surface are for testing. With a proper type schema, parameterized testing becomes feasible, and the total set of inputs and outputs becomes a much smaller, but likely still infinite set. Type systems also provide a great platform for documentation, IDEs, language servers, generating documentation, and evolving a schema over time. Avro is outstanding here, as is GraphQL as schema definition languages, the rest (all of them.) fall down on one or more vital area.
Proper application of an "aggregate root" pattern For example, avoiding the "active record" pattern, and grouping locally related objects such that data access is controlled by the "root" (i.e in a "invoice has line items" example, the invoice would be the entry-point for changes to lineitem quantity to apply business rules, calculate taxes, check customer standing, and stock control. Restricting a "model" to the core few objects of a system. ActiveRecord isn't an objectively bad pattern, but when all objects are addressable without constraint, it can be nearly impossible to maintain a strong aggregate-root based model, to leverage the often more powerful Repository Pattern, or Event Sourcing, and it's difficult to leverage the extensive capabilities of the underlying data stores.
Proper application of modelling tools and design strategies Framework-over-database is precisely the absence of software design, in today's environment the developers who operate in that level will soon find themselves in real trouble from low-and-no-code backed by a Firebase database or a Google Sheets sheet. Just as a highly professional building crew will not be enthused about trying to turn a Shanty Town shack into a built-to-code residence, software architects and highly professionalized teams will be reluctant to
Proper managing of side-effects and transactional guarantees An operation such as booking a cinema ticket, or securing a package holiday should have very clear guarantees about the transactional properties. It should not be possible to book 1 of n tickets in a mult-seat booking, or book a flight without a hotel if the customer tried to secure both. In the absence of support from your data store, some kind of "two-phase" transactions can be emulated for which established patterns exist. Controlling for side-effects (such as API calls, IO, reading random numbers, reading from external data sources) are harder to control for, but again practices are well established in some communities to have entirely deterministic execution of some or all codepaths, and simulation tools which will will rigorously "prove" a "pure" (no side effects) model, these tools are effectively unused entirely by business software, to the detriment of practically everyone.
Proper Auditability Probably through some kind of event-sourcing mechanism (e.g log everything that happens as the source of truth, and derive projections from it, PDFs for your stakeholders, and various read-only databases which can be hyper optimized for your use-cases). Event sourcing is the oldest record keeping strategy on the planet, and with the blistering power of modern computer systems almost any amount of data can be managed in an event-sourced fully audited manner, with proper caching strategies, careful treatment of read-cursors, and proper aggregate partitioning schemes this architecture can be potentially infinitely scalable. Regrettably it only works right if you apply it along side the other proper things enumerated in this list.
Proper user- and authentication-strategies Conflating a single human being with a single account is a sin committed by practically every piece of software on the planet, ever. That being said decoupling the human who is authorized from the "account", and permitting many-to-many-to-many relationships enables enormous agility in software. Such strategies would permit seamless "customer authorizes admin to log into their account" for support without requiring super-admins (violates the GDPR), would assist elders who are not intimately familiar with technology to delegate access to their accounts to trusted relatives temporarily or permanently, and allows developers under careful control to impersonate, or share accounts on testing environments. Here are mature and established standards such as OAuth 2.0 which have stood for a decade or more, applied correctly they are enormously liberating. Role-based-access-control (RBAC), super user "admin" accounts, and similar anti-patterns can be successfully avoided if OAuth is applied properly with a sympathetic understanding of the nuance in in the spec, however again this requires taste, and expertise, and must overcome the ever-present draw to grab the simplest library in the language's package registry and copy 10 lines from a README and move on to the next ticket.

Application of all of those patterns above, at precisely the right time, not too soon and not too late; which requires intuition and experience, something which is regrettably entirely subjective, it also inflates enormously the cost in some places, but radically reduces cost over time as a framework-over-relational-database model stagnates, and becomes intractable, a properly designed system can be an everlasting joy to maintain and offer an enormous competitive advantage.

I think Agile and XP practices, the easy accessibility of framework-first development, and the overwhelming ratio of inexperienced, to experienced programmers means that the ideas above will never reach mainstream adoption.

For every senior, experienced programmer who takes time to understand the needs of the stakeholder, the regulatory and environmental (i.e existing code, existing practices), and attempts to produce high quality, well-thought-out work, which is sympathetic to the entire nuanced multi-dimensional spectrum of needs.... there will be ten less experienced people who lack the experience necessary to build for maintenance and will through malice or inexperience (incompetence?) offer a "competing", but sub standard solution.

Sometimes you don't need top-of-the-line high skill plumbing, and you literally need the software equivalent of a bucket and hose, but business stakeholders often can't tell the difference, and if a more experienced engineer commits to a certain time frame, scope and cost, there will often be that less experienced engineer promising the business what they need, right this second, for half the price, with no up-front discussion. To both the inexperienced engineer and the business stakeholder, they are apparently offering the same thing as the more experienced engineer, but the more experienced engineer is offering a different, significantly higher quality product, so the "market" cannot behave rationally in the absence of regulation or objective standards.

A 4G home alarm for my garage

Lee Hambley — Thu, 22 Apr 2021 08:09:53 +0000

This isn't really a dev/tech post, but I wanted to be public somewhere with the information because I found nothing in Google for a problem.

A company called Digoo (seems to be part of a larger thing, since their app has support for hundreds and hundreds of home automation IoT thingies) makes an alarm called the "DG-ZXG30".

From what I can tell, they make it in 2G&GSM versions, and also the one I had to buy, the 4G&GSM one. A prior version of this alarm was the DG-HOSA which is packaged a bit differently and isn't quite as featureful.

I'm in Germany and our 2/3G cell networks here are shutting down in 2021, and that leaves only the 4G option.

So the alarm works on 4G frequencies 850/900/1800/1900Mhz, 2.4 and 5Ghz wifi, and uses 433mhz wifi for accessories (door and window sensors, and PIR motion detection)

I was disappointed to note when I tried to use this with a partner SIM from my telekom provider that it didn't achieve a network connectivity. (wifi, sure, but no cellular)

After some debugging, it occurred to me that maybe the PIN entry on the SIM card wasn't working. The device prompts to "dial" when inserting a SIM after booting, and appears to accept the input but it didn't work.

I removed the SIM PIN using another device (tricky, the alarm wants a full-size sim, and all my spare phones want micro- or nano-SIM), I have a laptop which takes a full-size SIM card, and on Windows it's quite easy to remove a SIM PIN in the cellular "Advanced options" settings dialog.

Now, within moments the Digoo alarm has a really strong signal, and I can finally confirm it all works as hoped.

Interesting, too - it's using a separate modem. At some point I tore it apart to check, I really expected to find a rebranded low-end mobile phone SOC, and I suppose maybe this is, but it's right next to an ARM chip I couldn't adequately identify which has the right format to be a little, probably fairly capable SOC, indeed a GigaDevice ARM32-M3 at least, though I didn't look more deeply into the possible specs.

For under 80 Euro, this alarm is crazy cheap, with all the accessories it has, I'm really pleased with the purchase, now that I got it working.

Aggregate Roots in action

Lee Hambley — Tue, 26 Jan 2021 16:23:07 +0000

In Domain Driven Design there is an entire taxonomy of terms for various kinds of objects, unfortunately it can be difficult to really apply DDD in the day to day, but recently an opportunity surfaced in a project I was working on, and I wanted to write it up.

The classic example of "Aggregate" and "Aggregate Root" is something like the line items on an invoice, and the invoice itself.

From a programming point of view, it doesn't make sense to implement changeQuantity or delete on a lineItem directly, if you do this, the invoice or shopping basket loses track of the items inside, there's no obvious place to recalculate taxes, check stock, etc.

Other examples might be that edits to forum comments should be applied to the forum thread individually, and not simply as a direct edit on the individual post. Notification rules, minimum waiting time between edits, anti-spam, and other considerations must be accounted for when planning the data flow.

From a DDD perspective, this isn't always obvious, stakeholders may talk about "changing the number of invoice items" (a hint that invoice and item are hierarchically owned), but in other cases such as the forum it's less clear, I certainly mentally model changing my comment in a forum as an action that instinctively feels like the comment is the recipient.

In contemporary web app frameworks, typically MVC and/or RDBMs it is really, really easy to accidentally to make your "nested" aggregates (line item, and forum posts/comments) public, the scaffolding generators will encourage you to model the CRUD verbs on the aggregate entities themselves, and of course the RDBMS is going to give everything a nice convenient handle in the form of an auto-incrementing primary key.

Defaults matter, and the prolific use of RAD MVC frameworks backed by classical relational databases sets a lot of teams up poorly for doing good domain design.

Enough pontificating, let's look at this example, something from the real world.

The application models logistics for fast food, groceries and retail. Drivers move around and complete Tasks at Waypoints, each task comprises one or more Steps

From a Driver's point of view at any given Waypoint they has a series of Tasks to complete, each of which has a linear flow of Steps to sign-off.

Although virtually all of the application is backed by MySQL, and all of these entities have primary key IDs, we deliberately kept Task.id and Step.id out of the public API in the latest redesign.

Client teams (internal) have questioned this decision, because they have been accustomed historically to being able to completeAgeCheckStep(stepId), completeSignaturePhotoStep(stepId, photoData), but I wanted to lay out some of the reasons why:

Steps, on Tasks must be completed in order. It is nonsense to try and complete step 2 on the 4th task in the Waypoint, so any public API that lets you address changes to tasks other than in the next/first position are misleading consumers.
The Waypoint contains a [] of Task, so again here, the Task must be completed in order, so there's no need to expose a unique identifier for those either. Knowing that a task is the nᵗʰ on a Waypoint is more than adequate for addressing it uniquely.
Completing a Step, or a Task changes the state of that Waypoint (e.g from in_progress to complete, when they're all done). To prevent having to signal up- and down- the hierarchy, the only way to change a Waypoint state is to make sure the Waypoint is the access point (Aggregate Root) for the things it contains.
In general this design is cleaner, because we reduce the surface area of the API from five or six verbs over three nouns, to a couple of different verbs over one noun. The complexity is not eliminated, just morphed, but into a more manageable one, we believe.
Leaking integer IDs in general is poor form, it exposes private implementation details, and is a (weak) proxy for the volume of business your platform is doing. It also sets you up for enumeration attacks in case someone is able to find a weakness in any part of your security.

Probably this isn't a slam-dunk argument and we expect the continue defending this decision in our API design team, as things like this are distinctly unusual in API design, we have become so accustomed to CRUD verbs over HTTP/REST that discussing domain-oriented RPC over HTTP type APIs feels like an up-hill battle.

Make no mistake though, such decisions, and battling for simpler APIs can pay huge dividends over time.

File-system performance with Docker for mac.

Lee Hambley — Mon, 12 Oct 2020 07:32:47 +0000

File-systems are tricky in general, that is a statement we need to make up-front and unambiguously.

POSIX file-systems (most of the ones you know and love) even moreso, as POSIX specifies some guarantees about what it means to take certain operations, and those requirements are fairly rigid.

In the contemporary age of computing, there's a handful of file-systems, and the most common features have more-or-less settled-in, and there's not nearly as much innovation as there used to be, and for most of us, most of the time, that's just fine. Solid State Drives (SSDs) cover all our our common use-cases, and by virtue of being absolutely screaming fast, file-systems in general were propelled a decade into the future "by magic" because of the underlying technology.

This post is a quick attempt to set-up some thoughts about file-systems, and the performance characteristics we expect, specifically looking at shared file-systems for development environments, Docker to be precise.

Docker on Linux is a very thin wrapper over CGroups and Kernel Namespaces, the performance is so close to the native performance of the underlying file-system, and the underlying hardware that there's barely anything to discuss. In that scenario the file-system stack looks something like this:

The story doesn't change much if using Docker shared volumes mapped between the container and the host because the partitioning of the filesystem that is visible to the container is all magic of kernel namespacing and clever use of "overlay" filesystems, which are a way to "stack" multiple, possible sparse directories on top of one another to give a union view of all the parts of the stack.

When you do a FROM someimage:latest, and then you mount, or copy in files into the container, you're creating a two or three layer deep stack, and Docker takes care of making sure all your pancake layers are there when you need them.

This, however isn't a story about Linux and Docker file sharing, this is about the situation on Mac, and possibly Windows (although I really have no idea about Windows).

Because macOS isn't Linux, they don't have the cgroups and namespace features which are what make Docker work, that makes it impossible to run Docker natively on any platform that is not Linux, to get around this limitation, Docker simply manages an "invisible" Linux virtual machine.

In theory there's no reason why this should prefer one hypervisor or another. (the tech that virtualizes hardware, examples would be VMWare Fusion, Parallels, VirtualBox, etc)

On macOS, it's xhyve, nothing spectacular or special about xhyve, as far as I'm aware, except it's bundled basically as part of macOS by Apple as Hypervisor.framework, and xhyve just uses that.

The important thing is though, that we no longer share any resources between the host and the VM. If we want to share a directory now, we need to share between macOS and the virtual machine, and from the virtual machine down to the Docker container.

Worse, because the invisible virtual machine (Linux), and the host use different file-systems (say EXT4 in the VM, and HFS+ in the host), not all concepts are portable. HFS+ for example is case insensitive (optionally), and the way file-system events (notifications about changes to files and directories) don't work quite the same, and probably other aspects I'm forgetting, such as whether or not files can be atomically replaced, or deleted whilst still being open, or whatever else on the long-tale of occasionally important behaviour.

Here's how our stack from before looks with two extra levels of VM, and file-sharing:

In the realms of shared file-systems, the bridge between our Docker VM (and the containers it runs) and the host system file-system must be some kind of network, or demi-network based file-system.

Common known solutions are NFS and CIFS (Samba), however Docker provides their own osxfs (closed source), and in their edge channel provide one by the name of gRPC-Fuse (also closed-source).

Common file-system benchmarks (including the extensive NFS benchmarking and tuning documentation) focus a great deal on optimizing for reads and writes, which historically makes sense, magnetic storage wasn't renowned for speed, so (the software part of) file-systems ran the risk of being a significant bottle-neck.

Indeed, for file browsing tools (Windows' Explorer, macOS' Finder, Gnome's Nautilus, etc) even some of those benchmarks can make sense, it's common to open and read from files to search for icons or render previews, etc. The actual read/write performance of a filesystem is a significant driver.

In the context then, again of Docker (for Mac), I did some profiling of the file-system performance but not a the level of raw read/write performance, but indeed a level lower.

Before actually reading a file, it must be opened, before even opening one can check for the existence with calls to specific system APIs which return meta-data.

The family of file-system related "system calls" (syscalls), runs a breadth of ~15 things, open, read, write, stat, lstat, access, and close being the most frequently called ones. (rename, symlink, etc being also common).

Syscalls represent a trivially small amount of the percentage of time spent reading a file, usually. An open may take 7ns
where a read may take somewhere in the order of 20ns to read 1024 bytes.

Take a look at this, after generating a 10MB file of zeros, we can ask cat to print it, and trace the underlying system-calls, note this only works on Linux so I'm doing it in Docker, on my Linux machine, but in an Alpine container with no shared file-system.

cat is using open and close taking 0.000061 and 0.000011 seconds respectively and the sendfile syscall which copies between the file and my terminal without having to do reads of the file, this actually makes cat about twice as fast as head for printing a file:

/ # strace -c -f cat 10meg
% time     seconds  usecs/call     calls    errors syscall
------ ----------- ----------- --------- --------- ----------------
 99.99    0.048492       24246         2           sendfile
  0.01    0.000004           4         1           close
  0.00    0.000000           0         1           open
  0.00    0.000000           0         2           mprotect
  0.00    0.000000           0         1           execve
  0.00    0.000000           0         1           getuid
  0.00    0.000000           0         1           arch_prctl
  0.00    0.000000           0         1           set_tid_address
------ ----------- ----------- --------- --------- ----------------
100.00    0.048496                    10           total
/ # strace -c -f head 10meg
% time     seconds  usecs/call     calls    errors syscall
------ ----------- ----------- --------- --------- ----------------
 64.48    0.051081           5      9757           writev
 35.29    0.027957           2      9767           readv
  0.07    0.000053          26         2           mprotect
  0.05    0.000036          36         1           open
  0.04    0.000028          28         1           execve
  0.02    0.000017          17         1           arch_prctl
  0.02    0.000016          16         1           ioctl
  0.02    0.000014          14         1           getuid
  0.02    0.000014          14         1           set_tid_address
  0.00    0.000003           3         1           close
------ ----------- ----------- --------- --------- ----------------
100.00    0.079219                 19533           total

OK, at the risk of having gotten lost then, let's say that programming languages, development tooling and the kinds of things we usually run in Docker have quite specific requirements.

The pyramid architecture

Lee Hambley — Wed, 05 Aug 2020 09:59:49 +0000

After more than a decade in the software industry, there are few conversations that never seem to get resolved, monolith vs. micro-services, or vi vs. emacs.

The micro vs. macro services thing is something I want to try to offer a different perspective on. I've noticed recently that the teams I work with have a hard time to estimate work, and set expectations with stakeholders, and I think that maybe we're missing a tool to help us frame the conversations.

Before getting into the details, I want to make an honorable mention for the OSI model, it defines a pyramid of layers that were instrumental in helping the build-out of the early internet and subsequently the world wide web.

The layers are:

1: Physical Link: Literally physics, electro-magnetic or mechanical transmission of signals.
2: Data Link: Transferring data point-to-point using Layer 1.
3: Network Link: A network of machines, packets, retransmission, etc lives here.
4: Transport Layer: Variable length, and bi-directional communication, etc on top of the packet/frames abstraction.
5: Session Layer: The concept of presence, correlating transport layer activity into coherent streams, or sessions.
6: Presentation Layer: The protocol layer, e.g MySQL connection activity look different to TLS sockets.
7: Application Layer: Applications don't need to know anything about how data gets from A/B, they get the socket read()/write() metaphor, and in theory, it "just works".

🛈 If you've ever heard terms such as "Layer 5/7 Load Balancer", then this is people referring to the OSI model, such a load balancer would be aware of application or session layers, "sticky" load balancers, for example that route all a user's HTTP requests to the same back-end server have to have a concept of sessions, and not care only about the network, or transport link layers.

This isn't an essay about the OSI model, but the key message I want to transmit is that this abstraction is so clean, that it "just works", most of us (humans for sure, but also engineers) can ignore the existence of this concept, and when we work with it, we intuitively gravitate to the appropriate level of abstraction.

The top of this pyramid (sometimes also shown as a stack) just says "application", but of course, inside application we have architectural concerns too. Maybe the logical "application" (e-commerce website, logistics platform?) comprises a database, and one or more services, some APIs, etc.

You see hints of this kinds of split in the regular team arrangements in most companies, we have developers who build software, operations people who run the infrastructure; even within the software developers, you can easily separate into "back-end", "front-end" and "mobile", easily different aspects of an extended stack.

OSI would group these all into the top one or two layers, so we leave our venerable friend behind, and invent a concept for ourself.

We can adopt a similar model in within the "Application Layer", and it can help us figure out where to "put" work, whether a feature we're building in a cross functional team, is actually risking their timeline by crossing "layers".

Looking at my own team, I came up with this, which I think fits, but you will need to discover what is right for your organization.

We can use this to answer some questions:

How should teams be organized? Is it reasonable for us to have a "cross-functional" team ship features across the top three layers?
Do lower-layers get handled by "service teams"? Feature teams campaign for the service/platform teams to provide infrastructure

We can also use this to avoid some pitfalls:

Cross-functional feature teams shouldn't plan to build a feature relying on new infrastructure until it's in place. You wouldn't build house on shaky foundations, the same applies here.

There's some risks involved with model, too. If you don't have a good mutually beneficial relationship between platform teams and feature teams, the feature teams may avoid building useful features, because the infrastructure just isn't there, or worse yet, those feature teams start introducing their own "infrastructure" that maybe isn't as aligned with the company philosophy or standards as it could be.

Hopefully too, this helps put to bed some of the arguments about microservices vs. monoliths, as long as your stack is cleanly arranged, you can figure out whether introducing a division in the "Application layer" unduly affects the layers on top, or whether it can be a smooth integration.

The biggest red-flag, should be when your architecture discussions start to devolve into this:

Please, share this around, get in touch, and let me know your thoughts on whether or not this bit of writing helped you, or the people you work with/