DEV Community: David Cantrell

Automatic cross-platform testing: part 7: 32 bit, again

David Cantrell — Fri, 27 Feb 2026 21:23:41 +0000

I have written about this before, but I've had to do it again. I noted previously that Github's actions for things like checking out a repository and for retrieving build artifacts are now 64-bit only. You've been able to work around that by using older versions of the actions but I didn't want to do that so I'm taking a different approach.

My new method builds on work I've done previously to pull all my testing for different Unixy platforms together into one workflow. This tests on 64-bit versions of NetBSD, FreeBSD, OpenBSD, IllumOS (that is, the modern version of Open Solaris), and Linux. And as of earlier today it also runs the tests on 32-bit Linux. As in my previous blog entry it's still really 64-bit hardware, with modern x86 ISA extensions, but it's a 32-bit OS image with 32-bit addressing, userland, compiler toolchain and so on. If you absolutely must have a pure 32-bit x86 environment then you'll need to acquire some obsolete e-waste and run it yourself. Something like this based on the Vortex86 chipset looks like a good choice.

There are only two interesting bits of the workflow. The first implements enough of Github's actions/download-artifact for my purposes. In the YAML file it's all one line because YAML and everything that uses it is hateful, I've tidied it up here:

curl -L -H "X-GitHub-Api-Version: 2022-11-28"
        -H "Authorization: Bearer ${GH_TOKEN}"
        -o dist-for-install.zip
  $(curl -L -s
         -H "Accept: application/vnd.github+json"
         -H "Authorization: Bearer ${GH_TOKEN}"
         -H "X-GitHub-Api-Version: 2022-11-28"
         https://api.github.com/repos/${GH_REPOSITORY}/actions/artifacts
         -o -
    |jq -r '
         [.artifacts.[]|select(.expired == false)]
         |max_by(.created_at).archive_download_url
    '
  )

The inner curl uses the Github API to fetch a list of all the build artifacts that have ever been created for this repository, as JSON, which it filters using jq to find the most recent one which hasn't expired, and then extract its archive_download_url field. The outer curl then fetches that.

The second bit interesting bit is specific to testing my perl code. Perl can be built on 32-bit platforms to support either 32 or 64 bit integers. On 64-bit machines perl integers are always 64-bit - and there's been a note in the docs for 20 years about how a build-time parameter for 32-bit ints may be added later! I need to test with 32 bit ints, but most OS packages, even on 32-bit machines, will provide a perl with 64-bit ints. So I need to build my own. This is dead simple if a little wordy:

curl -o perl-5.42.0-src.tar.gz https://cpan.metacpan.org/authors/id/B/BO/BOOK/perl-5.42.0.tar.gz
tar xzf perl-5.42.0-src.tar.gz
cd perl-5.42.0
sh Configure -de -Dprefix=$HOME/perl-5.42.0-installed
make -j $(nproc)
make install
$HOME/perl-5.42.0-installed/bin/perl -MCPAN -e 'install qw(App::cpanminus)'

The call to the Configure script is where the OS packaging people make it have 64-bit ints by adding the -Duse64bitint flag. Normally you would make test too but I've satisfied myself that it does indeed pass its tests using this Docker image, so I skip that step to save a lot of time.

Finally, I can extract my software and test it. It is bloody irritating that Github automatically wrap my tarball in the obsolete zip format. I blame Microsoft:

unzip dist-for-install.zip
mkdir dist-for-test
tar -C dist-for-test -xzf *.tar.gz
cd dist-for-test/*
$HOME/perl-5.42.0-installed/bin/cpanm --installdeps .
$HOME/perl-5.42.0-installed/bin/perl Makefile.PL
make test TEST_VERBOSE=1

And there you have it - how to test your code automatically, using Github actions, on a 32-bit platform.

Unicode-Search - my first Electron app!

David Cantrell — Sat, 23 Nov 2024 11:35:46 +0000

I am a Unix greybeard who mostly works in perl and shell programming, with a bit of dabbling in C and Rust, but for a long time I've wanted to do some Javascript. I just needed an interesting enough project to use it in. Well, at work I spend a lot of time toiling in the Unicode mines, fixing mojibÃ©ke errors in a somewhat elderly code base. As part of that I wrote a guide for my colleagues, which points them at a very useful reference website run by Xah Lee. That has been around for years, but as the single maintainer of several projects myself I am wary of relying too much on projects with a single maintainer. So I decided to write my own, and that this would be my interesting little Javascript project.

It seems like a sensible choice for learning a new language, as it's not doing much, just looking up data in a static structure and displaying it to the user. It's all synchronous code, there's no I/O beyond updating a web page, and the user interface can be dead simple.

Electron's quick start guide is excellent and gave me the basic boilerplate for a "Hello World" application, and then the rest was just lots (and lots, and lots) of looking things up on Stack Overflow and W3Schools. From start to finish it took about 6 hours, and my code is on Github.

I'm quite sure that those of you skilled in the ways of Javascript will find the code "idiosyncratic", but that's OK, these are baby's first steps. If you have any useful tips for improving it, and can explain them simply, then they would be most welcome. I would also welcome pull requests that make it stop looking ugly as sin. Learning how to make rounded corners and stuff in CSS wasn't in scope :-)

Update: because working on the command line is always better than using a GUI, it has sprouted a CLI tentacle. The same code does most of the work of parsing user input and looking up characters, it just has a different function for spitting the results back out to the user.

Programmable tab completion with bash

David Cantrell — Sun, 10 Nov 2024 23:27:15 +0000

We all use tab completion in the shell, but have you stopped to think about how it works? How with one command it will auto-complete only directory names, with another directories and filenames, but with another it will complete branch names for your VCS, for example? It's programmable! And so we can bend it to our will!

I recently had an annoyance. I use a tool called ts (task spooler) to run some CPU-intensive processes in the background, and to queue them so that only one such process runs at once. It is invoked like this:

$ ts command arg1 arg2 arg...

I wanted to be able to use tab completion for the command, and to then use that command's tab completion for subsequent arguments. After much swearing and cursing - the documentation in the bash manpage is not the best - and a bit of help from some nice people on Stack Overflow I came up with this:

complete -o bashdefault -o default -F __ts_bash_completions ts

__ts_bash_completions () {
    COMPREPLY=()

    if [ "$COMP_CWORD" -eq 1 ]; then
        COMPREPLY=($(compgen -c -- "${COMP_WORDS[COMP_CWORD]}"))
    else
        local command_completion_function="$(complete -p ${COMP_WORDS[1]} 2>/dev/null|sed 's/.*-F \([^ ]*\) .*/\1/')"

        if [ ! -z "$command_completion_function" ]; then
            COMP_CWORD=$(( COMP_CWORD - 1 ))
            COMP_LINE=$(echo $COMP_LINE|sed "s/^${COMP_WORDS[0]} //")
            COMP_WORDS=( "${COMP_WORDS[@]:1}" )

            $command_completion_function "${COMP_WORDS[0]}" "$2" "$3"
        fi
    fi
}

Let's go through it in detail. The first line tells the shell to use the function __ts_bash_completions when the user is typing the ts command and its subsequent arguments:

complete -F __ts_bash_completions ts

We then define that function.

__ts_bash_completions () {
    COMPREPLY=()

and the first thing we do is create an empty array COMPREPLY. bash completions populate this global variable to tell the shell what options are available. We then see how many complete words there are on the command line:

    if [ "$COMP_CWORD" -eq 1 ]; then

COMP_CWORD is another global variable that contains the number of complete words currently in the command. If that is 1 then the only word currently in the command is ts itself, we want to autocomplete the name of a command:

        COMPREPLY=($(compgen -c -- "${COMP_WORDS[COMP_CWORD]}"))

compgen (completion generator) generates a list of all the commands available in the $PATH which begin with ${COMP_WORDS[COMP_CWORD]}. This introduces yet another magic global variable, COMP_WORDS is an array (zero indexed) of all the words currently on the command line, including the one currently being typed, which may be empty. We pick the last one and pass that to compgen for it to use as a filter.

At this point we've tab completed the name of the command that ts is to run and deserve a beer.

Now we need to deal with tab completion of the arguments to the command that we want ts to run. First we need to find what function is used for completions for that command:

    else
        local command_completion_function="$(complete -p ${COMP_WORDS[1]} 2>/dev/null|sed 's/.*-F \([^ ]*\) .*/\1/')"

complete -p tells you exactly how tab completion is configured for a given command, and spits it out in the form of the command to use to configure it. The function will be the argument to the -F option, so we use a rather crude sed invocation to extract it, if it is present. The resulting string will be empty if no completion function is defined. We then check that it isn't empty and play around with the contents of the various COMP_* variables, setting them up to pretend that ts isn't involved:

        if [ ! -z "$command_completion_function" ]; then
            COMP_CWORD=$(( COMP_CWORD - 1 ))
            COMP_LINE=$(echo $COMP_LINE|sed "s/^${COMP_WORDS[0]} //")
            COMP_WORDS=( "${COMP_WORDS[@]:1}" )

We decrement COMP_CWORD because we want to ignore one of the completed words - that being ts itself. We remove ts from the start of COMP_LINE - which is a string containing the entire line of input. And we remove the first element from the COMP_WORDS array. Finally, we run the command's own completion function which will set COMPREPLY for us:

            $command_completion_function "${COMP_WORDS[0]}" "$2" "$3"

It appears that the calling convention for completion functions has changed over time. At some point in the past they were called with arguments, which only contain the command name and the last two arguments on the command line, then they were changed to instead use those global variables. In this case we can just use the arguments that were passed to our function altering only the first one, the name of the command whose arguments we are completing. Depending on how modern the completion function is for a given command, our ts completion needs to support both calling conventions.

There's one final wrinkle, which the observant amongst you may have noticed when I started going through this line by line. In my script I have -o bashdefault -o default when defining the relationship between ts and its completion function. Those are to cope with the case where the completion function leaves COMPREPLY empty. If that happens then -o bashdefault applies bash's own defaults for tab completion, ie helps you pick a filename, and if that doesn't return anything -o default applies readline's defaults. You can see this in action for git's completions. If you type git log <tab> in a git repo you get a list of branch names, and if you type git log foo<tab> you get a list of branches whose names begin with foo. But if there are no such branches you get a list of files whose names begin with foo, which is the shell's default action.

Anyway, after a great deal of wrestling with poor documentation, that's a little annoyance dealt with, and exactly the same code can also be used for other similar commands such as sudo and nohup. If you think you'll find it useful then the code is on Github and it's also in the ts mercurial repo so will no doubt be in a future release.

The typeface you didn't know you wanted and were trained to hate

David Cantrell — Tue, 05 Sep 2023 14:59:32 +0000

For the last several weeks I've been using Comic Mono in my terminal. It's a fixed width typeface based on the font that we've all been trained to despise and sneer at for almost 30 years, Comic Sans.

But you know what? Comic Mono works really well at small sizes even with my becoming-elderly failing eyes and my inability to always find my reading glasses. I can see plenty of code in my editor with lots of context. On a 40" screen at arm's length I can comfortably get 100 lines of text - so lots of context for the code I'm working on - in a terminal using 12pt text and read it easily.

Give it a go for a few days!

Number::Phone is now 64-bit only

David Cantrell — Tue, 20 Jun 2023 10:40:02 +0000

A couple of years ago I wrote that I would be dropping support for perls with 32-bit integers in Number::Phone. Well, that two year deprecation cycle is up, and last night I switched over to the new database format, the software for which requires 64-bit integers. As of the next release it will use under 10MB of disk space instead of about 100. An order of magnitude improvement, and it was fun to write as well.

Perl has supported 64-bit integers on all reasonable platforms for the last 20 years, even on those platforms which still use 32-bit pointers, so I don't expect that anyone will notice any change at all except that their hosted Docker containers will now be faster to deploy and cheaper to run.

cpulimit annoyed me so I improved it

David Cantrell — Sat, 25 Mar 2023 19:07:07 +0000

Previously | more previously

A few days ago I discovered cpulimit. It's a great tool that nicely (haha) complements nice. Where nice is normally used to reduce the amount of CPU a process uses by changing it priority, a niced process can still end up using more CPU than you want, and will of course use all that it wants if nothing with a higher priority comes along.

But sometimes you want to restrict a process to using no more than some particular fraction of CPU time regardless of priority. A good example is when you don't want those noisy PC fans to kick in and you don't care how long a job takes because whether it finishes in 15 minutes or 8 hours it's still going to finish while you're asleep.

cpulimit is perfect for this, and is simple to use. An invocation like this:

cpulimit -l 50 somecommand ...

will run somecommand but restrict it to only use 50% of a CPU. It does this by forking a watchdog process that periodically checks to see how much work somecommand is doing, and if it's used too much briefly pauses it by sending the STOP signal. After a little while it will un-pause it with the CONT signal. Of course, because the load on your machine from other processes will never quite be constant, cpulimit rarely hits the target exactly, but it gets close enough.

But I wanted to tweak it a bit. I wanted to be able to interactively "turn the volume knob" so that I could give somecommand more or less CPU whenever I fancied. The result is a pull request which unfortunately is unlikely to ever get merged, as the original author hasn't touched the project in years, but if any of you want the nifty new feature applying the patch and building your own custom cpulimit is pretty easy.

How the patch works is simple. It installs signal handlers for SIGUSR1 and SIGUSR2 which respectively increase and decrease the CPU allocation by 1%. Want to turn it up by 50%? Just write a little shell loop to send the signal 50 times:

for i in $(seq 1 50); do kill -SIGUSR1 $pid; done

Determining which process to send the signal to is a bit tricky, as there are two cpulimit processes running. There's the first one, which is just waiting in the background for somecommand to finish, then there's the watchdog that got forked off. It's the watchdog you want to send the signals to. You can tell which is the watchdog as it will generally have a higher PID and be using a little bit of CPU. If you are cpulimiting multiple processes then you can tell which watchdog is related to which process because the watchdog will have the command and its arguments on its command line. For example:

$ ps aux|grep ffmpeg|grep -v grep
david   90311 103.3  0.7 36105472 485448 s011  T     6:42pm   1:07.89 ffmpeg ...
david   90312   5.6  0.0 34221044    828 s011  S     6:42pm   0:02.82 cpulimit -l 100 ffmpeg ...
david   90310   0.0  0.0 34122740    796 s011  S     6:42pm   0:00.01 cpulimit -l 100 ffmpeg ...

You can see here that I asked cpulimit to allow ffmpeg to only use 1 CPU of the several available on this machine (ie to use 100% of a CPU - on modern machines the maximum allowed is the number of CPU cores * 100%). My shell accordingly started process 90310 which forked and execed ffmpeg with pid 90311 and forked the watchdog process as pid 90312. The watchdog is using a little bit of CPU. It is therefore to process 90312 that I should send signals.

$ for i in $(seq 1 400); do kill -SIGUSR1 90312; done

That will send the "turn it up a bit" signal 400 times, so ffmpeg is now limited to at most 500% of a CPU, and a few moments later:

$ ps aux|grep ffmpeg|grep -v grep
david   90311 497.0  0.8 36105472 507160 s011  T     6:42pm  12:42.37 ffmpeg ...
...

we can see that ffmpeg is running a lot harder, now taking just under 500% of the CPU.

TIL: diff-so-fancy; and some funky git config

David Cantrell — Wed, 25 Jan 2023 19:12:13 +0000

I just discovered diff-so-fancy, and very nice it is too. I immediately added it to my standard git config, which is semi-automatically installed on every machine I use. However, I've not (yet) installed diff-so-fancy on all the machines I use, and for those platforms for which it's not packaged I probably won't bother installing it from source.

But if I just follow the author's instructions which amount to adding this to my ~/.gitconfig:

[core]
    pager = "diff-so-fancy | less --tabs=4 -RFX"

then git diff will break:

$ git diff HEAD^
diff-so-fancy | less --tabs=4 -RFX: diff-so-fancy: command not found

but there's an easy fix! Whatever is in pager is just shell code, so this works:

[core]
    pager = "if [ ! -z \"$(which diff-so-fancy)\" ]; then diff-so-fancy | less --tabs=4 -RFX; else less; fi"

The output from git diff is piped into that little script. If diff-so-fancy is installed (ie if "$(which diff-so-fancy)" is not zero-length) then it does exactly what diff-so-fancy's author suggests. Otherwise, if diff-so-fancy isn't installed, just run less.

A Rant

David Cantrell — Fri, 20 Jan 2023 17:54:56 +0000

How the hell, given these interests:

is this post in any way relevant and deserving of being promoted to me on the front page of this site?

And no, negative weights being treated as "anti-follows" isn't enough. There are so many topics I'm not interested in that configuring them all is just too much work.

Number::Phone release candidate

David Cantrell — Thu, 12 Jan 2023 21:52:39 +0000

I've recently tackled a feature request that's been sitting in the backlog for several years, to use libphonenumber's data to validate numbers with non-geographic country codes. That's codes like +800 (international freephone), +870 (Inmarsat) and so on.

Previous versions would sort of work with these numbers, in that they could be instantiated as objects, but there was no information on validity or any other properties.

For example here it's looking at an Iridium number. First with the previous release:

Number::Phone->new("+881 672520333333333333333732")->format()

+8816 72520333333333333333732

Number::Phone->new("+881 672520732")->is_mobile()

undef (ie, dunno)

and with the release candidate:

Number::Phone->new("+881 672520333333333333333732")

undef (ie, not valid)

Number::Phone->new("+881 672520732")->format()

+881 6 725 20732

Number::Phone->new("+881 672520732")->is_mobile()

1

Much of the code is auto-generated, and this involved quite a bit of hacking around in the build system and object instantiation, so there's the potential for exciting new bugs even for numbers in boring old geographic country codes, although I didn't have to change any of the existing tests so I am hopeful that I avoided that. Here are the changes.

I'd be grateful if you could test this, and assuming that no-one finds any bugs it will be in either the March or June quarterly release. You can download it from here.

Automatic cross-platform testing: part 6: 32 bit Linux

David Cantrell — Sun, 17 Jul 2022 17:25:26 +0000

Previously on this channel

Testing on 32 bit Linux

Just about every virtual machine out there in the various CI platforms and hosting companies is a 64 bit x86 machine. But I wanted to test my code on a 32 bit machine as well. Unfortunately without having real 32 bit hardware available, or emulating it on more modern hardware, you can't quite do this, but you can at least run an OS and use libraries compiled for 32 bit hardware, 32 bit memory management, and with 32 bit pointers.

It's surprisingly easy, and lot less hackish than the solution I found for testing on OpenBSD.

Github workflows normally run directly on whatever host type you specify in the runs-on option. But you can instead tell them to run in a container, which can be anything available on Docker Hub. Jobs running in a container can have all the same steps that you would have in a non-containerized job, including using other Github workflow actions.

To test my code on 32-bit Linux I use the i386/ubuntu:latest image. I am also using the same technique to run tests for one of my projects, which is sensitive to exactly which Linux distribution you are using, on Arch Linux, using the archlinux:latest image.

Virtual machine efficiency

David Cantrell — Wed, 23 Feb 2022 21:27:22 +0000

I'm one of the CPAN testers, a bunch of people who test everything that gets uploaded to the CPAN. We test with multiple different builds of perl on multiple operating systems and different hardware platforms, and send reports of failing tests. When I'm wearing my Author Hat as opposed to my Tester Hat I very much appreciate test failure reports from platforms I don't have access to, or which are generated using perl builds with features I hadn't considered. I test on 5 different operating systems (6 if you count 32 bit x86 and 64 bit amd64 Linux separately) and 40 different builds of perl.

I mostly test in virtual machines, running on a single Intel Mac Mini. Using VMs means I don't need multiple noisy real machines of course, but it also makes them all easier to manage.

Until very recently I used Virtualbox for this, mostly because back when I started running VMs on a Mac it was the best free game in town. But it's not the most efficient and I have now switched to the much better, and still free, UTM, which is a simple GUI wrapper around the notoriously hard to configure QEMU, patched to use Apple's hypervisor.

This chart shows CPU usage over the last month. The first half is when I was using Virtualbox. Running all those test builds usage was a fairly constant 50% of what was available. The VMs could keep up with the workload, but there was little capacity for more despite all that unused CPU. No matter how many CPUs I allocated to the various VMs, it just wouldn't use all the resources. It's very clear in the chart when I started migrating VMs from Virtualbox to UTM. CPU usage starts spiking much higher - indeed, it reaches 100% and stays there when the VMs test a batch of uploads - but then drops to almost zero when that batch has finished. The work is done faster.

Virtualbox is also a memory hog. This chart shows that I couldn't really add any more Virtualbox VMs, as there was little memory left. And worse, most of what was in use (the blue section of the chart) is "wired" memory, in kernel-space. Yes, most of the memory Virtualbox was using was being used by its kernel extension. It is again clear when I started migrating VMs to UTM. Memory usage falls off a cliff, and "wired" memory usage falls even faster, with a far greater proportion of the lower total being just normal user-space memory (the red section of the chart).

Now that I've got back all that wasted memory I am, of course, beginning to fill it with more VMs, this time for development as opposed to testing. I have recently become a larval stage Rustacean and have dived headlong into reporting obscure platform-specific bugs.

But you can see from my dock that I've still got three Virtualbox VMs running. Why?

While UTM is much better at resource usage, it's not perfect. Virtualbox seems to emulate more of a system instead of passing it through to the hypervisor and so is better for running more unusual OSes. I have two VMs there running Illumos which I have yet to figure out how to boot in UTM. A problem that I think is related to UTM's greater use of the hypervisor is that you can't suspend and resume VMs that use it instead of being emulated, so I've still got a few VMs hanging around in Virtualbox which spend most of their time suspended. Finally, what stops me from using UTM at work is that you can't use it as a Vagrant provider. This is incredibly annoying, as the lack of a decent virtualization application makes the otherwise very nice M1 Macs nothing more than pretty toys. I expect that this glaring lack will be fixed within the next couple of years.

A brief guide to perl character encoding

David Cantrell — Mon, 31 Jan 2022 18:43:59 +0000

Credits

I originally wrote this at work, after my team spent far too many days yelling at the computer because of Mojibake. Thanks to my employer for allowing me to publish it, and the several colleagues who provided helpful feedback. Any errors are, naturally, not their fault.

12:45. Restate my assumptions
The Royal Road
- Characters, representations, and strings
- Source code encoding, the utf8 pragma, and why you shouldn’t use it
- Input and output
  - PerlIO layers
The Encode module
- Encode::encode
- Encode::decode
- Encode:: everything else
Debugging
- The UTF8 flag
- Devel::Peek
- hexdump
- PerlIO::get_layers
The many ways of writing a character
- String literals
- The chr function
- Octal
- Hexadecimal
- By codepoint name
- Other hexadecimal
- In regular expressions
- ASCII-encoded JSON strings in your code
- Accented character vs character + combining accent
Odds and ends

12:45. Re-state my assumptions

We will normally want to read and write UTF-8 encoded data. Therefore you should make sure that your terminal can handle it. While we will occasionally have to deal with other encodings, and will often want to look at the byte sequences that we are reading and writing and not just the characters they represent, your life will still be much easier if you have a UTF-8 capable terminal. You can test your terminal thus:

$ perl -E 'binmode(STDOUT, ":encoding(UTF-8)"); say "\N{GREEK SMALL LETTER LAMDA}"'

That should print λ, a letter that looks a bit like a lower-case y mirrored through the horizontal axis.

And if you pipe the output from that into hexdump -C you should see the byte sequence 0xce 0xbb 0x0a.

The Royal Road

Ideally, your code will only have to care about any of this at the edges - that is, where data enters and leaves the application. That could be when reading or writing a file, sending/receiving data across the network, making system calls, or talking to a database. And in many of these cases - especially talking to a database - you will be using a library which already handles everything for you. In a brand new code-base which doesn’t have to deal with any legacy baggage you should, in theory, only have to read this first section of this document.

Alas, most real programming is a habitation of devils, who will beset you from all around and make you have to care about the rest of it.

Characters, representations, and strings

Perl can work with strings containing any character in Unicode. Characters are written in source code either as a literal character such as "m" or in several other ways. These are all equivalent:

"m"
chr(0x6d) # or chr(109), of course
"\x{6d}"
"\N{U+6d}"
"\N{LATIN SMALL LETTER M}"

As are these:

chr(0x3bb)
"\x{3bb}"
"\N{U+3bb}"
"\N{GREEK SMALL LETTER LAMDA}"

Non-ASCII characters can also appear as literals in your code, for example "λ", but this is not recommended - see the discussion of the utf8 pragma below. You can also use octal - "\154" - but this too is not recommended as hexadecimal encodings are marginally more familiar and easier to read.

Internally, characters have a representation, a sequence of bytes that is unique for a particular combination of character and encoding. Most modern languages default to using UTF-8 for that representation, but perl is old enough to pre-date UTF-8 - and indeed to pre-date any concern for most character sets. For backward-compatibility reasons, and for compatibility with the many C libraries for which perl bindings exist, it was decided when perl sprouted its Unicode tentacle that the default representation should be ISO-Latin-1. This is a single-byte character set that covers most characters used in most modern Western European languages, and is a strict superset of ASCII.

Any string consisting solely of characters in ISO-Latin-1 will by default be represented internally in ISO-Latin-1. Consider these strings:

Release the raccoon! - consists solely of ASCII characters. ASCII is a subset of ISO-Latin-1, so the string’s internal representation is an ISO-Latin-1-encoded string of bytes.

Libérez le raton laveur! - consists solely of characters that exist in ISO-Latin-1, so the string’s internal representation is an ISO-Latin-1-encoded string of bytes. The "é" character has code point 0xe9 and is represented as the byte 0xe9 internally.

Rhyddhewch y racŵn! - the "ŵ" does not exist in ISO-Latin-1. But it does exist in Unicode, with code point 0x175. As soon as perl sees a non-ISO-Latin-1 character in a string, it switches to using something UTF-8-ish, so code point 0x175 is represented by byte sequence 0xc5 0xb5. Note that while valid characters’ internal representations are valid UTF-8 byte sequences, this can also encode invalid characters.

Libérez le raton laveur! Rhyddhewch y racŵn! - this contains both an "é" (which is in ISO-Latin-1) and a "ŵ" (which is not), so the whole string is UTF-8 encoded. The "ŵ" is as before encoded as byte sequence 0xc5 0xb5, but the "é" must also be UTF-8 encoded instead of ISO-Latin-1-encoded, so becomes byte sequence 0xc3 0xa9.

But notice that ISO-Latin-1 not only contains ASCII, and characters like "é" (at code point 0xe9, remember), it also contains characters "Ã" (capital A with a tilde, code point 0xc3) and "©" (copyright symbol, code point 0xa9). So how do we tell the difference between the ISO-Latin-1 byte sequence 0xc3 0xa9 representing "Ã©" and the UTF-8 byte sequence 0xc3 0xa9 representing "é"? Remember that a representation is "a sequence of bytes that is unique for a particular combination of character and encoding". So perl stores the encoding as well as the byte sequence. It is stored as a single bit flag. If the flag is unset then the sequence is ISO-Latin-1, if it is set then it is UTF-8.

Source code encoding, the utf8 pragma, and why you shouldn’t use it

It is possible to put non-ASCII characters into your source code. For example, consider this file:

my $string = "é";

print "$string contains ".length($string)." characters\n";

from which some problems arise. First, if the file is encoded in UTF-8, how can perl tell when it comes across the byte sequence 0xc3 0xa9 what encoding that is? Is it ISO-Latin-1? Well, it could be. Is it UTF-8? Again, it could be. In general, it isn’t possible to tell from a sequence of bytes what encoding is in use. For backward-compatibility reasons, perl assumes ISO-Latin-1.

If you save that file encoded in UTF-8, and have a UTF-8-savvy terminal, that code will output:

é contains 2 characters

which is quite clearly wrong. It interpreted the 0xc3 0xa9 as two characters, but then when it spat those two characters out your terminal treated them as one.

We can tell perl that the file contains UTF-8-encoded source code by adding a use utf8. We also need to fix the output encoding - use utf8 doesn’t do that for you, it only asserts that the source file is UTF-8 encoded:

use utf8;
binmode(STDOUT, ":encoding(UTF-8)");

my $string = "é";

print "$string contains ".length($string)." character\n";

(For more on output encoding see the next section)

And now we get this:

é contains 1 character

Hurrah!

At this point a second problem arises. Some editors aren’t very clever about encodings and even if they correctly read a file that is encoded in UTF-8, they will save it in ISO-Latin-1. VSCode for example is known to do this at least some of the time. If that happens, you’re still asserting via use utf8 that the file is UTF-8, but the "é" in the sample file will be encoded as byte 0xe9, and the following double-quote and semicolon as 0x22 0x3b. This results in a fatal error:

Malformed UTF-8 character: \xe9\x22\x3b (unexpected non-continuation byte 0x22,
immediately after start byte 0xe9; need 3 bytes, got 1) at ...

So given that you’re basically screwed if you have non-ASCII source code no matter whether you use utf8 or not, I recommend that you just don’t do it. If you need a non-ASCII character in your code, use any of the many other ways of specifying it, and if necessary put a comment nearby so that whoever next has to fiddle with the code knows what it is:

chr(0xe9);   # e-acute

Input and output

Strings aren’t the only things that have encodings. File handles do too. Just like how perl defaults to assuming that your source code is encoded in ISO-Latin-1, it assumes unless told otherwise that file handles similarly are ISO-Latin-1, and so if you try to print "é" to a a handle, what actually gets written is the byte 0xe9.

Even if your source code has the use utf8 pragma, and your code contains the byte sequence 0xc3 0xa9, which will internally by decoded as the character "é", your handles are still ISO-Latin-1 and you'll get a single byte for that character. For how this happens see "PerlIO layers" below.

Things get a bit more interesting if you try to send a non-ISO-Latin-1 character to an ISO-Latin-1 handle. Perl does the best it can and sends the internal representation - which is UTF-8, remember - to the handle and emits a warning "Wide character in print". Pay attention to the warnings!

This behaviour is another common source of bugs. If you send the two strings "Libérez le raton laveur!" followed by "Rhyddhewch y racŵn!" to an ISO-Latin-1 handle, then the first one will sail through, correctly encoded, but the second will also go through. You’ve now got two different character encodings in your output stream and no matter what encoding is expected at the other end you’ll get mojibake.

PerlIO layers

We’ve seen how by default input and output is assumed to be in ISO-Latin-1. But that can be changed. Perl has supported different encodings for I/O since the dawn of time - since at least perl 3.016. That’s when it started to automatically convert "\n" into "\r\n" and vice versa on MSDOS, and the binmode() function was introduced in case you wanted to open a file on DOS without any translation.

These days this is implemented via PerlIO layers, which allows you to open a file with all kinds of translation layers, including those which you write yourself or grab from the CPAN (see for example File::BOM). You can also add and remove layers from an already open handle.

In general these days, you always want to read/write UTF-8 or raw binary, so will open files something like this:

open(my $fh, ">:encoding(UTF-8)", "some.log")

open(my $fh, "<:raw", "image.jpg")

or to change the encoding of an already open handle:

binmode(STDOUT, ":encoding(UTF-8)")

(NB that encodings applied to bare-word file handles such as STDOUT have global effect!)

Provided that we don’t have to worry about Windows, we generally will only ever have one layer doing anything significant on a handle (on Windows the :crlf layer is useful in addition to any others, to cope with Windows’s endearing backward-compatibility with CP/M), but it's possible to have more. In general, when a handle is opened for reading, encodings are applied to data in the order that they are specified in the open() function call, from left to right. When writing, they are applied from right to left.

If you ever think you need more than one layer, or want a layer other than those in the examples above, see PerlIO.

The Encode module

The above explains the "royal road", where you are in complete control of how data gets into and out of your code. In that situation, you should never need to re-encode data, as it will always be Just A Bunch Of Characters whose underlying representation you don’t care about. That is, however, often not the case in the real world where we are beset by demons. We sometimes have to deal with libraries that do their own encoding/decoding and expect us to supply them with a byte stream (XML::LibXML, for example), or which have had incorrect or partial bug fixes applied for any of the problems mentioned above and for which we can’t easily provide a proper fix because of other code now relying on the buggy behaviour (by for example having work-arounds to correct badly-encoded data).

Encode::encode

The Encode::encode() function takes a string of characters and returns a string of bytes that represent that string in your desired encoding. For example:

my $string = "Libérez le raton laveur!";
encode("UTF-8", $string, Encode::FB_CROAK|Encode::LEAVE_SRC);

will return a string where the character "é" has been replaced by the two bytes 0xc3 0xa9. If the original string was encoded in UTF-8 then the underlying representation of the input and output strings will be the same, but their encodings (as stored in the single bit flag we mentioned earlier) will be different, and the output will be reported as being one character longer by the length() function.

Encode::encode can sometimes for Complicated Internals Optimisation Reasons modify its input. To avoid this set the Encode::LEAVE_SRC bit in its third argument.

If you are encoding to anything other than UTF-8 or your string may contain characters outside of Unicode then you should consider telling encode() to be strict about characters that it can't encode, such as if you try to encode "ŵ" into a ISO-Latin-1 byte sequence. That's what the Encode::FB_CROAK bit is about in the example - in real code the encode should be in a try/catch block to deal with the exception that may arise. Encode's documentation has a whole section on handling malformed data.

Encode::decode

It is quite common for us to receive data, either from a network connection or from a library, which is a UTF-8-encoded byte stream. Naively treating this as ISO-Latin-1 characters will lead to doom and disaster, as the byte sequence 0xc3 0xa9 will, as already explained, be interpreted as the characters "Ã" and "©". Encode::decode() takes a bunch of bytes and turns them into characters assuming that they are in a specified encoding. For example, this will return a "é" character:

decode("UTF-8", chr(0xc3).chr(0xa9), Encode::FB_CROAK)

You should consider how to handle a byte stream that turns out to not be valid in your desired encoding and again I recommend use of Encode::FB_CROAK.

Encode:: everything else

The "Encode" module provides some other functions that, on the surface, look useful. They are, mostly, not.

Remember how waaaay back I briefly mentioned that perl’s internal representation for non-ISO-Latin-1 characters was UTF-8-ish and how they could contain invalid characters? That’s why you shouldn’t use encode_utf8 or decode_utf8. You may be tempted to use Encode::is_utf8() to check a string's encoding. Don't, for the same reason.

You will generally not be calling encode() with a string literal as its input, but with a variable as its input. However, any errors like "Modification of a read-only value attempted" are your fault, you should have told it to Encode::LEAVE_SRC.

Don't even think about using the _utf8_on and _utf8_off functions. They are only useful for deliberately breaking things at a lower level than you should care about.

Debugging

the UTF8 flag

The UTF8 flag is a reliable indicator that the underlying representation uses multiple bytes per non-ASCII character, but that’s about it. It is not a reliable indicator whether a string’s underlying representation is valid UTF-8 or that the string is valid Unicode.

The result of this:

Encode::encode("UTF-8", chr(0xe9), 8)

is a string whose underlying representation is valid UTF-8 but the flag is off.

This, on the other hand has the flag on but the underlying representation is not valid UTF-8 because the character is out of range:

chr(2097153)

This is an invalid character in Unicode, but perl encodes it (it has to encode it so it can store it) and turns the UTF8 flag on (so that it knows how the underlying representation is encoded):

chr(0xfff8)

And finally, this variable that someone else’s broken code might pass to you contains an invalid encoding of a valid character:

my $str = chr(0xf0).chr(0x82).chr(0x82).chr(0x1c);
Encode::_utf8_on($str);

Devel::Peek

This is a very useful module for looking at the internals of perl variables, in particular for looking at what perl thinks the characters are and what their underlying representation is. It exports a Dump() function, which prints details about its argument’s internal structure to STDERR. For example:

$ perl -MDevel::Peek -E 'Dump(chr(0xe9))'
SV = PV(0x7fa98980b690) at 0x7fa98a00bf90
  REFCNT = 1
  FLAGS = (PADTMP,POK,READONLY,PROTECT,pPOK)
  PV = 0x7fa989408170 "\351"\0
  CUR = 1
  LEN = 10

For the purposes of debugging character encoding issues, the two important things to look at are the lines beginning with FLAGS = and PV =. Note that there is no UTF8 flag set, indicating that the string uses the single-byte ISO-Latin-1 encoding. And the string’s underlying representation is shown (in octal, annoyingly), as "\351".

And here’s what it looks like when the string contains code points outside ISO-Latin-1, or has been decoded from a byte stream into UTF-8:

$ perl -MDevel::Peek -E 'Dump(chr(0x3bb))'
SV = PV(0x7ff37e80b090) at 0x7ff388012390
  REFCNT = 1
  FLAGS = (PADTMP,POK,READONLY,PROTECT,pPOK,UTF8)
  PV = 0x7ff37f907350 "\316\273"\0 [UTF8 "\x{3bb}"]
  CUR = 2
  LEN = 10

Notice that the UTF8 flag has appeared, and that we are shown both the underlying representation as two octal bytes "\316\273" and the characters (in hexadecimal if necessary - mmm, consistency) that those bytes represent.

hexdump

For debugging input and output I recommend the external hexdump utility. Feed it a file and it will show you the bytes therein, avoiding any clever UTF-8-decoding that your terminal might do if you were to simply cat the file:

$ cat greek
αβγ
$ hexdump -C greek
00000000  ce b1 ce b2 ce b3 0a                              |.......|
00000007

It can of course also read from STDIN.

PerlIO::get_layers

Once you’re sure that your code isn’t doing anything perverse, but your data is still getting screwed up on input/output you can see what encoding layers are in use on a handle with the PerlIO::get_layers function. PerlIO is a Special built-in namespace, you don’t need to use it. Indeed, if you do try to use it you will fail, as it doesn’t exist as a module. Layers are returned in an array, in the order that you would tell open() about them.

Layers can apply to any handle, not just file handles. If you’re dealing with a socket then remember that they have both an input side and an output side which may have different layers - see the PerlIO manpage for details. And also see the doco if you care about the difference between :utf8 and :encoding(UTF-8) - although if you diligently follow the sage advice in this document you won’t care, because you won’t use :utf8.

The many ways of writing a character

There are numerous different ways of representing a character in your code.

String literals

"m"

For the reasons outlined above please only use this for ASCII characters.

The chr function

This function takes a number as its argument and returns the character with the corresponding codepoint. For example, chr(0x3bb) returns λ.

Octal

You can use up to three octal digits "\155" for ISO-Latin-1 characters only but please don’t. It’s a less familiar encoding than hexadecimal so hex is marginally easier to read, and it also suffers from the “how long is this number” problem described below.

Hexadecimal

"\x{e9}"

You can put any number of hexadecimal digits between the braces. There is also a version of this which doesn’t use braces: "\xe9". It can only take one or two hexadecimal digits and so is only valid for ISO-Latin-1 characters. The lack of delimiters can lead to confusion and error. Consider "\xa9". Brace-less \x can take one or two hex digits, so is that \xa (a line-feed character) followed by the digit 9, or is it \xa9, the copyright symbol? Brace-less \x is greedy, so if it looks like there are two hex digits it will assume that there are. Only if the first digit is followed by the end-of-string or by a non-hex-digit will it assume that you meant to use the single digit form. This means that \xap, for example, is a single hex digit, so is equivalent to \x{0a}p, a new line followed by the letter p. I think you will agree that use of braces makes things much clearer, so the brace-less variant is deprecated.

By codepoint name

"\N{GREEK SMALL LETTER LAMDA}"

This may sometimes be preferable to providing the (hexa)decimal codepoint with an associated comment, but it gets awful wordy awful fast. By default the name must correspond exactly to that in the Unicode standard. Shorter aliases are available if you ask for them, via the charnames pragma. The documentation only mentions this for the Greek and Cyrillic scripts, but they are available for all scripts which have letters. For example, these are equivalent:

"\x{5d0}"

\N{HEBREW LETTER ALEF}"

use charnames qw(hebrew);
"\N{ALEF}"                  # א

Be careful if you ask for character-set-specific aliases as there may be name clashes. Both Arabic and Hebrew have a letter called "alef", for example:

use charnames qw(arabic);
"\N{ALEF}"                  # ا

use charnames qw(arabic hebrew);
"\N{ALEF}"                  # Always Hebrew, no matter the order of the imports!

A happy medium ground is to ask for :short aliases:

use charnames qw(:short);
"\N{ALEF}"                           # error
"\N{hebrew:alef} \N{arabic:alef}"    # does what it says on the tin

Other hexadecimal

"\N{U+3bb}"

This notation looks a little bit more like the U-ish hexadecimal notations used in other languages while also being a bit like the \N{...} notation for codepoint names. Unless you want to mix hexadecimal along with codepoint names you should probably not use this, and prefer \x{...} which is more familiar to perl programmers.

In regular expressions

You can use any of the \x and \N{...} variants in regular expressions. You may also see \p, \P, and \X as well. See perlunicode and perlrebackslash. You should consider use of the /a modifier as that does things like force \d to only match ASCII and not, say, ৪ which looks like 8 but is actually BENGALI DIGIT FOUR.

ASCII-encoded JSON strings in your code

You may need to embed JSON strings in your code, especially in tests. I recommend that JSON should always be ASCII-encoded as this minimises the chances of it getting mangled anywhere. This introduces yet another annoying way of embedding a bunch of hex digits into text. This example:

use JSON;

to_json(chr(0x3c0), { ascii => 1 });

will produce the string "\u03c0". That’s the sequence of eight characters " \ u 0 3 c 0 ". The double quotes are how JSON says “this is a string”, and the two characters \ and u are how JSON says “here comes a hexadecimal code point”. If you want to put ASCII-encoded JSON in your code then you need to be careful about quoting and escaping.

Perl will treat the character sequence \ u as a real back-slash followed by the letter when it is single-quoted, but in general it is always good practice to escape a back-slash that you want to be a real back-slash, to avoid confusion to the reader who may not have been paying attention to whether you’re single- or double-quoting, or in case you later change the code to use double-quotes and interpolate some variable:

my $json = '"I like \\u03c0, especially Greek pie"';

# or double-quoted with interpolation
my $json = qq{"I like \\u03c0, especially $nationality pie"};

Accented character vs character + combining accent

For many characters there are two different valid ways of representing them. chr(0xe9) is LATIN SMALL LETTER E WITH ACUTE. The same character can be obtained with the two codepoints "e".chr(0x301) - that is LATIN SMALL LETTER E and COMBINING ACUTE ACCENT.

Whether those should sort the same, compare the same, or one should be converted to t’other will vary depending on your application, so the best I can do is point you at Unicode::Normalize.

Odds and Ends

This website is extremely useful for looking up characters' names, codepoints, and UTF-8 encoding:
- Xah Lee’s Unicode Search
Configure your editor to highlight non-ASCII characters!
- vim
- add this to your ~/.vimrc file:

" replace the * with *.pl,*.pm,*.js etc if you only want this for some file types
autocmd BufReadPost,BufNewFile * syntax match nonascii "[^\u0000-\u007F]" containedin=ALL
highlight nonascii guibg=Red ctermbg=1 term=standout