DEV Community: Mike Whitaker

Shell features you didn't know you needed (or possibly even existed) #9

Mike Whitaker — Thu, 02 Apr 2026 14:51:13 +0000

Substituting the contents of a variable in Bash.

I came across this one needing to turn a Git branch name into a Docker image tag, and having it trip over the fact that our branch naming convention is <user>/<issue>/<short description>, and Docker doesn't like / in tags.

Obviously there are many ways of stripping/substituting the /: for example,

using tr

IMAGE_TAG=$(echo $BRANCH | tr \/ \-)

using sed

IMAGE_TAG=$(echo $BRANCH | sed -e 's/\//-/g')

using perl

IMAGE_TAG=$(echo $BRANCH | perl -pe 's/\//-/g')

And there's probably others with awk, and who knows what else.

But why bother, when you can do it in bash itself.

IMAGE_TAG=${BRANCH//\//-}

To clarify, that's essentially:

${VARIABLE/from/to} # replace first occurrence of 'from' 
${VARIABLE//from/to} # replace all occurrences of 'from'

Simple, when you know how.

MariaDB FULLTEXT searches and transactions

Mike Whitaker — Wed, 01 Apr 2026 11:21:01 +0000

Another in a series of "things I had to learn the hard way, and you shouldn't".

The key takeaway from this is that MariaDB FULLTEXT search indices are not updated until a transaction is COMMIT'ed.

Allow me to demonstrate:

MariaDB [fulltext_demo]> create table text_data (
   -> id serial, info text);
Query OK, 0 rows affected (0.022 sec)

MariaDB [fulltext_demo]> CREATE FULLTEXT INDEX 
   -> ft_idx on text_data(info);

Now we have a table fulltext_demo with a row info indexed for a fulltext search. And just to prove it:

MariaDB [fulltext_demo]> insert into text_data set info="banana";
Query OK, 1 row affected (0.015 sec)

MariaDB [fulltext_demo]> select * from text_data where 
   -> match(info) against ('banana');
+----+--------+
| id | info   |
+----+--------+
|  1 | banana |
+----+--------+
1 row in set (0.000 sec)

So far, so good. Now let's do that inside a transaction.

MariaDB [fulltext_demo]> begin;
Query OK, 0 rows affected (0.000 sec)

MariaDB [fulltext_demo]> insert into text_data set info="orange";
Query OK, 1 row affected (0.000 sec)

MariaDB [fulltext_demo]> select * from text_data where 
   -> match(info) against ('orange');
Empty set (0.000 sec)

If like me, you are going "WTF" at this point, see above. The updated row does not get added to the fulltext search index until the commit. Allow me to demonstrate.

MariaDB [fulltext_demo]> commit;
Query OK, 0 rows affected (0.000 sec)

MariaDB [fulltext_demo]> select * from text_data where
   -> match(info) against ('orange');
+----+--------+
| id | info   |
+----+--------+
|  2 | orange |
+----+--------+
1 row in set (0.000 sec)

This makes testing things inside a transaction (where you intend to roll back the test fixtures afterwards) just a little bit more irritating than it could be. :D

Setting up SSH on Windows

Mike Whitaker — Mon, 30 Mar 2026 11:04:38 +0000

Every online tutorial for this seems to be way WAY too complicated for what is in fact a pretty basic process. This is targeted at people who (for example) need to set up a remote SSH session on a Linux server for development purposes).

We assume you have a username, password and hostname for the server you want to log in it.

install OpenSSH on your local Windows machine if it isn't already. You can find this under Settings, search for "Optional Features"

Click the checkbox, and if it's not installed, install it.

Fire up a command prompt.
Create yourself an SSH key pair, by running ssh-keygen

C:\Users\You>ssh-keygen
Generating public/private ed<something> key pair.
Enter file in which to save the key (C:\Users\You/.ssh/id_ed<something>):
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in C:\Users\You/.ssh/id_ed<something> 
Your public key has been saved in C:\Users\You/.ssh/id_<something>.pub
The key fingerprint is:
SHA256:<some random Text> you@YourMachine
The key's randomart image is:
+--[ED<something> 256]--+
| several lines  |
+----[SHA256]-----+****

This generates a secure public/private key pair. On older machines, it may be saved in .ssh/id_rsa<something> instead of .ssh/id_ed<something>. If you have provided a passphrase[1], don't forget it. :D

Now you need to copy the PUBLIC key (the one ending in .pub) to your remote server[2]. Run:

C:\Users\You>scp .ssh\id*.pub user@server:
you@server's password: <type your password here>   
id_ed<something>.pub            100%  110     1.9KB/s   00:00

Note the trailing colon on the command line: don't miss this off!

C:\Users\You>ssh user@server
you@server's password: <type your password here>
<lots of blurb>
you@server$

You are now logged into the remote server. Add your ssh public key to the list of permitted ones.

you@server$ cat id*.pub >> .ssh/authorized_keys
you@server$ <now hit control+D>

You will find yourself back at the Windows command prompt

Check you can log in without a password. If you originally provided a passphrase when you ran ssh-keygen, you will be prompted before you connect.

C:\Users\You>ssh user@server
<lots of blurb>
you@server$

[1] The difference between a password and a pass*phrase* is that the former gets transmitted across the network to your remote host, and can potentially be snooped by a malicious attacker. A passphrase is local to your machine, and therefore immune to being snooped.

[2] You can do this and the next step in one go: the command (on your Windows machine) is

type .ssh\id_*.pub" | ssh user@host "cat >> .ssh/authorized_keys"

Is your TextMate 2 folding corrupt?

Mike Whitaker — Wed, 21 Jan 2026 22:19:18 +0000

Having trouble with corrupted/misplaced fold markers in a TextMate 2 file? Has something changed the file under TextMate 2's feet while you have it open with some folds closed? (not looking at anyone in particular, Claude Code :D)

Simple solution:

close the file in TextMate 2
open Terminal
run the following: xattr -d com.macromates.folded <your filename here>
reopen the file in TextMate 2

Presto. Problem solved.

Catalyst::Request body issues with the file position pointer

Mike Whitaker — Fri, 28 Nov 2025 15:15:54 +0000

OK, so...

For those using the Perl Catalyst web framework in ways involving structured request bodies (e.g. API POSTs)...

$c->req->body is a string, unless Content-Type is application/x-www-form-urlencoded, text/xml, or multipart/form-data (or in fact application/json, which isn't in the docs), in which case it's a File::Temp (an overloaded file handle), and $c->req->body_data gets you the deserialised body.

For various reasons, largely to do with the idiosyncrasies of one particular module, I need to read the raw body data from the $c->req->body file handle to process a Stripe webhook payload. For various other reasons, as part of the API call logging, I need to call $c->req->body_data to get at the deserialised body in another module.

You may imagine my delight when adding the latter caused the former to fail.

An afternoon of bad language and extra debug eventually revealed that $c->req->body_data doesn't clean up after itself, and I have to seek( $c->req->body, 0, 0) before I can read any data from the body file handle.

If this is useful to anyone else, you have my sympathies.

How Philips Hue Bridge discovery actually works

Mike Whitaker — Sat, 06 May 2023 19:01:57 +0000

Having been bitten by a change in the Hue discovery protocol a while ago that I've only just got round to figuring out and fixing, I figured it would be useful to stick this up somewhere where Google can find it. So, here we go...

When a Hue bridge boots up, it has a brief interaction with the Hue servers at meethue.com and sends it the bridge ID, its IP address and port. This is stored under the IP address that the Hue servers have for you - essentially REMOTE_HOST in CGI terms (or 'the answer whatismyip.com gives you').

When an app (such as the Hue app or the Alexa skill) wants to know where your hub is, it connects to discovery.meethue.com, which will look at your IP, and if there's a record (or more than one if you have multiple bridges, I guess) for it, returns it/them, like so:

[
  {
    "id":"001788fff536899d",
    "internalipaddress":"192.168.0.14",
    "port":443
  }
]

The app then knows it can go find the bridge on the given internal IP, and you're all happily sorted.

Spotted the issue yet?

Yup - the wired sockets and one of the Wifi networks on my home network are from the /27 my ISP gave me two decades ago. The Hue is (obviously) on a different IP to my phone, and because there was no NAT involved, the bridge's info was stored on meethue.com under its actual routed IP.

Result, the answer from discovery.meethue.com is:

[]

Zip. Nada. Discovery failed.

To quote @tweethue:

That could be the issue. The bridge is designed to be used with a simple home router.

Well, duh.

Upgrading MySQL 5.7 to MariaDB 10 on Ubuntu 18

Mike Whitaker — Thu, 22 Dec 2022 11:25:56 +0000

FX: Stands up amid very large pile of yak hair

Right. If you're ever in the unfortunate space of needing to upgrade MySQL 5.7 to MariaDB 10 on Ubuntu 18.04 LTS, here's what to do and what not to do.

1) next time, upgrade to 20.04.5 LTS first, and most/all of this will go away :D

2) do NOT install the default mariadb-server-10.1 package, as its mysql_upgrade script does not know how to convert from mysql-5.7 tables.

3) Go here and select your Ubuntu version and a suitable MariaDB version.

4) Cut and paste the appropriate repo lines, and do what it says.

5) if it doesn't run mysql_upgrade for you, run it as root.

6) You may find that it hangs on starting the MariaDB server. This is an issue with apparmor, and can be fixed by following these instructions (reproduced here just in case of link rot).

sudo systemctl stop mariadb
echo "/usr/sbin/mysqld { }" | sudo tee /etc/apparmor.d/usr.sbin.mysqld
sudo apparmor_parser -v -R /etc/apparmor.d/usr.sbin.mysqld

The above should display Removal succeeded for "/usr/sbin/mysqld"

And then, to stop it reappearing on reboot:

sudo ln -s /etc/apparmor.d/usr.sbin.mysqld \
/etc/apparmor.d/disable/usr.sbin.mysqld

Finally:

sudo systemctl start mariadb

Design your own security vulnerability #2

Mike Whitaker — Thu, 14 Apr 2022 21:23:59 +0000

Passwords. Every CEO's nightmare, surely, is to be woken with the news that their company's password data is up for grabs on the Internet for a price.

OK - let's get down to brass tacks. How does this happen?

"Ahah", eager new developer with a brilliant idea for a site thinks. "Let's avoid the pitfalls of #1 in this series and make our users have passwords."

Fine. Quick bit of SQL and their web framework of choice, pop the user and their password in the appropriate table, and they can go SELECT * FROM users WHERE username="$user" AND password="$password" and check if it's valid. Simple, right?

Sidebar right here: the seasoned veterans screaming at the other issues with that snippet, just hush. We'll get to that in a couple of articles time, OK? :D

And here's problem #1. Their database contains the user's unencrypted password. In the event that someone outside the company gains access to the contents of that table, their CEO will be on the phone about 30 seconds after the events in the first paragraph. And they can't assume it won't happen. All the security in the world (and it's a fair bet so far they don't have that!) won't protect them from a disgruntled employee, a social engineering attack, or a misconfigured firewall.

"Easy", they think. "I'll just encrypt the password, then I can decrypt and compare when the user logs in." And worse, they'll probably invent some really clever (I really need the sarcasm emoji about now) code to do it.

And here we are with problems #2 and #3!

Let's deal with #3 first. Rolling your own encryption code. Just. Don't. The odds on our hero coming up with some thing that doesn't have a fundamental flaw with it are small. And if they don't understand the issues involved, what the risks are, and what the encryption function needs to do, their idea of an encryption algorithm might be a bit better than ROT-13.

#2 is the bigger problem, and it's an 'ahah' moment when I can get users to realise that it's connected to 'no, I can't tell you your password' and why sites force them to change it if they forgot it. If someone has access to your DB and your code, and therefore the decryption algorithm (which if they have access to your system you better assume they probably do), they have all it takes to crack all your passwords, and we're back to problem #1 and the phone ringing at 4am.

The key takeaway here is we don't need to know what the password is.

We just need to know that what the user typed encrypts to the same thing.

I need this to be an 'ahah' moment for you if you don't know it already.

Pick a good one-way algorithm, where turning the password into its 'encrypted' (actually, hashed) form is fast, and breaking the algorithm to get it back is prohibitively hard. Think of it as the algorithmic equivalent of tearing up a piece of paper into lots of pieces - that's quick. Putting it back into one whole piece is not..

All you have to do, then, is hash the user's password when they create it, pop it in your users table, and do the same to what they give you when they log in. Compare the two, and bingo.

But, as a commenter points out, and I did in fact know (insufficient caffeine, yer honour), if we don’t pick our hashing algorithm correctly, we have problem #4.

As it stands, two people with the same password have the same hash. Equally, if you know what the algorithm is you can turn up with a table of candidate hashes (basically, run a list of likely passwords through the hash algorithm, which is cheap) and look for matches.

Solution: pick an algorithm that uses a ‘salt’, a random addition to the password that’s hashed with, and stored with, the hashed password.

We are firmly in “do not roll your own“ territory here. Any library worth its salt (see what I did there?) here will have both hash and compare functions (e.g. bcrypt or PKBDF2) to save you getting it wrong. And maybe now your CEO can rest a little easier on this front.

Final lesson: don’t code tired, and have someone else review your code before it gets anywhere near production. :) (thanks ARMB)

As a corollary to this: if a site can give you your current password back if you've forgotten it?

Run.

Very fast.

In the other direction.

At the very least do NOT trust it with any data you care about being exposed to a wider audience.

Sidebar 2: yes, oh seasoned veterans. This is all an oversimplification. But clearly, some folks - lots of folks - don't even get this far.

It is a constant source of wonder and bewilderment to me how often haveibeenpwned gets updated with new leaked passwords that have clearly been leaked from large, popular sites that don't follow at the minimum these simple best practices. The negligence to let that happen probably should be a criminal offence, and CTOs who presided over it barred from ever holding such an office again.

Design your own security vulnerability #1

Mike Whitaker — Wed, 13 Apr 2022 15:36:40 +0000

The first in probably an even more occasional series... None of these are new. But people keep making the same tired old mistakes.

We begin, as ever, with the best of intentions. As part of our business, let's store some customer data by customer ID, where customer ID is a numeric key on our database of customers. Most databases will happily generate this for you and increment it by one when you add a new customer.

Let's have a web page so the customer can see his own data, say at https://my-safe-data.biz/customer/NNNNN where NNNNN is the customer ID. Fine and dandy. No-one but the customer and us know his ID, so he's quite safe. We can just pass that ID to the database, ask it for the data for that customer, and present it. Don't need to password protect it, because the ID is its own password, right - no-one knows anyone else's customer ID, do they, surely??

Wrong.

You don't have to know anyone else's customer ID. Just make an educated guess on how they're generated.

Enter J Random Cracker, who just happens to have seen one of our URLs because he's a customer. "I wonder", they think, "what happens if I change my customer ID by 1"...

"Oh look. Someone else's records." An hour and a bit of scripting later, and they've pulled out LOTS of records to which they have no right.

"But", you say, "no-one would be that stupid, would they?"

Well yes. They would (and yes, just one of many examples, before you ask).

One easy fix is to require a login before you can access your own data, but that opens more potential vulnerabilities. Of which more later.

Another is to use a different method of generating customer IDs.

Same applies.

Shell command options you didn't know you needed #8

Mike Whitaker — Wed, 13 Apr 2022 08:15:02 +0000

Another grep option!

This also tracks back to #6 and #7 in this series, and to be honest, it's more a shell option I really should have known existed. (I’m not proud: some of the reasoning behind these articles is so that you can learn from my “oh, duh” moments.)

As you may recall, we're tracking down a bunch of files with iso-8859-1 characters to convert to utf-8. Things weren't quite as I painted them in the previous posts, as I discovered the directory tree contained a bunch of binary files that I really didn't want to convert. Time to break out the brutal axe and chainsaw combo beloved of shell hackers everywhere, find and xargs.

find mydir -name '*.html' | xargs grep -l -avx '.*'

Upside, it works.
Massive downside for big directories (mine's probably several million files in a four or five deep tree), that's one process for every smallish group (see xargs --show-limits) of files. And it's freakin' SLOW (literally hours for me).

You may imagine me kicking myself (hey, I'm big enough to admit when I mess up!) when I was checking the man page for grep and found this FAR more elegant solution.

grep -l -r --include '*.html' -avx '.*' mydir

--include '<pattern>' makes grep only check files that match <pattern>. As with find -name, stick the pattern in quotes so the shell doesn't try and match it first. And yes, there's a --exclude as well :D.

Shell command options you didn't know you needed #7

Mike Whitaker — Tue, 12 Apr 2022 12:26:40 +0000

grep has lots of options. This one's really a combination of three to do something nifty.

On the previous post, I was converting a bunch of iso-8859-1 files to utf-8. Setting aside for the moment the fact that I was using iconv for this (read the fine man page!), you might perhaps be wondering, how did I know which files I wanted to convert?

First off, this is not magic. It presupposes that you know the files are iso-8859-1 and that your locale is set to utf-8. The latter is easy enough - check your LANG environment variable and set it to something suitable if it doesn't already end in '.UTF-8' or '.utf8' (detailed discussion of this not for this article).

The former is your problem. I can't really help you with this - data is a bunch of bytes plus an encoding you may or may not know :D So. Assuming you have a reasonable level of certainty these files are encoded to iso-8859-1, run this:

grep -axv '.*' mysuspectfile

It will return any lines with iso-8859-1 characters that are not legal utf-8, which is to say pretty much anything with its high bit set.

Sidebar: Before anyone jumps on me, yes there are obscure combinations of high-bit-set iso-8859-1 characters that are legal utf-8, but they are sufficiuently unlikely in any normal text written by people not on weird psychoactive substances for this test to be pretty reliable. Reference here for more details.

Why does this work?

-a says 'treat this file as printable text'
-v says 'invert this match'
-x says 'match the whole line against the pattern .*'

.* (because of -a and your locale) means 'any legal utf-8 character'. The -x requires every character in the line to match that pattern, and the -v will then spit out lines for which that is not the case.

To generate just a list of offending files from a directory, then, we can use -r to recurse down the directory, and -l to just report matching filenames.

grep -r -l -axv '.*' mydirectory

Bingo. All ready to throw at xargs -P :D

Shell command options you didn't know you needed #6

Mike Whitaker — Mon, 11 Apr 2022 16:22:41 +0000

Been a while, but here's a handy one I discovered over the weekend.

Back to the trusty xargs, that rather blunt and brutish chainsaw for processing a long list of files or whatever that someone gave you.

In this case, I had a list of files that I knew with 99%+ certainty were created on our old server and thus encoded in iso-8859-1, contained characters that were represented differently in utf-8 (which we had switched to on our new server) and needed converting, and a handy script wrapper around iconv to do one file at a time.

All 41,000 of them. The list took four hours to generate, during which time I was pondering the fact that I really should have taken advantage of the fact that, usefully, the new server has 40 cores of Xeon goodness. So we ought to be able to parallel process this list now we've got it, right? And ideally without bothering with GNU Parallel or Perl's Parallel::ForkManager?

Turns out we can!

xargs -P <n> (if supported on your OS) runs the commands generated by xargs in n-way parallel.

So:

cat <list of 41K files> | xargs -n 1 -P 100 <iconv wrapper>

We need the -n 1 as the wrapper only takes one file at a time, and this is how we tell xargs that. Deep breath. Hit RETURN.

Whoosh. Load on server briefly rockets to 45, then falls just as fast to its steady 1 and a bit. In about one minute flat, for all 41,000 files.

Not bad.