DEV Community

Gary Kramlich
Gary Kramlich

Posted on

1

Code signing HSMs for Pidgin2

We haven't done a Pidgin 2 release in awhile because aside from other things, I've been unable to sign the Windows binaries.

I bought a new code signing certificate in March of 2024 and it came on a Hardware Security Module (HSM). This is a USB device you plugin into your machine and it contains a private key that allows you to use it without exposing it. The main issue with the HSM was the complete and utter lack of documentation that came with it. So I messed around with it for awhile, but couldn't figure it out. We had no pressing reasons to release so it sat.

In October, rekkanoryo was visiting and with his help we identified that when plugging it in to a Windows host it would show up in the certificate store. We spent a lot of time trying to figure out how to get it to work with osslsigncode but apparently we needed to use pkcs11 and neither of us had any experience with that.

So we ended up trying signtool from the Windows SDK and were able to get it to sign a Pidgin executable, but only when calling it from command prompt. We tried integrating it into our build system but we had zero luck.

Before I go into more depth, lets describe our build system. Our build system on Windows is a collection static Makefile's that depends on a POSIX shell like Bash to work properly. As you might have guessed, these Makefile's can get pretty complicated. For the shell part, we used to recommend cygwin but I've been using Msys2 for awhile now.

So to make this work, we needed to be able to run signtool from a shell running under msys2. However, whenever we'd run the command it would complain that we didn't provide the /fd argument even though it was for sure there. We ended up trying multiple versions of signtool as well and still no luck.

At this point, we started trying to launch a command prompt process with the command using the /C flag to run it. But this wasn't running the command and just giving us a prompt. We gave up at this point as we were out of idea.

So some more time went by and more and more issues started pressing for a Pidgin 2.14.14 release. The certificate prompts, the GCC14 and GCC15 compile errors, as well as the libxml compile errors were all pushing for a new release. So I started looking at this again today (January 21st).

Everything was fuzzy as it had been 3 full months since we looked at this and rekkanoryo and I both forgot a lot of the finer details of what we attempted. Because of this I started trying to get this working with PKCS11 again.

After an hour or two of searching and trying random things I finally learned that the module piece of PKCS11 is specific to your HSM. So I spent about 30 minutes looking to see if the vendor for my HSM provided one. I didn't find anything. So back to signtool I went.

Originally I was testing by modifying our Makefile's and waiting for them to get to the failure point and then tinkering from there but that obviously took a lot of time and got old quick.

So I went back to trying to run the command manually from msys2. But this was just failure after failure. I then realized I should just try running something like cmd.exe /C /c/Program Files (x86)/.../signtool.exe /? and see if I could get the usage out of it. But to my surprise I just got a command prompt in my msys2 Window. So it was clear at that point that that method wasn't going to work no matter how hard I tried.

That's when I got the idea to just put the command in a batch file and to try running that from msys2. So I threw the /c/Program Files (x86)/.../signtool.exe /? into a file named sign.bat and ran it from msys2 with ./sign.bat. To my great surprise, I got the usage.

I then updated sign.bat to include the full command and holy crap it works! We'll almost, I was connected to the machine via RDP and the HSM doesn't work when it detects an RDP session as a security feature. So I quickly ran over to the machine and ran the command again. This time the HSM software popped up asking me for the password to unlock the key. I entered it and SUCCESS!!

I then went to verify the signature and noticed that it only included a signature with a SHA-384 digest.

A screen shot showing the digital signatures and cert information of the pidgin offline installer

So this should work fine with with Windows 10 and 11, but we have people running Pidgin 2 on Windows as old as XP (I know..). We need to keep supporting that or they will complain, loudly.

After some digging through the signtool documentation, I determined that we can add multiple digests by running signtool again and pass the /as argument and a different value for the the /fd argument.

So with that done, the signatures look a lot better and we should be able to release now and not break anyone on very old Windows.

pidgin.exe showing signatures with digest of sha1, sha256, and sha384

We still would like to have some people test on Windows XP, Vista, 7, and 8, so if you have any of those installed, please let me know so I can get you a file to try it out.

This has been ton of work, but it's finally done. I wrote a quick post in our discourse with the actual final setup in the event that they're useful to someone else and so that I have a backup in case I need it.

I hope you found this story interesting, but I hope to never go through it again :)

Heroku

Simplify your DevOps and maximize your time.

Since 2007, Heroku has been the go-to platform for developers as it monitors uptime, performance, and infrastructure concerns, allowing you to focus on writing code.

Learn More

Top comments (0)

Billboard image

The Next Generation Developer Platform

Coherence is the first Platform-as-a-Service you can control. Unlike "black-box" platforms that are opinionated about the infra you can deploy, Coherence is powered by CNC, the open-source IaC framework, which offers limitless customization.

Learn more

Retry later