DEV Community

The human toll of log4j maintenance

Yawar Amin on December 14, 2021

BY NOW, most of the internet knows about the famous Log4Shell exploit, and if you don't, it's easy to get a sense of how disastrous it's been. To d...
Collapse
 
goodevilgenius profile image
Dan Jones

Here's another suggestion:

What if the Alibaba engineers fixed it and sent the patch to the log4j engineers?

I've submitted patches to open source libraries that my company depended on, and did so during working hours, because it was critical to the work.

Why didn't those paid Alibaba engineers fix it themselves?

Collapse
 
yawaramin profile image
Yawar Amin

Yes, very good point. But also, what if the log4j maintainers charged money to merge the Alibaba patch? After all, properly reviewing a patch, ensuring it fixes the issue, and doesn't introduce a worse one, then publishing the fixed version, and all the work that entails–this is still hard work. Nothing in an OSS license prevents charging for it.

Collapse
 
goodevilgenius profile image
Dan Jones

Absolutely. With something this critical, if Alibaba discovered, they definitely should be putting in whatever resources is necessary to getting it fixed quickly.

Collapse
 
jayjeckel profile image
Jay Jeckel

There have been a lot of people coming out of the woodwork to make hyperbolic claims about the open source philosophy (like the ones you link to) and for the most part they've all been really bad takes that showed a distinct lack of understanding what open source actually is.

However, you basically nailed it with the quote from Rich Hickey, though I would quibble that open source is a philosophy about everyone having the freedom to use and modify source code, and the licenses are just tools to achieve and enforce that freedom.

Otherwise, you and Rich are dead right; open source doesn't give anything beyond free as in freedom and a loose implication of free as in beer, though the latter isn't absolute or all encompassing.

All that said, I would like to address the part about log4j:

the fact that the log4j developers had this massive security issue dumped in their laps, with the expectation that they were supposed to fix it. How did that happen? How did a group of smart, hard-working people get roped into a thankless, high-pressure situation with absolutely no upside for themselves?

Firstly, nothing was dumped in the log4j dev's lap. It's their project, their (frankly stupid) feature, and their vulnerability, so it was already in their lap. Why were they expected to fix it? Because it's their software and it's their boneheaded mistake that caused all this.

In most other cases I would completely agree with you, but in this case it's a feature that never should have been in the library in the first place, being in there it never should have been enabled by default, and being in there and enabled by default it should have been fixed a long time ago.

The log4j devs crapped on the floor, so yes, if they want to keep credibility with their peers, the community, and the industry as a whole, then they need to clean it up. Though, I agree with you, had it been me I would have asked Alibaba for a bunch of money or, barring that, at least asked them to submit a pull request with the required fixes.

As I've said plenty of times around here and other places, the open source community is not here to serve corporate interests. The exact opposite, in fact, we are here to take away as much power as possible from corporations and put it into the hands of everyone.

Collapse
 
yawaramin profile image
Yawar Amin

I agree with almost everything you are saying. But:

The log4j devs crapped on the floor, so yes, if they want to keep credibility with their peers, the community, and the industry as a whole

To me this attitude is part of the problem. It's part of the insidious culture we have in our industry of 'you must prove yourself good enough to be recognized as one of us' and oddly enough that social proof is done by having them do unpaid labour for others. What happened to the credibility they've had all these years by having their software be used by mega-corporations? It's quite funny how it's now gone with the snap of a finger and they are back to being perceived as n00bs, because of one stupid mistake that all the omniscient godlike engineers criticizing it now, never once brought up since the Year of our Lord 2013 when it was merged.

This is part of the problem, the mindset that leads to the brainwashing, that 'we have to fix this, it's on us'. This is exactly what I'm arguing against here.

Collapse
 
jayjeckel profile image
Jay Jeckel

I understand what you're saying, but I don't see it from that perspective and wasn't intending to portray my criticisms in that way.

How I see it, they have proven that they are "one of us". They made a functional library that is useful for many other developers, they've given to the open source community, they have credibility and they are by no means "n00bs", and, like all of us, they've made mistakes; they're developers, they are one of us and they qualify as much as any of us to pass any gate kept by coders. And because they are one of us, they have to fix their mistakes.

They don't have to fix it because other people rely on the software; they don't have to fix it because other people want them to; they don't have to fix it because of any misperceived notion of obligation to the open source community; and they definitely don't need to fix it because some suits treat their library as part of a revenue stream.

They have to fix this mistake because it isn't some bug that anyone could have overlooked (even if a lot of people did overlook it for too long), it is a fundamentally incorrect design decision that any self respecting developer would want to fix. They have to fix this mistake because not doing so would be like a chef not caring that the food they made you was burnt. They have to fix this mistake because they are professionals with integrity and cleaning up one's own messes is what a responsible, caring person does.

They don't have to fix it to prove they are competent developers, they have to fix because they are competent developers.

Thread Thread
 
yawaramin profile image
Yawar Amin • Edited

It's easy to say in hindsight that this is a 'fundamentally incorrect design', now that we know all this. My question is, where were all these brilliant insights a month, a year, five years ago, while the library was in production use throughout the Java ecosystem? Where were the critics who are crawling out of the woodwork now? Sorry, but imho, y'all don't get to show up eight years after the fact and criticize and dictate and try to pull mind games like this appeal to the sense of pride thing that you're trying to do. ('Where's your pride as a developer? Don't you want to provide this free patch for a library that you've been maintaining for years, to us who are profiting off it, on our urgent timeline, working through the weekend? Don't you feel any shame?')

They don't have to fix it to prove they are competent developers, they have to fix because they are competent developers.

And if they didn't want to, they're not competent developers? Slightly different words, same gatekeeper-y meaning.

Look, my argument is not that bugs/design flaws should not be fixed. It's that they should not be fixed on the high-pressure timelines of corporations which are benefiting off their unpaid labour. I say chew on it for one, two, three months. Don't let the urgency of others become your own. Don't let them dictate that you spend evenings and weekends away from your family. They'll be criticized no matter what they do–your comments prove that. So might as well preserve their mental health while they go through that. Or even let others contribute and try to fix this supposedly urgent bug!

Thread Thread
 
jayjeckel profile image
Jay Jeckel

It's easy to say in hindsight that this is a 'fundamentally incorrect design', now that we know all this.

That it is and I'm not claiming to speak with anything other than hindsight. But that doesn't change the situation. With current modern knowledge and eyes, there is no denying that a logging library executing code from strings is a major violation of basic design principles.

My question is, where were all these brilliant insights a month, a year, five years ago, while the library was in production use throughout the Java ecosystem? Where were the critics who are crawling out of the woodwork now?

You're not wrong. Present this scenario in any design class and the conclusion will be that a ridiculous number of eyes saw this problem and did nothing about it.

Sorry, but imho, y'all don't get to show up eight years after the fact and criticize

But we do get to do that. This is the topic of the day, it's both important to the industry and useful as a learning experience; developers are going to comment on it. Que Sera, Sera.

You keep bringing up gatekeeping. Well, telling developers when and where they can have an opinion on the work of their peers sounds to me like the bad kind of gatekeeping.

and dictate and try to pull mind games like this appeal to the sense of pride thing that you're trying to do. ('Where's your pride as a developer? Don't you want to provide this free patch for a library that you've been maintaining for years, to us who are profiting off it, on our urgent timeline, working through the weekend? Don't you feel any shame?')

I've dictated nothing. I didn't even imply that they had to follow anyone's time table, urgent or otherwise. I also didn't say anyone should feel shame about anything.

And it isn't a mind game. Maybe it doesn't apply to you, but a lot of people from all professions take pride in their work, the log4j devs among them it seems and rightly so.

And if they didn't want to, they're not competent developers? Slightly different words, same gatekeeper-y meaning.

No, that's not what I was saying. The implication is that, as competent developers that take pride in their work, they would want to fix the problem.

And judging people on their actions isn't gatekeeping. It's how one is supposed to judge other people.

Look, my argument is not that bugs/design flaws should not be fixed. It's that they should not be fixed on the high-pressure timelines of corporations which are benefiting off their unpaid labour. I say chew on it for one, two, three months. Don't let the urgency of others become your own. Don't let them dictate that you spend evenings and weekends away from your family.

On that we completely agree. Open source is volunteer work and no one can tell you what you have to do or when you have to do it. If one doesn't like how a dev runs their project, then they can always fork the repo and do the work themselves. That's the beauty of open source.

They'll be criticized no matter what they do–your comments prove that.

Pointing out a mistake isn't criticism in the negative sense of the word. No one I've seen is saying they are bad people or should be punished because they made a bad design decision, but them not being bad people also doesn't make the design decision any less bad.

Thread Thread
 
ianturton profile image
Ian Turton

doesn't mean they have to fix it over the weekend - if it is that urgent to someone who has spotted the issue then they can work over the weekend and submit a patch for the issue.

Collapse
 
pinotattari profile image
Riccardo Bernardini

log4j is not broken

Stop blaming log4j

Yes, you read it right. It works as documented. It is not a bug, it is a feature, really.

What you can blame on log4j is feature creep, that made log4j handling tasks that go beyond the duty of a logging library.

A logging library should do one thing: take a string, together with a "severity" and save it somewhere. The string should be an opaque, binary blob and no processing should be done by the library. Stop.

Who actually is to blame for the problem is who used this library

  • You decided to use a library that has a very dangerous feature (remember, not a bug). To make a simile: suppose I make an http server that with some "magic URL" gives you access to a shell on the server (e.g. example.com/.shell/?command="rm -rf /"). Clearly a convenient, but very dangerous feature. Would you install such a server? I wouldn't.
  • You decided anyway to use log4j (maybe you had some constraint that forced you using it). OK, then at least do some sanitation on the material that you receive from outside!! Come on! It is Good Practices 101. Remember Little bobby tables of xkcd?

Conclusion? Should I assign blame in this matter I would do as follows

  • 10% to log4j library for "extreme feature creep"
  • 90% to log4j users for choosing a library with dangerous features and using it carelessly.
Collapse
 
jedwidz profile image
Jonathan Michael Edwards

That's an interesting comment. As I explain in my post Log4j vulnerability: What the FAQ happened?, the feature was there and intentional (so yes, not broken in that sense), but was never properly documented.

And regarding sanitation/sanitization, OWASP is one of the best authorities on best practices, and they specifically did not recommend sanitization for log4j 2. You're cordially invited to provide a citation for Good Practices 101 that advises otherwise.

Collapse
 
yawaramin profile image
Yawar Amin

Yes very good point. Not really related to the point I am trying to make in my post, which is that people are expecting log4j maintainers to automatically jump into action during nights and weekends to save their hides–for free.

Collapse
 
bsdimp profile image
Warner Losh

I've done open source for the last 30-odd years. I'm happy to fix your bugs in code I've written for free. But on my timeline. If you want an expedited timeline, then that's done "one the clock" for a fee. And if you'd like me to spend time away from my family during the holidays, the fee can be quite large. Granted, it took 10-15 years of that to arrive at this point. In this situation, I'd have requested a patch from Alibaba since that would expedite the timeline of a fix. Especially since the language used sounds like a veiled threat (people already know about this, so we'll try to keep a lid on this, but because of our sloppy disclosure policies we've put you in a pickle. Please fix it soon). Not sure of a good way around this, though...

Collapse
 
leob profile image
leob • Edited

Dev.to should pin this article at the top of their listing, well at the least it should be in their "posts of the week" newsletter - you've put this so eloquently and powerfully ... this is a wake-up call if there ever was one ... legendary stuff!

Collapse
 
wirelessben profile image
wirelessben

$50,000 for a world-wide impact? $10,000,000 to fix all the world's servers seems more like it. That's a ransom, of course, but there has to be a middle road.

Collapse
 
yawaramin profile image
Yawar Amin

I know, right? I think I was being extremely conservative.

Collapse
 
valeriavg profile image
Valeria

Open source means you can make modifications. Can't code - help with debugging. Can't debug - write documentation. I would understand if said Alibaba engineer would urge to REVIEW and MERGE his urgent fix. But no, someone else must find it, fix it and test it. Open source is not broken, some people are.

Collapse
 
yawaramin profile image
Yawar Amin

Yeah the thing is, if there had been no patch, we would all still be able to do the mitigation of deleting the JndiLookup.class file from production JARs to stop this attack. So what did all this pressure on the maintainers achieve? A bunch of people upgrading, and many complaining. 🤷‍♂️

Collapse
 
tardisgallifrey profile image
Dave

Is it so impossible to believe that a company the size of AliBaba didn't have paid staff that could sit down, open the source, and fix it themselves? They could have submitted their fix to the maintainers along with other fixes to see if theirs be became a permanent patch.

Collapse
 
Sloan, the sloth mascot
Comment deleted
Collapse
 
yawaramin profile image
Yawar Amin

Maybe. The argument goes that it's easier to upstream a patch and let the maintainers take care of it for you rather than having to pin all your projects to a fork. So maybe they would just quietly contribute the fix instead of paying money.

Collapse
 
sirfartwaffle profile image
Anus McBeef

This is what you get when you bleat "code wants to be freeeeeeeeeeeeeeeeee!" Release stuff open source? Don't expect payment, acknowledgement, or anything in return.

Collapse
 
yawaramin profile image
Yawar Amin

Sorry but you've completely misunderstood Free Software/open source.