DEV Community: Leon Timmermans

A year of strife

Leon Timmermans — Wed, 14 Apr 2021 19:59:48 +0000

Disclaimer: this is based on my own personal experience and perception. It's not The Truth, but it is my truth.

Last year, in early May, I got an email that got me worried. Really really worried. Not because of disagreement (though I did disagree with it), but because I knew this would be the most controversial thing I've seen in my more than decade on perl5-porters (the mailing list where core contributors discuss the development of perl). If I was wrong it was only because controversial turned out not to be quite strong enough a word for what would happen.

In it, a radical change was proposed. Radical in technology, but I'm not going to focus on that today. Radical in values, which really means it's radical for the community as well. It immediately divided both core contributors and much of the community into many factions with two main contingents.

What may seem like a simple technical disagreement was everything but. It was a fight for the soul of the language and the community that we love. While there were a lot of (mostly silent) people in the middle, in the discourse there were two sides and both (or at least enough people on both sides) were entirely convinced that the language and community that we all love only has a future if their side wins, and that the other will drive it to ruin.

If that doesn't sound like the prologue of a Greek tragedy, then I don't know what does.

It led to vigorous discussion; initially behind the scenes, and after the Conference-in-the-Cloud also on perl5-porters and elsewhere. And quite frankly, this only stopped when people mostly stopped talking to each other after 2 weeks because there was little point to it.

The alarm bells really should have gone off here.

Soon enough the conflict moved to github. The unfortunate thing is that there were two groups of people: one with the will to make this happen but not quite the knowledge, and they started writing code. And one group that didn't want this to happen that did have the knowledge, who started reviewing said code. As you can probably guess, this soon became a mutually aggravating situation. After a few weeks of stepping on each other's toes, it became an open conflict.

The stakes were astronomical. I've heard several people say (in private) that they felt they would be needing to look for a different career and community over this. Combine this with a pandemic that has all of us stressed out, this led to a situation which I can only summarize as both sides kept pushing until the other side would give up. Given the sheer stubbornness of the average perl contributor this could only end when a sufficient number of people have burned out that neither side is capable of doing pretty much anything. It was a dynamic that was quite frankly burning me out too, to the point where several of my loved ones told me I should give up on this for sake of my own mental health (and I've heard similar stories from others).

It's easy to assume that the things that happened were a few abusive assholes, but in my experience it was mostly a vicious cycle of previously reasonable and kind people lashing out at the people that were burning them out.

And that's where the alarm bells became cold-war style air sirens, the kind you can hear from miles away (sorry gen-z, you probably have no idea what I'm talking about).

This was the moment when we (or actually a bunch of community elders) started up a process to set up a new governance, because any decision making process that can generate this amount of strife is in need of repairs. Many people saw it as a diversion, but I don't agree. I think it was the only way to restart this conversation, and to get to a point (currently still in the future) where we can heal again.

But when hell broke loose, another thing was finally spoken out, though it was easily missed in between all the unpleasantness, is a simple truth that there is a third option that had been unthinkable a year ago: forking. This changes everything about the equation, literally everything.

Ultimately I believe that is what ended the discussion. It ended with the simple knowledge that the opposition was able and willing to fork if that's what it takes to prevent their worst-case scenario. In a way it was a Solomon's judgement for the steering committee. It ended the discussion not just because almost no one wants to fork the community, but because perl7 as announced last summer wouldn't be viable given a competitor that actually supports all of CPAN today and will continue to do so for the foreseeable future (just look at how long it took the python3 transition to take over, and that's without people wanting to keep supporting python2).

It is a pyrrhic victory. Pyrrhic not just because "we" haven't really won anything. Everyone lost because friendships were damaged (and perhaps lost) over this. We lost because a bunch of the people who matter for moving perl forward are currently effectively burned out (a few loudly so, but many in painful silence). We lost because people stopped contributing over this (most prominently but not limited to former project lead Sawyer).

So what can we learn from this, and how can we move forward?

We needed different governance, for a multitude of reasons. The old pumpking model had a tendency to burn people out, that was not sustainable (to be honest, Sawyer seemed burned out to me even before any of this happened, I can only imagine where he is now). Having more people in our executive, and having them there for bound terms (it's easier to stop when it takes effort to continue) will hopefully prevent that in the future. But also because it will lead to more diverse points of view. I suspect we'll be tweaking our governance for a while we gain more experience with it, but at least we have a foundation to do that with now.
Going through this during a pandemic really didn't help. I honestly believe this situation would not have gotten so badly out of hand if we could meet up face to face and talk to each other with a beverage in our hands; if we weren't all so tense because we haven't seen many of our loved ones in ages. We should remember that. Some discussions should be done in real life; sometimes the right response to a conflict is "Ooh, I get the impression this may be a Chartreuse-level discussion. Are you up for that?"
We should have more attention for our contributors' mental health; we need to remember the human on the other side of the conversation. It's perhaps cliche to say we should act with more empathy for each other, but frankly we collectively also acted with too little empathy for ourselves.
It seems that the proponents initially mistook a lot of the grievances as mere bike-shedding, when they actually were meaningful concerns that wouldn't go away by themselves. They didn't seem to realize that they were crossing the Rubicon by actually moving forward with their plans without a consensus. We should not set ourselves up for failure like that, there is no way that could have ended well.
One thing we started doing much too late was mediation (Neil in particular has done great work there lately). We need people who can see all perspectives and bring them all back together. This is where we truly miss Larry more than anything, because bringing different needs together was his true talent. I don't think (or hope) we will be needing this often, but it's probably good to be prepared for such an eventuality.
We talk a lot about speaking with kindness, but we don't talk nearly enough about listening with kindness. Unlike speaking, listening kindly can't really be enforced, but the speaking without listening can easily become dysfunctional. Achieving this requires a cultural change. I don't know how to get us there, but I do know that it will require leadership.
The other thing that requires leadership is bringing us back together again. Perl is unique in the breadth of its user base and that is both Perl's greatest strength as its greatest weakness. These conflicting interests were at the root of this conflict, and finding a path that serves all those interests (including the less loud ones) will be the main challenge for the steering committee.

Better argument parsing with Getopt::Long

Leon Timmermans — Tue, 15 Dec 2020 14:17:54 +0000

The problem

Raku has a built-in argument parser. This is a really good idea, given how common argument parsing it, but I still ended up writing my own, and to explain why I did so, I should first explain what the built-in parser does.

The raku built-in parser is a blind parser. This means that it converts the input arguments into a Capture without any knowledge of the MAIN sub, and then tries to call MAIN with that capture. This has several implications.

The first is that the input syntax has to be context-free. It allows for -foo/--foo (meaning :foo), -/foo/--/foo (meaning :!foo) and -foo=bar/--foo=bar (meaning :foo(val("bar"))). It does not allow traditional unix syntaxes such as -j2 or --jobs 2 as that would require knowing in advance that those two options take an argument.

The second issue with is is that it fails in very confusing ways. To explain this I will give some examples using zef (chosen because of its ubiquity).

$ zef instal
Usage:
  zef [--force|--force-fetch] [--timeout|--fetch-timeout=<Int>] [--degree|--fetch-degree=<Int>] [--update=<Any>] fetch [<identities> ...] -- Download specific distributions
  zef [--force|--force-test] [--timeout|--test-timeout=<Int>] test [<paths> ...] -- Run tests
  zef [--force|--force-build] [--timeout|--build-timeout=<Int>] build [<paths> ...] -- Run Build.pm
  zef [--fetch] [--build] [--test] [--depends] [--test-depends] [--build-depends] [--force] [--force-resolve] [--force-fetch] [--force-extract] [--force-build] [--force-test] [--force-install] [--timeout=<Int>] [--fetch-timeout=<Int>] [--extract-timeout=<Int>] [--build-timeout=<Int>] [--test-timeout=<Int>] [--install-timeout=<Int>] [--degree=<Int>] [--fetch-degree=<Int>] [--test-degree=<Int>] [--dry] [--upgrade] [--deps-only] [--serial] [--contained] [--update=<Any>] [--exclude=<Any>] [--to|--install-to=<Any>] install [<wants> ...] -- Install
  zef [--from|--uninstall-from=<Any>] uninstall [<identities> ...] -- Uninstall
  zef [--wrap=<Int>] [--update=<Any>] search [<terms> ...] -- Get a list of possible distribution candidates for the given terms
  zef [--max=<Int>] [--update=<Any>] [-i|--installed] list [<at> ...] -- A list of available modules from enabled repositories
  zef [--fetch] [--build] [--test] [--depends] [--test-depends] [--build-depends] [--force] [--force-resolve] [--force-fetch] [--force-extract] [--force-build] [--force-test] [--force-install] [--timeout=<Int>] [--fetch-timeout=<Int>] [--extract-timeout=<Int>] [--build-timeout=<Int>] [--test-timeout=<Int>] [--install-timeout=<Int>] [--degree=<Int>] [--fetch-degree=<Int>] [--test-degree=<Int>] [--dry] [--update] [--serial] [--exclude=<Any>] [--to|--install-to=<Any>] upgrade [<identities> ...] -- Upgrade installed distributions (BETA)
  zef [--depends] [--test-depends] [--build-depends] depends <identity> -- View dependencies of a distribution
  zef [--depends] [--test-depends] [--build-depends] rdepends <identity> -- View direct reverse dependencies of a distribution
  zef [--sha1] locate <identity> -- Lookup locally installed distributions by short-name, name-path, or sha1 id
  zef [--update=<Any>] [--wrap=<Int>] info <identity> -- Detailed distribution information
  zef [--open] browse <identity> <url-type> -- Browse a distribution's available support urls (homepage, bugtracker, source)
  zef look <identity> -- Download a single module and change into its directory
  zef [--fetch] [--build] [--test] [--depends] [--test-depends] [--build-depends] [--force] [--force-resolve] [--force-fetch] [--force-extract] [--force-build] [--force-test] [--force-install] [--timeout=<Int>] [--fetch-timeout=<Int>] [--extract-timeout=<Int>] [--build-timeout=<Int>] [--test-timeout=<Int>] [--install-timeout=<Int>] [--degree=<Int>] [--fetch-degree=<Int>] [--test-degree=<Int>] [--update] [--upgrade] [--dry] [--serial] [--exclude=<Any>] [--to|--install-to=<Any>] smoke -- Smoke test
  zef update [<names> ...] -- Update package indexes
  zef [--confirm] nuke [<names> ...] -- Nuke module installations (site, home) and repositories from config (RootDir, StoreDir, TempDir)
  zef [--version] -- Detailed version information
  zef [-h|--help]

The problem here is a simple typo in the word "install", but nothing in this wall of output actually hints at that.

If it can match a subcommand the error message is better/shorter, but still not terribly helpful.

$ zef install
Usage:
  zef [--fetch] [--build] [--test] [--depends] [--test-depends] [--build-depends] [--force] [--force-resolve] [--force-fetch] [--force-extract] [--force-build] [--force-test] [--force-install] [--timeout=<Int>] [--fetch-timeout=<Int>] [--extract-timeout=<Int>] [--build-timeout=<Int>] [--test-timeout=<Int>] [--install-timeout=<Int>] [--degree=<Int>] [--fetch-degree=<Int>] [--test-degree=<Int>] [--dry] [--upgrade] [--deps-only] [--serial] [--contained] [--update=<Any>] [--exclude=<Any>] [--to|--install-to=<Any>] install [<wants> ...] -- Install

Instead of telling us to give a module to install, it lists all the possible arguments for this subcommand (though to be fair, this one is largely zef's fault for making that first MAIN argument mandatory).

The same error message is given for zef install Foo --timeout=3.5 (because a Rat is not an Int) and zef install Foo --timeout 10 --timeout 10 (it passes a two value list instead of an Int to :$timeout).

These error messages are not helpful (and in some cases, it being an error isn't either). The problem here is simple: Raku knows it can't dispatch the capture to any of the MAIN candidates, but it doesn't know why. Figuring out why requires exactly the sort of introspection that it tries to avoid so hard.

But the most confusing way the argument parsing fail has to be the way it handles enums. It will interpret any known enum literal in scope as an enum as a string, e.g.:

zef install True
Cannot resolve caller new(Zef::Identity:U: Bool:D); none of these signatures match:
    (Zef::Identity: Str :$name!, :ver(:$version), :$auth, :$api, :$from, *%_)
    (Zef::Identity: Str $id, *%_)

It's impossible to pass Raku programs using built-in argument parsing any of the strings True, False, Less, More, Same, a bunch of others and any enum literal you've defined in your script as strings because they'll be converted into something else entirely.

The solution

So instead of using the built-in argument parser, I wrote my own argument parsing module: Getopt::Long. Unlike the built-in one, it is contextual. It will first look at the sub MAIN to know what arguments to expect, than parse based on that, and then call MAIN. This way, one can parse -j2 to :j(2), and --jobs 2 to :jobs(2). This results in a far more unixish interface than what is possible using the default parsing.

But there's a second advantage: better error messages. It will try its hardest to give an informative error message. For example:

Unknown option --foo
Option --foo doesn't take arguments
Cannot convert --foo argument "10a" to number: trailing characters after number
Invalid Date '20-02-20' given as --foo argument; use yyyy-mm-dd instead

It tries very hard to detect any potential issue before dispatching is done, so that it can give an informative error message.

This should make it much easier for the user of a Raku program to figure out what they did wrong, while at the same time offering a much more standard interface to your users. It allows for a series of options to make the external interface either more unixish (the default) or more like raku's argument parsing (for backwards compatibility).

Getopt::Long offers both a functional interface (much like the Perl module that inspired it), and a MAIN wrapper that can function as a drop in replacement of the existing argument parser: all you have to do to benefit from this is add a use Getopt::Long; to your script!