DEV Community: Daniil Baturin

How many open source maintainers are there?

Daniil Baturin — Mon, 07 Dec 2020 18:39:09 +0000

There's a highly interesting post by Ben Frederickson, titled Why GitHub won't help you with hiring.

It makes some very good arguments against requiring a strong GitHub profile for job applicant screening. The main argument is that most great developers don't have one. I think the best argument against it is that unless you hire people to work on open source code, requiring them to have a track record of contributions to open source software is hypocritical.

However, it's not my point. My point is that his analysis is hinting at the true size of the open source developer community, and it's not as large as it seems.

From the GitHub archive data, he got these figures:

Only 17% of GitHub users pushed any code in the last year
Only 7.4% of GitHub users pushed more than 10 times
Only 1.4% of GitHub users pushed more than 100 times
Only 0.15% of GitHub users pushed more than 500 times

As of 2020, GitHub is the largest code hosting platform with about 40M users.

Now, let's convert Ben's relative numbers to absolute values:

Only 560 000 people pushed more than 100 times.
Only 2 960 000 people pushed more than 10 times.

So, if you somehow bring them all together, the community of people who ever made some code dumps or drive-by contributions is about as large as the populations of Amsterdam, Athens, or Chicago.

The people who actively maintain projects could fill a city like Dresden or Las Vegas.

That's compared to 40M registered accounts. It's hard to tell how many of those accounts are duplicates, or how many programmers are in the world.

Now, let's make an incredibly conservative estimate: most accounts are duplicates and there are 10M programmers in the world. It means only one in three ever helped an open source projects, and one in five maintains a project of some kind.

And that's when almost every programmer depends on at least some open source code for their daily work.

I think this leaves no open questions about whether companies should allow employees work on open source in their paid time, or whether we should advocate for participation. The answers to all those questions is "yes".

Is "written in $lang" a feature or an implementation detail?

Daniil Baturin — Sun, 06 Dec 2020 08:45:26 +0000

Some keep arguing that "written in $someLanguage" is a minor implementation detail that should not matter to users. This can only possibly be true for applications (as opposed to libraries). However, is it actually true?

On the surface is sounds plausible: you can write good software in any language, so if you aren't going to modify the program, it shouldn't matter.

However, in practice there may be very valid reasons.

Performance and startup time

It's no longer true that "interpreted" languages are orders of magnitude slower than ones that are compiled to native code. JIT compilation and modern VM designs can make them almost as fast to run as native code.

However, those improvements have done almost nothing to the startup time. Interpreter/runtime startup cost still ranges from "non-trivial" to "atrocious".

For web backend and for most desktop apps, startup time isn't very important. For CLI tools and programs designed as building blocks for scripting (like jq) it may be crucial.

Safety guarantees

For some use cases, memory safety provided by a garbage collector or a borrow checker is important enough to sacrifice something else for it. Security-sensitive public-facing services can benefit from a rewrite in a memory-safe language just because that way you know that a next Heartbleed is not lurking in your codebase.

For a larger set of use cases, "does not segfault" is a useful property too, since it's easier to avoid data corruption in that case.

Ease of installation and maintenance

If a program is written in Go, you know it's most likely self-contained and statically linked. You also know that it doesn't support plugins or live patching.

If it's in Python, you know that special effort is required to create a self-contained package. However, you also know that you can load code run time, or edit a program on disk.

Platform support also varies. If your target platform is an IBM mainframe, and a program you want is written in a language that only has an interpreter for x86 and ARM so far, you know you are out of luck.

Conclusion

"Written in $language" can be a valuable piece of information at the very least.

It's not to say that there are no projects whose only "advantage" over predecessors is different language—not even always a suitable one. But dismissing the language argument uncritically can be equally silly.

TOML first class dates is a mistake

Daniil Baturin — Sat, 05 Dec 2020 14:37:06 +0000

Making something a first-class entity may seem like an unquestinably good idea, but it's not always the case. Sometimes it creates more problems than it solves.

I fear TOML's first-class dates is one of those cases. Here's why.

Behaviour is ambiguous

The TOML spec does a good job at specifying acceptable date and time formats. However, it doesn't say what implementations are supposed to do with them.

What it says:

To unambiguously represent a specific instant in time, you may use an RFC 3339 formatted date-time with offset.

RFC3339 dates indeed specify a specific instant in time in the past. They aren't good for things like "an offer valid until 2038-01-19 in Nebraska" because the UTC offset of the Nebraska time zone isn't guaranteed to stay the same until 2038. Timezone changes aren't even that rare and haven't all occured decades ago: for example, Brazil stopped using DST in 2019.

But, that's not really my point. My point is that the spec doesn't say whether the result of deserializing a TOML date should be a timezone-aware object or converting it to a UNIX timestamp is acceptable. In practice... it varies.

Why this matters? If an implementation doesn't preserve timezone information (which it isn't required to do), then dump(load(toml)) is a lossy operation. It can be quite a serious issue for tools that automatically manipulate TOML files.

There's no natural datetime type choice for every language

Some languages, like Python or Go have datetime types and functions in their standard library. Whether a "batteries included" is a good approach or not is debatable.

What isn't debatable is that other languages like Rust or OCaml do not have it built-in—they leave choice of a datetime library to use developer.

However, it means that a TOML parsing library will have to either force a datatime library choice on developers who want to use it, or provide an abstraction mechanism.

Conclusion

All in all, I think TOML would really be better off without first-class dates, which aren't even used all that often in the wild.

General relativity vs complex analysis

Daniil Baturin — Mon, 30 Nov 2020 08:47:08 +0000

There are two distinct categories of scientific discoveries and inventions: facts and methods.

The great thing about "fact" discoveries is that they are often easy to give a one-line, catchy summary.

If someone asks what's the big discovery of the general relativity, you can say "it's that massive bodies bend the spacetime around them". It will be a correcty, even if simplistic explanation. The best thing is that it can even be confirmed visually, with pictures of stars during a total solar eclipse and spectacular effects of gravitational lensing.

There's another class of discoveries that can be described as "methods". Complex analysis is a great example.

High school students are sometimes taught complex numbers. Or rather a set of meaningless rules for manipulating imaginary numbers. Let's pretend there's a number i such that i^2 = -1. Very funny game, isn't it?

The real big idea of complex analysis sounds vague and ambiguous to the uninitiated: you can discover properties of real functions by studying their complex-valued versions.

You can classify singularities using a complex-valued equivalent of Taylor series, calculate contour integrals using just one point and find the factorial of -3/2.

It also takes a few months of study to understand the power of complex analysis.

What does it have to do with programming? Technically, in computer science and software engineering, every discovery is a "method".

However, some discoveries are much closer to facts. For example, the Dijkstra's algorithm is a fact: there's a quick way to find shortest paths in a graph. You can even make an animated demonstration.

Many programming techniques, however, are exactly methods—not unlike complex analysis or group theory. Higher order functions, dependent types, formal verification and so on: they are useful and important, but don't allow for catchy and correct descriptions.

Even relatively simple concepts like the World Wide Web can fall into this category. I'm pretty sure in 1990, a description like "it allows everyone to publish documents online and link to any other document" prompted a lot of "so what?" reactions.
You have to see the Web in its advanced state to realize that it enables completely different usage patterns than any closed hypertext environment.

Thus, if you see a description that seems vague, it may not be a communication failure. Maybe its very nature makes a consice description impossible.

How to tell if it's the case, or if a proposed method is any good (or if it even works) is another question.

How to make an informative project README

Daniil Baturin — Fri, 27 Nov 2020 08:10:41 +0000

There are lots of projects with rudimentary READMEs. It's sad because many of those are good, promising projects, but they make it hard for prospective users to assess them and try them out.
There are also people who advocate flashy READMEs with images, animated GIFs and lots of shields. Being flashy isn't always a bad idea by itself, but a README must be informative first.

Here's my checklist for making READMEs informative.

Add usage examples

A bunch of examples cannot replace proper reference documentation, but reference documentation cannot replace introductory examples either.

As a potential user, I want to see what is it like to use your project. For a GUI tool, screenshot is an obvious way to show what it looks like. Libraries and CLI tools benefit from it even more though.

For libraries, show a self-contained program

When I'm looking for libraries, I find a demonstration in a REPL or a small self-contained program very helpful.

If your library is hard to demo that way, maybe the library itself needs work. For example, when I wanted to sort dependencies, one thing I found annoying about existing graph manipulation libraries is how much initialization they need, when I just wanted to sort dependencies.

So one of my goals for ocaml-tsort was to make it easy to show in a one-line example:

# #require "tsort";;

# Tsort.sort [
  ("roof", ["walls"]);
  ("foundation", []);
  ("walls", ["foundation"])]
;;
- : string Tsort.sort_result = Tsort.Sorted ["foundation"; "walls"; "roof"]

Some projects, like MVC frameworks, are genuinely difficult to showcase in a README. In that case a "starter project" repo can be a good alternative.

For CLI tools, show output or use animated demos

CLI tools can benefit from some examples with output. Man pages usually just give you command examples with descriptions, but if you can show how the tool changes its input data, that's much better.

Tools that work with text are usually easy to showcase. Suppose you were to tell a novice about sed.

sed is a tool for non-interactive text editing.

Example: replace every occurence of "hello" with "hi".

$ echo "hello world" | sed -e 's/hello/hi/g'
hi world

If your tool works with files and directories, you can show before/after tree outputs.

Interactive tools can be difficult to showcase, but if you aren't limited to plain text, you can do it with an animated GIF or a video. I love how thefuck did it, for example.

Add build/installation instructions

Even if it's just the "usual procedure", say what that procedure is!

One reason is that a user may be coming from another language and isn't familiar with your language and ecosystem convention.
C programmers who suddenly need to install a JS application may have never encountered npm before, while JS programmers may not know that ./configure && make && make install is a usual procedure for a C project.

Another reason is to reassure users that there's no unusual procedure. I see ./configure && make && make install and I know that I will not need to install a fashionable build tool of the week and purchase black candles to build your project.

You should make sure that your stated procedure actually works from a clean repo state. For example, some users of GNU autotools don't include autogenerated configure scripts and makefiles in the git repos and don't tell the user to run autoreconf -i in their README.
Then users who never created their own autotools projects see "Run ./configure" and wonder how to run it if there's no ./configure to begin with.
You can avoid that confusion with just one additional line:

$ autoreconf -i
$ ./configure && make

Specify your dependencies

Some people just cannot use latest versions of everything, no matter how much they want it. There are many reasons people may not be able to "just" upgrade.

For those people, it's really important to know if they can even use your project at all. So, if your project needs Python 3.7, then people stuck with 3.6 will know they cannot use it.

If you don't know your minimum requirements, then you should find out.

Specify expected level of maintenance and community interaction

Every open source license comes with a clause that says there's no warranty. We all understand that in practice, that situation is very different for, say, the Linux kernel and a script I cobbled together for a one-off task and uploaded in hope someone else some day finds it useful.

Sadly there's no commonly accepted way to specify how much effort you intend to put into handling bug reports and interacting with the community.

There's a "no maintenance intended" shield meant for completely abandoned projects.

Some informal statements may help, like "I'm using this in production and I'm ready to fix bugs" or "I'm not planning to add support for use cases I don't have, but I'm ready to merge your patches".

Make sure you actually follow your own policy of course. If your README says "Bug reports and contributions are welcome", but in reality you auto-close them for "inactivity", you need to reconsider it.

Tidy up your GitHub account

Daniil Baturin — Thu, 19 Nov 2020 09:06:21 +0000

End of the year is a designated special time. One of the things I try to do is cleaning up my GitHub account. Here's my checklist.

Archive inactive repositories

Some people advocate deleting "unneeded" repositories. That advice was in the widely popular GitHub is your resume now post for example, and many repeated it since then.

However, if you aren't actively working on something now and/or see it as "low value", it doesn't mean you will never work on it again, or that no one is using it.

You may want to mark then as inactive by archiving them instead. When you archive a repository, it's made read-only, and visitors see a warning that it was archived by the owner. You can un-archive it at any time later.

...or make them private.

Now that GitHub allows creating private repositories for free, you may want to make use of it too. If you want to keep an old project for the reference or nostalgia, but don't want it to "pollute" your GitHub profile, you can hide it from the public view instead.

Update your pinned repositories

Whether peers and recruiters should treat one's GitHub account as a resume and portfolio substitute or not is a serious question. The truth is that many people do treat it as such, so how it looks to them is a real concern.

When the (in)famous "GitHub is your resume now" post was written, GitHub would only show your most popular repositories on the profile page.

Luckily, GitHub developers added more ways to customize your profile page. Now you can pin up to six repos on the profile page. This makes having a lot of repos much less of a problem.

If that isn't enough, you can also create a profile README by creating a repository that matches your username and adding a README.md file to it.

No need to resort to book^W repository burning to make your profile serve as a better resume/portfolio page.

Delete genuinely unused forks

There's often a tendency to keep forks of other people's repos for way too long after your pull requests are merged.

If you aren't working on a repo fork, it may really be better to delete it. You can always fork it again.

If you are concerned about repo's fate, like it happened with youtube-dl lately, it's better to maintain mirrors on independent code hosting services that are concerned with digital freedom and privacy, like Codeberg or SourceHut.

Walk through your pull requests

We all want things to move fast, but it's still a fact of life: sometimes pull requests linger for months or even years.

And sometimes it's we who forgot to merge PRs, or make changes when repo owners request them.

If you are tidying up your account, it's a good idea to walk through both incoming and outgoing PRs. Merge or reject incoming PRs. If your own PRs don't get any attention, ping the repo owner. If they want you to change something, make the change.

TOML: the annoying parts

Daniil Baturin — Wed, 18 Nov 2020 01:57:37 +0000

If your project needs a configuration file, TOML is usually the best format.
It's definitely easier to read and write than JSON, and much less fragile and ambiguous than YAML. There are comments, real booleans, and multi-line string literals.
It also offers better flexibility than classic INI-like formats, and there's a spec where users can learn the syntax.

I went for TOML when I wrote the Soupault static site generator/HTML processor, and in these 1.5 years that I've been using it and working on it, I never thought "I wish I'd chosen something else". None of the users told me they wish it used something else either.
However, there are still annoying points that I hope will be improved some day.

Paste-ability

Don't we all like to paste configuration snippets from tutorials and other people's configs?

When nested tables are involved, TOML doesn't make it especially easy to paste, or even discuss configuration options.

Consider this config:

[settings]
  verbose  = true
  debug    = false

How are we going to talk about those options in the docs?
Let's try this: "When settings.verbose = true, build progress output will be printed to the console".
Users cannot paste settings.verbose = true into that config because TOML doesn't allow re-defining options.

This is an invalid config because the verbose option is re-defined:

settings.verbose = true

[settings]
  verbose  = false
  debug = false

It gets funnier. This is technically valid TOML:

[settings]
  verbose  = false
  debug = false

settings.verbose = true

It just doesn't mean what one may think it means: the last option is interpeted as settings.settings.verbose.

So, best you can do when talking about options is to use lengthy descriptions like "Option verbose in the [settings] table".
Well, or avoid using tables, but it defeats the purpose.

If I were designing a format from scratch, I would likely avoid having both [table]\n key and table.key notations because they clearly don't compose.

Unordered tables

Sometimes I wish the standard mandated that tables were ordered dicts. If you need order, you can only use lists of tables (and their syntax and behaviour are rather clumsy and not obvious), or provide a different way to specify the order.

When I wanted to make it possible to use output of one "widget" as an input for another, I chose to add an explicit after option.

# Highlights the link to the current section in the navigation menu
[widgets.section-link-highlight]
  after = "insert-nav-menu"
  widget = "section-link-highlight"

  active_link_class = "nav-active"
  selector = "div#nav"

Maybe the fact that users can specify multiple dependencies with after = ["widget-1", "widget-2"] is a good thing, but it may also make complex configs hard to follow.

I suppose one reason TOML designers made tables unordered is that it's how hash tables/dicts work in most languages. In Python you have OrderedDict, and in OCaml or Haskell you can use a (string × α) list. There's no equally common and straightforward way to do that in JS or many other languages.

Automatic manipulation

Unordered tables and "more than one way to do it" mean parsing and formatting TOML objects isn't a lossless operation. None of existing libraries can parse a TOML file and print it back as it came.

It's hard to blame them. Preserving comments, line numbers, and whether a table was normal or inline for example—these are all pretty hard tasks. They are also unnecessary if you only want to de/serialize TOML.

If you are writing a validator or convertor, then things are different.

When I had to make some breaking changes to the config syntax, I made a convertor from the old format to new to simplify migration for users.

However, what it outputs is quite far from the config that comes in. You'll lose all your comments at the very least.
I tried to alleviate it by making the convertor output a detailed log of the changes it's making so that users can replicate them by hand.

Conclusion

I don't regret going with TOML for the configuration format. There's still room for improvement in its tooling and usage conventions though, and I hope that I or someone else will be able to make those improvements.

HTML5 details element: equally great and disappointing

Daniil Baturin — Mon, 16 Nov 2020 01:15:05 +0000

I love making websites that work without JS. That's why I also love all web standards that make it easier.

HTML5 <details> element promises to make collapsible blocks and tree views first class entities in web pages. All in all it feels like a great success—with a few frustrating shortcomings when it comes to styling.

Compatibility and graceful degradation

It works in every browser released in 2017 and later.

Since web browsers simply ignore unknown elements, it also degrades gracefully: the content of <details> is just shown by default.

Styling

One issue is that people who aren't well-familiar with this element often don't realize that the element is clickable and that the triangle is a part of the "button".

It's not surprising, since that element breaks the desktop convention. You can alleviate it with a bit of CSS though.

summary { cursor: pointer; }

However, I'm still surprised why it's not the default.

Styling the button itself

What's more surprising is that there's no easy, or even standard way to style the "expand" button itself.

This feels especially weird. If HTML form elements can be styled in arbitrary ways, why should <summary> be an exception?

In any case, the standard doesn't define that, and different browsers do it differently.

In Firefox, you can easily style the button with list-style-type or list-style-image properties. Sadly, this only works in Firefox.

In Chrome and Edge you can only disable that button with this pseudo-element rule:

summary::-webkit-details-marker { display: none; }

Then you can insert a symbol or an image with a ::before selector.

So, the only somewhat cross-browser way is to disable its display with list-style-type: none (for Firefox) and with above rule for Chrome, then insert your own "button" one way or another.

Still even that won't work in Opera and Safari.

Accessibility

That's the frequently overlooked part! There are some impressive hacks that emulate collapsible elements with heavily-styled radio buttons and CSS, but it's a nightmare for screen reader users because the semantics of it all are opaque.

With <details> and <summary>, user agents know what it is and can present it in an appropriate way in each medium.

Multi-line string literals

Daniil Baturin — Mon, 09 Nov 2020 21:52:47 +0000

Python

I suppose most people know that Python allows writing multi-line string literals in triple quotes. This feature relatively hard to miss because it's used in docstrings.

def hello():
  """ Prints a greeting """
  print("hello world")

Using it for variables is also common.

s = '''
hello
world
'''

print(s)

All in all, it's an easily discoverable feature.

Lua

It's less widely known that Lua supports multi-line strings because it uses unusual delimiters: double square brackets.

s = [[
hello
world
]]

print(s)

Let's see if it works:

$ lua ./test.lua 
hello
world

Once you remember the delimiter, it's not harder to use than Python's triple quotes.

This syntax is also used by the TOML serialization format.

OCaml

OCaml's approach is the easiest of all. However, it's also the least easily discoverable.

To write a multi-line string, you... need not do anything special. You can use add line breaks to normal strings and the lexer will understand them.

let s = "
hello
world
"

let () = Printf.printf "%s" s

Let's test it:

 ocamlopt -o test ./test.ml 

$ ./test 

hello
world

Can we make searching for Hacktoberfest projects to contribute easier next year?

Daniil Baturin — Wed, 04 Nov 2020 19:01:39 +0000

It was the first time I participated. I've heard something about the Hacktobefest before, but never paid attention—not least because contributing to various projects is a part of my daily work and life and I never looked for events encouraging people to contribute.

I have to admit the PR spam drama was what got me curious enough to actually take a look. Initially I thought I'll use the challenge to make some PRs I was going to make but kept leaving for later.

Then came the rules change. I thought if I registered for that thing anyway, I can as well use it as a challenge to find new interesting projects. Indeed, I discovered TL;DR pages and added some OCaml tools to their collection.

It may actually be good that Hacktoberfest 2020 essentially forced people to discover new projects if they want to participate.
However, I found searching for projects much more difficult than it should be. The /topics/hacktoberfest page doesn't provide an obvious way to search for more than one topic. I hope there is a way to do that, but I couldn't find it it.
Without an easy way to customize the search query, all you can do is to "load more" and hope for something that interests you.

If next year's Hacktoberfest will also be opt-in, I hope searching for projects will be made easier.

If you like a project, test its development versions

Daniil Baturin — Wed, 04 Nov 2020 13:06:50 +0000

Bugs in open source projects are often highly visible and create a lot of trouble for maintainers and users alike.

Bugs in libraries can require both upstream and downstream maintainers make difficult decisions, too. Suppose an unintended breaking change is found in libfoo 1.5.0. Now maintainers not only need to fix that bug, they also need to decide whether to try "un-releasing" the broken version, how to notify the users of the problem and so on.

Myself I always test betas and code from the master branch whenever I can, with projects I use often. This behaviour is not entirely altruistic: I know that if a bug makes it into a released version before I see it, I will be in trouble as much as the maintainers.
As an open source maintainer, I also know that my downstream users will be in trouble if my dependencies have bugs.

It can be hard to find time for testing. However, I often see that people underestimate the importance of it. They think it wouldn't make a difference and they aren't good testers.

In reality, no one is a bad tester. Any testing is important, and every maintainer will (at least, should) appreciate any testing effort.

Here are some common misconeptions.

It works for me, thus the maintainers already know it works

They may know that it works for them. They have no way to know that it works for you until you tell them.

It's very nice and reassuring to hear from users that a new feature works for them as expected.

Someone else is already testing it

There's often a bizzare situation when everyone thinks someone else is already doing it, only to find out that no one did.

Even if someone is really testing it already, doesn't mean you should not. Chances are what they are doing isn't exactly the same, or their environment isn't exactly the same as yours.

Many bugs require specific environment settings to trigger: certain hardware, combination of options, dependency versions or something else. Even the largest and well-funded projects don't have resources to test every combination, so community help is required to catch and reproduce them.

The project has automated tests

Unit tests are great, but they can only test functions in isolation. Even automated integration/acceptance tests only catch problems that developers know may happen. Nothing catches the unknowns.

Even formal methods can only prove that the code matches developers' assumptions.

While some bugs are indeed simply programming errors, many more are caused by missing or incorrect assumptions about the ways the code interacts with the environment, or the ways different parts of the code interact with one another.

My workflow is so ordinary, how can my testing help?

You may think your setup and workflow is completely ordinary, and you need to be doing something special, outside of your normal workflow, to be a useful tester.

It isn't true! First, your workflow is almost surely different from that of the developers at least in some ways. Second, you have an excellent intuition for the right and wrong, and can spot even slightly odd behaviour easily.

Any maintainer will appreciate a dedicated tester who goes to great lengths to try the code in unusual settings, test uncommon paths, and push the code to its extremes.

However, even just doing what you are always doing, just with a development version instead of a stable one, and telling developers that it works for you is already valuable.