DEV Community: Mathieu Ferment

Beyond processes

Mathieu Ferment — Sat, 28 Jun 2025 20:27:20 +0000

4 years ago I became an engineering manager.

When I started, I thought processes were the solution to everything. One problem? Design the right process - it won't happen again. Bad work? One process will fix it.

Well, you can guess, it didn't work.

For a process to be effective, it requires constant monitoring. People need to understand it, adopt it, and follow it. It is never perfect. You will always be fine-tuning it. And it is expensive - in time, in energy, in attention.

Also, beware of overreacting. If every problem leads to a new process, soon you're buried under a mountain of them. Instead of helping, they slow you down and it becomes hard to keep track of all of them.

Today, I have a different view on process and how to use it. Before explaining I'd like to ask the question: but what is actually a process?

What is a process?

Let's look at a concrete example.

Imagine a 100-person company with one HR manager. Employees want to take vacation, so they call him, email him, drop by his desk... He's drowning in requests. Context-switching constantly. Struggling to track anything.

So he creates a form: "This is a form that I need you to fill out if you want to request time off." That is great, it centralizes all requests, he can transfer them to a data structure or a software! He designed a process.

For me, this is a process: a goal, a method, someone who applies the method and someone who benefits from the goal. For the above example:

Goal: Save HR's time
Method: Fill the form
Who applies it: The employees
Who benefits: The HR manager

But here's the kick: the people who apply the process are not always the ones who benefit from it. The employees pay the cost - less flexibility, more effort - while the HR manager benefits from it.

This asymmetry can make some processes disliked. You could say one pays the price, the other gains it, so this is a fertile situation to create frustration. Yet the alternative - the previous mess for the HR manager — is also unacceptable.

Here is another example.

In a small software startup, bugs are leaking into production. The developers decide that every change must be reviewed. They actually "invented" code review.

Goal: Fewer bugs
Method: Have the code written by someone reviewed by someone else
Who applies it: the developer team
Who benefits: the developer team

Same group, different mindset. They see the pain of bugs, so they value the process. They're more likely to adopt it because they feel the benefits directly.

One thing to be aware of is that most of the processes have a cost, the one paid by applying the method.

It could be time. Money. Complexity. Less flexibility.
Can be small, can be big, but it’s almost always there.

In the first example with the HR form, the cost is paid by employees:

if they are not sure how to fill the form, they cannot rely on the HR manager help (or they must reach for it) unlike before: less user-friendly
the form has a set structure that could not accommodate some cases: less flexibility

When designing a process, you must weigh the cost versus benefit. And when applying a process, that equation still matters.

Spectrum of process mindset

In a company, one could find a range of attitudes regarding processes:

On one end, people who dislike processes: they see mostly the cost and quite often are the ones applying the method (hence paying the cost) but not benefitting from the goal. Sometimes this is because the process is poorly designed, so these people have valid reasons to dislike it. But there is also people who, no matter the process, simply do not want to do something that feels slow or pointless.

On the other end, people who follow every rule, every time. They value the outcomes - less bugs, fewer mistakes - and in order to maximizing the chances of achieving it, they do not want to risk skipping steps.

I find myself in the middle.

My approach to processes

When I see a process that I have to apply I always try to remember the goal, the method, who applies it and who benefits it. It then helps me answer the question "should I apply it or not?". And asking myself, "If I don't apply the process, are we still achieving the goal?" usually gives me the answer.

If yes, I may choose to skip the process - and save the cost.
If not, I follow it - and pay the cost aware of the benefits.

If I reuse the second example with the startup: the developer team has now decided that every Pull Request must be reviewed by two other people.

Then someone fixes a typo in the README and submits a PR for it.

Do you really need two reviewers? Probably not: the goal of the process is to catch bugs and code issues - but here, there is no code. No risk. You can skip the method and still hit the goal.

But now imagine the PR modifies a config file that affects payment processing. The change looks simple: one line... But skipping review here could be risky.

In this case, applying the process is worth it as the cost is justified by the potential impact.

Going beyond the process

I remember reading a book about SCRUM and learning about the Shuhari.

Quote from the Wikipedia page:

"It is known that, when we learn or train in something, we pass through the stages of shu, ha, and ri. These stages are explained as follows. In shu, we repeat the forms and discipline ourselves so that our bodies absorb the forms that our forebears created. We remain faithful to these forms with no deviation. Next, in the stage of ha, once we have disciplined ourselves to acquire the forms and movements, we make innovations. In this process the forms may be broken and discarded. Finally, in ri, we completely depart from the forms, open the door to creative technique, and arrive in a place where we act in accordance with what our heart/mind desires, unhindered while not overstepping laws."

The book was using the Shuhari to explain why it is important to stick tightly to the SCRUM book when a team starts applying it because experiencing it is important to understand what it brings, what it changes, what it costs. And only after they realize what SCRUM brings to the table should a team start deviating from it and define its own process.

I think we can apply Shuhari to processes. Not familiar with a situation or a topic? Stick to the process: it was designed by people who understood the situation better than you, you should be safe applying it. But once you've understood the stakes - the goal, the method, who applies it, and who benefits - then you can deviate... and choose to apply the process fully, partially or not at all.

Time or features? A false project dilemna

Mathieu Ferment — Sat, 08 Mar 2025 20:11:31 +0000

The Scope Triangle

As a developer you have probably seen this figure before:

This is the Scope Triangle, which highlights the relationship when building a (software) project between time, scope and cost. I'm not going to explain it, I'm sure you saw it before (and if you did not here is the Wikipedia page 😉).

One of the statements we get from this figure is "The Scope Triangle means choosing two fixed constraints and adjusting the third". It sound wise, but does it really work?

Following the above statement, you've probably heard or even said something like this before: 'Do you want it fast or do you want it good?' or 'Pick two: speed, quality, or cost.'

Unfortunately things are rarely this simple. Actually all three constraints are flexible to some extent. This is why, when developers ask their boss to choose between two, they often hear... 'Both.' 😅 And it's frustrating! But it reflects how decisions are often made in software companies. Here is why.

The house buying dilemma

Imagine a couple looking to buy their first home. They meet with a real estate agent and explain their priorities:

Budget: "The most important thing is that we don’t overspend"
Quality: "But we also want a nice and cosy house"
Timing: "And of course, we want to move in quickly"

At first, the agent shows them houses within their budget. But these houses are small, outdated, or in inconvenient locations. They realize that sticking to their budget means giving up on quality and location.

So they adjust their expectations: "Actually, house quality is the most important." The agent then shows them nicer homes but they're far more expensive than expected. Now they’re stuck again. If they prioritize quality, they either need to spend more or wait longer to save up.

Eventually, they reach a conclusion: what they really want is a reasonable compromise. None of the constraints is truly fixed, but at the same none of the constraints can altered too much. They’re willing to increase their budget slightly if they find a house that meets most of their expectations, and they’re willing to adjust their quality expectations if it means staying within their means. They don’t want to fully sacrifice one item: budget, quality, or timing ; they just search for the right balance.

Same goes for scope, time, and cost

Same applies for software development works. A customer might say, "We have a strict deadline and we need to deliver 100% of the features" . This can sound unrealistic and you could be tempted to say "please choose one" but the truth is, the customer is looking for the best possible compromise.

Instead of forcing a choice between two constraints, developers should focus on understanding priorities and finding the right balance. The deadline might actually be pushed a little, or the features could be delivered in 2 batches? If you can demonstrate that the outcome will be good, you will be able to get some flexibility from the customer. But remember: he won't fully sacrifice one.

Back to the Scope Triangle

"The Scope Triangle means choosing two fixed constraints and adjusting the third" - well actually no. In my opinion the triangle shows something else: the three constraints are equally important for the customer or the boss. Nobody sane will sacrifice completely one. Everybody will be willing to find the right adjustment, the right cursor, which provides the best outcome. So try to understand what the outcome should look like for your customer/boss and you'll know what you can change.

One possible evolution of software development industry thanks to LLM

Mathieu Ferment — Tue, 23 Jul 2024 16:42:14 +0000

Since ChatGPT launch in 2022, we have been bombarded by statements and opinions regarding the future of work being altered by AI, software development included.

It is very hard to sort the crazy from the likely in this tsunami of predictions. Today I would like to add my own to this tsunami 😄 , based on previous evolutions of our industry.

ChatGPT revolutions

First of all I think it's important to clarify what text-based LLM can do and what they cannot. I'll focus on text-based LLM and ignore other LLM like Dall-E because we're interested in AI writing code. I'll refer to LLM by using ChatGPT as an example, to simplify things, but you can replace ChatGPT in my blabbering with any text-based LLM.

In my opinion, ChatGPT and its friends have brought in this world two amazing features.

The first feature is ChatGPT being able to generate relevant text. It's the most obvious one because this is the one that made it famous. After all we're referring to them as "generative AI" so it is natural we focus on their ability to generate text.

ChatGPT is capable of generating huge amounts of text, most of the time very relevant, from a given prompt. We've seen it writing books, speeches, discussions, and obviously some programming code, code that can be run. In case you don't know, how ChatGPT generates text is actually it making a prediction of what the next word should be. So if you ask ChatGPT to complete the sentence "Happy new ..." it'll probably complete it with "year" because out-of-context, the most likely word to be written after "Happy new ..." is "year". And this, ChatGPT did learn it from reading and consuming billions and billions of documents scattered all over the Internet. When it generates text, ChatGPT is "simply" (I know, I know, there's nothing simple in that but you get what I mean) using probabilities. It outputs the most probable word, one after another. If someone asks him to solve "2+2 = ?" it will write 4 not because it has computed the result but because, thanks to it reading millions of documents about mathematics, it knows the most likely answer is 4. That is how ChatGPT works: it writes text which has the highest probability to be true / relevant / what you wanted. But it's doing it in a very smart way.

That's for the first feature. But for me the second feature is 10 times more impressive.

The second feature brought by ChatGPT is its ability to infer meaning from text. For example, if I give ChatGPT the following prompt "I'm in Paris and I want to visit it, what should I do?" ChatGPT is actually able to "understand" what I said even though I was very brief. It can "understand" (this word is not accurate because ChatGPT does not perform understanding operations) that my current location is Paris city, France, that I want to tour the city, and that I am looking for places / things to see. And to achieve this it broke my sentence down into small tokens and then matched them to its gigantic database to find out the relationship between them and finally understand what I was trying to say. And it was able to link "I do" with "visit it" because it has read the entire Internet, and he has already seen these tokens or equivalent used together. ChatGPT is learning by imitation: it knows "yes" and "no" are opposite because it has read documents where the two were used in opposite ways.

In my opinion this is the real game-changer. To generate relevant text output using probabilities is something we've been doing for years, ChatGPT is simply doing it at a level never seen before. But its ability to infer meaning to sentences not even written in a specific manner is a lot more important. We had text recognition systems before, but they were limited. ChatGPT has outmatched all of them. It can even understand sentences with grammar issues in it, something a "standard" text recognition system is not able to because it was built to recognize well written text, not gibberish.

I will come back to how this ability is a game-changer later.

ChatGPT writing code

Now let's compare two scenarii:

in the first scenario, a skilled PHP developer will be tasked to write a PHP code script that does fetch the top ten contributors from a GitHub repository
in the second scenario, ChatGPT is tasked to do the same

In the first scenario, let's assume this developer never used GitHub API before. So he's going to look at the API documentation, read it, find the endpoints he can use, understand the expected payload format and response format, and then finally write the code to parse the answer from GitHub to extract the data he is looking for. It will take him between 10 and 30 minutes to write this:

<?php
require 'vendor/autoload.php';

use GuzzleHttp\Client;
use GuzzleHttp\Exception\RequestException;

function getTopContributors($owner, $repo, $token = null) {
    $client = new Client([
        'base_uri' => 'https://api.github.com/',
        'headers' => [
            'User-Agent' => 'PHP Guzzle Client',
            'Accept'     => 'application/vnd.github.v3+json',
        ]
    ]);

    if ($token) {
        $client = new Client([
            'base_uri' => 'https://api.github.com/',
            'headers' => [
                'User-Agent' => 'PHP Guzzle Client',
                'Accept'     => 'application/vnd.github.v3+json',
                'Authorization' => 'token ' . $token,
            ]
        ]);
    }

    try {
        $response = $client->request('GET', "repos/$owner/$repo/contributors", [
            'query' => ['per_page' => 10]
        ]);

        $statusCode = $response->getStatusCode();
        $body = $response->getBody();
        $contributors = json_decode($body, true);

        return $contributors;

    } catch (RequestException $e) {
        if ($e->hasResponse()) {
            $errorResponse = $e->getResponse();
            $errorStatusCode = $errorResponse->getStatusCode();
            $errorBody = $errorResponse->getBody();
            $errorMessage = json_decode($errorBody, true);
            return [
                'error' => true,
                'status' => $errorStatusCode,
                'message' => $errorMessage['message'] ?? 'An error occurred'
            ];
        } else {
            return [
                'error' => true,
                'message' => $e->getMessage()
            ];
        }
    }
}

// Example usage
$owner = 'octocat';
$repo = 'Hello-World';
$token = 'your_github_personal_access_token';

$contributors = getTopContributors($owner, $repo, $token);

if (isset($contributors['error'])) {
    echo "Error: " . $contributors['message'] . "\n";
} else {
    echo "Top 10 Contributors:\n";
    foreach ($contributors as $contributor) {
        echo $contributor['login'] . " (" . $contributor['contributions'] . " contributions)\n";
    }
}

In the second scenario, ChatGPT is given the same assignment and prompt the exact above solution in 10 seconds.

Is it code or..?

Sounds impressive, right? 10 seconds instead of 30 minutes? But let's not pretend otherwise: we all know here 😄 on dev.to that this simple task is not representative of a developer's work. Most developers don't get to write code from scratch everyday, most developer missions have a lot more context and constraints, so we all know it was a very simple scenario and ChatGPT being 60 times faster than a human on this case means nothing regarding ChatGPT ability to replace developers.

That is not what interests me here. Let's look at the code that was written.

In this code, some PHP code has been written, it uses the popular PHP HttpClient Guzzle, and it also uses PHP built-in JSON parsing function json_decode() to parse the output before writing the desired answer.

In the first scenario, did the developer really write some "PHP code"... or has he been rather using multiple things and gluing them together to obtain what he needed?

What I mean is that in the first scenario, the developer did not write code to send raw HTTP requests. He also did not write code a computer can understand: he wrote PHP code that was compiled down to bytecode before being interpreted by the Zend engine. And he did not write the logic to transform a JSON string into PHP-usable data: he used PHP built-in JSON parsing capabilities. In fact the code he wrote is mostly acting as a carrier, using one piece, fetching the output, transforming it a little, then passing it to another piece that returns another output that is then given to another piece for another operation before finally outputting something. This is what I call using multiple things and gluing them together to obtain what I need and... that is what most developers do today.

Unless you're working in a specific industry, chances are high that everyday the code you are writing is glue. Glue that uses code and systems (frameworks, libraries, applications...) written by other developers, that you can leverage to build something else. And your code is mostly passing things in and out from these multiple systems to obtain the behavior you wanted to achieve for the end-user. This is typical of a modern code project.

So ChatGPT has not written code either. What he did was glue multiple things in order to obtain what I asked him for, exactly like the human developer.

The evolution of code

Alan Turing created one of the first computers in 1947, and it ran punch cards. Punch card were used as data storage for its computer. Can you imagine how tedious it must have been to write a program using punch cards? You had to translate what you wanted to achieve into operations that could be implemented by the right holes in a card. A very different method that the ones we use today.

Punch cards were later replaced by vacuum tubes and magnetic tape, much more efficient. A lot better, but I guess "writing code" at that times was still tedious and very time-consuming. Still no keyboard or screens, still no IDE or cloud.

Fast-forward we get then the first functioning programming languages in 1950 with machine code and Autocode. Then came FORTRAN and a lot others until we got modern languages such as PHP in 1990.

Each of these programming languages is an abstraction. At the end, the computer can only run machine code. We have created plenty tools (compilers, interpreters...) that transform our fancy programming languages into machine code, because fancy programming languages are much easier and faster to write.

This is why we create programming languages: because writing

foreach($data as $item) { echo $item->id; }

is so much easier and faster than writing the equivalent Assembly code

section .data
    data_items dq item1, item2, item3, 0 ; Array of pointers to items, terminated by 0
    format db "%d", 10, 0                ; Format string for printf: "%d\n"

section .bss
    item1 resq 2 ; Reserve space for item1 (id, next)
    item2 resq 2 ; Reserve space for item2 (id, next)
    item3 resq 2 ; Reserve space for item3 (id, next)

section .text
    extern printf
    global _start

_start:
    ; Initialize items
    mov rax, 1
    mov [item1], rax
    mov rax, 2
    mov [item2], rax
    mov rax, 3
    mov [item3], rax

    ; Point to the start of the data_items array
    mov rsi, data_items

.loop:
    ; Load the current item pointer
    mov rdi, [rsi]
    ; Check if we've reached the end (NULL pointer)
    test rdi, rdi
    jz .done

    ; Load the id from the current item
    mov rax, [rdi]

    ; Print the id
    mov rdi, format
    mov rsi, rax
    xor rax, rax
    call printf

    ; Move to the next item
    add rsi, 8
    jmp .loop

.done:
    ; Exit the program
    mov rax, 60
    xor rdi, rdi
    syscall

(I have no idea how to write Assembly, I asked ChatGPT to write this one)

Prompts are an abstraction

If I write the PHP code

foreach($data as $item) { echo $item->id; }

then it will be compiled / interpreted / transformed into opcode resembling the Assembly code above. I wrote something, I wrote the behavior I wanted to happen, the input and PHP and the Zend Engine can convert it something the machine can run, the output.

If we take a step back, we might notice that it is somehow similar with asking ChatGPT "Please write me some PHP code that fetches the top ten contributors from a GitHub repository" to ChatGPT, and having ChatGPT transform this input into PHP code that can be run, an output?

Just like using to knowledge of PHP I can write the same thing with 1 line of PHP versus 50 lines of Assembly, likewise I can write the same PHP code by myself in 30 minutes or in 10 seconds with ChatGPT. And the reason ChatGPT can do this is because

It has this amazing capability to infer meaning from my prompt
It can use the knowledge learnt from billions of documents to generate the right PHP code, implementing the behavior I want to happen
ChatGPT is not writing code, he's doing a developer's work which is using multiple things and gluing them together to obtain what he needs.

I like to imagine ChatGPT as the next iteration of a compiler. But instead of converting programming code into machine code, it converts text into code. Or better than text: behavior. Intent. The general principle remains the same: I am responsible for describing what I want to happen using a specific syntax (human speech for ChatGPT, PHP for the Zend Engine) and it is able to translate it into code that can, at the end, be run by a computer. ChatGPT is an abstraction layer.

A possible future, what happens to developers

ChatGPT is impressive if you ask him to write some code from scratch because there is no context. But give it a medium-sized codebase and it's completely useless (as of today), because there is way too much things to analyze. ChatGPT does well with small prompts, and codebases are not small prompts. So for now developers can only rely on ChatGPT to write small chunks of code, well isolated. It's better than nothing, but we're far from being able to stop writing the code ourselves.

Also remember that ChatGPT is using the knowledge he gained from billions of documentations and code examples from the Internet (including GitHub). Meaning: the code he writes is sometimes... garbage. Not because he's stupid, or smart, he's neither: he's simply replicating what he has learnt. So ChatGPT developer skills are probably the "average" programmer skills, meaning it will (just like all of us) make mistakes that become bugs.

But let's imagine that AI companies are able to solve / mitigate these two issues. What would become of us, developers, who are being paid for our ability to read and write code?

When FORTRAN was released in 1954, I can imagine that at the beginning there was 2 groups of developers. Some decided to stick with what they knew: Assembly. They were skilled at it, they were able to write very optimized code because it was very low level, and they did not understand the benefits of using FORTRAN. Early versions of FORTRAN also had issues when compiling the code into machine code, making Assembly developers dubious of it. But the abstraction capability of FORTRAN became a catalyst that allowed the developers who decided to use it to write programs much faster and more complex than before... and history teaches us FORTRAN emerged victorious. At this time, most developers, I guess, stopped writing Assembly code, and started using only FORTRAN. They only used the abstraction, not the layer under.

I think we might be in a similar situation. Our "modern" programming languages, which are still fancy line-by-line instructions for the computer, are the equivalent of Assembly. To write a program, we split its behavior into small operations that become instructions that we write, one by one. Just like the first developers used to create punch cards, one by one. ChatGPT prompts could be the equivalent of FORTRAN. With a programming language, we have an accurate capability to dictate exactly what we want to be run. But with the abstraction of ChatGPT, we can obtain the same, less controlled, less accurate, but so much faster and high level. Today we continue to use ChatGPT to help us write code, I think at some point we'll simply stop writing the code and let ChatGPT writes all of it, taking care of the implementation details.

In the future we might have conversations like this:

"Hello John, I'm back from a meeting with the customer. I wrote down two new features he'd like us to build next week. How are you doing on this week mission?

Well, I finished designing the API component. I had to adjust a few prompts I did for the new usecase, but the API component is now ready. Testing is finished and complete. I will then design the UI component - I think 20 prompt iterations will be enough, it should be ready before tomorrow.
Did you see that ChatGPT 23.3 has been published? They introduced something like prompt-based cloud hosting. Prompts are now fit with a specific architecture to make it easier to deploy with prompts. They call it infrastructure as prompts. Also they fixed a few exotic usecases where prompts were being corrupted when dealing with databases too large.
Cool, I will check that this afternoon. We should really migrate our current applications to ChatGPTCode 23 as soon as we can, the version 22 is EOL at the end of the year and then it'll become harder to re-run past prompts and obtain the same code."

In this imaginary scenario, the developer is not writing code directly anymore. Instead, when being given a new feature, he transforms the feature into a list of changes to be done in the codebase, and each change is implemented by providing the right instructions to ChatGPT. The right prompts.

Even if one day ChatGPT is able to "digest" a codebase of millions of lines, I think it will be inefficient to ask it to maintain it. My feeling is that it will be much more efficient to ask ChatGPT to write small, configurable and reusable components that can be connected together. A database component, a UI component, a session component... small components that ChatGPT can easily create and maintain (we need ChatGPT to be able to alter them later for new needs) and that can be coupled together to build an end-user application. Maybe a first candidate for ChatGPT-generated components would be applications built on AWS Lambda functions? They might be small enough to be entirely digested by ChatGPT.

Conclusion

The first developers "wrote programs" using punch cards. Later developers "wrote programs" in machine code, describing computing operations one by one. Further developers "wrote programs" using programming languages. Each time, we have added an abstraction level, pushing us farther from bytes operations, memory swaps and buffers, and focusing more on the end result and less on the implementation details. These abstractions have been tools allowing developers to build things more and more awesome. ChatGPT might be the next addition to these abstractions.

If you're worried about what'll happen to your job as a developer, don't. The code that will be run by ChatGPT will still run on a computer. It will need CPU and memory, it will access the network, it will read and write from a disk. People who understand how code is run on a computer will always be needed, to design it well. Asking ChatGPT to write code sounds easy, asking ChatGPT to write enterprise-level code that runs efficiently and securely in the long term is a job. It will need computing skills and logic, just like it does today.

Large migration projects are complex battles

Mathieu Ferment — Sat, 06 Jan 2024 21:16:03 +0000

After working as a developer for 9 years, I switched tracks to a manager position in 2021 (which is not very unusual in this industry). It’s been a bumpy road, but after 3 years I start understanding things and concepts that, retrospectively, were affecting me as a developer before. It looks like I'm starting to grasp them thanks to the fact that being a technical manager forces you to understand both software development and software business.

Today I would like to share some of these things with you.

Business and software development

Let’s start with one: “most enterprise software serves business”.

The very first reason why professional developers write code is to implement a business usecase. Maybe it’s creating a website to provide information about a company. Maybe it’s building a ticket selling platform for an event. Maybe it’s simply a data repository tool, like the data of a company's employees, and the company needs a basic CRUD interface for this database to help its HR department.

But the scenario behind all of these software development projects is identical:

A company has a business need
The business need can be addressed by building the right piece of software
One or many developers (and other people like designers, project managers…) are hired to build the piece of software.

And this little story above drives many developers' teams lifes. Business need, piece of software, developer team. It’s a cycle. As long as it stays healthy. By this I mean that it works if the value delivered by the piece of software is higher than the amount of money spent on step 3. If the company paid 200k$ to build a software that saves 300k$ yearly, it’s a good return on investment.

Things however can go wrong when software must undergo a migration.

A migration, in this blog post, is a software project where a piece of software is modified but the end user does not notice the change as the behavior is kept identical. The consequence being that the end user does not perceive any benefits from this migration. It breaks the healthy scenario mentioned above at least if you just look at the direct and visible costs.

As an example, let's say company A has 4 web python applications powered by framework B, and decides to migrate the 4 applications to use another framework C. End users will hopefully not notice any change, as the goal of a successful migration is to keep everything identical for end users. Only the developer / administrator team of the 4 applications will be aware of the migration outcomes.

There are many reasons why a company might need to perform a migration of one or many of its software applications. It can be switching providers, it can be switching technology, it can be upgrading an old piece of code, it can be partnerships changes, it can even be politics. But it happens all the time.

Today I would like to discuss the business challenge of large migration projects and how leading them to completion is a complex battle.

Three strategies

In order to achieve a migration project, I can think of 3 software migration strategies to lead it, with their benefits and drawbacks.

The big bang strategy

The big bang strategy is the simplest strategy. Let’s go back to the python scenario: company A has an application called FullSwitch, which uses the framework B. The goal is to migrate the application from framework B to framework C instead. We’ll call the current codebase version 1.0, and the codebase, once migrated to framework C will be version 2.0.

Applying the big bang strategy principle, the developer team starts the tedious work of building version 2.0, migrating the codebase from framework B to C. During this time, they completely stop working on version 1.0 (or almost completely). After many days/weeks/months of work (depending on the codebase size and complexity) the codebase version 2.0, running on framework C is written! Now comes the time to perform the switch in production: it’s deploy time.

The switch is performed on Monday (and not on Friday): the application running on version 1.0 is shut down, the data is migrated, and the version 2.0 is deployed and takes over. The replacement is done and everybody’s happy.

Happy? Not quite all the time. Because what I mentioned above is the happy scenario. In less optimistic scenarii, things can go wrong as the big bang strategy has 2 specific flaws.

The first flaw is that the v2.0 deployment happens at the very end. Which means that if the development phase duration was 6 months (not an usual timeline for a large codebase), during this phase the version 2.0 was solely running on development environments. The software industry has learnt the hard way that waiting for the very end of a software project to deploy is a very bad idea.

The longer the time between when code has been merged and when it is released increases the chance of an oversight. The only real feedback is received in production. Waiting and accumulating all of these code changes to deploy them all at once is a guaranteed way to have multiple major bugs in production to fix at the same time, and probably a painful deployment process. History has taught us that bigger changes are harder to review, debug and fix.

The second flaw is that during the development phase, the version 1.0 was completely frozen as the whole developer team is focused on building version 2.0. The longer the version 2.0 is built, the stronger the pressure will be from business to resume operations on version 1.0 because they simply need it - and I mean it. The ability of a company to rely and adapt on its software is key to staying a successful competitor in its market. A too long software freeze period can be the downfall of the business, and even if the version 2.0 is finished, if the business is sinking, the whole company crumbling, the version 2.0 will soon rot in the cemetery of unused software. A company needs its software to evolve and adapt to its business needs. It can wait for a few weeks, months hopefully, but it cannot wait forever without risking going bankrupt.

That being said, it is unlikely that the company would wait for its downfall. If the version 2.0 development phase is too long, what is most likely to happen is that the company will decide to halt the version 2.0 project. Because the lack of investment on version 1.0 had too much impact, because the business was at risk. And unfortunately it was probably the right call. The sad consequence is that the version 2.0 and all of the time invested by the developer team in it will be, in lucky cases, postponed, in less lucky cases, canceled. This is the full failure scenario for a migration, where the version 2.0 is never launched.

The cars race strategy

In order to address the big bang strategy flaw about business pressure, an obvious idea comes to the mind: continue maintaining version 1.0 . This means that instead of focusing the full developer team on building version 2.0, the team is split between two groups, one continuing working on the v1.0 codebase while the other builds the v2.0. With a reduced workforce dedicated to it, version 2.0 will be longer to build but we avoid the grim business downfall scenario of the big bang strategy.

Two other problems do arise.

The first is that all of the work done on v1.0 directly serves the business needs. It helps employees be more productive, it fixes pain points for company employees, it allows the company to reach new customers and markets and the new features added help convince new customers to adopt the application. It serves the company well. On the other hand, the work on v2.0 stays hidden in the shadows and, on paper, does not produce any added visible value.

This situation can be fine for a few months, but the longer it stays like this, the more likely the day will come where a director of the company says out loud “Why are we spending so much on v2.0 when the return on investment is zero? We could do so much more if the whole developer team was working on v1.0”.

You might think that this director is stupid and completely misses the point, that the migration to 2.0 is needed, that he’s focusing on short term benefits instead of long term… but doing so you would be overlooking how humans minds work.

If the 2.0 project was started yesterday, the idea is “fresh” in everyone’s mind. It’s clear why it has to be built. And most people in the company will understand - and agree - that it is necessary to perform the migration from v1.0 to v2.0 . But as time passes, that message will fade. As time passes, benefits of working on 1.0 will be tangible while the promises of the 2.0 benefits will become more distant, more abstract. This is a natural human mind's behavior, similar to how the media cover topics. When a new war begins somewhere in the world, the topic is broadly covered, and people are very interested in being informed and how it’s going. But if the war drags on, the people’s attention will shift to something else. The loss of interest of the public in that war will be seen in how the media cover the topic: less and less screen time will be dedicated to it until it becomes a thin line at the bottom of the newspaper.

So just like with the big bang strategy, the longer the version 2.0 is built, the higher the probability that the project might be halted ; this time because the perceived benefits have reduced while the tangible cost - in missed opportunities - is very real.

A skilled technical director might be able to fight this trend, by communicating often on version 2.0 progress, demonstrating the new version outcomes and benefits, but he can only delay the problem. He’s still in a race against time.

The second flaw makes the first even more difficult to deal with, and it’s because of this flaw I named this strategy the “cars race” strategy.

In the cars race strategy, two developer teams work on two versions of the same application. The first team continues working on v1.0, and constantly adapts it to new business needs: they ship bug fixes, new features, improvements. The second team works on v2.0 and wants to achieve the very same behavior and features of v1.0 but with a different framework. We have 2 cars racing against each other.

However the first developer team is leading: because they keep adding new features to the application perimeter. So they constantly push forward the limit that the second developer team must reach, because the v2.0 can only replace the v1.0 if they address the same business needs. So the v1.0 car is ahead, and keeps accelerating, and the v2.0 car must follow and catch up to win the race. There is unfortunately a scenario where the v2.0 car never catches up and loses the race, meaning it is never launched. It ran out of gas, because gas is expensive and the company did not want to put any more gas at this point of the race.

Finally the cars race strategy keeps the technical pitfall of the big bang strategy: the deployment of the new codebase happens at the end of the development phase, with all the drawbacks mentioned previously.

The trickle strategy

The trickle strategy aims to fix the above issues by avoiding the idea of competing v1.0 and v2.0 versions.

Instead, the migration is performed by iteration. A small part of v1.0 is migrated from framework B to framework C, and once this part is done and functional, a deployment of this codebase, let’s call it a version 1.1, is done. Version 1.1 is an hybrid: part of it runs on framework B and part of it runs on framework C. Then another iteration can start, leading to a release of version 1.2, with a higher part of the software migrated to framework C. The work continues, iterations continue, regular deployments continue until 100% of the codebase has been migrated to framework C, at this point we can call it a v2.0.

This strategy provides 2 obvious benefits:
First it avoids the scary one-shot deployment time of the 2 previous strategies and instead allows regular deployments of the solution. Deploying small changes is the first thing to start doing to make deployments less scary and diminish the risk.
Second, there is only 1 application running, not two: business cannot “shut down” the v2.0 because it does not exist. And the 1.0 does not exist anymore. What the business is using is the 1.3 or the 1.4, the only version that exists, that both serves the business needs and grows closer to the target migrated codebase.

Is that the perfect solution then? Unfortunately no. The trickle strategy comes with multiple costs that the previous strategies did not have.

The first is a cost paid by the codebase. Having 2 frameworks in one codebase is a technical design issue. It increases complexity, it requires some bridge code to be written that allows the 2 parts of the application to communicate and interact, it makes the application more complex and expensive to run and tangible side-effects like performance issues can emerge.

Having to continuously adapt and maintain that bridge between the two frameworks is an expensive operation, and the longer the project development duration, the higher the cost, quite simply. After a few months, there will be one day where a developer will state out loud the obvious “It’s so complex! We should have done this using the big bang strategy, we are wasting so much time doing this iteratively! This bridge code is a mess and we’ll throw it away one day, what a waste!”.

He’ll say that, because he’s paying actively the cost of that strategy: it’s the developer team that suffers from that hybrid situation, not the business. On the contrary, the 2 other strategies above did not push any added complexity to the developer team. We can also guess that because of the hybrid nature of the codebase, and of the regular deployments of each complete iteration, the development phase will be a lot longer than the previous strategies. But at least this time the team is not in a race against time.

Or is it? There’s another danger related to time. As the software continues to run in production through iterations v1.1, v1.2, v1.3 and so on, it’s likely that these interactions will contain migrated code, but also code improvements, bug fixes, and other code changes aimed to help the business. So these iterations will contain a mix: code changes aimed at achieving the migration and code changes aimed at helping the business.

Let’s say the development phase goes on for months. Maybe years. Just like with the cars race strategy, the benefits of the migration, which were clear in everybody’s mind in company A at the beginning of the migration project, do fade in time. As time passes, company employees outside of the migration project look at the developer team, shipping iterations after iterations, and think “I get it that every iteration allows the migration to advance, but well maybe we could do a little more of business code changes, and less migration code changes? I mean it’s OK if we finish the migration a little later, nobody’s counting on that, and on the other hand we have this really important project for the customer that we need to ship fast and…”

I guess you get it: the risk with the trickle strategy is not that business would halt the project completely one day, but slowly invests less in its migration part, more in other projects, and maybe one day they even halt it and think “well, it’s true our software is an hybrid between 2 frameworks but is it really an issue? I mean it runs, doesn’t it? We can live with that”.

Conclusion

“It depends”: for the majority of questions in this world, that is the only true answer. And that is the answer I give if someone asks me which of the 3 strategies is the best. Each comes with its benefits, each comes with its risks, and each comes with its costs.The one you should choose for your migration project depends on your context, your codebase, your developer team and the business behind it.

The only thing that these 3 strategies do share is: if it goes for too long, you risk a complete halt of the project. It is important to contain the migration to a set timeframe, and make sure deadlines don’t slip too much. You need to do it as fast as possible, you need to deliver it as fast as possible, before human minds do what humans minds do and change their minds as time passes.

Acknowledgement

This article that focuses on different migration types and the technical risk they carry has been a great help for writing this blog post.

Why is this feature taking so long?

Mathieu Ferment — Mon, 02 Jan 2023 20:35:17 +0000

This blog posts aims to provide an answer to the following question: 'But WHY does it takes you 2 weeks to implement this basic button?' 😆

As a software developer for 10 years, I've heard this question multiple times, and I guess if you too are a developer then you'll also hear it. This question is actually a cry for help, the request from someone trying to understand how something that looks simple in his opinion requires multiple days or weeks of a skilled developer to implement. This person is confused by the mismatch between the feature 'simplicity' and the time needed to deliver it.

Let's take the example of SonaPlay, a new company making computer games. Today, the company is actually 3 people, 3 friends who love computer games and decided to start a software company to pursue their passion. They create their very first game: Times of Heroes, a multiplayer RTS game that happens in medieval times. Right now, the game allows players to fight each other with multiple units: soldiers, archers, knights and catapults. It's still an early phase in the game development so game mechanics are limited.

I would like to explain software complexity by comparing two usecases: adding a new player unit now, or later when other features have been added.

Usecase 1

For this usecase we're willing to add a new unit to the game: spies. Spies are special units able to 'convince' enemy units to switch from one player to another player. But this only applies to 'human' units: soldiers, archers, knights. Catapults cannot be convinced to switch sides.

At this point in the game, units can simply move and attack so adding the new spy unit is not an expensive addition to the game. As mentioned before, only catapults are exceptions to the 'convince' capability of spies. Implementing the 'spy' unit in the game can be translated to:

add a new unit with 120 health points
similarly to other units, this unit can move in any direction but cannot attack
this new unit has a 'convince' ability with a cooldown of 60 seconds
'convince' can be targeted against enemy soldiers, archers and knights
spies cannot 'convince' each other

So this is a fairly simple feature and, looking at the description, we can guess this should not take long to implement for our 3 gamer friends.

Usecase 2

In this second usecase, let's say the spy idea has been postponed. The game then evolves and so does the company. The team grows which allow to ship more complex features.

The game now features horse archers, boats, swordsmen, axemen, elephants, ballista. Boats and elephants can 'carry' other units. Units can not only move and attack, but also defend, jump, eat, be healed at a hospital and teach other units to upgrade them. Units still have a health bar but also a stamina bar and a hunger bar. Players can also construct buildings. Additionally units behave differently whether they are alone or in a group, near a friendly building or near an enemy building. The game has become more complex (and hopefully funnier to play).

We're willing to add the new spy unit to the game at this moment. The description of this new feature has become longer as the game is more complex. Adding a new unit inside that complex system requires more thinking. Let's see what it means:

add a new unit with 120 health points, 50 stamina points and 20 hunger points, unable to attack other units
similarly to other units, this unit can move in any direction, but also jump, eat, be healed and teach other spies
this new unit has a 'convince' ability with a cooldown of 60 seconds
'convince' can be targeted against enemy soldiers, archers, knights, swordsmen, axemen, elephants (but NOT ballista, catapults, boats, buildings)
spies cannot 'convince' each other
a group of spy does not behave differently than a spy alone

As you can see the description of the behavior is now longer and needs to answer more situations. Not only do we need to implement the new unit behavior but we might also need to update the behavior of already implemented units.

Consequently we can expect that the time to implement usecase 2 will be longer than usecase 1.
And this is only about the rules! Remember that any character on screen needs to be animated. In usecase 1, only 'move', 'convince' and 'die' animations were needed but now additional animations 'teach', 'heal', 'eat' and 'jump' need to be drawn. Let's not forget the game has sound so more audio need to be recorded. Animations and sound are also more expensive to create in usecase 2 compared to usecase 1.

Real-life example: translation

A popular example of this problem is translation. Let's say you have a e-commerce website which can be browsed in English today.

Usecase 1: adding a new button. Fairly simple: choose location, design it, implement its action, choose the English text to be displayed on top of it.

What if the e-commerce website starts to sell in multiple countries, and to do so new languages need to be supported? The website can now be browsed in English, Spanish, Greek and Moroccan. It means that any text displayed on the website must now support 4 versions, and visitors can choose which version they want to read.

Usecase 2: adding a new button. The button must still be designed and the action implemented, but now 4 texts must be defined instead of 1, one for each supported language. Also the same sentence / phrase in one language can be longer or shorter when translated to another language, so the button must accommodate multiple text lengths. Same feature, longer to deliver. And as the number of supported languages grow, so does the cost of adding new tests to the website.

Conclusion

As you can see, the very same feature implemented in two different existing systems, one simpler and one more complex, takes more time to be implemented. This is the cost of systems complexity: adding a new feature or behavior in a already complex system can be very expensive even though the feature itself is simple. The cost lies in making it work with the whole existing system, making it fit what is already built.

EDIT on 2023-02-22: I found a tweet that highlights the above statement: jesseddy/status/1627923011490975746

Using a programming language is not (entirely) free

Mathieu Ferment — Tue, 16 Nov 2021 10:54:05 +0000

During the latest PHP storm, I was participating in the chaos on Twitter and mentioned something that I think is worth exploring.

So today I wanted to say that using a programming language is not entirely free.

Nowadays, plenty of amazing programming languages are available to build applications. You can pick your favorite! I can mention Python, Go, Ruby and my personal favorite PHP. There are dozens of others, naturally. Each of the languages I listed is free, which has two meanings

free as in freedom: these programming languages are developed under an open source license, making them freely usable and distributable
free as in no money: you can get these programming languages source and executables for zero dollar, no need to purchase them

Did you consider how amazing this is? Not only are these programming languages awesome, they are also available for free and even better: regular new releases are published providing new bug fixes and features, again for free! So... the cost of building an application on top of one of these programming language is zero dollar, right?

This is what I want to correct today.

Using a programming language is not entirely free.

Let us follow the journey of Matt. Matt is a developer and wants to launch a startup. He wants to build an amazing SaaS product on PHP. He starts today, so he builds his application on the latest stable version of PHP: PHP 8.0.
The SaaS product of Matt launches and he gets a few customers so his business can start growing, that is nice! And so far the cost of building this product on PHP is zero dollars (hosting provider fees excluded).

Now fast forward to October 2023. This is an important date because in November 2023, PHP 8.0 will be truly End-of-life. Truly, because it was End-of-life from the 26th of November 2022, but security patches were still being released until November 2023.

Matt’s business grew, and I'm really happy for him! He thinks it's a good time to upgrade from PHP 8.0 to PHP8.2 as 8.2 was released a few months ago. But... Now that Matt analyzes the changelog, it looks like between PHP 8.0 and PHP 8.2, some breaking changes have been introduced and the consequence is that Matt’s codebase, which is now quite large, does not run on PHP8.2.
Matt now needs to invest time (thus money) in exploring these changes in order to make his SaaS product run on PHP8.2 . Woops. The cost of building his product on PHP is not zero dollars, now.

But Matt’s business is still in the early years, each dollar is important. So he really doesn't want to spend money on this upgrade if he doesn't see a clear benefit. He doesn't need the new features of PHP8.2 and he sees no bug fixes in the changelog that apply to his use case. So... upgrading from PHP8.0 to PHP8.2 looks like a clear loss of money for Matt.

He tries to understand, but really, this seems weird. What's the point of upgrading to a new version if it costs money but brings nothing in return? That's a bad ROI, and in business, bad ROI means bad decision.

This situation happened to Matt because of the illusion that using a programming PHP was free.

What happens if Matt does not upgrade? If Matt stays on PHP8.0 past November 2023?

Well, consequences will start to line up slowly, one after the other.

First, if a security issue is reported in PHP8.0 after November 2023, it will not be patched by the official PHP team. For example, if a vulnerability is found in PHP8.0, 8.1 and 8.2, only the supported versions will be patched. So by staying on PHP8.0, an unsupported version, Matt puts himself in a dangerous situation as, if a security issue is reported, there will be no official patch he can apply. Of course, he could hire a PHP expert to build the patch, but the goal was to avoid spending money, not to spend a huge amount of it.

Second, by staying behind on PHP8.0 Matt simply loses opportunities. PHP9 will arrive after PHP8, and then PHP10. These versions will bring new features, performance improvements and bug fixes. By staying on PHP8.0 Matt puts himself in a situation where he cannot benefit from the latest improvements of the language. He’s missing opportunities.

Third, as years pass, 2024 then 2025, and Matt’s application continues to run on PHP8.0, it becomes harder for Matt to hire. Developers do prefer to use the latest versions of their favorite programming languages. Similarly new libraries and tools only work with PHP9 and higher so once again Matt missed opportunities as his PHP8.0 application is not compatible with them. It becomes harder to find developers proficient in PHP8.0 so Matt needs to pay them more. Talk about saving money, right?

Fourth, very bad news! Year is now 2026 and the hosting provider of Matt tells him PHP8.0 is too old and it won't be supported anymore as a standard package. Well, if Matt pays for an upgraded - and more expensive - plan he can continue using PHP8.0, but once again that is money not well invested.

We can resume this journey: by not upgrading his codebase to more recent versions, Matt slowly dug himself into a hole. And the longer he stays in this hole, the more money he’s spending, and the harder it will be to catch up as the upgrade becomes more difficult, day after day.

That is why using a programming language is not completely free.

You can build an application using a programming language for free, sure, but once you start running this thing and you need to keep it running, you need to regularly invest in staying up-to-date with the evolution of this language in order to stay on a version that is:

secure
receives regular bug fixes
relevant for the developer market as you will need to hire someday

That is true for actually any technology, not only programming languages. Operating systems, frameworks, libraries, CMS... Failure to regularly update will become a threat to your business, first as missed opportunities, and finally as a maintenance burden.

So software, like everything in life, needs maintenance.

I was often puzzled by the idea that software rots as code is intemporal and can run forever, until I understood that technology is a never-finished product and that by staying in place you put yourself behind the others. So, yes, software does need maintenance and versions upgrade is a major maintenance budget.

Your Pull Request is not a gift

Mathieu Ferment — Thu, 28 Oct 2021 08:43:06 +0000

As a maintainer of several open source projects for now 3 years, I regularly meet people, mostly about an open source contribution. This allows me to experience all kinds of interactions.
Most of them are collaborative and great, and keep me thrilled to work in open source. But some of them are spoiled by common beliefs or preconceived ideas about open source. So I'd like to bust one today.

Today I would like to tell you that, I’m really sorry, but your Pull Request is not a gift. And by ‘gift’ I mean something given by someone to another person out of kindness.

Before going any further, I want to highlight that this post is my opinion, not an absolute truth. Open source is broad, open source projects come in all shapes and sizes. What is true for some projects might be false for others.

Back to the story, it all starts here:

The background of this story is that a contributor sent many pull requests to a project I maintain - as a member of a large team - and the contributor was disappointed to see that his contributions were not being reviewed fast enough for him. Unfortunately, when these contributions were submitted, the maintainer team was overworked to push a release out, so we slowed down the reviewing work in order to get this release out and be able to resume normal work later, freed from the release stress.

The contributor was really frustrated. This was because, from his point of view, what happened was that he used his own time, his personal time, to find and solve many issues of our project, and submitted the solutions to us. He saw his Pull Requests like many presents sent to us, him gifting us bug fixes or improvements 'for free' so we would not have to do it ourselves. He expected us to welcome these gifts warmly, like solutions and answers we did not have to find, and process them quick and fast. Because... this is what you do with gifts, right? Thank the giver greatly, take the gift and enjoy it fast.

Unfortunately Pull Requests are not gifts.

This is an unfortunate misunderstanding. When a contributor submits a Pull Request that does fix a bug, they might think they are doing us a favor. I understand why some people think of it like this: this person used his/her time to build something and send it to us. This really looks like a favor.

But sending a Pull Request to an open source project is not a favor - at least not for all projects. Nor a gift, or a noble gesture.

When someone submits a contribution to an open source project, the very first person who benefits from this action is... themselves (most of the time). That is because, most of the time, this person is first a user of this project, so for example by fixing a known bug, they actually make the software better for them first - and then share their solution with every other user.

As maintainers of an open source project, we have our own roadmap to push. We have projects, goals, milestones and actively work on them. If someone finds an issue on the project and sends a Pull Request, we’ll be happy to review it so that the whole user community of the project can benefit from it, but we’ll see it as one more item we have to process, not as a gift.

For the projects I maintain, we do not ask people to submit Pull Requests ‘to be kind’. We do not request people to dedicate some of their time to the project ‘out of kindness’, using their time and skills for us. We do not want people to spend time contributing on our projects ‘to help people’ or ‘to be a nice developer’. What we need is people that use the project and, if they see they can improve it, do submit their improvements as Pull Requests. Kindness and friendship is good. But that's not what contributing to open source is about. It's about collaboration.

People sometimes think that a maintainer team should be happy to receive contributions, should be happy for all of this time strangers dedicated to build and send Pull Requests and we should welcome these tokens of appreciation as gifts. We do not see this picture this way. Actually more Pull Requests is... more work for us?

Yes, more work for us. Understanding, validating and merging your Pull Request actually takes a lot of work.

When receiving a contribution to the project, we have to review it. We apply the very same review process that we follow between maintainers, we verify the same items, we look for the same defects. So we spend time on the issue.
When dealing with tricky changes, we might trust people whose skills and knowledge we know well. But people we don't know? We have to be even more careful.

Open source contributions can be submitted by people from literally anywhere, and this is great. They can come from experienced developers or junior developers. They can come from people who carefully analyzed every aspect of the problem they were trying to solve, or they can come from someone who saw a bug, saw a quick fix and submitted it without realizing that his quick fix actually introduced another bug.
Since I have no idea what group the contributor comes from - because we get contributions from complete strangers - I must prepare for the worst so I actually double-check everything in the Pull Request. Sometimes, in order to verify the Pull Request correctly solves an issue, I will go as far as trying to solve the issue myself to verify how I would solve it and compare it to the submitted solution, which means the work has been done twice: once by the contributor and once by me. But this is what I need to be able to approve the Pull Request and state that it is valid and can be merged. After all, maintainers are gatekeepers and guardians of the quality of an open source project.

Moreover, reviewing and exploring this Pull Request can require even more work. I might request the help of an UX designer to validate some UX choices. I could also request a QA analyst to verify that not only the Pull Request correctly fixes an issue but it also does not introduce other bugs. In the end, for a Pull Request that was submitted after one hour of work by a contributor, maybe five or six hours of work will be required to validate it is correct. So... someone’s Pull Request, worth 1 hour of their time, might require 6 hours (and you can convert this in dollars spent) on our side to validate it. That makes it an expensive gift to receive.

Overall, receiving, validating and merging a Pull Request requires maintainers some work. It can even happen that the quantity of work needed to validate is higher than the quantity of work needed to build the Pull Request in the first place.

So you might be wondering... ‘if reviewing and exploring external contributions is so expensive, wouldn't it be more cost-effective to simply ignore external contributions and only work between experienced and approved developers?’

There was a recent example of a project that followed this strategy: Litestream. Initially, Litestream was closed to external contributions because it was too expensive for the maintainer team to handle them. As I said before: reviewing Pull Requests costs time and money. If there is not enough time or money, a project can simply choose to not accept external contributions.
Fast forward, Litestream now accepts bug fixes contributions, but no features. Bug fixes are slightly easier to handle as they are clearly scoped and most of the time the number of changes is small.

For the projects I maintain, rejecting external contributions would probably kill the community so it’s never going to happen. It’s expensive, it’s long, we have a huge backlog of Pull Requests waiting for review but we’ll deal with them, no matter how long it takes because this is key to keeping an open source project healthy. This costly process of validation is what makes it possible for anyone to contribute to the project, making it truly open. So we’ll happily pay the price and welcome and review incoming contributions.

Whoops! I just painted a very grim picture of one of the aspects of an open source project. Please let me brighten it a bit before the end.

If you regularly contribute to an open source project, you will become more experienced about it, and the maintainers team will start to know you. This will make the reviewing process smoother and fairly more enjoyable on both sides as trust will start to grow between you and the project’s team.

Then the next step becomes possible: when trust is firmly anchored, you could even apply to become a maintainer of the project. Yes, you. If your application is approved, you would be able to help the project team handle the reviewing Pull Request workload and will definitely make the project healthier, for both sides once again.
If you’re curious about how this happens, have a look at the maintainer application process of the Wordpress project as an example. Each project can have a different process, but most of the time the main requirement is trust, because welcoming a new maintainer in a team is like giving them a duplicate of the house keys.

So now you know :) are you willing to make open source better?

Why should I contribute to Open Source?

Mathieu Ferment — Fri, 13 Nov 2020 14:39:37 +0000

I work as a maintainer for the open source project PrestaShop. When a user submits a bug report, we thank him for the report and ask if he is willing to submit a Pull Request to fix it. To which some people answers "I'm sorry? Why should I fix the bug myself? Why should I do your job? 🤔" and that is actually a very important and relevant question.

If you are a developer, chances are high that you are using everyday open source technologies that you will never contribute to. For example many people will use Linux 🐧 all their life without ever contributing to the Linux project. Never submit a bug report. Never submit a patch. Never test the beta or the release candidates.

I am one of these person 😅 I use Linux and I never did any of these things.

And it seems to be alright. The project continues to live, and I continue to use it for free without any issues. So... why should I (or you) bother? Even if I do nothing, I still get this awesome OS for free, so why should I change?

There are already a few answers on dev.to about "why contribute to open source": some will tell you that contributing to open source is a great way to learn. Others will tell you it helps building your developer reputation. Some developers also think of contributing as "giving back", like saying thank you.

But all of these reasons to contribute share one flaw in my opinion: they rely on developers good will or self-interest. This means that someone who has no free time nor motivation will not contribute for these reasons.

So today I would like to provide a completely different answer. And to introduce it I would like to compare it to the opposite of open source software: proprietary software.

Balance for proprietary software

Proprietary software is the opposite of free software. It is software that you need to buy to be able to use it.

In proprietary software, there is a balance between two sides: customers and seller. Or in other words, users and vendor company.

On the first side are users. Users want to use the software so they pay for it. In exchange for their money, they can use the software either on-premise or hosted. The software is probably being used to achieve something else: if it's eCommerce software, the purpose is to sell online. If it's a CRM, it's for marketing purpose. If it's an ERP, it's to perform entreprise management tasks. The software is a product purchased for a purpose, quite simply.

On the other side is the company selling the software. They are given money by customers, and in exchange they grant access to the software, to the product.
Obviously in order for customers to continue buying it, they must maintain it, which means fixing bugs and enhancing its features. They must address the needs of the customers in order for the software to stay relevant.

So each side trades something with the other to obtain something.

This balance is sustainable if both sides benefit from the trade.
The company must earn more money than it spends maintaining the software while the customer must obtain more value from its purchase (meaning cash) than what he spent on the software license.

Balance of responsibilities for open source software

In open source software, money is obviously out of the picture. But there is also a balance. However the currency being traded is effort, not money.

On the first side is the project whose representatives are the maintainer team. The team spends effort maintaining the software but also managing the project (for example handling incoming bug reports: reproduce, classify, sort) and deliver regular releases for free.

On the other side are users, and obviously what they get in the balance is the software, for free. Users get full and free access to the project.

If some people ask themselves "Why should I submit bug reports? Why should I contribute? I can sit here and get it for free while doing nothing" they must understand that this attitude ... will result in the project eventually dying (if a majority of users do so).
The project would die because the relationship would be unbalanced and consequently unsustainable. It cannot be one-sided.

So... the balance need to be restored. In exchange for the free software given to them, open source users must use the currency for open source projects: effort.

Effort by submitting bug reports.
Effort by suggesting feature ideas.
Effort by contributing to the source code and the documentation.
Effort by testing the betas and the release candidates.

Because maintainers will never be able to manage the project all by themselves. In order for the community to continue being delivered the software for free, they must act as testers and owners for it.

So here is the balance of responsibilities for open source software:

While the balance of responsibilities for proprietary software is profit-driven (each side aims to win more than it spends), the balance of responsibilities for open source software is rather a dependency relationship. Both sides must invest in each other because if one decides to cut the effort, the other will collapse. It can be seen as a prisoner's dilemna where maximum benefit is achieved when both sides cooperate.

The power of multiplication

Presented like this, I'm painting a very poor picture of open source projects 😅! Luckily, the sustainability model of open source is way more robust and complex than just this balance, I over-simplified the concept to explain it.

What however stays true in this over-simplified model is how much more an open source project can achieve than a proprietary software.

In a proprietary balance of responsibilities, the user "only" spends money. All of the development effort is to be spent by the company, which means it is limited by the company size, however rich it is.

In an open source balance of responsibilities, the development effort is spent on both sides, and that is a wonderful thing.

Consider the following situation: you are a user of an open source project A.
While using it, you identify a bug. You report it on GitHub (and that is a first valuable contribution) but as it is not critical, you are being told by maintainers that it wont be fixed anytime soon. No worries, you have some time and money so you spend them fixing the bug yourself and you submit the bugfix as a Pull Request after 3 hours.

Once the Pull Request has been merged, at the end of the day, the outcome looks like this:

you spent 3 hours worth of your time
maintainers spent some time reviewing/testing/merging your Pull Request
one bugfix was delivered to the project

So the balance looks quite bad for you, right? 3 hours for one bugfix of a project you don't even own. Meh. 😒

But the very interesting thing is that, while you were exploring your bug today, twenty other users of the same project A have been exploring twenty other bugs and submitted Pull Requests too. At the end of the day all of the Pull Requests have been reviewed, tested, merged ; and a new patch release is delivered with these twenty-one bugfixes.

So from a group point of view:

each member of the users group spent 3 hours worth of its time
maintainers spent some time reviewing/testing/merging all Pull Requests
twenty-one bugfixes were delivered to the project
a new release was delivered and can now be used by the twenty-one users

So the real balance looks a lot better now: 3 hours of your time for contributing to a greater effort that landed twenty-one bugfixes worth sixty-three hours of work. In other words, you traded 3 hours worth of your time in exchange for sixty. And that is the power of open source.

Just like in the prisoner's dilemna, if one decides to be selfish and not the others, he gets all benefits while spending nothing. But if everybody decides to keep its effort (= to not contribute), everybody lose. Finally, everybody gets the same amount of effort at the end of the day, so it's a very smart strategy for everybody to maximize this amount of effort.

Maintenance burden

As mentioned by @ttoine, another benefit you get by contributing is passing the burden of maintenance.

Let's say for example you need GIMP to support a new image format TZL. You implement a driver for this. It takes you 3 days but you do it.

Great! Now you have it. But you know that software ages. You will need to spend more effort on your driver in the coming years to fix bugs, to keep it working with new releases of GIMP and the evolution of TZL format.

If however you submit your work as a contribution to GIMP, and it is accepted and merged, you pass the burden of maintenance to the maintainer team. And that is very valuable because it saves your time in the long run. The future years of maintenance will be done by the maintainers, not you.

Another thing you will get from submitting your work as a contribution is free feedback: when you submit a Pull Request, your code will be tested, reviewed, explored, analyzed. It is likely you will receive valuable feedbacks to raise the quality of your work so it can be merged in the project. This feedback is very valuable, it is like free consulting! Your code will gain in quality and you will learn a lot from this process.

Conclusion

Open Source is, to the core, driven by community. Its strength lie in the many, its richness in the diversity of its users. When combined and aligned, this can move mountains (think of Linus's law for example). When misunderstood and forgotten, it just gets abandoned.

That is both a lot harder and a lot more interesting than proprietary software where money rules it all. But what is really important is the multiplication effect, without equivalent in a standard proprietary balance.

So, back to the question, this is why you should contribute to Open Source. Because if nobody else does, the project dies. And because if you do, and same for other users, everybody gets so much at the end of the day.

Where do I start?

If you are now willing to contribute after reading this article, this is great! I would suggest you start contributing on projects you know well and you use everyday. What about testing the next beta or release candidate of your favorite programming language? You can also explore the issue tracker of your favorite framework. Remember that by investing effort in a project, you are also (and mainly) benefiting yourself.

What is magic in code and why avoid it

Mathieu Ferment — Fri, 04 Sep 2020 20:47:24 +0000

I have been building software for more than 8 years now and one lesson I have learnt the hard way was to avoid "using magic code" as much as possible.

So... what a strange statement, right ? "Avoid magic" ? What does that even mean ?

Please allow me to explain what these words mean to me.

What is magic in code

What I refer to as "magic code" is the concept of something happening inside your software without a direct, meaningful statement written by you, the developer.

Here is an example:

databaseManager = new DatabaseManager();
databaseManager->persist();

In this little snippet of code, I have written a direct function call persist(). Consequently I expect that the computer will perform a call to persist() when the snippet is run. The behavior and how the code can be read are closely related. I could read it aloud and what I would say and what happened would match.

Now what about this:

databaseManager = new DatabaseManager();
magician.call('persist', 'all');

Here is a first example of magic: the above object magician has the ability to look for all existing objects and call the chosen function on them. So the function persist() will be called, just like in previous snippet, however ; in the code above ; the relationship between what is written and what happens is not clear. The two snippets of code produce the same outcome, but they are written very differently.

A second example will be handy:

public class MyClass {
  int x = 5;

  public static void main(String[] args) {
    MyClass myObj = new MyClass();
    System.out.println(myObj.x);
  }
}

The class above is pretty straightforward to read. This is a class which holds an x integer property, initialized to 5. The main function will print it.

What about this trick:

public class MyClassWithXFive {
}

Believe it or not, but the empty class MyClassWithXFive behaves exactly like the above MyClass. It holds an x integer property, initialized to 5 and it has a main function that prints it.

This is because in some software I am working on, there is a rule (a magic rule ?) that when a classname ends with "WithXFive" then the software automatically generates a x property initialized to 5 in it! This property can be used just the same as the previous snippet.

This is a typical example of magic behavior in code: some properties, some definitions, some rules that happen not because of the code you have written but rather because of the names you have chosen for your classes or properties.

Some more examples of the same kind:

declaring a property id and this property is automatically used as main identifier for given class
declaring a class with suffix Test and this class is automatically considered and run as a test

Different types of magic

Here is a not-exhaustive list of what I would consider as "magic" in the code:

1) As mentioned before, behaviors that happen because of the name of your classes, properties, files

This kind of magic would happen thanks to a parser that would extract your classes / properties / files names and then look for specific patterns. Upon finding such pattern, it would trigger some behaviors.

This means that the name of your classes is determining for your application behavior. When modifying the name of your classes you might then introduce bugs.

2) Behaviors that happen because your code is located into a specific directory

This kind of magic would happen thanks to a finder that would look at a given directory and trigger some behaviors on all findings of this directory.

This means that the location of your classes is determining for your application behavior.
Moving your source code files might then introduce bugs.

3) Artifacts automatically generated

This kind of magic parses your code and uses it to build other chunks of code that are computed and written on disk. The code that is, eventually, processed at runtime is the generated code, not the one you have written.

This one is the less dangerous of all and it is used quite often for cache purposes.

Ironically, there is a concept called magic strings but I would not consider it to be "magic" 😄 it's another kind of dangerous magic though.

Why software projects use magic

I believe that most of the time, magic is being built into a software for the purpose of simplicity. It allows the developer to avoid writing some steps, some artifacts, some boilerplate code. It reduces duplication and "no brainer" code to write. When I tell you that "the only thing you need to do is to add your file in folder Z and you're done", it sounds appealing, right ?

"Magic code" might appear, at a given time, as an elegant solution to a real problem. There are well-known best practices in software that urge us not to waste time on meaningless tasks. The "Automate everything" moto is one of them.

Magic code looks like a good solution to such problems at a given time. The drawbacks, however, come later.

The issues of magic code

The reason I am strongly suggesting to avoid magic as much as possible is how hard it is to deal with it when things go wrong.

In order to illustrate my statement, please allow me to share a story, starring Bob, a random developer.

Story starts, Bob is hired to work on a codebase where there is one magic behavior implemented. This codebase is a web application, and Bob is told by the team leader that, although the application is expected to be used in a browser by most users, it does provide a REST API. Good news: the REST API is built in a very smart way: it's generated by a system that automatically expose all entities whose name ends with "Exposed".

So in the codebase, Bob looks at some entities:

There is an entity UserExposed, and because of its name, it is consequently in the scope of the REST API. Indeed Bob can see there is a /users REST API endpoint available on the application.
ProductExposed is in the scope too, there is a /products API endpoint Bob can use too.
There is a Supplier entity and it is not in the scope, because its name does not match the rule.
There is also a Brand entity, and it is not in the scope either.

Now that Bob is more familiar with the codebase and how the API is built, here comes the Product Owner. He asks Bob to create two new endpoints in the REST API: /suppliers and /brands. Cool, it seems quite easy, right ? Bob only needs to rename the entities to SupplierExposed and BrandExposed and "everything will be done for you by the API system". Sounds magic!

So Bob starts working, opens his IDE and modifies the classnames. Bob starts with Brand and he renames the class to BrandExposed. Oh! The API endpoint /brands is immediately available, even the documentation is up-to-date! It can immediately be used, and Bob did not need to write a single new line of code. This is truly awesome!

Then Bob modifies Supplier and rename it to SupplierExposed. Bob saves the file, commits the changes. But... nothing happens. Weird. The API endpoint should be implemented. Did he miss something ? He double-checks and double-saves the file, waits a little... But still no /suppliers endpoint. Nothing. Nothing comes out of the void.

And this... is where the bad things start. Because Bob needs now to understand why it does not work (and fix it). 😱

Because this system is "magic" it is very hard for Bob to track where the functions in charge of the REST API generation are. It's very hard to find why something is not triggered. Bob does not even know when it's supposed to be triggered.

In traditional imperative programming statements, you can follow easily how functions and changes occur. Debugger can even print it out for you! But now that magic is in the place, you (and Bob) cannot rely on it anymore. Magic code makes your software hard to navigate.

Let's say that after one hour and some sweat, Bob has found the area in the codebase responsible for the REST API endpoints generation. It gets worse: the same code is responsible for generating /products, /users, /suppliers endpoints! So the code is full of things like generateName(), loopThroughProperties, loopThroughEligibleObjects(). The code is very abstract, it's hard to track what's going on and relate it to the bug.

And if Bob finally finds what he believes to be the root cause of the issue, he has yet to make sure that his changes do fix the /suppliers endpoint without modifying/breaking all the other endpoints, generated by the same code! 😱

I think the story of Bob illustrates well the pitfalls of magic in code: generic code, abstract code, hard to relate with an actual bug scenario, and hard to navigate, debug and maintain.

This is, by the way, one of the drawbacks that is sometimes mentioned to discourage usage of Frameworks. It's not unusual for frameworks to rely on magic. As mentioned before, this can become a very handy asset in the hands of developers, but when there is a need to debug/expand/update the magic behavior, things can get nasty.

So avoid magic code

Magic code is a double-edged sword. It has tremendous potential and can allow building very powerful systems. It can also boost performances in some situations, like caching.

However these benefits come with a (very) high maintenance cost, which is why I advise you to avoid it in most situations.

Following the blog post I wrote 3 years ago "Basic code is easy to maintain" I still believe that most of us should choose simplicity over efficiency.

Unless you're Google or Facebook. Then you can choose efficiency and complexity 😁 and you'll be fine!

What it means to rebase a Pull Request submitted on GitHub

Mathieu Ferment — Mon, 20 Jul 2020 08:26:23 +0000

Dear reader,

This blog post is intended to help people willing to understand what it means to rebase a Pull Request submitted on GitHub. It does not explain "how" (this part has been addressed already by multiple posts, you'll find links about this at the end) to perform the rebase but rather what happens behind the curtain and why it is needed.

So I'll continue this post while considering that

you are willing to contribute to an open source project hosted on GitHub
you need to rebase one Pull Request you have submitted and you do not know how or why

Please note that, in order to make this post accessible to people still not comfortable with git concepts, I will simplify and twist a little how git rebases work, in order to make them easier to understand. The main idea however should be accurately presented.

How rebases come into your life

If you are reading this, congratulations! Because it means you have submitted a Pull Request to an open source project hosted on GitHub and that is already remarkable.

However it might happen that your PR (Pull Request) becomes out-of-date. This can happen for several reasons, the most frequent one being that files modified in your PR are in conflict with changes that have already been merged.

This will be indicated by GitHub as such:

And even if your PR meets all the other criteria to be merged, the merge cannot happen. GitHub will block your PR.

How git conflicts happen

How did we end up here ? Here is probably what happened :

1) In order to create the PR, you forked the main repository and created a branch of yours. This branch was issued from the repository main branch HEAD, the latest commit available on the main branch. We'll say the main branch is called develop and your branch is my-branch.

Here is a small graph presenting the situation:

The blue commits are the develop commits, the red commits are the commits you added on your branch. Your branch my-branch is issued from the last blue commit, the HEAD of develop.

Your Pull Request goal is to merge the 2 red commits into the develop branch.

2) At the time you created your branch, everything was fine. Your PR could have been merged without issues. However your PR was not merged immediately.

Maybe it needed some adjustments, required by the maintainers. Maybe it needed some advanced exploration and testing to make sure it was free of bugs. Maybe you got feedbacks and could not handle them sooner because you were busy...

Whatever the circonstances, while your PR was idle, the work on the main repository continued! Other Pull Requests were merged, which means new blue commits were added on develop branch.

3) Unfortunately, because a Pull Request always target the latest commit on targeted branch, it might not be mergeable anymore. If for example the 2 new blue commits on develop modify a file also modified by your PR, then this might introduce a git conflict that prevents the merge of your PR.

And this is it: your PR cannot be merged.

At this point, one project maintainer is likely to ask you to rebase your PR on top of the targeted branch. Rebasing the branch used to create your PR will resolve the issue preventing the merge.

Time to rebase

The solution is consequently to "rebase your PR on develop". But what does it mean ?

Here is what is going to happen:

4) When you start the rebase, git is going to modify your branch. First, it will take all your commits and put them aside, like this:

Your red commits are removed from the branch. Gone. The branch looks like it went back into the past.

5) Git is going to bring the new blue commits of the develop branch inside your branch. This is the first main concept behind the rebase. Git is going to bring the latest changes of develop into your branch.

6) This situation is now equivalent to having started your branch from the latest commits. From git point of view, since your branch contains all the blue commits from develop, it's just like if your branch was started from the current HEAD, and not the old one, two commits before.

7) It is now time for git to bring back the red commits put aside. Now git will apply them back on the branch. However the branch has changed, the red commits will be added onto the new HEAD of your branch.

8) Git is going to apply the commits one by one. For each commit, git will check whether there is a git conflict and, if there is one, will ask you to resolve it.

9) Finally, after a few resolution of conflicts, all your red commits will be put back into your branch. This is it. Your branch is now rebased, up-to-date with develop.

10) Now you can synchronize your fork with your local repository by pushing your changes and GitHub is going to update the Pull Request status, acknowledging its new content. The PR should now be mergeable.

Conclusion

The git rebase operation is actually a rewrite of your branch history, whose goal is to bring into your branch the changes that happened on the main repository branch. The strategy behind removing your own commits to re-apply them onto the updated HEAD aims to create a git history as clean as possible.

Some considerations

There is another way to update your PR to fix the conflicts: that is to merge develop into my-branch. A rebase is however usually preferred because merging develop into my-branch create merge commits that make the git history a lot harder to read.
Git rebase will re-apply your commits one after the other on the updated branch. This might be a very long operation if each commit has git conflicts with main branch. If you have modified 15 times the same line that was in conflict, you will have 15 conflict resolutions to do! In this usecase, merging develop into my-branch is an interesting solution because you solve the conflicts only once.
Since you have rewritten your git history, when you want to synchronize your fork with your local repository, you will not be able to use git push. You will need to use --force option.
When submitting a PR, you can grant maintainers the right to modify the branch that you used to submit the PR. Maintainers granted this right have the ability to perform the rebase themselves, but this is a lot more complex for them because they are not the PR author. When resolving git conflicts, it is very useful to know how the PR was built to understand how the conflict should be resolved.

After theory comes practice

Here are some great tutorials explaining the "how" to rebase a PR submitted on GitHub:

How to Rebase a Pull Request on edx-platform
How To Rebase and Update a Pull Request by Lisa Tagliaferri on digitalocean.com
How to rebase a GitHub pull request by Aurelien Navarre

Tips from a lead developer: my own version of Rapid Decision Making

Mathieu Ferment — Mon, 18 May 2020 13:30:09 +0000

Introduction

In 2016 I was given the opportunity to work as a lead developer in a small tech startup. This was the first time I was given the responsibility of a developer team of three people and I have learnt thousand of lessons from this experience, some the good way, by reading books on the topic, and some the hard way, by failing at doing my job properly and having to fix my mess.

I kept this position for two years and a hundred blog posts would not be enough to write all the wonderful lessons I was taught by this experience. Managing humans (even harder: managing developers !) is tough and you learn very fast.

Today I'd like to share one of these lessons with you ; I call it RDM: Rapid Decision Making.

The one hundred decisions challenge

So, if one day you start working as a team leader, one of the first thing you will notice is that suddenly you have to take hundred of decisions per day.

Yes, hundred.

Because not only your team member have lot of questions for you, other teams also consider you are the main point of contact when they need to collaborate with your team. So the day starts, you get up to grab a coffee, and in twenty seconds people come at you and ask you to tell them 1) customer A is angry and wants to kill the project, what do we do ? 2) the new intern starts today, who onboards him and what topic should he work on 3) project B is late, how can we fix that ?

At the beginning I thought that I should handle all these decisions very carefully. I thought that as the lead developer, I was responsible for giving the best answer to solve the problem. That is why I wanted, for each decision, to think it over, gather valuable informations and use these findings to craft and deliver the very best answer.

Very fast, my days were spent only doing decision making. No code review, no bugfix, nothing else than making decisions, from 9am to 7pm. Then I would rush my other assignments until 8pm or 9pm sometimes because I had my own share of work to deliver.

Obviously this strategy was not going to work on an extended period of time.

Rapid Decision Making to save the day

So I came up with the RDM strategy: Rapid Decision Making.

Whenever someone comes at me with a need, asking me to decide what he or we should do about any topic, I give myself a timebox to answer. I will schedule a 30 seconds timebox for trivial/minor matters, 1 minute for medium matters, 1 hour for important matters and one day for critical matters.

Then I try to see if I can think of a "default valid answer". What I mean is that I try to check whether there is an easy answer, not the best one, but satisfactory enough, to this problem.

For example "what should the new intern do ?" ; a default valid answer would be "read our documentation with fresh newcomer eyes and tell us what looks wrong". It might not be the best way to use his time but

this is a task that does not require to be onboarded by a team member
this is something useful (his feedback will improve the documentation)
this is easy to do
there is no risk doing it

So this solution is not bad. It's a default valid answer.

Then I try to allocate, following the criticality of the matter, time to find a better answer. However if at the end of my timebox I have not found a better answer, I give the "default valid answer".

RDM in practice at the restaurant

Here is a full example: I go to a restaurant with a group of people. We sit and are given the menu. The waiter go and could come back at any time asking what we want to eat, and I don't want to answer "sorry I have not chosen yet" (I hate having people waiting for me).

So when I am given the menu I immediately scroll through it fast, looking for something I would be happy eating. It might not be the best dish I could find, it might not be what I really wish to eat today, but it is something that I will not be disappointed of. It will be my "default valid answer". For example a default valid answer could be a burger.

When I have found my default valid dish, then I start reading the menu again slowly, looking for something that I'd really want. If the waiter comes before I find something better, I will answer my default valid dish. If I find, in the menu, a better choice, I will answer this choice.

Since I have started using RDM, I have never again answered "I'm sorry I don't know yet what I want to eat" to a waiter !

Good is almost always enough

The main idea behind the RDM strategy is that most decisions you have to make, as a team leader, do not require the very best answer. Most of the issues we encounter daily at work are solved fine when given a "good enough" answer. Keep your time and your brain juice for critical topics, the topics that will decide if your project lives or dies.

Since 2016 I keep using RDM in my professional life whenever it is relevant. Which is actually most of the daily questions I face ! And I think a lot of developers could do the same.

Sometimes we developers can spend hours on topics without much value. Arguing whether is_valid or isValid is more readable or discussing whether folder path should be App/User/Preference or App/UserPreferences for example 😄

Maybe after 10 hours exploring the question and reading books about it one could find an argument that makes answer A better than answer B. But was it worth it ? I think for most topics it was not. RDM is one way to avoid wasting time on meaningless topics. If you cannot find a great answer within the timebox, just go with the default valid answer. And most of the time it will work fine. The idea is to stay pragmatic.

I hope this blog will be useful for some struggling developers or team leaders 😊 .

Introduction to github fork workflow - why is it so complex ?

Mathieu Ferment — Tue, 14 Jan 2020 08:05:59 +0000

Dear reader,

This blog post is intended to help people willing to understand better the process to contribute to a github opensource project for the 1st time. It does not explain "how" (this part has been addressed already by multiple posts, you'll find links about this at the end) to use the github fork workflow but rather "why" it is used and why it seems so complex the 1st time you look at it.

So I'll continue this post while considering that

you are willing to contribute to an opensource project hosted on github and using the standard github fork workflow
you have already learnt about it (and maybe used it, but not mastered it)
you feel like this system is unnecessary complex to setup and use

It's about people

Github fork workflow is actually quite a nice answer to a complex topic. This topic is "how to make multiple developers work together on a codebase without hindering each other's work". Because this is what OSS* projects are at the very core: multiple developers working together on a codebase.

*OSS = Open Source Software

Please let that sink in. Have you ever considered how complex it would be to write a book together with multiple people ?

Well, if you let everybody free of modifying any part of the book, this is going to be messy ! People erasing other people's parts (to replace it with their own), people disagreeing on sentences, or people who write duplicate sentences because they did not agree beforehand on who would write which part ...

In order to avoid a mess, you need to manage how people work together. You need to provide people a process, a workflow that allows them to focus on the business value of the book (writing it !) rather than wasting time synchronizing with everybody's agenda.

Obviously someone will need to be in charge of this workflow, carefully checking each person whereabouts, and your workflow needs to allow this person to handle the contribution of each member of the group in a manner that provides him

a way to review the contributions
a way to merge contributions into the book
and it must be easy to use (so handling everything with emails is a no-go) as it is expected there will be a lot of contributors, but very few maintainers

So github fork system aims to solve this topic, although the subject is source code, and not a book... even if github fork system could very well be used to structure how a book is written.

Let's break github workflow down

Now that we know what problem the github fork workflow aims to solve, let's see how it unfolds.

Let's consider the following scenario : a contributor considers the OSS project PrestaShop but thinks the logo is actually quite bad and would look better in black&white. He wants then to suggest a better version of this logo.

Please meet Preston, mascot and logo of PrestaShop

In the beginning of this story, you have the official repository alone, the authority where the official source code belongs to. It is a git repository, hosted on github.com .

Now comes the contributor who wishes to suggest a better version of the logo. He is forbidden to do this on the official repository (else it would mean anybody can modify anything in the project and this would lead to chaos).

So github requires him to create a copy of the official repository, which is called a fork.

This copy is, at first, identical to the official repository. But this fork is by no means subjugated to the official repository, it has its own life. The contributor can do whatever he wants with this fork: modify all files, delete all files, use it as a project ... this fork can live a life on its own, and belongs to the contributor only.

Back to our usecase. The contributor now wants to modify the logo on his fork rather than the official repository. Since his fork is "his property" he can do whatever he wants on it, including altering the logo to make it black&white.

But his fork is a repository on github. This is not a convenient environment to do some modifications (although github provides quite a nice online editor), especially for images. The contributor will consequently create a git copy, a clone, of his github repository on his own computer.

At this point there are 3 versions of the PrestaShop OSS project:

the official main repository, with the standard logo ; hosted on github
the contributor's fork, with the standard logo so far ; hosted on github
the contributor's fork copy, with the standard logo so far ; hosted on his computer

Since the local copy is on the contributor's computer, he can edit it using his favorite image editor (someone said GIMP ?) and improve the logo. He does so by altering the logo to make it more "classy" ... meaning black&white.

Now there are 3 versions of the PrestaShop OSS project:

the official main repository, with the standard logo ; hosted on github
the contributor's fork, with the standard logo so far ; hosted on github
the contributor's fork copy, with the modified logo ; hosted on his computer

The contributor then updates his fork by pushing his changes from his computer copy into github. This is a standard git process where you host the code online and edit it on your computer. Usually you mirror the branches of the github repository and your local copy.

Now there are 3 versions of the PrestaShop OSS project:

the official main repository, with the standard logo ; hosted on github
the contributor's fork, with the modified logo ; hosted on github
the contributor's fork copy, with the modified logo ; hosted on his computer

Finally, the contributor can submit a Pull Request, to request the official repository to acknowledge the modifications he did on his fork ... and make it a part of the official repository.

The official repository maintainers will be able to look at his Pull Request, review it, suggest modifications and either accept or refuse it.

Summary

When you are being told for the 1st time about github fork workflow, you might think "why ? why does this contribution process involves 3 git repositories, me pushing my work from my computer to a fork then the fork requesting a merge into the official repository, this is such a waste of time ..." . If the contribution was a quick fix (like a missing comma), you might spend more time applying the workflow* than making the necessary code changes !

*For these kind of usecases, github online editor is quite nice as it allows you

to modify directly your github fork repository (no need for a local copy)
to create a Pull Request automatically from a file of the official repository you want to modify (no need for a fork !)

But when you consider the big picture, it actually makes sense. Each of these 3 repositories serve a purpose.

The official repository is the "location" of the official source code and must be modified only by approved contributions.

Forks allow developers to do whatever they want and allow specific code changes to be submitted for merge into the official repository.

Since forks are hosted on github, they are not easy to use, which is why it is better to make a local git clone of your fork on your computer and work on this copy. On your computer you have access to all your developer tools and environment.

The "hard work" of doing an opensource Pull Request actually comes from passing and pushing changes between these three.

Also, keep in mind that in our scenario we focused on the contributor point of view:

But for the project maintainers, the situation rather looks like this !

Github fork workflow also allows opensource maintainers to keep track of all incoming contributions and efficiently manage them. An efficient tool is necessary for a quality project, which is why so many projects use github fork workflow.

Some parts that I have not covered

In this blog post, I did not talk about

the different branches and branching strategy, that can add another layer of complexity to this workflow
the issues of keeping a fork up-to-date with the original repository, be it the main branches or rebasing pull request branches

I might cover them in another blog post, as this one is already quite long to read for an "introduction".

After theory comes practice

Here are some great tutorials explaining the "how" to contribute to a github opensource project.