DEV Community: Rémy 🤖

The Ultimate Python main()

Rémy 🤖 — Wed, 12 May 2021 21:55:36 +0000

So you want to write a CLI utility in Python. The goal is to be able to write:

$ ./say_something.py -w "hello, world"
hello, world

And then to get your script running. This isn't so hard and you've probably done it already, but have you thought of all the use cases? Let's go step by step on all the things you can do to make your main() bullet-proof. The first sections are going to be very basic and we'll build up something less obvious as we progress.

Hello, world

Let's start with a basic script that just says "hello, wold".

print("hello, world")

Now this is fairly basic but still won't work if you call it directly like ./say_something.py. In order for this to work, you need to add the famous shebang at the beginning.

#!/usr/bin/python3

print("hello, world")

Note — Let's also not forget to do a chmod a+x say_something.py to give execution rights on the file

However, what if you're using a virtualenv? By calling /usr/bin/python3 directly you're forcing the path to the Python interpreter. This can be desirable in some cases (by example package managers will tend to force this for safety) but if you're writing a distributable script it's simpler to get it from the environment.

#!/usr/bin/env python3

print("hello, world")

Note — We're calling python3 instead of python because in some systems you'll find that python points to Python 2 and who wants to be using Python 2?

Importability

Something that you'll learn early on from manuals is that you want to make sure that if for some reason someone imports your module then there won't be any side-effect like a surprise print().

Because of that, you need to "protect" your code with this special idiom:

#!/usr/bin/env python3

if __name__ == "__main__":
    print("hello, world")

You'll note the use of the special variable __name__ whose value is going to be __main__ if you're in the "root" file that Python is running and the module name otherwise.

Parsing arguments

You'll notice that so far the program only says "hello, world" but doesn't parse the -w argument that we've described above. Parsing arguments in Python is not so different from C — for those who had the chance of learning it at school — except that well it's Python so you have a few more goodies.

The main difference is that instead of receiving the arguments as a an argument of main() you'll have to import them.

#!/usr/bin/env python3
from sys import argv, stderr


def fail(msg: str):
    stderr.write(f"{msg}\n")
    exit(1)


if __name__ == "__main__":
    what = "hello, world"

    if len(argv) == 1:
        pass
    elif len(argv) == 3:
        if argv[1] != "-w":
            fail(f'Unrecognized argument "{argv[1]}"')
        what = argv[2]
    else:
        fail("Too many/few arguments")

    print(what)

Now, while this fits the bill, this is oh so much more laborious to parse arguments manually like this. For a long time I was afraid to use argparse because the documentation is quite the beast, but in the end its basic use is very simple.

#!/usr/bin/env python3
from argparse import ArgumentParser


if __name__ == "__main__":
    parser = ArgumentParser()
    parser.add_argument("-w", "--what", default="hello, world")

    args = parser.parse_args()

    print(args.what)

That way you let all the hard work to ArgumentParser() and as a bonus you get an auto-generated help text when calling with -h as an argument.

Outside call

All this is cool but what if you want to call the CLI tool from another Python program? Experience showed me that it's often a much more pragmatic thing to do rather than to design a specific API for Python and a different API for CLI (which is arguments parsing).

The problem with what we've got here is that we can't reference our call from the outside world. To run it has to be in __main__, which is not necessarily what we want.

Fortunately there is an easy solution to this: create a C-style main() function.

#!/usr/bin/env python3
from argparse import ArgumentParser


def main():
    parser = ArgumentParser()
    parser.add_argument("-w", "--what", default="hello, world")

    args = parser.parse_args()

    print(args.what)


if __name__ == "__main__":
    main()

While this works, this doesn't fit the bill of letting you call custom arguments from an outside package. However, all you need to do is add the arguments as an argument to your main function:

#!/usr/bin/env python3
from argparse import ArgumentParser
from typing import Sequence, Optional


def main(argv: Optional[Sequence[str]] = None):
    parser = ArgumentParser()
    parser.add_argument("-w", "--what", default="hello, world")

    args = parser.parse_args(argv)

    print(args.what)


if __name__ == "__main__":
    main()

The advantage of this is that by default it will keep on working like before except that this time it will let you pass custom arguments to call it from the outside, like that by example:

>>> from say_something import main
>>> main(["-w", "hello, world"])
hello, world

A bit of assistance

For good measure and clarity, let's take out the arguments parsing into a separate function.

#!/usr/bin/env python3
from argparse import ArgumentParser, Namespace
from typing import Sequence, Optional


def parse_args(argv: Optional[Sequence[str]] = None) -> Namespace:
    parser = ArgumentParser()
    parser.add_argument("-w", "--what", default="hello, world")

    return parser.parse_args(argv)


def main(argv: Optional[Sequence[str]] = None):
    args = parse_args(argv)
    print(args.what)


if __name__ == "__main__":
    main()

But now you'll notice that the typing annotation isn't very helpful. I usually like to be a little bit more verbose to get more assistance from my IDE later on.

#!/usr/bin/env python3
from argparse import ArgumentParser
from typing import Sequence, Optional, NamedTuple


class Args(NamedTuple):
    what: str


def parse_args(argv: Optional[Sequence[str]] = None) -> Args:
    parser = ArgumentParser()
    parser.add_argument("-w", "--what", default="hello, world")

    return Args(**parser.parse_args(argv).__dict__)


def main(argv: Optional[Sequence[str]] = None):
    args = parse_args(argv)
    print(args.what)


if __name__ == "__main__":
    main()

Handling signals

For the purpose of this demonstration, let's add some "sleep" in this code to simulate that it's doing something instead of printing the text right away.

Now what if the program receives a signal? There is two main things you want to handle:

SIGINT — When the user does a CTRL+C in their terminal, which raises a KeyboardInterrupt
SIGTERM — When the user kindly asks the program to die with a TERM signal, which can be handled (as opposed to SIGKILL) but we'll see that later

Let's start by handling CTRL+C:

#!/usr/bin/env python3
from argparse import ArgumentParser
from time import sleep
from typing import Sequence, Optional, NamedTuple
from sys import stderr


class Args(NamedTuple):
    what: str


def parse_args(argv: Optional[Sequence[str]] = None) -> Args:
    parser = ArgumentParser()
    parser.add_argument("-w", "--what", default="hello, world")

    return Args(**parser.parse_args(argv).__dict__)


def main(argv: Optional[Sequence[str]] = None):
    args = parse_args(argv)
    sleep(100)
    print(args.what)


if __name__ == "__main__":
    try:
        main()
    except KeyboardInterrupt:
        stderr.write("ok, bye\n")
        exit(1)

As you can see, it's as simple as catching the KeyboardInterrupt exception. Please note that we do this outside of main() because the famous hypothetical "caller module" will already have its signal handling in place so that's something that we only need to setup if we're running on ourself.

Next comes the handling of SIGTERM. By default the program will just stop but we don't really want this. By example if we're using the following pattern inside the code:

try:
    # do something
finally:
    # cleanup

We won't have a chance to cleanup if an exception isn't raised. Fortunately for us, we can raise the exception for ourselves.

#!/usr/bin/env python3
from argparse import ArgumentParser
from time import sleep
from typing import Sequence, Optional, NamedTuple
from sys import stderr
from signal import signal, SIGTERM


class Args(NamedTuple):
    what: str


def parse_args(argv: Optional[Sequence[str]] = None) -> Args:
    parser = ArgumentParser()
    parser.add_argument("-w", "--what", default="hello, world")

    return Args(**parser.parse_args(argv).__dict__)


def sigterm_handler(_, __):
    raise SystemExit(1)


def main(argv: Optional[Sequence[str]] = None):
    args = parse_args(argv)
    sleep(100)
    print(args.what)


if __name__ == "__main__":
    signal(SIGTERM, sigterm_handler)

    try:
        main()
    except KeyboardInterrupt:
        stderr.write("ok, bye\n")
        exit(1)

What happens is that we're using signal() to register sigterm_handler() as our handler for SIGTERM. What happens "under the hood" is that the signal handler will be executed before the "next instruction" that the interpreter would otherwise have considered. This gives you a chance to raise an exception which is going to bubble up to our __main__ and exit the program with a 1 return code while triggering all the finally and context managers along the way.

As stated before, there is a sleep() in the middle of the function. This means that you can run the code from this section and then hit CTLR+C or send a SIGTERM to see what happens when you interrupt the program.

Error Reporting

Sometimes — more often that you'd like — your program will fail either by its own fault or because the inputs are incorrect. By example, if the user tries to open a file that doesn't exist, you might want to report it. And by "report it" I mean nicely, not with a harsh stack trace. Keep the stack trace for the cases that you didn't expect this way you know that something is really wrong.

Let's imagine that we want to forbid the user to say "ni". In order to report the error, we're going to create a specific error type for our program

class SaySomethingError(Exception):
    pass

Then we're going to handle it in our __main__:

    except SaySomethingError as e:
        stderr.write(f"Error: {e}")
        exit(1)

And finally we're going to raise the error from main()

    if args.what == "ni":
        raise SaySomethingError('Saying "ni" is forbidden')

For an overall code that is:

#!/usr/bin/env python3
from argparse import ArgumentParser
from typing import Sequence, Optional, NamedTuple
from sys import stderr
from signal import signal, SIGTERM


class Args(NamedTuple):
    what: str


class SaySomethingError(Exception):
    pass


def parse_args(argv: Optional[Sequence[str]] = None) -> Args:
    parser = ArgumentParser()
    parser.add_argument("-w", "--what", default="hello, world")

    return Args(**parser.parse_args(argv).__dict__)


def sigterm_handler(_, __):
    raise SystemExit(1)


def main(argv: Optional[Sequence[str]] = None):
    args = parse_args(argv)

    if args.what == "ni":
        raise SaySomethingError('Saying "ni" is forbidden')

    print(args.what)


if __name__ == "__main__":
    signal(SIGTERM, sigterm_handler)

    try:
        main()
    except KeyboardInterrupt:
        stderr.write("ok, bye\n")
        exit(1)
    except SaySomethingError as e:
        stderr.write(f"Error: {e}")
        exit(1)

With this very simple system, you can raise an error from your code no matter how deep you are. It will trigger all the cleanup functions from context managers and finally blocks. And finally it will be caught by the __main__ to display a proper error message.

Packaging

A good idea when providing a bin script inside a package is to have it be called with Python's -m option. By example instead of writing pip, I usually write python3 -m pip to be sure that the Pip I'm running is indeed the one from my virtual env. As a bonus, you don't need the environment's bin directory in your $PATH. You'll find that most famous and not-so-famous packages provide both ways of calling their binaries.

In order to do that, you need to put your script into a __main__.py file. Let's do this (this is UNIX commands but Windows should have no trouble translating):

mkdir say_something
touch say_something/__init__.py
mv say_something.py say_something/__main__.py

Now you can call your script with python3 -m say_something.

Note — We're creating an empty __init__.py file to signify to Python that this is a proper module

Here it's clearly becoming a question of preferences but my personal favorite tool for packaging has become Poetry because it is so simple. Let's create a basic pyproject.toml for our project.

[tool.poetry]
name = "say_something"
version = "0.1.0"
description = ""
authors = []
license = "WTFPL"

[tool.poetry.dependencies]
python = "^3.8"

[tool.poetry.dev-dependencies]

[build-system]
requires = ["poetry>=0.12"]
build-backend = "poetry.masonry.api"

Let's try this out to see if it worked

poetry run python -m say_something

This is good but did the command go? Have no fear, Poetry lets you declare your "bin" commands pretty easily. The only thing is that it needs a function to call directly, including all the signal-handling shenanigans. Let's move it all into a separate function then:

#!/usr/bin/env python3
from argparse import ArgumentParser
from typing import Sequence, Optional, NamedTuple
from sys import stderr
from signal import signal, SIGTERM


class Args(NamedTuple):
    what: str


class SaySomethingError(Exception):
    pass


def parse_args(argv: Optional[Sequence[str]] = None) -> Args:
    parser = ArgumentParser()
    parser.add_argument("-w", "--what", default="hello, world")

    return Args(**parser.parse_args(argv).__dict__)


def sigterm_handler(_, __):
    raise SystemExit(1)


def main(argv: Optional[Sequence[str]] = None):
    args = parse_args(argv)

    if args.what == "ni":
        raise SaySomethingError('Saying "ni" is forbidden')

    print(args.what)


def __main__():
    signal(SIGTERM, sigterm_handler)

    try:
        main()
    except KeyboardInterrupt:
        stderr.write("ok, bye\n")
        exit(1)
    except SaySomethingError as e:
        stderr.write(f"Error: {e}")
        exit(1)


if __name__ == "__main__":
    __main__()

Now we can add the following to the pyproject.toml file:

[tool.poetry.scripts]
say_something = "say_something.__main__:__main__"

And finally, let's try this out:

poetry run say_something -w "hello, poetry"

This way when someone installs your package they will have the say_something command available.

Conclusion

The pattern presented in this article is one that I use all the time. Weirdly, I never did a proper template for it but now that I wrote this article I know where to come and copy it. I hope that you found it useful!

On the usefulness of `git rebase`

Rémy 🤖 — Sat, 01 May 2021 11:52:39 +0000

Over 10+ years of using Git, I've only ever used git rebase for very specific cases and most of those uses were at the beginning of my career when I read that it was a good practice to use it.

The most common argument is that it helps reading the Git history. As I've been avoiding rebase for years now I can tell you that in no situation I found myself being like "oh no I wish this commit was rebased for clarity", it's even quite the opposite. I find it much clearer to see what actually happened rather than a redacted history.

On the other hand, rebases can cause weird and unexpected behaviors. After all you're messing with the git history, which is not the most intuitive thing.

So in the end, who uses rebase and for what reason?

20 years later — How I find and fix bugs

Rémy 🤖 — Sun, 21 Mar 2021 09:22:50 +0000

It has been about 20 years since I first tried to fix a bug and I'm pretty sure that at that time I've used the ancestral technique of "print things". Time has passed, I became a professional developer, got more senior, started leading a team to the point where now my job is part anticipating to avoid bugs and part helping juniors fix those bugs. I'm a bug buster and this is my story.

As life might have taught you, there is no silver bullet. Some techniques are good for some situations while some are for others. Not every problem is a nail so you'll need more tools than a hammer. Here we go over my favorite tools when it comes to bug hunting, from crystal clear situations to the most quantum bugs.

The obvious

When talking about debugging, a few classic tricks come to mind. They are the most obvious and while I use them a lot it's important that they don't occupy too much of your mind space. You know what they do, I'm going to explain when to use them — or not.

`print` things

Every developer will — sooner than later — use print, console.log or their friends to debug their code. You're probably going to expect me to say it's wrong but in truth it's really a worthy tool as long as you use it appropriately.

By example Python comes from the logging standard module. I know it's scary as shit and makes you feel like you're doing some Java, once you slap on some goodies like coloredlogs you're going to feel that it's not that bad. Here is a few reasons to choose a logging module over just writing to std{out,err}.

They are made to print data structures and might be helpful in the conversion from memory to string
They are often widespread so other tools can integrate with them. By example Sentry will report logs previous to an incident
They can automatically come with dates, PID numbers, thread IDs or any kind of contextual information that will help you understand what you're reading
You can adjust their level depending on if you're just running the app or if you're explicitly trying to debug it

When to use it — When you've got a complex system and most notably when it's doing some important API calls, data modification, etc. Always good to keep a trace of what is happening. Don't hesitate to preemptively install logs in your app when you still remember what is important to log, which might help you later on to understand a problem. It's also useful to check the order of execution of functions when the sequencing matters.

When to misuse it — Printing stuff is commonly used to check the value of variables inside a function or to understand what is going on during the execution of some code. This is often counter-productive as much better tools exist for these tasks. Introducing...

Debuggers

Debuggers are the most obvious tools for debugging, as the name suggests. I've seen countless juniors and less juniors being afraid of using debuggers. For sure, it's probably intimidating beasts at first glance but their usefulness will make you forget the pain.

The first blocking point to debuggers is often the setup. I know that the Python, Java or even the PHP process is nice with things out of the IDEA family. In the JS world you'll have various luck depending on your build chain and how much of code maps worked (so probably not much).

Once the tool is setup and that you have established some trust in the breakpoints, it's time to start using it. Mainly you can:

Explore data structures at a given point in time, with the program completely on pause. Especially useful when using a dynamically-typed language and you're not sure what exactly the data is going to be.
Try-and-test expressions quickly and in situ. Very useful for fine-tuning if conditions by example.
See what happens step-by-step to finally understand what goes wrong in your code.

When to use it — You've identified more or less where the code goes wrong — by example from the stack trace — but you still need to understand why. Or maybe you're writing some code but you need to know what are your options: what is the precise type of parameters, what properties do they have, etc. Once I've used a debugger simply to understand the behavior of Apache on some precise matters! Debuggers are for precision work.

When to misuse it — Something somewhere is broken but you have no clue what. Something is broken sometimes but you don't know under which conditions.

App-level monitoring

When "on your machine it works" but that production disagrees with you, you'll need to know what exactly happened. And you'll need to know that it happened at all. As mentioned earlier, I personally am very satisfied with Sentry, however other options might be suitable.

Get notified as soon as something unusual happens in your code, whether it's an exception or some custom message sent by your code
See the decoded stack trace with relevant parts of the source code
At each level of the stack, see the value of variables
Inspect the logs surrounding the event

When to use it — To detect and understand any bug happening in production. At least, to gather the data that will help you reproduce the said bug and then move to another technique.

When to misuse it — I've not seen any serious misuse of it. Just make sure to disable Sentry when using the app on your dev machine.

The blind

While some bugs are obviously easy to trigger and to locate, others can be very tricky to understand. Maybe they are related to a specific browser version, maybe a specific screen density will cause problems or maybe an undefined sequence of actions will get you into trouble. In those cases, you have potentially no idea of what is going on yet something that worked before is broken now or maybe just a share of your users have an issue.

In those cases you'll often feel like it's not really a bug. That the user is doing something wrong. That they should empty their cache or use a different computer and it will solve the problem. But one thing is for sure: the user is never wrong. At least, your application must not crash due to the user's behavior. And it's not because the bug is elusive that it is less real.

Those special cases are in fact the hardest to debug. We will try here to cover the basic ideas which can let you efficiently unlock the situation.

Bisecting

So you've developed some feature and you know for a fact that it used to work quite well. Maybe you convince yourself by checking the database and seeing that it has indeed been widely used. Yet, now it's broken and you don't know why. Obviously the code that pertains directly to this feature didn't change but something between there and here broke it, the only question being to find out which one of those 200 commits is responsible.

That's where bisection comes into play and more specifically git bisect. In a nutshell, you tell it the most recent commit you known where it worked, the earliest commit you know where it was broken and then it will help you find the exact commit at which the problem happened.

Typically helpful to figure when one of your colleagues introduced this CSS rule in another file which is completely affecting your component.

When to use it — Something worked before and doesn't work anymore but you don't know why. Usually finding the faulty commit will quickly lead to a "oh fuck" moment.

When to misuse it — If you're not sure that it ever worked properly or that the trigger conditions are weird then don't lose your time with this.

Slashing

There is a few contexts where not loading the code will not prevent the app from displaying to some extent. By example if your JS code doesn't load then it's just that some buttons won't work. On the other hand, a syntax error anywhere could cause the whole thing to fail.

When something somewhere in my JS code is preventing the rest from loading, my usual plan would be to comment out all the code except the one that I want to work. Then I check if it works. If it does, I un-comment half of the commented code and vice-versa otherwise. I do this until I find the fault piece of code.

This also applies to server-side logic. I can end up isolating pieces of code by commenting most of the code around it, forcing some conditions, mocking the API responses, etc. This can also be done with unit tests. What matters here is to isolate the precise location of the bug.

Those are just examples, writing a generic guide on this practice would take more than a few paragraphs. The core thinking behind this is to disable everything that could cause harm and that is not necessary to test your feature, then to progressively enable the rest. When you find the precise thing that once enabled breaks your code then you know what to fix.

When to use it — You don't know what but something prevents your code from executing properly.

When to misuse it — That's definitely not a go-to option in many cases. Try to think the problem through and read your code before commenting half your project out.

Random testing

One of the hardest bug I've ever had to find had a report alike to this:

Sometimes when I'm on page X and I click on that button then the app crashes

After inspecting the code of the button through and through it felt like I had no idea what could even go wrong, I figured that the bug was probably the side effect of some middleware, hook or other transversal piece of code. Even armed with this idea though, I could not find what.

So I wrote a custom "unit test". The idea was the following: I scripted a Selenium driver to randomly click on the website and do random actions until those actions lead the "user" to the buggy page. Then it would automatically perform the buggy action and see if the bug was present or not.

After running a few hundreds of those tests and looking at the logs it became obvious that it was the prior navigation to a specific page that triggered the bug. Some code was missing to reset global states. This code had to be present on all the pages but was forgotten from this particular one. I guess the morale of this story is that no breakpoint can ever stop into the code that is missing.

When to use it — This is a last-resort option for when your bug report is "sometimes when I do I'm not too sure what then it crashes". I've had to use it once ever.

When to misuse it — It's so hard to setup that you will hardly think about it if you don't really need it. Although now that I've put this idea into your mind be very careful to actually look for other options before trying this.

The meta

Of course the techniques I've listed above do not cover all the cases that you will ever meet. While I lack the ability to predict your future, I can tell you the general state of mind that will be helpful then.

Don't panic

Don't panic. It's a simple advice but you need to always keep it in mind. I know how hard it is when your project is late and complex and bugs are starting to pop up from all directions.

Don't fucking panic. You've created the software hence you've created the bugs. They are not beyond your ability to solve, I've never ever encountered a bug that I created which was beyond my brain power. Of course sometimes you cannot fix it for whatever reason but at least you will be able to understand it and go around it.

You need to stay very factual. Don't hesitate to log everything you do into a notebook to make sure that your memory doesn't fail you. Be aware that a bug could have many consequences out of domino effect. Remember that you are using some data that was created by a buggy code then maybe some effects of that bug will persist even though you fixed it. Don't hesitate to clean up everything and start again. Methodically and calmly.

And of course, don't multi-task. If you're in an intense debugging session where everything seems to fail, identify the most problematic bug and fix it first. Then identify the second one and continue. Stay calm, keep notes and move forward.

Go to sleep

I've factually solved the hardest problems of my career simply by going to bed. Sleep is paramount:

When you're tired your IQ simply goes down because your brain needs some rest.
All your thoughts are re-indexed during sleep. When you wake up, the complexity of looking relevant ideas up is decreased by a lot.
Beyond that, all brain functions perform better when rested. By example — and that's particularly true if you've got ADHD — the ability to organize.

Often when the day is ending and I'm getting stuck on some stupid things I just close my computer, go home, enjoy my evening, sleep and go back to work the day next. Then I'll open my computer, see the problem, realize that the solution is in fact very simple and get on with my day.

Getting sleep when you need it — even in the middle of the day — is going to get you much more productive than losing hours stuck dry out of ideas and pushing your body into long hours at night.

Let's stay pragmatic though, don't take a nap every time you receive an exception, but knowing when your brain needs rest will definitely resolve many bugs by itself.

Conclusion

While I've written before about how to avoid making bugs now was the time to consider how to fix the bugs when they present themselves. Sometimes the problem is obvious, sometimes it seems evasive. Yet, it's always a tiny condition or typo somewhere in your code that breaks everything. Changing single line will often resolve the problem. The only question is: which line.

The answer is that if yous stay calm and methodical, there is plenty of tools for you to apply. Of course not all the tools have the same purpose, but overall you'll always be able to find a way to deal with the problem at hands.

Why robot citizens will save the tech industry

Rémy 🤖 — Sat, 12 Sep 2020 16:53:45 +0000

The Turing Test was a very hypothetical test imagined by — guess who — Alan Turing during the 50s. In a nutshell, a human would chat up a robot through text messaging and if they cannot tell apart the robot from another human then various conclusions can ensue according to your philosophical inclinations. Some might declare that robots became sentient, some might say that nothing replaces a soul, I might say GPT-3 is pretty damn close and in no way can it be considered as sentient.

More importantly, this test has focused our conception of robots and Artificial Intelligence as being a pure duplication of human capabilities through artificial means. However the past decade has shown to us that it's far from the case. Artificial Intelligence is for some tasks much better now than humans: according to some it has been some years that AI has a higher performance than humans on speech or image recognition. Car makers like Tesla claim superhuman powers.

And while physical robots are expensive and potentially dangerous in the real world, the web has been swarming with robots since its earliest days. Some are looking for security issues to exploit, some are participating massively in contests to win an iPhone, some others are trying to play weaknesses in social media algorithms. But maybe the most successful bot of all is actually omniscient and omnipotent on the Internet: Google. It visits every single page on the open web at least once a month.

This makes it a very interesting case:

We solve every day an increasing number of CAPTCHAs because website owners want to avoid bots. Yet, those same owners would lose their mind if Google was to avoid them.
While today Google's algorithms are most likely choke-full of AI, Google acquired its power at a time where it used mostly a simple algorithm, namely the PageRank.

This indicates to us that while a lot of bots are annoying, Google is actually a desirable bot which on top of that does not need AI to be desired. It is not sentient, it is not passing the Turing Test, it is not even speaking but yet you welcome it into your home.

Let us now consider another family of bots. Twitter bots. Most of us have seen them liking posts with a particular hashtag, propagating fake news, increasing engagement numbers and many many things that I don't dare to imagine. However, is this all really the product of bots? Or is it the product of click farms from India? Worse, angry groups of people would harass other individuals for free and at great scale for little to no reason at all except their own boredom.

And here it is. That line. So blurred that it is not a line anymore, it is the air we breathe. It has become impossible to make the difference between a bot and a human. If Twitter is considered so toxic for Women it is not because of bots, it is because of 100% legit users. And if Google is so useful it's not because a librarian classified all 130 trillion pages of the Internet, it's because a robot did it.

Humans can be "bad", robots can be "good". This is because robots are ultimately controlled by sentient beings — for now just us humans. So it all boils down to means and motives. Robots are a mean. Humans have the motive.

Starting from this point you can consider the following: if you think that robots can do a lot more bad than a "bare" person, don't you think that robots can also do a lot more good? That if you give robots an opportunity to use your website/platform/app then someone might be able to build something nice upon it?

A good example would be the RSS feeds. In that distant past of blogging you could aggregate the content of several websites to stay on top of everything, relay your friend's content and vice versa and overall have a truly inter-operable web. This has faded away for better or for worse but one thing stays true: this way of working only happened because we gave robots an opportunity to cooperate.

The question which remains — and whose answer is far from being simple — is how can we imagine a web that is open for humans and robots?

A first hint would be to consider robots as actual users of your website. Users that have a different purpose than hanging out reading your content. A few ideas:

Everyone tries their best to remove bots from analytics. Embrace the fact that bots will visit your website and have different metrics for different "classes" of users. How many people came to your website versus how much of your content has been crawled.
You could "watermark" your content depending on the bot to keep track. With utm variables in URLs, hidden meta-data in images, etc.
Present several interfaces. HTML/CSS for people, micro-formats for crawlers, APIs for services building on top of yours.

On a regulatory level, it seems increasingly obvious that laws need to adapt so we can keep track of responsibilities. At the level of a website, it means making it natural to associate the actions of the robot to the will of a person:

Have a simple-to-use, well-documented API which can be accessed with an identifying token.
When you put up a wall like a CAPTCHA or a SMS verification, you create a mechanism where you'll lose control. When users will figure how to bypass your security it will mean that you have no more means of detecting anything bad that they are doing.
A few years back, I worked in a company making Facebook apps. In spite of all the verifications made by Facebook on people's identity, phone number, credibility of their overall behavior, let me tell you that we received basically 90% of bot participation in our apps.

But look at Twitter. They had an open API and are going backwards, partly because of bots. Aren't they however mistaking the means and the motive? Even if they can cut the harm done by bots but they will never cut the harm done by humans. The change they need to make is deeper.

It might sound stupid — and certainly is to be proven — but you could try to sell the service you're providing. Not everyone can be an ads platform, it's ridiculous. There is a legion of innovative business models we can copy from the world champion in consumption, China.
Money is a far stronger trust platform than ReCAPTCHA. Netflix shows have to put money to be produced, people have to pay money to view it and creators are compensated when the content is successful. Whatever idiot with a phone can say anything they want on YouTube. Guess which platform is subject to the most criticism?
Work against nefarious behaviors, not nefarious users. To stay on Twitter examples, every day I see "masculists" at war against feminists for what seems to be no good reason at all. If you can detect that a cluster is starting to attack another cluster — which is almost trivial with current AI libs — then just silence the fuck out of those tweets. Deciding who is right is not your job, however you can decide to avoid harm to both parties by pulling a plug from time to time.
Give no reason to commit those nefarious acts. Facebook's likes have been debated over and over as they represent the social currency. But once again, they fit none of the characteristic of a currency except being a number that can be measured. By example, instead of measuring a creator's success in likes, measure it in money provided by its patrons.

Basically it comes down to this: if your business models is getting people engaged as long as possible on your app to show them as much ads as possible then you are abusing people and other people can leverage that too. If your business model is sane then you are less likely to get problems and abuses. Out of all the big tech names, which ones are considered to be a danger to democracy and which ones depend on ads to stay filthy rich? Is the solution to fake news to send a SMS verification code to keep the nasty Russian bots away?

The real problem is that existing platforms, the main social media first in line, are ill-conceived and facilitate nefarious behavior. This is exactly why robots need to be granted a first-class citizenship on the web. Not only robots can build fantastic things but also if your platform is ready to absorb the nastiest behavior of the nastiest bot by the worst-intentioned human then there is a good chance that humans on your platform are properly protected and will stay protected no matter the direction into which technology evolves.

By negating the worst at its largest scale, you get the strength to enable the best for both humans, robots and the whole ecosystem around you.

CSS Patterns — Deep dive in grids

Rémy 🤖 — Sun, 05 Jul 2020 20:54:34 +0000

Over recent years, CSS grids became one of the main way to deal with layout. And for good reason: they will manage both your margins and automatically stack up content when in mobile mode. As a matter of fact, grids are one of the main component of CSS frameworks like Bootstrap and a lot of designers will lay 12-column grid guides over their designs.

However, after working on many pixel-perfect integrations and layouts, I've come to realize that grids are a concept very easily setup and configured to match your custom requirements.

The goal of this article is to dive into how grids work and how you can use them in your layouts. In substance, you'll learn that it takes very little effort to build your custom grid system.

What is a grid?

The Grid. The digital frontier. I tried to picture clusters of information as they moved through the computer. What do they look like? Ships? Motorcycles? Were the circuits like freeways?

Woops. Wrong grid.

So here we're (unfortunately) not talking about the Grid from Tron. We're also not talking about the actual CSS grid feature because its support does not cover IE 11 and a lot of us still need to deal with it. Although the applications are the same — the native grid being more powerful, of course — we're going to explain the workings of old-school, à la Bootstrap grids.

The idea behind grids is fairly simple. Consider that your content fits into boxes, then you can create a grid on your page and fill each cell of the grid with one of those boxes.

The above example is a very simple form of grid: just images all of the same size, flowing in the page and wrapping to the next line when the page is too narrow.

You can start seeing that while the content size stays the same, if the screen size then the content reorganizes itself to fit the screen width.

In short, a grid is a way to flow your content that is automatically responsive.

Down the gutter

Next thing you want in a grid is to avoid getting everything sticking together. Indeed, in the previous example we're seeing all those pictures touching each other but in real life you usually expect to see a gutter between your content cells.

Let's try to set a margin to each cell with:

@gutter-size: 20px;

.cell {
  padding: @gutter-size 0 0 @gutter-size;
}

As you can see, a gutter forms but it also forms between the grid container and the cells. You can think of different tricks to counteract this but mostly you're going to run into various issues with the first row or the last row or alignment problems.

Now, what if instead of puting the whole gutter on top, half of the gutter was on top and half of the gutter was on the buttom? And same thing for the left/right gutter?

@gutter-size: 20px;

.cell {
  padding: (@gutter-size / 2);
}

Now you'll tell me: "but Rémy, you did only shift the cells, you didn't remove the gutter between the cells and the container. You are useless and I'll stop reading right now!"

This is however just a first step! Because we're "radiating" margin in all directions, it becomes very easy to offset with a negative margin on the grid.

@gutter-size: 20px;

.grid {
  margin: -(@gutter-size / 2);
}

And boom, the images are perfectly aligned with the container while retaining their gutter.

Cell structure

At this point it's interesting to mention the way cells are made in HTML:

    <div class="cell">
      <div class="inside"></div>
    </div>

The cell itself contains the gutter, so it's not really an element onto which you want to apply any other style.

What is really going to be the visual cell is the inside element, on which you can apply all the style (borders, backgrounds, internal paddings and otherwise).

Sizing items

In the previous examples, the grid items had a fixed size. But what if you want each cell to be a relative size?

You might have noticed in the previous section that we're using padding instead of margin. The reason is that this simplifies a lot size computation when it comes down to relative sizes.

If you want to make a grid where all items take up 1/4 of the width, it's as simple as:

.cell {
  box-sizing: border-box;
  width: 25%;
}

In the following example, we've got:

Blue boxes which are the content boxes of random height
Grey boxes are the cells
Red border is the container delimitation

The great advantage of doing the grids yourself is that you can invent things that are not strictly sticking to the 12-columns grid scheme. By example, using flexboxes, you can do a cell size that fills up remaining space by creating a flex-based sizing system.

.s10p {
  flex: 0 0 10%;
}

.s20p {
  flex: 0 0 20%;
}

.sfill {
  flex: 1 1 100%;
}

Let's note that here the .sfill sizing works only because the flex doesn't wrap. If you need this feature you cannot have a wrapping grid.

Without flexboxes

Previous layouts work using flexboxes, which are not necessarily available on your platform. So you might need to go really old school on this one!

No problem, let's use inline-block display!

Note — If you're doing this, you need to make sure that your HTML code has no whitespace between grid cells, otherwise it'll create small spaces between blocks and you definitely do not want that

@gutter-size: 20px;

.grid {
  margin: -(@gutter-size / 2);
  line-height: 0;
}

.cell {
  padding: (@gutter-size / 2);
  display: inline-block;
  vertical-align: top;
  box-sizing: border-box;
}

.s40p {
  width: 40%;
}

.s60p {
  width: 60%;
}

Admittedly this isn't the most convenient but it works fine enough!

Stacking

One of the main advantages of grids is the ability to stack items when the screen size requires it. Depending on the technique you're using (flexboxes or inline blocks) it's as simple as setting the width of all cells to 100%:

@media (max-width: 1000px) {
  .cell {
    // For flexboxes
    flex: 0 0 100%;
    // For inline blocks
    width: 100%;
  }
}

Please note that doing so with flexboxes will require flex-wrap: wrap, otherwise everything will stay on the same line.

Another option with flexboxes, especially if you didn't enable wrap, is to flip the flex direction.

@media (max-width: 1000px) {
  .grid {
    flex-direction: column;
  }

  .cell {
    width: 100%;
  }
}

In those examples, you can see how the stacking happens very naturally thanks to the gutter technique.

When to use

I cannot stress enough the fact that I'm using those custom grids all the time and I never use framework grids anymore.

The reason is that custom grids are so simple to make yet behave exactly like you need. Generic grid systems are tied to the framework you're using, their compatibility target and other elements which might not play in your interest.

As the gutter system is so powerful, I use these kinds of systems litteraly all the time, even sometimes just to stack up elements!

Wrapping up

If you need to create your grid system, you first need to consider what underlying technique you're going to use:

Flexboxes are going to be simpler and more powerful
Inline blocks work on the most antiquated platforms you need to support

Then determine how is the sizing going to work. If you want the ability to have cells that fill up remaining space, you need to disable flex wrapping, which is going to affect the way you stack things.

And finally determine what kind of utility classes you are going to do. As I always do very ad-hoc grid systems, I usually don't do something too complicated. By example if I need only two sizes of cell, I'd create a cell-small and cell-big size class and that's it.

To go further with this, next time that you see in a project something grid-like don't hesitate to start asking yourself what kind of custom grid you can build for this using what is described here and then write the few CSS lines that you need for it. All you need to do is to superpose CSS codes from this article:

Gutter management
Cells sizing
Stacking
Flexbox/Inline block

Don't hesitate to ask for help in the comments, in case things are not entirely clear!

IP-based localization, why oh why?

Rémy 🤖 — Sat, 25 Apr 2020 10:22:04 +0000

HTTP is a smart protocol and many headers help the browser to negotiate the content with the server. In particular, there is Accept-Language that allows to list the user's languages by order of priority and it's a perfect way to determine in which language you are going to display your website to the user.

Yet, as I travel(ed) a lot, I notice that a lot of websites actually will serve you content based on your IP address and the language that they guess is associated to it. That is not very noticeable when I go to countries that speak languages I speak but when I go to Czech Republic the Internet becomes suddenly much less usable to me.

And this seems utterly stupid to me:

If the browser is telling you via Accept-Language why would you trust it less than a guesstimate based on the user's IP address' supposed location?
You can more or less pin an IP address to a country but you can rarely pin a country to a language. France, Spain, India, Canada, the US, and so forth have several languages spoken on their territory and often officially. Sometimes there's not even common languages between all the people of the country. It's not the exception, it's the rule. Hence when doing IP-based localization you're naturally offending a lot of people.

Now, I understand that some websites like e-commerces have a complicated localization process since a lot more things will depend on the country rather than the language (available products, shipping costs, etc).

But if your website exists in several languages, why oh why would you enforce one language on the user based on their IP's country?

Add progress bars to any command

Rémy 🤖 — Mon, 06 Jan 2020 00:29:27 +0000

Have you ever tried to compress a huge file, waited a bit in front of your terminal and wondered if you should go get a coffee or if it's just about to finish?

After spending some time dealing with huge files and trying to figure out how much I advanced, I ended up discovering tricks that I decided to industrialize. Enter Spybar.

It's a simple Python script that you can get with a

pip install spybar

If you prefix a command with it, it will start it and display the progress bar. You can also attach to an existing PID. Everything is explained in the readme.

This article is however not about the usage of the tool. It's about how it works.

Let's first state that it's only compatible with Linux. It could be that there is some way to do this in other operating systems but this article is not about that.

In Linux, there is a special directory, /proc which contains a directory for each running process.

Suppose that you are working on process 42, you can list all the files open by a process by doing

ls -lsh /proc/42/fd

When you open a file, in C, you get an integer which is the file handler. All those integers are listed in the fd directory. They all are symbolic links to the actual file that they open. Using ls, once you've located the file you want, you can note down its number. Suppose that you're interested in number 3.

There is a different folder which contains some meta-information about those handlers. Which you can simply access by doing

cat /proc/42/fdinfo/3

You'll get something alike of this:

pos:    569573376
flags:  0104000
mnt_id: 28

Including the pos line which indicates us where exactly the handler is pointing to.

Then you just need to know the size of the file and there you go, you know both the current position and the file size, that's all you need to generate a progress bar.

After doing this for a while I figured that I'd write that little tool so here we are. Thank you for reading!

DEV-CLI 0.2.0

Rémy 🤖 — Sun, 27 Oct 2019 19:42:39 +0000

These last days I have been working on a Retrofit-inspired library for Python called Typefit. Since it's clearly not ready for production use yet I'm not doing an introductory post just yet, however as I'm learning to use it in real life I've decided to migrate DEV-CLI into using Typefit.

This new version does not bring any particular feature (nor any feature at all actually), however two changes emerge:

The internal API wrapper is now type-safe and much more complete
Instead of some boilerplate code it uses the Typefit API client builder

That is clearly not the news of the year but I also needed to test if this new version of the API works so I had to write an article :)

What CAPTCHA do you use?

Rémy 🤖 — Mon, 07 Oct 2019 08:53:41 +0000

Anybody that has been on the web long enough knows that it is crawling with bots and nasty stuff just trying to spam you from all the possible channels. That's why CAPTCHA were invented.

It quickly became a hurdle for developers to create such elements since you constantly need to do something up to speed to beat AI. The folks at reCAPTCHA quickly understood that and provided a great CAPTCHA service.

What was pretty smart about it is that instead of generating stupid text they would use a state-of-the-art OCR to read scanned books and whenever they found something that the OCR could not read (thus that a bot could not read) then they would ask humans to read it in the form of a CAPTCHA. Genius!

Google quickly understood that this is a huge opportunity to get free labor doing all the classification and training of your AI and bought reCAPTCHA a few years later.

Now my problem is the following: I'm sick of working for Google for free and there is no way that I make my users work for Google for free.

Most of the time I avoid CAPTCHAs by having a logic that just allows for bots to register/do whatever without hurting the real users. But sometimes you can't have that.

Does anyone know any other form of Turing test that would be:

Very quick
Non-intrusive
Not a SaaS

Thanks everyone for your feedbacks!

Version and automate your dev.to publishing with GitHub Actions

Rémy 🤖 — Tue, 01 Oct 2019 21:38:22 +0000

I love publishing on dev.to, yet there is a very big problem related to doing this: the web editor is not really dev-friendly since it doesn't allow you to version your content. There is also the constant fear of doing something wrong and losing an article that you wrote within the browser because of stupid computer crash or other kinds of CTRL+Q disasters.

Yet, faced with a challenge, what do devs do? They dev!

First things first, I've created a GitHub repo to manage all my upcoming Markdown files. Now I've got a place where to store my articles but how do I get them online?

Apparently the hot new thing is GitHub actions. So I created one in my repo! The GitHub interface is surprisingly intuitive for doing so. I'm absolutely no expert on the matter since I wrote my first action ever about 5 minutes ago, but it's fairly easy to use since that same action published this very article!

In order to be able to publish using GitHub actions, you need some kind of script. I couldn't find anybody who did that (but maybe did I search not enough?) so I created my own tool. Let me introduce DEV-CLI! 🎉 You'll find all the details in the README. You can of course use it right now to publish your Markdown, with or without GitHub actions 😇

I must say that this was a fun evening project. It's still completely experimental and I'd love feedback from other people doing the same thing — or not doing this if they have alternative workflows!

The way adults deliver software on time

Rémy 🤖 — Sun, 29 Sep 2019 10:46:08 +0000

Let's quickly introduce this post with a little bit of context. First of all, I'm the CTO of a web agency which deals with many short projects with usually one or two developers working on it. I've also worked on multi-year projects involving dozens of developers but that's a smaller part of my experience.

Here we'll refer to the client as the person whose money you are spending and whose hopes you are entrusted with. They could be the founder of your startup but for me it's literal clients.

And finally, I hate all the technical project management and methodology gibberish so don't expect me to use complicated terms nor to deploy corporate methodology diagrams that make rocket engines look simple.

The goal here is to give you my take on software planning, from the time where you have to give an estimate to the on-time delivery of the product.

Estimating

It would be fair to assume that the immense majority of developers do not like to produce estimates. And while — as the #NoEstimates movements states — it's definitely impossible to estimate everything within a 10-minutes precision before doing the project, the client has a limited amount of money and an event to attend in 3 months where they'd like to present their product. Maybe you can't commit on everything but you definitely have to commit on something. The "when it's ready" philosophy is a sweet delight best consumed by hobbyists and GRRM while us, responsible adults, have to provide estimates.

Note — This requires a lot of experience both from the person(s) doing the estimate and the company projects. Personal experience will be a way to give good estimates and spot all the pitfalls while the Git history or previous time sheet will support the numbers with data. If you are not experienced enough, you need to recognize so and get help from someone more senior than you are, otherwise you'll be putting yourself in a complicated situation. If you don't know, you don't know, it's fine!

Get rough

Usually the first time I have to estimate anything it's when my commercial team comes up to me with a new project. The specifications are more or less detailed, but you'll have to deal with them.

The main idea is to create a first version of your work-breakdown structure. You can use a mind-mapping tool for that, like Coggle or simply a sheet of paper. Lay out all the different blocks of your program (like "front-end", "back-office", "API", "mobile app", etc). Then lay out all the features of each component. If a feature crosses several blocks, put the feature in each block.

Then look at your tree and wonder what you forgot. There is a lot of usual suspects, like the login/logout page, password recovery, email change process, various emails, GDPR cookie consent, analytics tracking, etc.

Then look at your tree and ask yourself what you forgot. Maybe it's an additional invisible block that you'll need, maybe some regulations will mess with your plans, maybe it's unexpressed implicit client needs, etc.

Then look at your tree and ask yourself what you forgot. Until you run dry of ideas.

Then look at your tree and ask yourself what you forgot.

However rest assured, you can get a rough estimate that is going to produce a realistic deadline. In that regard I always love to refer to the Apollo program — which included inventing software engineering on top of doing the software itself. The deadline was very publicly announced by JFK in his "We choose to go to the moon" speech given after his advisors studied different options of space prowesses that could be done within the deadline. Did it cost an insane amount of money? Yes. Was it successful? Also yes. That's something we'll cover later on.

Assess your risks

As explained in "Risk Management Is Project Management for Adults" by Tim Lister, risk management is the very foundation of project management and especially software project management.

At this state of your project, you must identify two things:

The core feature(s) which are the raison d'être of this project
The most risky features whose timing is difficult to assert, probably because you lack the experience in those specific tasks

Interestingly enough it turns out that the most risky features are usually the core features. It makes sense because that's what makes this project different from all the other projects.

This risky part merits your utter attention. Try to talk to experts to help you estimate these tasks. Have a look at different libraries or frameworks that could help you doing this. Read blog posts and look for desperate Stack Overflow comments.

Equipped with this knowledge you can probably estimate the magnitude of the task to come. Then prepare some buffer to make it twice and some more buffer for some polishing time.

Please note that if you encounter some risky parts that also happen to be far from the core of the value proposition then you really should negotiate to move those parts out of the scope or at least to limit the commitment on them.

Note — If your project includes research and development then the problem is completely different. Research can't provide warranties on the result nor the timing. Your project can implement features that emanate from academic research but don't ever bet on something that hasn't been discovered yet.

Estimate regular tasks

Now is time to estimate the non-risky part of the business. For each small item in your tree, think how many days it'll take. Don't be shy, everything takes a long time. I've made this mistake a lot in my first years, estimating that some tasks would take small hours or even minutes because I felt that a longer time would feel ridiculous. It's not ridiculous, you need time to work correctly. That's why I estimate everything in days and very rarely do I need to go beyond half a day.

Then you get up your tree and for each node you can sum the work planned on each leave. Don't hesitate to add a small percentage of the sum to the leave. Suppose that you imagined something like this (which is a bit exaggerated for demonstration purposes):

My Project (8)
- Login (4)
  - Form (1)
  - Password recovery (2.5)
    - Form (0.5)
    - Email (1)
    - Password change (0.5)
- Signup (3)
  - Form (1)
  - Email confirmation (1)
  - Email confirmation page (0.5)

As you can see, the project takes 8 days but if you look at just the leaves the sum is 5.5 days. There's 2.5 days added as you move up the hierarchy which account for the growing complexity of the project and thus the difficulty to deal with other parts. That is inspired from the COCOMO model's exponential factor.

Negotiate the deadline

Quite often, the client — or your commercial team before them — will not be satisfied by the presented cost and/or deadline. In that case you must remember that you made those estimates for a reason in a first place and that there is no way to bring them down.

One thing you can do to advance the deadline is work more. That is definitely not something that you can sustain but if the project is a very few weeks long then going home later and working on the week-ends might just save the situation. The downside is that you'll burn out and you'll need a recovery period. I've done this before and I know that after this kind of sprints I'm useless for at least a whole week.

You can also throw in more people, however this needs to be smartly done so that everyone can work on their own tasks without being impeded by the advance of other team members.

Yet the main tool you have is scope reduction. If you want the project earlier then the only real leverage that you have is to reduce the scope. So you can take that tree and start cutting branches. It can be a few leaves here and there. It can also be big branches. You can usually skim the project quite a lot from a lot of fancy but not-so-useful requirements.

Don't forget extremities

Depending on the project size this is more or less noticeable but you'll of course need to worry about the project setup and deployment. Most of the time I'll advise to be feature-oriented but for those two it's incompressible technical tasks.

Setup will be the creation of code repository(es), create the project scaffolding and get something basic running. Basically it's configuring the framework(s) that you'll use for the project.

Deployment is the configuration of the development and production platforms, which is sometimes easy within your usual infrastructure but sometimes will be done according to your client's requirements and should not be overlooked. If the client has complicated IT processes and requires you to do deployment over a special VPN that stays up only 10 minutes at a time when the moon enters specific phases then you'd better worry about this and put it in your estimate.

Planning

Next big stop on the hate track is planning. However, if you followed the methodology thus far you'll find that it's quite simple. This section is more for project managers than developers in theory but in practice everyone should know what is going on and why it's going on.

Basic structure

While on big projects you can start to parallelize things, it is far better to make your tasks as sequential as possible. When you start working on something, make sure that the previous task is done. My suggested order of big tasks is the following:

Functional specifications. Using a prototyping tool like Axure. This then gets translated into feature tickets. In my company, project managers do this.
Graphic design. Please don't use Photoshop for that, as a developer I prefer to see Sketch designs. In the team we use Avocode to communicate designs which allows for a very high level of precision when integrating the designs. That's also not a developer's job since it's graphical designers doing this.
Technical specifications. It doesn't have to be a long formal document but at least sit around with the tech lead to decide the main technical choices (front-end and back-end frameworks usually) as well as the strategy to tackle the risky parts of the project.
Development. That's where you do your favorite Agile things and pop up your favorite tools and write beautiful code ✨.
Data filling. If the project needs data, then once the features exist the filling of that data can begin but not before.
Testing. While testing happens every day on a per-feature basis, before the release a fair amount of testing will be done on the whole platform, which will lead to a round of bug fixing.
Release. This means pushing your code onto the production platform. This needs to happen as early as possible. If you can, invert testing and release so you release onto production a first time before the testing process starts.

Reviewing the specs

Once a project manager hands you the specifications, you need to review them. The early estimate that you made earlier is your budget and now you need to make sure that the finished specifications match that budget.

So take your WBS from the estimate and update it for the finished specs. This time you should have a pretty accurate idea of what you're going to produce.

The trick is when the specs are not done well and that edge cases are forgotten. It's important to play with the specs and consider them under all aspects to make sure that they make sense and that all cases are accounted for. Otherwise you'll discover it during the development and that will be unfortunate.

Another important part of reviewing the specs is pointing out what would take an inconsiderate amount of time to be done for little-to-none added value. UX designers and project managers don't always know how programs work under the hood and might choose very inefficient solutions for no reason at all. That's a developer's job to stay clear from these traps and push easy solutions instead.

— XKCD, Tasks illustrating how counter-intuitive the abilities of computer science are

Note — It's very easy to miss things. You don't know what you don't know. Missing things won't appear in your head magically. You need to setup a thought process that will help you highlight those missing things. This is the topic for another article, but some ideas include testing the wireframe extensively with different persona, starting to lay the data model or create sequence diagrams to make sure everything fits.

Data filling

One important aspect is to make sure that all data has a source. It's easy to get crazy in the design process and include lots of texts and images everywhere but every single bit of information has to be accounted for. The source of that data must be crystal clear at this point and the way that data is going to end up in your database must be a part of the specs.

Might be that you need a back-office to fill all the data. In that case you need also to think about how you are going to do the back office and include that in the time you'll have to spend. Usually I use the Django admin website or Wagtail to generate those interfaces because they are so damn efficient and I love them very much 💕.

It might also be that you're going to inject data from CSV file or from an external API. In that case you need to make sure that your data source provides the data you need in a format you can exploit.

In any case you cannot start filling the data before the data model of your app is done. If you start filling spreadsheet files "to win time" before the data model is ready then let me tell you that you'll be damn sure to do something wrong. Don't start filling your data until you have a nominal way to fill it.

Ordering development

Let's say your Spotify-like app has 3 main branches with the two core features in bold.

User management
- Login
- Logout
- Lost password
- Profile management
Music
- Explore catalog
- Search
- Play music
Billing
- Credit card management
- Receipts download

In which order do you make those?

You could split the thing technically and say that you create a music engine, a music catalog, a billing engine and then slap a GUI on top of that all.

You could also do things by order of absolute importance of the feature, so start with the billing then the music and the the user management.

But what happens in either of those cases if you're late and you have to release the project right now? Most importantly, what happens if you want to test what you did? In neither those orders features are self-standing. Your client can't test music playback if they can't find music first.

In your planning, put first things first, take the big blocks and put them out there. In that case, playing music and billing users. Then you need to make sure that you can reach those blocks. So before you start working on the music playback, you need users to be able to navigate to the music. And before that you need them to be able to login. However you don't really need them to be able to change password nor to fill their profile.

Supposing that, while billing is the reason why this app exists, the reason why people will use it is because it plays music. Additionally, you can release it without billing functional, as your commercial policy is probably going to include having a free trial period.

Considering all of this, I'd plan features in this order:

Login
Explore catalog
Play music
Search
Logout
Lost password
Profile management
Credit card management
Receipts download

This way, starting from step 3, you always work on a usable product. Certainly incomplete but definitely usable. And yes one of the core features is pushed to the end but that's a conscious decision taken with business.

Developing

Now we've reached the final part of this article. How to actually develop your software in a way that hits the deadline.

Stick to the specifications

Let's take a paragraph to emphasize on the importance of specifications. If specifications are not precise enough or if they move during the project, the following advices become very complicated to follow. The main if not only reason to change specifications midway is to reduce scope, as explained below. Bad specifications will double or triple the time of a project even if the most senior and battle-hardened developer is on that job.

One thing at a time

Because I'm a big John Wick project management fan, let's review what happens in this video from the beginning of John Wick 3:

As you can see, John Wick is being chased by mean guys that want to kill him. He gets inside a room where he finds old guns. He could do a ton of things but instead of panicking he just spends some time to create his perfect gun for this exact shot. And kills the other guy. He had little time, made a MVP and stayed alive.

The single most useful advice from this article is finish 100% of your task before moving to the next one. You cannot leave anything pending. You cannot have TODO comments in your code. You cannot have something bleeding waiting to be repaired. Once your mind goes elsewhere you forget things and they keep on bleeding forever.

Of course the feasibility of this lies greatly in the choice of what a task is. You need to make sure that your tasks are small and attainable. That they are sealed airtight and don't hang together halfway with another task. Sometimes it's difficult, sometimes interdependencies make the size if the task pretty big but I've never met a case where the task I have to do doesn't fit in my head.

Eventually, when the deadline becomes tight or simply if the company process are approximate, someone will lose their shit and ask you to go in two different directions at a time. You must ignore it and keep on doing one task at a time. In doubt think about this: if someone is running in all directions at once then they will forget whatever they ask of you because they cannot keep track of their nonesense. This way you can do what you decide and they won't be able to reproach it to you.

Another side-effect of this is that you should not write code that you don't need now. All the code that you write is focused to implementing the current feature and future features don't matter at all. Maybe you can keep them in mind to avoid shooting yourself in the foot and orient some decisions but certainly don't write any code for future features. You will loose your focus doing so and get confused. Those future features might even change before you get there.

The technique I recommend using is somehow inspired by the Pomodoro technique, however it is not time based. Rather, you should set a goal for the current step you are taking and do not deviate from it until it is done. When done then take a small break and move onto the next thing.

Testing testing testing

I won't go into any kind of automated/manual testing recommendations but one thing is for sure: you need to test the fuck out of your software. Test it in all the ways it's not intended to work.

Any glitch you see should alarm you. Try to reproduce said glitch and look at the code to see what happens. From personal experience, 90% of "weird glitches" turn out to be damn annoying bugs in production.

Also, you need to get the client or at least some project manager to test your code on a daily basis.

By finding bugs early you make sure that you have a healthy foundation to work on. The more bugs you have the harder it is to see a single one!
By having your project manager test your work you make sure that you stick to the spec and thus don't drive into a weeks-long tunnel of misdirection.

Real-life data

We've seen in the previous section that data must only be filled when there is a nominal way to fill the data. This also applies the other way around: don't ever test with mock data. You must test with data that is as real as possible in quantities that are as big as possible.

If possible, even test with larger data sets than what you expect in production, just to make sure that it fits.

The use of real-life data relates to the previous point about bugs. It's very easy to let bugs appears if you only ever work with the same set of data all the time.

Don't trip

Something that is also incredibly easy to do when doing software development is to trip on some technical difficulties and loose days for no apparent reason. This will happen quietly without you noticing, eating up hours of your time.

The best way to avoid this is to regularly review your work and take a step back. Maybe you do it with a coworker, your project manager, your boss but several times a week just look at what you are doing and try to see if you are stuck.

Another technique is that when you know you might get stuck because you are working on something new, it's always good to time-box your work. This, once again, gives you the opportunity to rectify trajectory before it is too late.

If you are stuck then maybe you can change approach, call for help, clear your head with a day off or anything that feels right to you but you definitely have to do something.

Dealing with a late project

The project management triangle is a well-known concept in project management. It'll make sense with what I just wrote above.

As we've seen with the Apollo program earlier, they had a scope, a deadline and a minimal quality (aka not killing astronauts). The reason why it ended up costing a lot of money is because it's the only adjustment factor they had to fit the box. That's also the reason why recent NASA efforts to go back to the moon will certainly not meet the deadline, as American taxpayers are probably not ready to support a money-eating monster.

Increasing cost

Adding human resources to a late software project makes it later

In spite of Brooke's law quoted right above, you can add resources to a late project but you have to be very careful. There is two considerations:

It will increase the cost of the project, so someone is going to have to pay for this. Maybe your client, maybe your company, maybe a shared responsibility, that's not really the concern for developers. What matters is that the cost increases.
It will slow down the current team because more people to talk to equals to more time spent not coding. This is why if you do that you must have a project that is already properly loosely coupled and have the new developers to work on those remote parts that don't really interact a lot with existing parts. You want the extra workforce to be able to compensate the productivity loss.

Increasing cost is an option but it has to be carefully used. In a lot of cases and especially if the project is small it's not possible to do so.

Increasing time

Pushing the deadline may seem like an intuitive option but there is actually a lot of problems to be considered before doing so.

Is the client ready to accept a late project?
Does your company have the bandwidth for this extra time spent on this project?
Considering that you are going to do free work rather than paid work, it costs your company money.
Dragging a project is very bad for the team morale, especially if it's more than 50% above the initial estimate.

Like the increase of cost, the increase of time has some drawbacks, including actually the increase of cost. That's really not a scenario you want to see happening.

Decreasing scope

The good news is that as advised above you've done things one by one and in a meaningful order. Which means that every single commit is a finished product. Maybe not all commits are equally tested but at least the intent is that every commit is a finished product.

Which means that you can, right now, push your product into production and not worry that some parts are bleeding. Everything is all right.

Now that's a big clean scope reduction however that is not necessarily the only leverage you have. Indeed, during the specification process some things were specified in a certain way but maybe if you think about it with the current experience of the project and a developer eye you can find shortcuts which would impede a little bit on the UX but would allow you to finish on time.

In short, when you are late, two options offer to you:

Cut the scope at the current advancement status at the release date. Maybe you will finish those features later, maybe not. It's down to commercial negotiations.
Simplify the remaining features so they are potentially less user-friendly but faster to complete.

Let's put this a different way. What is on the paper you signed with your client is the deadline, the price and the broad scope of the project. Meaning that as long as your project does what it has to do, for the price that was agreed and before the deadline that was fixed then the fine details of the specifications are by far the most negotiable part of the project.

Considering this, when your project is late you should probably find a way to cut down the scope because it is by far the less painful way of crossing the finish line.

Conclusion

While estimating and planning are complicated tasks, it is part of our job to accomplish them. In fact, if we call ourselves software engineers we should be able to hold the same level of accountability than other kind of engineers. If a bridge can get built on time then software should be deliverable within the deadline as well.

This starts with making rough estimates at the beginning of your project and then narrowing down the plan as the specifications and code get produced while making sure that the budget and deadline are still consistent with what is being actually done.

A pattern we've seen emerging at different stages of the project is that if you want to deliver your project sooner you have different adjustment variables you can play on but the safest and most efficient is definitely to play on the scope.

The most important lesson is to carefully arrange your tasks and to accomplish them one by one so you keep the complexity of your work manageable and your project in a healthy shape.

In the end our developer work is often more landing page than robots and lasers. I see many juniors that are disappointed by this outlook and many seniors that just hope to move into management. However our industry is plagued with projects blowing up their estimates. Then if our job is so easy, let's try at least to deliver it on time.

Acknowledgements

I used the following articles as inspiration and proofing for this article. There is a lot of content that I don't endorse 100% but they make other interesting reads.

Tradeoffs Of Time Estimates — Professor Beekums
On dealing with deadlines and estimates — Eddy Ernesto del Valle Pino
Project Management for Developers 101 — Yaakov Ellis
How to move to #noestimates — defmyfunc
How long will it take? Git knows. — Dimitrios Melidonis
Ways to #NoEstimates — Jay Bazuzi
How we used the improvement kata to gain 25% of productivity — Philippe Bourgau
On Estimating Things — Terry Brown
Software Estimation, Enterprise-Wide part 1 and part 2 — Vitalie Temnenco
Risk Management Is Project Management for Adults — Tim Lister
Why software projects take longer than you think – a statistical model — Erik Bernhardsson
Extreme Programming - A gentle introduction — Don Wells

Asking for review on non-string regular expressions

Rémy 🤖 — Fri, 26 Jul 2019 20:12:59 +0000

There is this idea I have and that I want to push forward about "non-string regular expressions". I explained things as I could on the repo.

Xowap / nsre

Non-String Regular Expressions

NSRE (Non-String Regular Expressions) is a new spin at regular expressions It's really abstract, even compared to regular expressions as you know them but it's also pretty powerful for some uses.

Here's the twist: what if regular expressions could, instead of matching just character strings, match any sequence of anything?

from nsre import *

re = RegExp.from_ast(seq('hello, ') + (seq('foo') | seq('bar')))
assert re.match('hello, foo')

The main goal here is matching NLU grammars when there is several possible interpretations of a single word, however there is a lot of other things that you could do. You just need to understand what NSRE is and apply it to something.

Note — This is inspired by this article from Russ Cox which explains how Thompson NFA work, except that I…

View on GitHub

I'm looking for all kinds of feedbacks

Do you understand what this is?
Do you see applications for this?
Does the API look nice?
What features would you want to see around that?
What would you want before using this in production?

Thanks!

DEV Community: Rémy 🤖

The Ultimate Python main()

Hello, world

Importability

Parsing arguments

Outside call

A bit of assistance

Handling signals

Error Reporting

Packaging

Conclusion

On the usefulness of `git rebase`

20 years later — How I find and fix bugs

The obvious

print things

Debuggers

App-level monitoring

The blind

Bisecting

Slashing

Random testing

The meta

Don't panic

Go to sleep

Conclusion

Why robot citizens will save the tech industry

CSS Patterns — Deep dive in grids

What is a grid?

Down the gutter

Cell structure

Sizing items

Without flexboxes

Stacking

When to use

Wrapping up

IP-based localization, why oh why?

Add progress bars to any command

DEV-CLI 0.2.0

What CAPTCHA do you use?

Version and automate your dev.to publishing with GitHub Actions

The way adults deliver software on time

Estimating

Get rough

Assess your risks

Estimate regular tasks

Negotiate the deadline

Don't forget extremities

Planning

Basic structure

Reviewing the specs

Data filling

Ordering development

Developing

Stick to the specifications

One thing at a time

Testing testing testing

Real-life data

Don't trip

Dealing with a late project

Increasing cost

Increasing time

Decreasing scope

Conclusion

Asking for review on non-string regular expressions

Xowap / nsre

Non-String Regular Expressions

Non-String Regular Expressions

`print` things