DEV Community: Ines

How do I measure the execution time of Celery tasks?

Ines — Mon, 13 Jan 2025 12:18:00 +0000

The next item in my collection of "Aren't we all just constantly re-creating the same bits of code?"
is how to track the execution time of Celery tasks.

Firstly, it could be argued that there are 2 different "exec" times for every Celery task:

the real execution time - the time when the code was running
the "time to done"-execution time - which includes the time spent in the queue waiting for a free worker

The reason both are important is: our real motivation is understanding when the thing is done.

We trigger a task, we want to know when that job is done and when to expect the results.

It's like project estimation. The manager truly wants to know when the project will be out of the house and not
really, that it can be done in 1 week, but nobody is free to do it for the next 6 months. 👷‍♀️

Signals 🎺

As far as I can see, we just need to time the tasks ourselves. Our weapon of choice are Celery signals.

Hint 1: Celery signals get all their arguments as keyword arguments. This means that we can just type out the kwargs we are interested in and pack the rest into **kwargs.
Also: This is the best thing ever. All signals all over the world should adopt this practice!!

Hint 2: We can store the exec start and end times on the task object itself among the other attributes, which are called "headers".

Start of queuing

Just as a new Celery task is put into a queue, let's record the current time like so:

from celery import signals
from dateutil.parser import isoparse
from datetime import datetime, timezone

@signals.before_task_publish.connect
def before_task_publish(*, headers: dict, **kwargs):
    raw_eta: str | None = headers.get("eta")
    if raw_eta:
        # the task is scheduled for later, so, let's start the countdown from then
        publish_time = isoparse(raw_eta) 
    else:
        publish_time = datetime.now(tz=timezone.utc).isoformat()
    headers["__publish_time"] = publish_time.isoformat()

Start of execution

When the task is picked up by a worker, let's record the current time like so:

from celery import signals
from datetime import datetime, timezone

@signals.task_prerun.connect
def task_prerun(*, task: Task, **kwargs):
    setattr(task.request, "__prerun_time", datetime.now(tz=timezone.utc).isoformat())

End of execution

When the task is done, we can calculate both exec times and store them .. somewhere. Probably in StatsD or some other monitoring tool.

StatsD is an industry-standard technology stack for monitoring applications and instrumenting any piece of software to deliver custom metrics.

- Netdata: Introduction to StatsD [1]

from celery import signals, Task
from dateutil.parser import isoparse
from datetime import datetime, timezone, timedelta

@signals.task_postrun.connect
def task_postrun(*, task: Task, **kwargs):
    now = datetime.now(tz=timezone.utc)

    publish_time: datetime | None
    try:
        publish_time = isoparse(getattr(task.request, "__publish_time", ""))
    except Exception:
        publish_time = None

    prerun_time: datetime | None
    try:
        prerun_time = isoparse(getattr(task.request, "__prerun_time", ""))
    except Exception:
        prerun_time = None

    exec_time: timedelta = now - prerun_time if prerun_time else timedelta(0)
    waiting_time: timedelta = prerun_time - publish_time if publish_time and prerun_time else timedelta(0)
    waiting_and_exec_time: timedelta = now - publish_time if publish_time else timedelta(0)

    stats = {
        "exec_time_ms": to_milliseconds(exec_time),
        "waiting_time_ms": to_milliseconds(waiting_time),
        "waiting_and_exec_time_ms": to_milliseconds(waiting_and_exec_time),
    }
    # TODO: log somehow to somewhere ¯\_(ツ)_/¯ 
    statsd.timing(f"celery.task.exec_time_ms", ..., tags=[f"task:{task.name}"])
    statsd.timing(....

Bonus: And once you are tracking how long tasks take, you can set up alerts for when they take too long. 🛎️

It can be as simple as adding a hard coded threshold to the above function:

if exec_time > timedelta(hour=1):
  logger.error(f"Task {task.name} took too long: {exec_time}. Check it out!")

Or you can have layers of thresholds or thresholds as settings on the task definitions or whatever you can express in code.

Isolate code blocks from each other?

Ines — Thu, 02 Jan 2025 18:01:18 +0000

Aren't we all just constantly re-creating the same bits of code?

This goes beyond boilerplate code. We are adding the same bits of code to every project, the same
git shortcuts, the same logs formatters, the same permissions decorators, ...

Here I've started putting together a personal collection of building blocks.

And I'm starting with: Code that isolates (insulates) code blocks.

Hint: The full function is on my blog post

Example: I'm sending emails to all team members.

If 1 person's email causes a bug, I want the code to skip this person and continue sending emails to others.

Naive - no error handling:

def send_all_emails(users: list[User]):
    for user in users:
        send_report_email_to_user(user)

If we have a list of 10 users, but we encounter an unexpected error
with the report for user num 3, then only the first 2 users will get the email, others will not.

With error handling:

def send_all_emails(users: list[User]):
    for user in users:
        with suppress_and_log_exc():
            # ↑ Will catch any Exception, log it correctly
            # and then continue with the next user 
            send_report_email_to_user(user)

What `suppress_and_log_exc` does

it catches some Exception class
logs it properly
lets the code continue

So, something like this:

@contextmanager
def suppress_and_log_exc(
    *,
    action_desc: str,
    # ↑ Let's require an identifier. The error msg will be more helpful this way.
):
    try:
        yield
    except Exception as exc:
        logger.error(
            f"Error `{exc.__class__.__name__}` occurred while {action_desc}",
            exc_info=exc,
        )

Possible use cases

This code comes in handy whenever you have a list of actions that are independent of each other.

Like for example: we are triggering various side effects after some event.

Maybe a new user has registered, so, we want to:

send a Slack high-five to the dev team and also
create a ticket for the customer success team to contact them and also
... .

If any of these side effects fail, the others must still be triggered.

Another example: we have a multi-tenant system and want to trigger one Celery task for each tenant.

If the code for creating a Celery task for customer number 5 has a problem, we still want to create the tasks for customers 6 to 1.000.000.

It would be silly, if our code were to fail at say... sending out the monthly bill, with customer number 5 and then not even try to send it to customer number 6 and 7 and so on.

Adding more settings to the contextmanager

We can make the function more customizable by adding a setting for:

the log level - some things are in reality just a warning or an info
a map of log levels - a specific log level per exception class
the exception class that we want to catch - maybe we just care about EmailSendingException
more log data - so we can better understand what went wrong when we see this msg in Sentry
an error callback - a function that is called, when the error happens, which can be used for custom error-cleanup
.. whatever your heart desires .. 💖

So, here is now the full code.

Fin: All together now

Our Example code could now look like this:

def send_all_emails(users: list[User]):
    for user in users:
        with suppress_and_log_exc(
            action_desc="Sending my very special report email",
            extra={"user": user.id}
        ):
            # ↑ Will catch any Exception, log it correctly and then. 
            send_report_email_to_user(user)

Or it could be crazy complicated like so:

import logging


def send_all_emails(users: list[User]):
    for user in users:
        with suppress_and_log_exc(
            action_desc="Sending my very special report email",
            log_level=logging.WARNING, # <- default log level
            log_level_maps={EmailIsInvalidException: logging.INFO},
            exc_types_to_catch=(ReportException, EmailException,),
            extra={"user": user.id}
        ):
            # ↑ Will catch any Exception, log it correctly and then. 
            send_report_email_to_user(user)

How to encourage developers to fix Python warnings for deprecated features

Ines — Wed, 25 Dec 2024 12:50:01 +0000

Our code is so interdependent! Adding dependencies to projects is just how code is done.

Every time somebody sneezes a new dependency is added to a project somewhere.

It is kinda great that we are on this side of the copy-right debate, where the default is building things on top of other people's work, and not shooting at everybody, who dares to even look in the direction of my code. But... who is going to upgrade all these dependencies all the time?

In the perfect world, we are a community and we all keep the libs upgraded to the latest versions, right?

Wrong! Devs need to be softly pushed to do the right thing. Yes, softly, so they don't notice they are being pushed 😉🚧🚧🚧.

Step 1: First we need to listen to deprecation warnings

We are using pytest, so this step is easy-peasy for us.

Starting from version 3.1, pytest now automatically catches warnings during test execution and displays them at the end of the session:

- Pytest: How to capture warnings

like this:

============================= warnings summary =============================
test_bla_bla_bla.py::test_test
  .../some_file.py:65: PydanticDeprecatedSince20: `pydantic.config.Extra` 
    is deprecated, use literal values instead (e.g. `extra='allow'`). 
    Deprecated in Pydantic V2.0 to be removed in V3.0. See Pydantic V2 Migration Guide 
    at https://errors.pydantic.dev/2.9/migration/
      extra=Extra.allow,

  .../marshmallow/fields.py:1186: RemovedInMarshammlow4Warning: The 'default' argument
    to fields is deprecated. Use 'dump_default' instead.
      super().__init__(**kwargs)

======================= 1 passed, 2 warnings in 0.12s =======================

Step 2: Pick the warnings you want to get rid of

I saw the marshmallow warning, looked at the code and saw that it was a simple fix.

So, I'm choosing RemovedInMarshammlow4Warning.

Step 3: Make deprecation warnings fail the tests

Again, we are using pytest, so this step is also easy-peasy for us.

Just add this to the config file:

[tool.pytest.ini_options]
...
filterwarnings = [
    "error::marshmallow.warnings.RemovedInMarshmallow4Warning"
]

- Pytest: Controlling warnings

Now every test that triggers this deprecation warning will fail.

Step 4: Fix the code

Unfortunately, you have to take one for the team here. You have to go and fix all the places where this warning is triggered.

I know.. it's unfair.

It's a lot of work, it's tedious work and you won't even be able to brag to your boss about it, because it is too technical and generally perceived as irrelevant from their standpoint.

And everything this does is help the next person (who won't be you), who will upgrade marshmallow and nothing in code will break. So, they won't thank you either, because they won't even notice anything has been done.

So, ... what I'm saying is: sometimes you need to do the right thing, and nobody will thank you for it.

But I will know. And you will know. 🤝

Step 5: Lean back and listen to Slack convos and GitHub comments

Eventually, there should be one message somewhere of somebody being confused: they wrote the code the same as last time, but this time, the tests fail, what's going on?

And you can say: "oh, hard to say, but I do see this is a deprecated feature, that must be the reason". 🍹

External links

Why is sooooo.. much documentation a pain to read?

Ines — Tue, 24 Dec 2024 09:10:04 +0000

Because it is written in a boring, counterproductive style. But why is that?

Because it copied the style of academic papers. But why are academic papers dry and boring?

Because most schools have since forever talked to their pupils in a dry and boring manner. But why is that?

Because until recently human society has been managed in an authoritarian manner. Everybody knew exactly who was above them and who was below them, who they have to obey and who they can give orders to. For the most part, fun was reserved for the afterlife and the rich. But come-on! we can do better, here are a few pointers.

(I utterly love this 5-whys interrogation technique! I recommend trying it out.)

Learning is easiest through experience, thus let's start with a practical exercise. We will write an email.

Imagine the following scenario:

It is Thursday, the 4th day of your 1-week sprint. You have just finished a ticket and have opened up a pull request.

The ticket was about developing a new way to create CLI commands. The existing commands were all built on a now-defunct library.

You are happy with your work, it took a lot of coordination between departments. However, one thing is still left to do: you have to tell everybody about this new way of writing commands.

How do you do it?

Attempt 1: Write a boring, old email

Let's say you decide to write an email.

How better to make everybody understand this new approach, then to tell them everything, the whole story, from start to finish:

Everybody,

I’ve been consulting with DevOps a lot lately because I was writing a new command. And it turns out DevOps has a lot of problems running our existing commands, they can’t do … and … and they are frustrated with … . Another problem is the fact that our library …. is not maintained anymore, so we shouldn’t use it anymore. It is not super dangerous to use it, but we should switch to something more stable soon. Together we’ve come to the conclusion that the best way forward is to use approach …. . I've written documentation about how to use the new system for CLI commands. So the old way is now deprecated, please do not use the old way anymore. If you need any help with the new way, you can ping me.

Thank you,

Me

Great, done! Let's send it out.

You have done everything and more. You've finished the ticket, you've communicated with everybody, you've even written docs and an email and everything. You are one great developer 😌 !

But, are you really done?

The crucial part of email sending comes after the email has been sent. Will anybody read it? And will the recipient act on it?

Why no one is reading your emails

Let's switch perspectives.

You are a different developer now.

It is still Thursday, the 4th day of your 1-week sprint. You are down to 1 un-finished ticket.

Truth be told, you have solved this last ticket as well, but the pesky end-to-end tests are failing. You've been staring at the failing tests for 1 hour now, trying to figure out how could your change have caused this, maybe the tests are just flaky.

Should you take some time and make them stable? Do you even have the time? There is this important meeting tomorrow, you should start preparing for.

Hm... oh, look an email has arrived.

What will happen now? Will this second developer read our email? Will she read it thoroughly?

Of course not.

The second developer has her own stuff to worry about. She'll skip over the text and forget it.

What did we expect to happen?

People are not machines, we cannot insert a new if-statement into their decision mechanisms via one email.

People remember through repetition, not by reading emails. So... what exactly was our goal with the email? We never specified it.

Attempt 2: A fun, short email

We should have started with instructions. First the TL;DR, then the explanation:

Everybody,

Please, read the following docs: http://....

We have defined a new procedure for creating CLI commands. We must all adhere to it.

Thank you,
Me

This way of writing feels unnatural at the beginning.

Intuitively, we would first explain the situation and then ask people to act. But we are all flooded with messages daily.

I have recently realized, that I am being paid to read. I read emails, blogs, docs, specs, Slack, Confluence, JIRA, ...

There is always more to read. The logical consequence is skimming all text.

We no longer read, we skim, searching for a glimpse of relevance.

Know your audience. Because they will only skim, be relevant and concise.

What would have been the reaction to the above, shortened email? I guess that 70% of its recipients would skim it and 20% would read the docs.

Is this it? Have we given it our best? Is this the best possible outcome?

Of course not. We went at this from the wrong angle. Sending out an email would always have been futile.

Email has its limitations, but this does not mean we cannot still achieve our goal.

Attempt 3: Secret messages, that trigger when somebody moves a lever

First, what do we really want?

We want that all future commands are written in the new way.

We must switch perspectives. We must think like "the other". It's like a marketing ploy: How do I make everybody buy my product?

Who will need to know about the new way of writing commands?

Definitely a developer.

She might be a senior developer, who has been at the company for 10 years, or she might be a junior developer in her 2nd week. It might be somebody from another team or another department, it could be a DevOps or it could be a QA person.

Maybe you'll know them, maybe you won't. Thus all hints and instructions need to be written down and must be very visible.

What will this developer want?

They will want to quickly and efficiently write a command.

They will absolutely NOT want to know the history of your meetings, what who said, what who thought, what day of the week it was, what anybody had for breakfast, none of these things. They will want to quickly and efficiently write a command.

Their first question might be: are there any existing commands and can I just do the same.

The will either search the docs or the source. So... you need to fix both paths.

If you search for "command" in docs, do you find docs about the old approach? Change this.

The docs need to reference the new approach, delete old docs, if possible, or add big bold "Deprecated" signs.

If you search for "command" in source, what do you find?

Is there a folder called commands in your app, but all the commands inside are done in the deprecated way? Rename the folder. Add "Deprecation" notices to all the old commands' classes. There are tools in many languages, with which you mark deprecated-code.

Leave breadcrumbs everywhere!

Archetype

The best documentation follows the same principle. Think about your audience and state the things that will be important to them.

A while ago I was searching for a Git-history drawing tool. I stumbled upon GitGraphJS and its utterly awesome docs:

The docs are concise, short and cover all the important first questions I had. Once I had set up the library with a simple example, I could look into any details.

Infamous Webpack

Webpack is infamously difficult to set up and infamously difficult to understand. What problem is it even solving?

Here is the top of their current home page. After reading this, I still don't know what Webpack even is, I don't know why I should use it, how it will help me and, honestly, I had to stare at those 4 file examples for a very long time before I understood what they are trying to tell me.

Lots of start-up products lack a clear description of what they do. Their pages talk about improving performance, collaboration, bundling or some other aloft subject. But read as I may, I still can't find a clear answer to: "What does this product actually do?".

I would need to invest weeks into understanding Webpack, who has that time?!

Several wrapper-libraries have been created around Webpack. Libraries, because nobody has time for the official docs.

And if these libraries have found a way to make Webpack easy to set up, why wasn't Webpack able to do the same?

One of these wrapper libraries is Laravel Mix. To set up and use Laravel Mix, it can be enough to read its 350 words long quick guide. I've tried it, it works! There is more documentation than this, you can go into depth on many topics, but you do not need to and that is the point.

If Webpack were meant for CS researchers to study it, then its documentation would be just perfect. If it was meant for us, however, then it missed the target. But it did provoke a bunch of memes.

Why is Stack Overflow so immensely popular?

Because most documentation is just not good at answering questions.

When you have a problem, how often do you go check the official documentation and how often do you check random blogs or Stack Overflow? I should trust the official docs of any product more than a random blog post from a random person/bot(?) on the internet. But I can't, because docs so rarely explain the most common use cases.

I would love it if most docs would include a FAQ section. If you are writing some docs right now, please, add a FAQ section. This will be a chance for you to see us: what will we do with your library, how will we use your code, what will be most confusing about your approach, ... .

To all future docs authors

To all you, future authors of documentations.

Please,

write a short and a long version of your docs (a tl;dr section is greatly appreciated)
consider the user experience of your readers
include a FAQ section and expand it with time.

Thank you! We will repay you with likes and shares and subscribes and also by cloning your approach.

External sources

Software Estimation Is A Delusion

Ines — Thu, 31 Dec 2020 16:25:46 +0000

You will never be able to accurately estimate the time and effort it will take you to build a piece of software .. for as long as you keep doing new things. As long as your projects don't resemble each other as your signatures do, you will not know how much effort it will take to finish a new project. A lot has been said and written about software estimation, but one thing is clear: predictions of future effort are always based on the amounts of past effort.

2 distinct approaches exist for measuring the future time and effort of software development:

asking a human expert for her opinion or
building a mathematical model and condensing the requirements of the software project to only the relevant parameters

The approaches might seem radically different, but the 2nd one is just trying to create a formal model of the 1st one. It is trying to cut out the human component of software estimation. It is trying to reduce an expert's decision process into a relatively simple and highly portable mathematical formula. And while it is doing this it is also often pretending to be searching for a law of nature, for the natural relationship between software building and time, which is independent of the organization or the customer that the project is built for or by.

However, the best mathematical models for software estimations are those that are highly calibrated to the team they are measuring. By "calibrated" I mean that the function for calculating the required effort is fitted to data produced by this team in past projects. And so we are back to experiences. If you have done this project before, then you will be able to estimate accurately how much effort it will take to do it again.

But of course, life is a moving target. As you do the same thing again and again, you get better and faster at it. As the world around you changes: new versions of OSs, new browsers, new programming-language versions, ... you have to change too, which makes you slower again. As you lose or gain members on your project, the effort and time needed are also changed. And I haven't even begun listing any big changes: like having a harassing customer, getting a disruptive team member, having to comply with a new law, ... .

It all boils down to whether or not you see the world as ever-unpredictable or ever-predictable. If it is ever predictable, then software estimation is perfectly possible and also works perfectly brilliantly. If, however, you see the world as ever-changing or you yourself are inclined to take on ever-changing projects, then any time or effort estimations are useless and deceiving. Because: nobody has yet come up with a prediction system that is not based on existing past data.

Is it even important?

Why are people even asking this question: "How long will it take to create this piece of software"? Because they don't have the resources to work on this forever. And this is understandable.

But there is a secret message in that question as well. It is something that we understand so very, very well that we don't even think about it. The proposed piece of software doesn't need to be so complicated. It could be much simpler and still do its job. And simpler software will take fewer resources to produce. But people don't want the simple software, they want the most that they can get for their budget.

The true question isn't: How long will this take?, but What can you build for this much money?.

2 types of people ask this question: managers and customers. These 2 groups are pretty different, we communicate with them differently. But the core answer to say to both of them is something along the lines of: "Let's build it iteratively". Let's first organize the requirements by priority: what is core functionality and what is optional. Let's then build first the foundations and then the core functionality. And if we will then still have time and money left, let's continue. And if we will then still have time and money left, let's continue. And if we will then still have time and money left, ...

For instance, let's say I am a freelancer creating simple web pages. A customer asks me to create their first web page, a simple presentation of who they are and what they do. They might come with a well-defined idea of what they like, with all kinds of details and requirements and a 100-page description document. And they want to know how much this will cost and how long it will take. For me, this is nearly impossible to estimate. But let's say I still give them a number, which might be too big for them. So they might continue: "What about if we take these 20 pages out. How much does it cost then?". This doesn't make the job of estimating any easier. Even if I know exactly how much time it saves me to ignore those 20 pages, my estimation of the full 100 pages was wonky. So.. wonky - 20 = still wonky. It will be much easier to start with a simpler page that can stand by itself, preferably one similar to something I've already built. After that, we can add more fancy functionalities if we will still want to.

And what if the task cannot be split up? Because such things exist too: we either switch to a new system or we don't.

The answer is simple but unpopular: We don't know how long something big that has not been done before will take. We have no experience doing it. But here is a list of things we will gain and here is a list of risks we will take. And one of the risks is also that all this time and effort will go to waste. If this project is important enough, let's allocate a few resources to it so that somebody can start working on it. If their work goes well, then let's continue, if it will still go well, then let's continue, if it will still go well, then let's continue gradually increasing that somebody's resources. But it could also not go well, in which case, we have to be prepared to stop work on this project and forget about it. Just because we've already invested 20 person-months into this project, doesn't necessarily mean it makes sense to finish it. Maybe it will be better to drop it. And maybe it will all go very well and we'll soon reap the rewards of starting this project.

The importance of this question is questionable. If you are asking your developers how long something will take because you will secretly send this number over to the customer as a binding offer, then stop doing it. Your developers are not freelancers, they don't need to haggle with your customers over how much they pay you. If you are asking your developers for numbers to report to your superiors, then only ask for estimates of projects that are similar to existing projects they did. If you ask for estimates for projects they have no experience with, you will get made-up numbers, that you will have trouble defending later on. If you are asked this question by a customer and you are the one, who has to come up with answers to it, then try to split up the project into sub-projects that are similar to your experience.

There is no point in giving estimations that you can't stick to with a reasonable error margin.

But what if your projects are similar and you want to give estimation a go, what are your options?

What are your options for estimation?

All estimates usually take 2 things into account:

the size and complexity of the project
the resources provided to the project - the team

Unfortunately, none of these attributes is easy to assess. It would be great if I could say: "I have 4 developers and a project of size 75200. How long will it take to build the project?". But, what is 4 and what is 75200? Which leaves you with 2 choices: either you spend a looong time putting the 4 and the 75200 into some kind of perspective, some kind of mathematical function, a prediction machine or you ask a human to feel it, to look at it and then say a number, from the gut, "What is your gut telling you?". Yes, these are your options. You can rely on a human or on a machine, each comes with a set of advantages and disadvantages.

There have been studies comparing which is better: human or machine. The results are ... depends on who you ask. Sometimes it is the models that are better other times it is the humans.

Human Expert

An expert would be someone, who knows how to build your project and has done it before and also knows the team. It is the complexity of the project in relation to the capabilities of the team that define the effort needed to finish the project. This is a difficult person to find.

It has been known since the 70s that developers tend to give very optimistic estimations. Nothing has changed since then.

One of the reasons is that developers have to give lots of estimates, but then rarely track the actual effort that went into the tasks. And if they don't track, then they can't review the initial estimate after the project is done.

Another is, that developers often estimate by breaking up the task into smaller tasks, estimate the sub-tasks and then just sum up the numbers. But this way inevitably misses some tasks, while giving the estimator a really good feeling of having thought of everything.

It is nearly impossible to come up with a complete list of every sub-task that will need to be done. We also tend to forget to include any unexpected problems. Problems we will only understand once we are staring them down.

The researchers M. Jørgensen, K. Teigen, K. J. Moløkken-Østvold did a marvelous study of how developers badly under-estimate the required effort titled Better Sure Than Safe? Overconfidence in Judgment Based Software Development Effort Prediction Intervals. They observed how estimates are asked for, but then rarely reviewed after the project:

We investigated more than 100 projects from 6 development organizations and found that about 30% of the projects had applied effort prediction intervals on the activity level. However, only 3 of these projects had logged the actual effort applying the same work break-down structure as used when estimating the effort. In fact, even these three projects had to split and combine some of the logged effort data to enable an analysis of the accuracy of the effort of PIs (tn: PI = prediction interval, a minimum - maximum effort).

They also created 5 teams of developers and asked them to estimate the effort needed for a project. The project was an actual past project of the company they worked for. The original estimation for the project was 1240 hours, but the project took 2400 hours to be finished. The teams produced the following values as the Estimate of the most likely effort: 1100, 1500, 1550, 1339, 2251. Just one of the teams came close to the actual number of hours spent, the other 4 had incredibly optimistic numbers.

Another interesting part of the experiment was that the worse estimations were done by developers and project managers. (So, please, project managers, stop pressuring the developers with statements like: "How can this take more than 1 day to create?". Such statements are just pure ignorance 🙄 )

The lowest effort estimates were provided by the developers and the project managers, whereas the user interaction designers and the engagement managers gave generally higher estimates.

...

As a result, technical background did not lead to better effort PIs (tn: PI = prediction interval, a minimum - maximum effort), only to more confidence in the estimates. This is in line with the distinction between an "inside" versus an "outside" view in predictions (Kahneman and Lovallo 1993).

So.. what did Kahneman and Lovallo say about inside and outside predictions? Here is an excerpt of their paper Timid Choices and Bold Forecasts: A Cognitive Perspective on Risk Taking:

An inside view forecast is generated by focusing on the case at hand, by considering the plan and the obstacles to its completion, by constructing scenarios of future progress, and by extrapolating current trends. The outside view is the one that the curriculum expert was encouraged to adopt. It essentially ignores the details of the case at hand, and involves no attempt at detailed forecasting of the future history of the project. Instead, it focuses on the statistics of a class of cases chosen to be similar in relevant respects to the present one.

Another powerful reason for over-optimistic developer-estimates I would like to emphasize is the social component of estimation. When a developer is asked to give an estimate, 2 social constraints guide her estimate. Firstly, the estimate cannot be huge, because that effectively stops the project immediately. And secondly, she cannot come up with a very wide estimate. An estimate is wide when the expected min and max values are far apart. Providing a wide estimate or an unexpectedly big one is considered the same as providing a useless estimate.

I still very vividly remember an event in my past. I was the head of a delicate effort to replace a patched-up version of something that was very basic to our app with a new, cleaner and more complete implementation. It was a tricky job, we didn't know what we were getting into simply because this project wasn't similar to any other project we did. But it was important. We couldn't live with the old implementation anymore and we had finally convinced the management to give us resources to replace it. I didn't give any estimate of the project at the beginning, but at some point during the project, I was convinced that I do have to provide some estimate. "It doesn't have to be correct", I was told, but I do have to give a number. And so I sat down, opened the form and picked a date 6 months into the future. My manager was sitting next to me and said something like: "What?! No, we can't put that in.". But I knew that 6 months was a perfectly optimistic number. Our app was huge and a lot of work will need to be done. And so, I and my manager looked at each other. We knew that this project needs to be done, the engineering department is convinced of it, but the management is difficult to persuade. We've already come very far, the finish line is very close. The +6-months number will stir up a lot of anxiety, but the project still needs to be done, it will be great, once it is done and anxiety is not helping anyone. And so we said: "Could we do it in 4 months?", "If we skip a few things and smash up a few others, maybe..probably..", "Ok, let's put +3 months then".

M. Jørgensen, K. Teigen and K. J. Moløkken-Østvold saw something similar in their research. Developers admitted that they believe their managers will see them as less competent if they provide estimates with huge margins. To test this thinking they created an experiment in which they asked 37 software professionals to evaluate some hypothetical project estimations. All 37 participants had some experience with project management. They were presented with estimates for 5 hypothetical tasks given by 2 hypothetical developers: D1 and D2. Both developers gave the same "Most likely effort", but different "Estimated min" and "Estimated max". D1 would give a narrow min-max interval, while D2 would give a much wider min-max interval. The participants were also provided the actual effort needed for each task.

Mathematically speaking providing a wider min-max interval means you will be right more often. D2's estimates were correct for 4 tasks out of 5, whereas D1 was right only 3 times.

The participants were then asked 3 questions. Here is how they answered the first 2:

"As a project manager, would you prefer the estimates of D1 or D2?" - 35 out of 37 participants prefer D1, even though D1's estimates are objectively less accurate than D2's. The other 2 choose D2.
"Do you believe D1 or D2 knew more about how to solve the development tasks?" - 29 participants choose D1, 1 participant choose D2.

The preference for D1 in the (a)-question means that nearly all the software professionals preferred the much too narrow intervals of D1 to the statistically more appropriate intervals of D2!

...

Answers to question (b) revealed that wider intervals were believed to indicate less task knowledge.

From all this, we can say that a lot goes into producing an estimate. It is not just the number of resources that need to be reserved. And for some reason, we prefer to have exact numbers, even if that means they are wrong most of the time. We want to have certainty, the feeling that somebody knows, what is going on, somebody else has things under control. Or at least somebody should pretend to have things under control and if they slip out of control, this somebody should take care of taking back control over them.

We are definitely not rational beings, but emotional ones :)

Mathematical models

There is an incredible variety of mathematical models available:

regression-based approaches: That is when you want to create a formula (ie. $ Y = ? * X ^ ? + ? $) that expresses the relationship between a dependent variable ($Y$), also called the "outcome variable" and one or more independent variables ($X$).
classification and regression trees: That is when you want to create a decision tree where each node represents a question and the node's branches represent possible answers to the question. Traversing through the tree eventually leads us to a leaf, which represents the outcome, our prediction. If the leaves represent a class (a discreet value), then the tree is called a classification tree. If the leaves can take any number in a continuum, then the tree is called a regression tree.
neural networks: That is when you create a huge, directed and weighted graph, and teach it to return a specific output given a specific input. And then feed it new input data and use its output as predictions of the future. The internal workings of this system become completely unknown to you, they become a black box of decision making.
Bayesian statistics: That is when you calculate the probability of an event happening given some prior knowledge of other conditions that might be relevant. $ P(A|B) = \frac{P(B|A) P(A)}{P(B)} $
lexical analyses of requirement specifications: That is when you analyze the words used in the documents describing the new software.
case-based reasoning
fuzzy logic modeling
simulation based probability
...

All of them have been studied by researchers, but I could find no data about how often a mathematical approach is used in software companies. Even worse, I did find this 2007 review of existing studies(A Systematic Review of Software Development Cost Estimation Studies by M.Jørgensen and M.J.Shepperd), which found very few case studies of how estimation is actually performed in companies:

The proportion of estimation studies where estimation methods are studied or evaluated in real-life situations is low. We could not, for example, find a single study on how software companies actually use formal estimation models. Our knowledge of the performance of formal estimation models is, therefore, limited to laboratory settings

It has been shown that a model that is better fitted to the organization using it will produce better estimates. (By C.S.Murali and C.Sankar in "Issues in estimating real-time data communications software projects" in 1997 and R.Jeffery, M.Ruhe and I.Wieczorek in "Comparative study of two software development cost modeling techniques using multi-organizational and company-specific data" in 2000.) But this again means that your organization must be producing similar projects with similar people, that you have to first meticulously track the requirements and the effort invested into several projects and that you must be careful to not create a model that is over-fitted to your data.

Estimation variables

Producing a good model for your organization may legitimately be too complex or too time-consuming. The biggest issue is how to get all the contextual information that is currently stored in the heads of your team members into the model. Which information is even relative?

Given that there is no one estimation method that had been proven best, it seems safe to say that it is not the method itself that produces accurate results, but what information you feed it with. Figuring out which information influences produces the most accurate result is a difficult job and it is made even worse by the fact that all your numbers are fabricated. You don't know how big a project will be once it is finished: how many lines of code you will create, how many tests, how many files. Even if you have finished a very similar project, you still don't know how much more time you will need to add that 1 "tiny" change that is in the requirements.

In 1994 N. Fenton published an elegant paper titled "Software Measurement: A Necessary Scientific Basis", in which he emphasized that we have to define metrics in an empirical way. A metric has to be defined together with the procedure for determining the value. It is not enough to say that a project's size can be small, medium or big. We must also define how to measure the project's size. Do we look for the number of files, the number of lines, is 1000 lines a small project, is it medium? This is especially important for predictions. Some mathematical models still exist that need the LOC (=lines of code) number to predict the time and effort for a software project. But it is impossible to know the LOC of a future project, so we are replacing one guess with another:

.. size is defined as the number of delivered source statements (tn: LOC), which is an attribute of the final implemented system. Since the prediction system is used at the specification phase, we have to predict the product attribute size in order to plug it into the model. This means that we are replacing one difficult prediction problem (effort prediction) with another prediction problem which may be no easier (size prediction).

A way to estimate the size and complexity of a project is via function points (=FPs), which express the amount of functionality a program provides to its users. Once 1 FP is defined, it is theoretically easy to count all FPs a project will create by reading through the requirements. The time it takes to deliver 1 FP is calculated from previous projects. The theory has been developed into a few ISO standards, but it has also seen a lot of criticism and quite a few improvement attempts. Whether it will work for you or not is ... unknown 😛

Conclusion

It is easy to get pulled into the details of how these approaches work and completely forget that we haven't yet proven that they even will work, that they will give you an accurate number. The research does not favor one approach over the others.

B.Boehm and C. Abts say in Software Development Cost Estimation Approaches – A Survey:

The important lesson to take from this paper is that no one method or model should be preferred over all others. The key to arriving at sound estimates is to use a variety of methods and tools and then investigating the reasons why the estimates provided by one might differ significantly from those provided by another.

But as we've seen, maybe it isn't really accuracy that businesses and people are looking for. Maybe estimates are needed for the sole purpose of risk aversion. People what to know what the risks of starting this project are. But risk can be measured in other ways.

At my current stage in life, I find that everybody involved in a project is much happier if we go through the project in iterations. We don't have to decide everything at the beginning of a project. We should first concentrate on the essential bits. Things that can be done in a few days, a week at most and review the result. It is much less risky to see somebody spend a few days on a project and then review the results than it is to wait for a few months. An amusing consequence of this approach is that the developer will then also first concentrate on the essential bits. Maybe the first goal will be a proof of concept, something ugly and hacky, but something that works. Or maybe the first goal will be bits and pieces of code and in-depth research of the requirements and limitations and then comes the proof of concept. Either way, we will be kept in the loop on the progress. And being kept in the loop is what makes the risk lesser.

Maybe it is time we stop estimating tasks left and right and instead start managing the project's risk and customer's expectations.

External sources:

Celery: A few gotchas explained

Ines — Mon, 02 Nov 2020 08:22:13 +0000

Have you ever heard of the continuum of theory-before-practice VS. practice-before-theory? Probably not, since I created the name just now 😏. But, though the name is new, the continuum is old. The question is simple: should I first study, study, study the documentation and then only after I presumably fully understand the library and its logic start using it in my code, or should I first dive into it, use it and abuse it before going back and reading the documentation of it.

Children vector created by freepik - www.freepik.com

We all float in the continuum, none of us is stationary. Life-events nudge us to the left and to the right and sometimes fiercely sling us into one of the extremes as if we were pink-pong balls. Often we only want to study as much as is absolutely needed, because we equate Practice with joy and Theory with tediousness. And we are right to a degree: how much of a foreign language can you remember if you don't use it regularly. But then, sometimes, it turns out that we badly underestimate how much theory is "absolutely needed". And we have to go back, just like I had to go back to figure out Celery. My strategy of broadly getting it was only broadly enough. Now I had to go back and read all the Theory.

Celery is actually full of gotcha-s. Partly because we are dealing with processes, concurrencies, threads, .. and most of the time such details are abstracted away and a developer doesn't need to think about them and thus has little experience with them. And partly because Celery does here and there perform in unexpected ways. Some documentation-reading is definitely required.

So, here it is, all kinds of basic and advanced concepts around Celery.

Workers and brokers

First, let me explain some basic concepts, under which Celery operates.

Celery is a "Task Queue". Yeah, I also didn't know this was an actual term, I just thought it was a description of what it is: a queue of tasks that will eventually be executed (Maybe our Slovene universities should start using English terms after all. I don't want us to sacrifice regional languages, but doctors also have to understand the Latin terminology. And English has become our de facto Latin, for better or for worse. It is just embarrassing to not know basic English terms after 5 years of study and 10 years of work...). So, Celery is essentially a program that keeps track of tasks that need to be run and keeps a group of workers, which will execute the tasks. Its main points are that it can execute several tasks in parallel and that it is not blocking the independent applications(=Producers), which are giving it tasks.

But, Celery doesn't actually store the queue of tasks in its memory. It needs somebody else to store the tasks, it needs a Message Broker (or Broker for short), which is a fancy term for a program that can store and handle a queue 🙃. These are usually either Redis or RabbitMQ. So, Celery understands and controls the queue, but the queue is stored inside Redis/RabbitMQ.

On to the workers..

When you start Celery (celery -A tasks worker) 1 worker is created. This worker is actually a supervisor process that will spawn child-processes or threads which will execute the tasks. By default, the worker will create child-processes, not threads, and it will create as many concurrent child-processes as there are CPUs on the machine. The supervisor process will keep tabs on what is going on with the tasks and the processes/threads, but it will not run the tasks itself. This group of child-processes or threads, which is waiting for tasks to execute, is called an execution pool or a thread pool.

Queues

Yes, I deliberately used the plural for queues, because there is more than one type of queue 🧙🏽‍⚗️.

First, there is the main queue, which accepts tasks from the producers as they come in and passes them on to workers as the workers ask for them. By default, you have only 1 such queue. All workers take tasks from the same queue. But you can also specify a few such queues and limit specific workers to specific queues. The default queue is called celery.

To see the first 100 tasks in the queue in Redis, run:

redis-cli lrange celery 0 100

These queues are more or less, but not precisely FIFO (if the priority of all tasks is the same). The tasks that are put into the queue first, get taken out of the queue first, BUT they are not necessarily executed first. When workers fetch new tasks from the queue, they usually (and by default) do not fetch only as many tasks as they have processes, they fetch more. By default, they fetch 4 times as many as they have processed. They do this because it saves them time. Communicating with the broker takes some time and if the tasks that need to be run are quick to execute, then the workers will ask for more tasks again and again and again in very quick successions. To avoid this, they ask for X-times as many tasks as they have processes (=worker_prefetch_multiplier).

But there are tasks that never make it into the queue and still get executed by the workers. How is that possible, you ask me? I was asking myself and google the very same question. And let me tell you, Google had very little to say about it. I found just scraps of information. But taking Celery and Redis apart for a few hours (or was it days??), here is what I discovered.

Tasks with an ETA are never put into the main queue. They are put directly into the half-queue-half-list of "unacknowledged tasks", which they named unacked. And I do agree that "unacknowledged" is a very long word with a good amount of silenced letters sprinkled in, but it is very easy to miss something named unacked when you are trying to understand how some tasks have just disappeared. So, a note for next time I or you need to name something: all user-facing names should be spelled out completely.

So what are ETA tasks? They are scheduled tasks. ETA stands for "estimated time of arrival". All tasks that have ETA or Countdown specified (ie. my_task.apply_async((1, 2), countdown=3), my_task.apply_async((1, 2), eta=tomorrow_datetime)) are kept in this other type of queue-list. This also includes all task retries, because when a task is retried, it is retried after a specific number of seconds, which means it has an ETA.

To see which tasks are in the ETA-queue in Redis, run:

redis-cli HGETAL unacked

You will get a list of keys and their values alternating, like this:

1) "46165d9f-cf45-4a75-ace1-44443337e000"
2) "[{\"body\": \"W1swXSwge30sIHsiY2FsbGJhY2tzIjogbnVsbCwgImVycmJhY2tzIj\", \"content-encoding\": \"utf-8\", \"content-type\": \"application/json\", \"priority\": 0, \"body_encoding\": ...
3) "d91e8c77-25c0-497f-9969-0ccce000c6667"
4) "[{\"body\": \"W1s0XSwge30sIHsiY2FsbGJhY2tzIjogbnVsbCwgI\", \"content-encoding\": \"utf-8\", ...
...

Tasks

Tasks are sometimes also called messages. At its core, the message broker is just something that passes messages from one system to another. In our case, the message is a description of tasks: the task: the task name (a unique identifier), the input parameters, the ETA, the number of retries, ... .

In Celery the task is actually a class. So every time you decorate a function to make it a task, a class is created in the background. This means that each task has a self, unto which a lot of things are appended (i.e. name, request, status, priority, retries, and more). Sometimes we need access to these properties. In those cases we use bind=True:

@shared_tas(bind=True,...)
def _send_one_email(self, email_type, user_id):
    ...
    num_of_retries = self.request.retries
    ...

Task acknowledgment

Previously we said that when workers are free, they go and fetch some more tasks from the broker. But it is a bit more nuanced. When a worker "takes" a task, the task moved from the main queue to the unacked queue-list. The task is completely removed from the broker only once the worker acknowledges it. This means that when the worker "prefetches" a number of tasks, what really happens is that those tasks are only marked as his (reserved). They are put into the unacked queue, so other workers won't take them. If the worker dies, then those tasks are made available to other workers.

So, when does a worker acknowledge a task? By default Celery assumes that it is dangerous to run tasks more than once, consequently it acknowledges tasks just before they are executed. You can change this by setting the famous acks_late. In this case, a task has the slight possibility of being run more than once, if the worker running it dies in the middle of the execution. And with "dies", I literally mean die. A Python exception in the task code will not kill the worker. Such a task will still be acknowledged, but its state will be set to FAILURE. Something has to happen so that the worker never reaches the code self.acknowledge(). And this is rare. For this reason, I suspect that setting acks_late or not setting it has little bearing.

ETA

As I already mentioned ETA tasks are ... hard to find. They never make it to the main queue. They are immediately assigned to a worker and put into the unacked queue. I suspect that it was not intentional that the ETA tasks immediately get assigned to a specific worker. I suspect this was just a consequence of the existing code. An ETA task can't go into a general queue, which works almost as FIFO. The only other place is among the unacknowledged tasks in which case it needs to be reserved by one worker.

Interestingly, the ETA time is not the exact time this task will run. Instead, this is the earliest time this task will run. Once the ETA time comes around, the task must wait for the worker to be free.

Retry Tasks

Celery doesn't perform any retry logic by default. Mostly because it assumes that tasks are not idempotent, that it is not safe to run more than once. Retrying a task does, however, have full support in Celery, but it has to be set up explicitly and separately for every task.

One way of triggering a retry is by calling self.retry() in a task. What happens after this is triggered? An ETA time is calculated, some new metadata is put together and then the task is sent to the broker, where it falls into the unacked queue and is assigned to the same worker that already ran this task. This is how retry-tasks become ETA tasks and are therefore never seen in the main broker queue. It is a very sleek, but unexpected system. And again, Google has very little to say about this.

Learn more about retries in my Celery task retry guide.

CPU bound or I/O bound and processes vs threads

As we already said, by default Celery executes tasks in separate processes, not threads. But you can make it switch to threads, by starting the workers with either --poll eventlet or --pool gevent. Both eventlet and gevent actually create greenlets, not threads. Greenlets (or green threads) are thread-like, but they are not threads, because by definition threads are controlled by the OS. Greenlets do not rely on the OS to provide thread support, instead, they emulate multithreading. They are managed in application space and not in OS space. There is no pre-emptive switching between the threads at any given moment. Instead, the greenlets voluntarily or explicitly give up control to one another at specified points in your code.

If your tasks are heavy on CPU usage: if they do a lot of computing (=are CPU bound), then you should keep using processes. If, on the other hand, your tasks are mostly doing HTTP requests (=I/O bound), then you can profit from using threads. The reason for this is that while your task is waiting for the HTTP request to return a result, it is not doing anything, it is not using the CPU and would thus not mind if another thread would make use of it.

There is a lot more to Celery

and the documentation is not perfect. Many features have their description split up and dotted around the web page. It is difficult to find details of the implementation. But it is also a complicated subject matter. I don't know how Celery will behave outside of the few scenarios I have literally created and experimented on. Sure, after a few years of intensive work I might have a good understanding of how it works, but Celery lives on the fringes of my day to day. I set it up, but then it disappears into async-land. It behaves radically different when on the server and when on my computer. I can see which tasks were done, but I can't see how well they were done. Transparency is very difficult with something that runs in parallel, possibly in threads and semi-independent of the application. I don't trust it, I don't trust that I understand its settings correctly or I don't trust that I know how to set them correctly. Celery is like a spirit, it comes and goes, sometimes it breaks, but most of the time, it just works. Hopefully, it works on the tasks we assigned it, but if that is not the case, it will be equally silent.

External Sources:

What are UTC, ISO time format and UNIX time and how does Python handle them

Ines — Fri, 30 Oct 2020 13:23:22 +0000

"If my time is 16:44, what is that in UTC time?" is what I asked myself last week.

"Do we use UTC dates or do we store timezones into the database? And by the way, if I call datetime.now(), am I getting the correct time or should I adjust it with a timezone suffix?".

This time-business is so delicate. I never had much trouble with date-times, until last week, when I had to create a humble Celery task that needs to be a master of time, that needs to understand how all the dates on all these objects relate to one another. And suddenly I find myself appalled by all datetimes with no timezone info. What a grievous mistake it was to allow programmers to forget about timezones, to just call now() and hope for the best. How is my super-duper, time-lord of a Celery task supposed to lord the time if a single measly timezone-lacking datetime knocks it out of its balance?

So I had to read a few docs and a few standards to really, truly understand what to do with missing timezones and whether the datetime I am creating is really and truly the exact datetime I want and not one that is 1h before or ahead (my timezone is +01:00, which is very easy to mistake for +00:00).

What is UTC time

UTC is a global time standard. It defined how time will be measured and coordinated between everybody. But UTC is also used as a successor of GMT (=Greenwich Mean Time) to describe the time at the 0° meridian.

In code, we usually denote it with +00:00 or with a Z. The letter Z is used for historical reasons because Z was the name of the timezone using this time. This time is also sometimes called the Zulu time since NATO's phonetic alphabet for Z is "Zulu".

The daylight saving time (=DST) does not affect UTC. DST simply modifies the ±[hh]:[mm]-part of the date-time: Europe's CTE (=Central European Time) is usually described as UTC+1, but during the summer, when DST kicks in, it simply becomes UTC+2.

What is the ISO format / ISO 8601 format

This ISO standard defines how to format dates and date-times without confusion and misunderstandings.

Date format:

2020-10-26

Date and time formats:

2020-10-26T08:15:30+02:00
2020-10-26T08:15:30Z
20201026T081530Z
2020-10-26T08:15:30.456+02:00 <- with microseconds

If the timezone is omitted, the datetime is usually interpreted to be recording the locale time. But of course, this is up to every programming team to define for themselves.

The T is also often omitted. The ISO standard does allow for this, but only if the 2 parties communicating with such dates have agreed to allow it. However, there is no mention of substituting T with a space. But I think we all see that our community has decided that it can correctly understand a datetime with a space and is actively using it everywhere.

What is UNIX time / Epoch time / POSIX time

Unix time is an integer, it's the number of seconds passed since 1st January 1970 UTC - a capriciously chosen date. This date is called the Epoch.

UNIX time excludes all leap seconds. It pretends that every day has exactly 24 * 60 * 60 (=86 400) seconds.

Timezones and UTC offsets

UTC offsets are the ±[hh]:[mm] parts of date-time formats. They are a multiple of 15 minutes and they describe the difference between UTC and the locale time.

Python in the world of datetime

Python differentiates between naive and aware datetimes. The first one has no timezone, while the second does have a timezone. Each datetime object has an optional attribute tzinfo, which can hold timezone information.

How to get the current time

My current time is 2020-10-26T12:30:10Z, my timezone is +1. Here I get my locale time:

In : from datetime import datetime

In : datetime.now()  # produces a naive datetime
Out: datetime.datetime(2020, 10, 26, 13, 30, 10, 933315)


In : datetime.utcnow()  # produces a naive datetime
Out: datetime.datetime(2020, 10, 26, 12, 30, 10, 933315)

Both above functions create a naive datetime.

How to get the current time with timezone

In : from datetime import datetime, timezone

In : datetime.now(tz=timezone.utc)
Out: datetime.datetime(2020, 10, 26, 12, 30, 10, 933315, tzinfo=datetime.timezone.utc)

How to create an exact datetime

Here I create the 31st Dec 2020, at midnight UTC time:

In : from datetime import datetime, timezone

In : datetime(2020, 12, 31, 0, 0, 0, tzinfo=timezone.utc)
Out: datetime.datetime(2020, 12, 31, 0, 0, 0, tzinfo=datetime.timezone.utc)

And here I create the 31st Dec 2020, at midnight in the timezone +03:00:

In : from datetime import datetime, timedelta, timezone

In : datetime(2020, 12, 31, 0, 0, 0, tzinfo=timezone(timedelta(minutes=3 * 60)))
Out: datetime.datetime(2020, 12, 31, 0, 0, 0, tzinfo=datetime.timezone(datetime.timedelta(seconds=3600))

Timezones are set with timedelta to describe the UTC offset.

How to get the UNIX time of any date / how to turn UNIX time to date

In Python, this format is called "timestamp". And it is not an integer, but a float, so that microseconds can be represented.

In : from datetime import datetime, timezone

# turn a date into a timestamp:
In : d = datetime(2020, 10, ...)
In : d.timestamp()
Out: 1603711810.933315  # the num of seconds since Epoch

# turn the timestamp into a naive date
In : datetime.fromtimestamp(1603711810.933315)
Out: datetime.datetime(2020, 10, 26, 11, 30, 10, 933315)
# turn the timestamp into an aware date
In : datetime.fromtimestamp(1603711810.933315, tz=timezone.utc)
Out: datetime.datetime(2020, 10, 26, 11, 30, 10, 933315, tzinfo=datetime.timezone.utc)
# turn the timestamp into an aware date at UTC+3
In : datetime.fromtimestamp(1603711810.933315, tz=timezone(timedelta(minutes=3 * 60)))
Out: datetime.datetime(2020, 10, 26, 14, 30, 10, 933315, tzinfo=datetime.timezone(datetime.timedelta(seconds=10800)))

The POSIX time is always the time at UTC+0, which means that when we call fromtimestamp with the timezone parameter we define only what the timezone of the result will be, not the timezone of the input value. When we passed in UTC+0, it produced the time 11:30, when we passed in UTC+3 it produced 14:30.

How to parse a string into a `datetime`

Use the datetime.fromisoformat for this, BUT: not all ISO 8601 strings can be parsed. Only strings of this format can be parsed:

YYYY-MM-DD[*HH[:MM[:SS[.fff[fff]]]][+HH:MM[:SS[.ffffff]]]]

Using Z as the timezone is not supported.

In : datetime.fromisoformat("2020-10-26T12:30+02:00")
Out: datetime.datetime(2020, 10, 26, 12, 30, tzinfo=datetime.timezone(datetime.timedelta(seconds=7200)))

In : datetime.fromisoformat("2020-10-26 12:30:10.998")
Out: datetime.datetime(2020, 10, 26, 12, 30, 10, 998000)

How to add or remove days/months/years to a datetime

One way is to use timedelta:

In : d = datetime.fromisoformat("2020-10-26 12:30:10.555+01:00")
Out: datetime.datetime(2020, 10, 26, 12, 30, 10, 555000, tzinfo=datetime.timezone(datetime.timedelta(seconds=3600)))

# add 1 day
In : d + timedelta(days=1)
Out: datetime.datetime(2020, 10, 27, 12, 30, 10, 555000, tzinfo=datetime.timezone(datetime.timedelta(seconds=3600)))

# add 4 hours
In : d + timedelta(minutes=4 * 60)
Out: datetime.datetime(2020, 10, 26, 16, 30, 10, 555000, tzinfo=datetime.timezone(datetime.timedelta(seconds=3600)))

# subtract a year (given that this is not a leap year)
In : d + timedelta(minutes=365)
Out: datetime.datetime(2019, 10, 27, 12, 30, 10, 555000, tzinfo=datetime.timezone(datetime.timedelta(seconds=3600)))

The other is to replace parts of the date, but this can trigger ValueErrors:

In : d = datetime.fromisoformat("2020-02-29 00:00:00+00:00")
Out: datetime.datetime(2020, 2, 29, 0, 0, tzinfo=datetime.timezone.utc)

# this date 20 years ago:
In : d.replace(year=2000)
Out: datetime.datetime(2000, 2, 29, 0, 0, tzinfo=datetime.timezone.utc)

# change year to 2019, which didn't have Feb 29th:
In: d.replace(year=2019)
---------------------------------------------------------------------------
ValueError
----> 1 d.replace(year=2019)

ValueError: day is out of range for month

The replace function can be used to replace any part of the datetime: minutes, hours, days, timezone, ...:

In : d = datetime.fromisoformat("2020-02-29 00:00:00+00:00")
Out: datetime.datetime(2020, 2, 29, 0, 0, tzinfo=datetime.timezone.utc)

In : d.replace(year=2019, day=28, tzinfo=timezone(timedelta(minutes=-8 * 60)))
Out: datetime.datetime(2019, 2, 28, 0, 0, tzinfo=datetime.timezone(datetime.timedelta(days=-1, seconds=57600)))

DEV Community: Ines

How do I measure the execution time of Celery tasks?

Signals 🎺

Start of queuing

Start of execution

End of execution

Bonus: And once you are tracking how long tasks take, you can set up alerts for when they take too long. 🛎️

Isolate code blocks from each other?

Example: I'm sending emails to all team members.

What suppress_and_log_exc does

Possible use cases

Adding more settings to the contextmanager

Fin: All together now

How to encourage developers to fix Python warnings for deprecated features

Step 1: First we need to listen to deprecation warnings

Step 2: Pick the warnings you want to get rid of

Step 3: Make deprecation warnings fail the tests

Step 4: Fix the code

Step 5: Lean back and listen to Slack convos and GitHub comments

External links

Why is sooooo.. much documentation a pain to read?

Imagine the following scenario:

Attempt 1: Write a boring, old email

Why no one is reading your emails

Attempt 2: A fun, short email

Attempt 3: Secret messages, that trigger when somebody moves a lever

Archetype

Infamous Webpack

Why is Stack Overflow so immensely popular?

To all future docs authors

Software Estimation Is A Delusion

Is it even important?

What are your options for estimation?

Human Expert

Mathematical models

Estimation variables

Conclusion

External sources:

Celery: A few gotchas explained

Workers and brokers

Queues

Tasks

Task acknowledgment

ETA

Retry Tasks

CPU bound or I/O bound and processes vs threads

There is a lot more to Celery

External Sources:

What are UTC, ISO time format and UNIX time and how does Python handle them

What is UTC time

What is the ISO format / ISO 8601 format

What is UNIX time / Epoch time / POSIX time

Timezones and UTC offsets

Python in the world of datetime

How to get the current time

How to get the current time with timezone

How to create an exact datetime

How to get the UNIX time of any date / how to turn UNIX time to date

How to parse a string into a datetime

How to add or remove days/months/years to a datetime

External Sources:

What `suppress_and_log_exc` does

How to parse a string into a `datetime`