[Python] How to begin your new Python project?

TL;DR

I use those following Python tools for a new project

• package manager: poetry
• formatter: black
• linter: ruff
• unit testing: pytest

Introduction

So say you start a new Python project(whatever it might be!) from scratch. You create a new directory on your local machine and create a new project from there(and almost certainly upstream it to a corresponding GitHub repository). Okay, just to make sure, you'll also have .gitignore file dedicated to Python projects.

But now what? What would you do? Creating a virtual environment for this project via python3 -m venv venv, and next? Sounds like something you'll find from most of the Python tutorials and books out there? But haven't you ever felt so bored from typing source venv/bin/activate all the time? Or haven't you felt your code chunks that have been written are so ugly that you just wanna throw them away?

So what do you need for productivity?

From the hint given by the VS Code Python tutorial, we could now make and develop a project in more convenient, faster, and neat ways. There are several tools that will help you with productivity. Largely, we need the following functionalities to improve our work productivity.

• Python package management: it is painful to do Python packages only with venv and requirements.txt. You need a more organized way of managing, using pyproject.toml.

• code formatting: I know, you are horrible at writing neat code. Would you just check a style guide all the time? That should be onerous.

• code linting: not only formatting, but also some of your code logic might be wrong, such as declaring unused variables. Would you just manually check all those bad habits?

• unit testing: before integrating into a whole system, you need to make sure the small parts that you just implemented are working just as expected. I am not saying that you should adopt a TDD methodology, but somehow you need to do some unit testing before pushing your code into the remote repository.

Now I'd like to introduce briefly a few of such tools that I am actually using for my Python projects.

1. Package Management - poetry

poetry is one of the famous Python package manager that is fast growing. If you are from JS/Node.js, you would have been missing so much about npm or yarn. As far as I know, poetry acts almost in the same way as npm or yarn. The documentation is super clear so you can just dive into it.

One thing to point out: however, unlike npm or yarn, poetry keeps all poetry-related projects' virtual environment directories in a separate directory, and doesn't make a virtual environment directory within the project's root directory. In order to change this, you have to change its configuration.

2. Code Formatting - black

black is a tool that makes your code look neat and pretty. You'll probably writing to long a function's declaration part, the line of which which would go beyond your screen. Or you may not have consistent number of blanks between each classes or functions. Black deals with these ugly parts from your code and correct them, according to its own opinionated style following PEP 8.

Of course, there are other tools like yapf or autopep8. However, I find black is the most convenient tools among these, considering that you don’t have to set up some customized configurations. Unless you are a Python programmer being very serious about your code style, I think you’ll do quite okay only with the default settings done by black.

3. Code Linting - ruff

I used to think of pylint above anything else for linting. However, recently I discovered another tool written in Rust, which means your linting job will become blazingly faster than before. ruff is still in its beta phase, but I find it quite useful. Since a linter is just a static code analyzer, I don’t think you need to be that much careful about the tool, unless it raises unexpected exception that will break our CI/CD pipeline.

4. Unit Testing - pytest

Nowadays it seems like there is no tools that could match against pytest as a Python unit testing tool. Its concise grammar and fast speed attracts most of the famous Python open source projects. Of course, if you’re from JS, you’ll probably see pytest as being a little bit rough compared to frameworks like Jest. But unit testing is unit testing, and I think the most important part in any unit testing, is how you can easily write a test but run it fast at the same time. Regarding that, pytest is doing its job pretty well.

Conclusion

Of course, these tools are not everything. For example, you could arrange these tools to work just before making a commit or within a CI/CD pipeline using tools like Github Actions. Tools are tools, and these are constantly changing. What you must have in your mind, I believe, is that you should keep your balance between tools and your project so that you don’t spend too much time on investigating and configuring tools, rather than focusing on the project itself. In the end, tools are for improving your productivity, not for deprive you of your time.