DEV Community: Victor Silva

Setup Ubuntu for Python development

Victor Silva — Thu, 13 Jun 2019 01:58:29 +0000

In this tutorial I'll show you how I prepare my machine for Python development after installing Ubuntu or one of its derivatives. There are many ways to do this, with different tools and techniques. Here I am sharing the way that works best for me.

These are the two goals of this process:

not to mess with the operating system's Python interpreter;
be able to work on projects that use different Python versions and different Python libraries.

As a practical example, we will create three Python projects:

A Django project with Python 3.7;
A Mezzanine project with Python 3.7 too;
A Flask project with Python 2.7.

We will begin with a fresh installation of Ubuntu 19.04.

System dependencies

We will build some Python versions, and for that we will need to install some system dependencies. We'll follow the recommendation of the subsection "Ubuntu/Debian/Mint" of pyenv's wiki's "Suggested build environment" section. As of 2019-06-11, they are:



$ sudo apt-get update; sudo apt-get install --no-install-recommends make build-essential libssl-dev zlib1g-dev libbz2-dev libreadline-dev libsqlite3-dev wget curl llvm libncurses5-dev xz-utils tk-dev libxml2-dev libxmlsec1-dev libffi-dev liblzma-dev

So we open a Terminal (Ctrl + Alt + T) and install the dependencies:

We'll also need Git:



$ sudo apt-get install git

Python version management

After the system dependencies we'll install pyenv. Among other things, pyenv lets us install different Python versions in our system.

Below, we're following the "installation" section of pyenv's GitHub page. Be sure to check that page for updated instructions.

Clone pyenv into `$HOME/.pyenv`



$ git clone https://github.com/pyenv/pyenv.git ~/.pyenv

Update `$HOME/.bashrc` with the lines below, and restart the shell



export PYENV_ROOT="$HOME/.pyenv"
export PATH="$PYENV_ROOT/bin:$PATH"
if command -v pyenv 1>/dev/null 2>&1; then
  eval "$(pyenv init -)"
fi

Install Python 2.7 and Python 3.7

Note: these may take a long time to complete, for pyenv will download and build CPython from source for each version.



$ pyenv install 2.7.16
$ pyenv install 3.7.3

As we can see in the screenshot, the Python versions have been installed in $HOME/.pyenv/versions. We can run both interpreters directly to verify that they are working properly:

However, we will not be installing libraries in these Python interpreters. They will be the base interpreters for our projects. Actually, each project will have its own isolated Python environment, with its own Python interpreter and its own libraries.

Python isolated environments

virtualenv is a great tool for creating these isolated environments. Installing it on Ubuntu is straightforward:



$ sudo apt-get install virtualenv

The creation of the environments is easy too. We'll create them on the next sessions.

Create the Django project

Let's create a directory $HOME/projects/django_project and then create an isolated Python environment for this project in $HOME/projects/django_project/.venv_django using the Python 3.7.3 that we installed earlier.



$ mkdir ~/projects
$ mkdir ~/projects/django_project
$ cd ~/projects/django_project
$ virtualenv .venv_django --python=$HOME/.pyenv/versions/3.7.3/bin/python

In the screenshot, we can see that an environment was created in the new .venv_django directory, and it contains its own Python interpreter, along with some other tools.

Activate the environment for the Django project

We need to activate this environment so we can use its Python interpreter and install libraries on it. After the activation, executing python (the Python interpreter) or pip (the Python package manager) will use the versions of these tools that are inside the environment.

The activation is essential. If we skip it, we will end up messing with the operating system's Python interpreter, and that's exactly what we don't want to do.

To perform the environment activation, we need to source the script bin/activate located inside the environment directory.



$ source .venv_django/bin/activate

Notice that, when the environment is activated, the prompt is prefixed with the name of the environment's directory.

Install Django and create and run the project in the environment

Now, with the environment properly activated, we can install Django with pip, create the Django project and run it.



$ pip install Django
$ django-admin.py startproject django_project .
$ python manage.py runserver

To quit the Django development server, just hit Ctrl + C.

Now we will create a Python environment for a Mezzanine project, so we must deactivate the currently activated environment (the Django one) and leave the Django project directory:



$ deactivate
$ cd ..

Create the Mezzanine project

In a similar fashion, now we will

create the environment for the Mezzanine project (based on Python 3.7.3 again);
activate the environment;
install the libraries that we need;
create and run the project.



$ mkdir mezzanine_project
$ cd mezzanine_project
$ virtualenv .venv_mezzanine --python=$HOME/.pyenv/versions/3.7.3/bin/python
$ source .venv_mezzanine/bin/activate
$ pip install mezzanine
$ mezzanine-project myproject .
$ python manage.py createdb
$ python manage.py runserver

To quit the Django development server (Mezzanine is built on top of Django), just hit Ctrl + C again.

Before moving to the last project (the Flask one), let's deactivate the Mezzanine environment and leave the project directory:



$ deactivate
$ cd ..

Create the Flask project

For the Flask project, the process will be similar, but the environment will be based on Python 2.7.16 instead. Flask doesn't have an autogenerator of basic projects like Django and Mezzanine do, therefore we'll need to write a little bit of code in order to have a minimal Flask application.



$ mkdir flask_project
$ cd flask_project
$ virtualenv .venv_flask --python=$HOME/.pyenv/versions/2.7.16/bin/python
$ source .venv_flask/bin/activate
$ pip install flask
$ touch app.py

Let's enter the following code in the file $HOME/projects/flask_project/app.py:



from flask import Flask
app = Flask(__name__)

@app.route('/')
def hello_world():
    return 'Hello, World'

Now we can run the Flask app to make sure that it works as expected.



$ flask run

To quit the Flask development server, just hit Ctrl + C. Then, let's deactivate the environment that we created for the Flask project and leave its directory.



$ deactivate
$ cd ..

Wrapping it up

We use Pyenv to install different Python interpreters
We use virtualenv to create isolated environments for different Python projects
We activate each environment before installing the libraries they need
When an environment is activated, the prompt is prefixed with the name of the environment's directory
We deactivate an environment when we don't need to use it
We never touch the operating system's Python interpreter
Each environment has its own isolated set of installed libraries, as shown below:

That's it! Do you have a different approach which works best for you?

My case against the "ternary operator" in Python

Victor Silva — Sun, 19 May 2019 23:45:25 +0000

Ternary what?

Ternary operators, also known as conditional expressions in Python, can be used to make our code shorter:

But i don't like them for two reasons:

1. They often go against the Zen of Python

If you don't know the Zen of Python, start a Python interpreter and type import this. The Zen is a set of guiding principles for writing Python code.

Arguably, most of the time conditional expressions are likely to break 3 of these principles:

Beautiful is better than ugly.
- E.g.: 'Odd' if n % 2 else 'Even' is anything but beautiful
Sparse is better than dense.
- E.g.: 'Odd' if n % 2 else 'Even' is more dense than an if/else statement
Readability counts.
- E.g.: 'Odd' if n % 2 else 'Even' is not as readable as an if/else statement

And, more importantly...

2. They often trick our test coverage

Conditional expressions may make us think that we've covered all possible branches with tests when we actually have not. Check the following example:

If I run the tests with coverage, this is the output:

According to the image, I covered 100% of the lines, without missing any branch. But looking again at the code, we can see that there is no test for months of the second semester of the year.

If I use the classic if statement instead, the lack of coverage will be correctly detected:

If I run the coverage exporting the results to HTML, I can actually see which statement was skipped and which branch was partially run during the test:

Now I know that I should write a test that calls semester with a value greater than 6 if I want the function to be 100% covered.

Conclusion

These are the reasons why I avoid conditional expressions and advise people to avoid them too: they tend to break the Zen of Python and they can fool our code coverage tools.

Granted, not every piece of code does need to be 100% covered by tests, but the ones that do should be measurable in the best way possible, and we've just seen that conditional expressions can get in our way when we are pursuing that.

Make simple Python types rich

Victor Silva — Fri, 17 May 2019 23:42:23 +0000

In the previous post of this series, I created a fictional example that uses a dict to map the installments of a loan to their corresponding due dates, in the following structure:

Suppose that we pass that dict all over the place in our code, and that in some places we need to calculate the difference between the first and the last due date.
We could create a function that makes the calculation, and use it wherever we need it, right? And we could be nice to the callers by providing helpful type hints, of course:

But now we have to import that function in every place we deal with the dict and need the difference between the dates. It would be good if we could encapsulate both the function and the data in the same object without having to create a complex class.
Well, it turns out that we can do that by subclassing Python's built-in dict, and our subclass will behave like a standard dict because it will actually be one:

Note that, unless we read the class's code, we have no clue that the keys and values are supposed to be integers and dates respectively. Not even PyCharm can help us:

We can fix that by inheriting from typing.Dict instead, which is a generic version of dict that can be annotated:

Look at the autocomplete that we have now:

Sweet!

Git surgery #1

Victor Silva — Sun, 28 Apr 2019 17:39:36 +0000

The context

At work we use Vincent Driessen's Git Flow branching model. Recently we found ourselves in the following situation:

feature A had been finished and merged into develop
release 1.2.3, containing feature A, had been started
feature B had been started after the start of the release 1.2.3
there were some adjustment commits for feature A in the release 1.2.3 branch
feature B had been finished and merged into develop
feature B didn't depend on any code added by feature A.

Visually, that's what we had (each column represents a branch; and each point represents a commit, with a single letter instead of its hash):

Now we just had to test and finish the release 1.2.3 (merge it into master, create a tag 1.2.3 at the merge, and then merge the tag into develop in order to integrate commits F, G and J) and deploy the newly created tag into production. Then we would create another release for feature B (probably 1.2.4), test, finish and deploy it.

The problem

For reasons that are beside the point of this post, it was decided that feature B should make it into production before feature A. In order to not mess up our version numbers:

release 1.2.3 now should contain feature B, but
- without the code of feature A commits (B, C, D and E)
- without the code of the adjustment commits F, G and J
the "new" release 1.2.3 should be tested, finished and deployed
feature A and its adjustment commits should be in a new release (1.2.4)
commit history of develop should not be rewritten, for it's a shared branch and other features may have been started based on it.

The solution

Note: for simplicity, in the Git commands below I'll use the letter that was used to represent each commit in the images instead of the commits' hashes.

We took note of the hashes of the feature A adjustment commits (F, G and J)
On branch develop, we reverted the commit that merged feature/A into develop (the merge commit E), so that code related to feature A was "removed" from develop:
- <master> $ git checkout develop
- <develop> $ git revert -m 1 E # creates commit M
We used reset --hard to reset the branch release/1.2.3 to develop, so that release/1.2.3 pointed to commit M
- <develop> $ git checkout release/1.2.3
- <release/1.2.3> $ git reset --hard develop
Still on release/1.2.3, we made a commit to bump the version number (a change required by Git Flow)
- <release/1.2.3> $ echo '1.2.3' > version.txt
- <release/1.2.3> $ git add version.txt && git commit -m "Bump version number" # creates commit N Note that, since the reset --hard, the commits F, G and J have been "lost". That's why we took note of their hashes in the beggining - we'll bring them back later.
After testing release 1.2.3, which now only contained code related to feature B, we finished (creating its tag and merging the tag into develop) and deployed it
- <release/1.2.3> $ git checkout master
- <master> $ git merge --no-ff release/1.2.3 # creates commit O
- <master> $ git tag -a 1.2.3
- ./fictional-deploy.sh
- <master> $ git checkout develop
- <develop> $ git merge --no-ff 1.2.3 # creates commit P At this point, we have released only feature B, but now we must bring back the code of feature A, create a new release with it, and also bring back its adjustments commits.
On develop, we reverted the commit M, which was a revert commit itself. Yes, we "reverted a revert".
- <develop> $ git revert M # creates commit Q In practice, that brought the code of feature A (inserted in commits B, C, D and E) back to develop.
Finally, we created the new release (1.2.4) and brought the adjustments previously made to feature A into it
- <develop> $ git checkout -b release/1.2.4
- <release/1.2.4> $ git cherry-pick F # creates commit R
- <release/1.2.4> $ git cherry-pick G # creates commit S
- <release/1.2.4> $ git cherry-pick J # creates commit T

Conclusion

With a few basic Git commands (reset, revert, cherry-pick, etc.) we were able to solve our problem:

only code related to feature B has been deployed
feature A and its adjustment commits have not been lost
we haven't messed up our versioning numbers
the history of the branch develop has not been rewritten.

Avoid dict overuse in Python

Victor Silva — Tue, 14 Aug 2018 04:02:07 +0000

Python's dicts are great and we all love them, but on large projects we can easily end up with dozens of functions like this:

There's nothing inherently wrong with the function above, but there are some little things that may complicate our lives as the project grows.

We may mistype the value of a key when accessing the returned data (like typing daily_interest_rates instead of daily_interest_rate). Because we're dealing with a dict, we would only find this error (that is, if we find it) during run time.

Or maybe we could start calling this function and using its returned dict in a lot of places. In this case, we would find ourselves repeatedly coming back to the implementation just to remember the dict's keys, or to check the due dates' structure.

We can get rid of these potential problems in advance, and make this code a lot more explicit, if we

Use proper classes instead of using dicts everywhere

Let's try it:

Note: below, I will use dataclasses, added in Python 3.7, to avoid some boilerplate __init__ code.

Now our function returns instances of a class, and thanks to the type hints that were added, we know the exact structure of these instances.

Note: if you never heard of type hints, check my really short introdution to them:

Python's type hints

Victor Silva ・ Aug 12 '18

#python #python3 #typehint

By using any Python-aware IDE, the two problems that I described before are no more. Because of the new class CalculationResult combined with type hints, PyCharm will warn me if I mistype a field name:

If I forget the structure of the due dates, type hints got me covered. PyCharm will rely on them to remind me that the dates are the values of a dict whose keys are integers:

Now, I'm not saying we should stop using dicts altogether. My point is that, for larger projects, maybe it's better to represent our data as proper classes. With the new Python dataclasses, we can do this in a jiffy.

Nowadays I tend to prefer dicts only in places like the field CalculationResult.due_dates above: when all keys are of the same type and represent the same thing (e.g.: the installments) and all the values are of the same type and represent the same thing (e.g.: the due date of the corresponding installment).

I find that my code is getting more expressive, explicit and maintainable by adopting this strategy.

Python's type hints

Victor Silva — Sun, 12 Aug 2018 02:12:26 +0000

If you've been working with Python for a little time, chances are that you've heard that explicit is better than implicit. If you've been working with Python for a long time, chances are that you strongly agree with that statement.

Today, I wanna share with you one of my favorite ways to make my Python code more explicit:

Use type hints

Let's consider the following function:

Before reading its body (which could be extremely complex), one knows neither the expected parameter type nor the return type.

Of course, we could name the function get_due_dates_dict and the parameter start_date, but that would be extremely anti-pythonic, specially when we have a much better option: type hinting via function annotations. Check it out:

Ok, now by just looking at the function's heading, we know that it expects a date and returns a dict. And just as important, any Python-aware IDE will know this as well, and will give us some nice autocompletion features.

But we can do better. We can annotate our function is such a way that we (and our IDEs) know even the structure of its return without having to read its body:

Now, it's explicit that our function returns a dict in which the keys are integers and the values are dates. In the screenshots below, we can see that PyCharm understands this perfectly, and gives us precise tips and autocompletions:

Type hints make everyone's life easier down the road.

That's it for today, happy coding!

DEV Community: Victor Silva

Setup Ubuntu for Python development

System dependencies

Python version management

Clone pyenv into $HOME/.pyenv

Update $HOME/.bashrc with the lines below, and restart the shell

Install Python 2.7 and Python 3.7

Python isolated environments

Create the Django project

Activate the environment for the Django project

Install Django and create and run the project in the environment

Create the Mezzanine project

Create the Flask project

Wrapping it up

My case against the "ternary operator" in Python

Ternary what?

1. They often go against the Zen of Python

2. They often trick our test coverage

Conclusion

Make simple Python types rich

Git surgery #1

The context

The problem

The solution

Conclusion

Avoid dict overuse in Python

Use proper classes instead of using dicts everywhere

Python's type hints

Victor Silva ・ Aug 12 '18

Python's type hints

Use type hints

Clone pyenv into `$HOME/.pyenv`

Update `$HOME/.bashrc` with the lines below, and restart the shell