2018 brought us a plethora of new features with the release of Python 3.7, followed by 3.8 in 2019, and 3.9 in 2020.
Many of those changes were behind the scenes. Optimizations and upgrades that the vast majority of us will never notice, despite their benefits.
Others are more obvious, additions to syntax or functionality that can change how we write our code. But even these visible changes can be hard to keep up with.
In this article, we will run through the more apparent upgrades to provide a brief but hopefully invaluable refresher on everything new to Python from the past few years.
> Python 3.7
- Breakpoints
> Python 3.8
- Walrus Operator
- F-string '=' Specifier
- Positional-only Parameters
> Python 3.9
- New Parser
- More Type Hinting
- Dictionary Unions
If you prefer video, I've run through everything (code included) here:
Python 3.7
Breakpoints
Sometimes, or realistically, almost always - code debugging is frustrating. Python 3.7 made it slightly less annoying though with the inclusion of breakpoint().
This new function allowed us to add breakpoints directly into our scripts, if we see an error that we simply cannot figure out - we can add breakpoint() to stop the code and allow us to try and figure out what is causing the error.
We can then run the code, which will break at our breakpoint and provide us with an interactive shell to test our code with.
It allows us to print, test our functions with different datatypes/variables - and simply do what we need to do to debug our code.
Read more about this in PEP 553.
Python 3.8
Walrus Operator
Most people have already seen the walrus operator, which was the go-to for anyone covering new features in 3.8.
As well as being a fun addition to Python syntax, the operator is genuinely very useful. It allows us to assign a value to a variable 'on the fly'.
This means that we can write more compact code. Although the following is not a good use-case, it makes it very clear what the operator can do.
Here we need to check for the length of a list, and if it is over a certain length, print a statement about its length being greater than three.
The walrus operator allows us to combine the variable assignment and if-statement into a single line like so:
Or perhaps a more useful example is the use of this with RegEx, where we need to check for the existence of a pattern and - if it matches - returns the matched pattern:
PEP 572 covers everything you could ever want to know about the walrus operator.
F-string '=' Specifier
Another good feature for debugging. When printing f-strings we can add = to print both the variable name and value, like so:
A smaller, but still useful addition. Covered comprehensively in bpo-36817.
Positional-only Parameters
Our final feature from 3.8 is the inclusion of syntax to specify function input parameters that cannot be called by name, but instead by position. For example:
Here we can specify the input parameters by their given names - and in most cases this makes sense.
However, sometimes this might be a behavior that we would like to block. For example, if we are coding a function wherein the name of the input parameters is likely to change in the future.
If we and other users of our functions specify the input parameter names, and we then update that function to use a different variable name - we could break all other code-bases using that function.
To stop users from using input parameter names, we can add the positional-only syntax /. Let's see how this behaves when added to our original code:
We see that a TypeError is raised for all parameters defined before /. This prevents users from using these variable names from the start. Now, users will have to use the position of op rather than the keyword:
So that when we do change op to method in the future, our users are protected:
Read all about it in PEP 570.
Python 3.9
I have written more about Python 3.9 in this article, but I will summarize everything here.
New Parser
Although this doesn't bring in any new syntax immediately, it is still significant and can lead to some big changes further down the road.
Guido van Rossum wrote the previous Python parser pgen 30 years ago. It was one of the first pieces of code ever written for Python.
Pgen uses a variant of LL(1)-based grammar. Meaning the parser reads the code top-down, left-to-right, with a lookahead of just one token.
This creates a few limitations:
- A one-token lookahead limits the expressiveness of grammar rules.
- Non-LL(1) grammar implementation requires a lot of messy workarounds for the Python devs.
- Left-recursive syntax can cause an infinite loop in the parse tree, causing stack overflow (explained here).
From Python 3.9 onwards. CPython will use the PEG-based parser, which we will begin to see the impact of from Python 3.10 onwards. You can read more about it in PEP 617 here.
More Type Hinting
The steady introduction and improvement of type hints in Python are a recurring theme in the past few years of Python.
In short, they introduce an optional layer of syntactical sugar that helps us understand which datatypes are expected in our code.
In the image above, we have code without type hints (left), and with (right). The Python linter will also read through this code and identify where there seems to be a mismatch between the datatype defined by our type hint, and the actual data types being used.
We can also use the -> type syntax to determine function output datatypes too.
Or we can define more complex datatypes too. PEP 585 covers type hinting in Python 3.9.
Dictionary Unions
Another interesting addition to Python operators. Both are used for dictionary unions. We have the merge operator |:
a = {1: 'a', 2: 'b', 3: 'c'}
b = {4: 'd', 5: 'e'}
c = a | b
print(c)
[Out]: {1: 'a', 2: 'b', 3: 'c', 4: 'd', 5: 'e'}
And the update operator |=, which allows us to do the merge in-place:
a = {1: 'a', 2: 'b', 3: 'c'}
b = {4: 'd', 5: 'e'}
a |= b
print(a)
[Out]: {1: 'a', 2: 'b', 3: 'c', 4: 'd', 5: 'e'}
Read more about dictionary union operators in PEP 584 here.
And that is it for this article covering some of the biggest changes and upgrades to Python over the past few years.
Of course, there is a huge range of features that we haven't covered here. But, these are those that are most obvious and create the biggest difference when working with Python.
Nonetheless, this selection of new features is more than enough to keep you updated with the growth and evolution of Python. And amidst the upcoming Python 3.10 changes, there is plenty to learn.
I hope you enjoyed this article! If you have any questions, let me know via Twitter or in the comments below. If you'd like more content like this, I post on YouTube too.
Thanks for reading!
Resources
[1] Guido van Rossum, PEG Parsers (2019).
Top comments (1)
Thanks for sharing.
By the way, before breakpoints you could use pdb - python debugger.
The position only syntax with
/is a feature that comes from C.I use F strings all the time.
I don't get the significance of the Walrus operator. It's really rare that I get to writing a line that I wish could be simplified using the Walrus operator. Maybe knowing it exists means I'll eventually start finding uses for it.
Regarding dictionaries merging, there is some older syntax to compare with. Python 3 only.