DEV Community: Nick Thapen

Python Refactorings - Part 1

Nick Thapen — Tue, 12 May 2020 11:10:00 +0000

Writing clean, Pythonic code is all about making it as understandable, yet concise, as possible. This is the first part of a series on Python refactorings, based on those that can be done automatically by Sourcery. Here I am focusing on why these changes are good ideas, not just on how to do them.

1. Merge nested if conditions

Too much nesting can make code difficult to understand, and this is especially true in Python, where there are no brackets to help out with the delineation of different nesting levels.

Reading deeply nested code is confusing, since you have to keep track of which conditions relate to which levels. We therefore strive to reduce nesting where possible, and the situation where two if conditions can be combined using and is an easy win.

Before:

if a:
    if b:
        return c

After:

if a and b:
    return c

2. Hoist repeated code outside conditional statement

We should always be on the lookout for ways to remove duplicated code.
An opportunity for code hoisting is a nice way of doing so.

Sometimes code is repeated on both branches of a conditional. This means that the code will always execute. The duplicate lines can be hoisted out of the conditional and replaced with a single line.

if sold > DISCOUNT_AMOUNT:
    total = sold * DISCOUNT_PRICE
    label = f'Total: {total}'
else:
    total = sold * PRICE
    label = f'Total: {total}'

By taking the assignment to label outside of the conditional we have removed a duplicate line of code, and made it clearer what the conditional is actually controlling, which is the total.

if sold > DISCOUNT_AMOUNT:
    total = sold * DISCOUNT_PRICE
else:
    total = sold * PRICE
label = f'Total: {total}'

3. Replace yield inside for loop with yield from

One little trick that often gets missed is that Python's yield keyword has a corresponding yield from for collections, so there's no need to iterate over a collection with a for loop. This makes the code slightly shorter and removes the mental overhead and extra variable used by the for loop. Eliminating the for loop also makes the yield from version about 15% faster.

Before:

def get_content(entry):
    for block in entry.get_blocks():
        yield block

After:

def get_content(entry):
    yield from entry.get_blocks()

4. Use any() instead of for loop

A common pattern is that we need to find if some condition holds for one or all of the items in a collection. This can be done with a for loop such as this:

found = False
for thing in things:
    if thing == other_thing:
        found = True
        break

A more concise way, that clearly shows the intentions of the code, is to use Python's any() and all() built in functions.

found = any(thing == other_thing for thing in things)

any() will return True when at least one of the elements evaluates to True,
all() will return True only when all the elements evaluate to True.

These will also short-circuit execution where possible. If the call to any() finds an element that evalutes to True it can return immediately. This can lead to performance improvements if the code wasn't already short-circuiting.

5. Replace list() with []

The most concise and Pythonic way to create a list is to use the [] notation.

x = []

This fits in with the way we create lists with elements, saving a bit of mental energy that might be taken up with thinking about two different ways of creating lists.

x = ['first', 'second']

Doing things this way has the added advantage of being a nice little performance improvement.

Here are the timings before and after the change:

$ python3 -m timeit "x = list()"
5000000 loops, best of 5: 63.3 nsec per loop

$ python3 -m timeit "x = []"
20000000 loops, best of 5: 15.8 nsec per loop

Similar reasoning and performance results hold for replacing dict() with {}.

6. Hoist statements out of for/while loops

Another type of hoisting is pulling invariant statements out of loops. If a statement just sets up some variables for use in the loop, it doesn't need to be inside it.
Loops are inherently complex, so making them shorter and easier to understand should be on your mind while writing them.

In this example the city variable gets assigned inside the loop, but it is only read and not altered.

for building in buildings:
    city = 'London'
    addresses.append(building.street_address, city)

It's therefore safe to hoist it out, and this makes it clearer that the same city value will apply to every building.

city = 'London'
for building in buildings:
    addresses.append(building.street_address, city)

This also improves performance - any statement in a loop is going to be executed every time the loop runs. The time spent on these multiple executions is being wasted, since it only needs to be executed once. This saving can be significant if the statements involve calls to databases or other time-consuming tasks.

Conclusion

As mentioned, each of these is a refactoring that Sourcery can automatically perform for you. We're planning on expanding this blog series out and linking them in as additional documentation, with the aim of turning Sourcery into a great resource for learning how to improve your Python skills. If you have any thoughts on how to improve Sourcery or its documentation please do email us or hit me up on Twitter

How do you test code written by code?

Nick Thapen — Tue, 28 Apr 2020 00:00:00 +0000

Refactoring is the process of improving the quality of code without changing what it does.

Our challenge in building Sourcery has been to automate this, and we've made some big strides which you can try out here. A question we're often asked is how we ensure that the changes Sourcery makes haven't affected the code's functionality. This is key - having trust that Sourcery won't break anything allows users to accept the refactorings with confidence.

The approach we take relies on three main pillars:

Static analysis of code to identify dependencies, types and variable usage
Extensive unit tests of code snippets before and after each type of refactoring
Running Sourcery over a suite of open source libraries and ensuring their tests still pass

Static analysis

Our static analysis toolset borrows heavily from compiler theory. Initial analysis of each piece of code to be refactored generates an extended reaching definition
to identify where variables are read and written.

a = {'example': 1}
b = a
this_could_add_to(a)
print(b)

For example this analysis picks up that the variable b has been linked to the variable a.
This means that the print statement and the call to this_could_add_to cannot be swapped without possibly changing the code's functionality.

Other types of analysis we do include type inference, analysis of logical statements to infer which conditions are true at each statement, and analysis of control-flow and loops.

All of this underpins Sourcery's refactoring engine, allowing us to determine when it is safe to make changes and which changes are safe to make.

Automated Tests

We believe unit testing to be an essential feature of a good quality codebase, and we use them extensively.

For each refactoring we generate many variations of positive and negative tests.

Positive tests are of the form: given this source code, it should be refactored to produce this expected source code.

Negative tests are of the form: this source code should not be changed by this type of refactoring.

For example - here we know that the inner a is truthy:

a = get_a()
if a:
    b = func()
    if a:
        print(b)

so we can remove the inner conditional:

a = get_a()
if a:
    b = func()
    print(b)

On the other hand there are many similar pieces of source code that should not be refactored:

a = get_a()
if a:
    a = func()
    if a:
        print(b)

Even though the outer a was truthy, we can say nothing about the value of the inner a and so cannot remove the inner if.

These negative tests are just as important, as they capture all the patterns that would results in functionality changing.
These include properties like: changing or using non local state, variables not having the required type, and dependencies between statements not allowing reordering.
This type of test ensures that our static analysis is functioning correctly.

We rely extensively on our analysis and unit testing, but operate under the principle of trust, but verify.
No matter how extensive our analysis and comprehensive our testing, it's still possible for changes in functionality to slip through the gaps.

That's why we also test Sourcery on open source libraries.

Open source library testing

As part of our Continuous Integration pipeline (run using GitHub actions) we test all refactoring changes on a suite of open source libraries.
The steps are the same for all of them:

Clone the latest stable version of the open source code repository
Run Sourcery on the entire code base, automatically applying all refactorings
Run the libraries' full test suite and check that all the tests pass

Each test failure that is found is reviewed to identify which refactoring caused it.
This always results in an improvement to our static analysis or refactoring definitions, and new tests are added so that the problem cannot reoccur.

Conclusion

As we stated at the start, refactoring is the art of improving code without changing the functionality. We consider it a major bug if Sourcery ever breaks working code and so we never release a version that does not pass all the above testing.

If you ever discover an example where Sourcery changes the functionality of source code, please raise an issue. We will fix it and add it to our test suite so it doesn't happen again.

We hope you found this interesting and it gives you some extra confidence. You can use Sourcery for free on GitHub or through your IDE!

Use Sourcery to review your Python code on GitHub!

Nick Thapen — Fri, 13 Mar 2020 00:00:00 +0000

We're excited to announce that Sourcery is now available as a GitHub app!

It is currently available for public repos - and will always be totally free for open source.

Easy setup - Get started quickly with out-of-the-box GitHub integration.
Check your whole codebase - On installation Sourcery will open a pull request for each of the installed repos suggesting changes across the whole codebase.
Ensure great quality - Sourcery will review every pull request from then on and suggest improvements.
Apply the changes instantly - Just merge the Sourcery pull requests to boost your code quality straight away.

To get it for free just sign up here!

Sourcery now available in VS Code!

Nick Thapen — Tue, 10 Mar 2020 00:00:00 +0000

We're excited to announce that Sourcery is now available as a VS Code extension.

This means you can use all of the great features of Sourcery directly in your VS Code editor:

Instant refactoring suggestions. We're the only tool that will refactor your code for you!
As you type Sourcery will analyse your code and suggest improvements
which you can add with one click.
We don't break your code. Sourcery uses extensive static analysis to ensure that its refactorings
don't change the existing functionality - backed up by testing on open source repositories.
Runs locally. Sourcery runs completely locally on your machine - no code is ever sent to the cloud.

Here's an example of a Sourcery refactoring from inside VS Code:

To get it for free just follow these steps:

Open VS Code and press Ctrl+P (Cmd+P on Mac) then paste in ext install sourcery.sourcery and press Enter.
Click here to get a free token.
Enter the token into the provided input, or search for sourcery in the VS Code settings and enter it into the Sourcery Token field.

Getting the most out of Python collections

Nick Thapen — Tue, 18 Feb 2020 00:00:00 +0000

One of Python's best features is that it has awesome capabilities for creating and handling collections. Building a good understanding of these will help you write clean Pythonic code.

To get you started this guide covers comprehensions, generators, and some really useful built-in functions and collection types.

Create and filter collections with Comprehensions

List, Dictionary and Set comprehensions are a powerful feature of the Python language that enable you to declare these collection types and fill them with values at the same time. This lets you do all sorts of filtering and mapping operations in a simple and readable way.

List Comprehensions

Without comprehensions a common pattern used to create a list and fill it with values is as follows:

cubes = []
for i in range(20):
  cubes.append(i ** 3)

Each operation is carried out separately - an assignment to create the empty list, and then a for loop iterating over a sequence within which the list is appended to. We end up with a list of cubed numbers sitting in the cubes variable.

This can be shortened to the following using a list comprehension:

cubes = [i ** 3 for i in range(20)]

Let's unpack what's going on. At first glance it can be confusing, with lots of the elements from the for loop version moved into different places in the code.

We know the purpose of the code is to create a list of cubed numbers. We therefore have an assignment using square brackets to signify that we are creating a list. Inside the comprehension we can start with the second half: for i in range(20). This is a standard for loop, but instead of a for body we put an expression in front of it giving the element that we will create at each iteration. In this case we have i ** 3, cubing the target variable i. The expression can be simple, or can involve any sorts of function calls or operations that you like. It does not even have to use the target variable.

We can also add an if clause at the end to filter the list:

odd_cubes = [i ** 3 for i in range(20) if i % 2 != 0]

Often squeezing code onto one line can make it more difficult to read. In the case of comprehensions all the elements that you need are nicely presented, and once you are used to the syntax it is actually more readable than the for loop version of the same code.

Dictionary Comprehensions

You can also directly create dictionaries using a dictionary comprehension. Here you set the keys and values at once using the following syntax:

squares = {x: x ** 2 for x in range(100)}

This creates a dictionary mapping numbers to their squared values. As you can see you set the keys and values while iterating over a sequence. Again you can use if clauses in your dictionary comprehensions in order to perform filtering at the same time.

Filtering would look like this:

even_squares = {x: x ** 2 for x in range(100) if x % 2 == 0}

Set Comprehensions

Moving on, there are also set comprehensions, which create a set, so all values are unique.

square_roots = {math.sqrt(number) for number in number_list}

Again these can be filtered. If you want to filter using an if expression then the position of the conditional in the comprehension changes like this:

positive_square_roots = {math.sqrt(number) if number > 0 else 0 for number in number_list}

More complex comprehensions

Comprehensions can also replace nested for loops. Take the example of flattening a list of lists:

list_of_lists = [[1, 2], [3, 4]]
flat = []
for sub_list in list_of_lists:
  for element in sub_list:
    flat.append(element)

This produces [1, 2, 3, 4]. As a comprehension this turns into:

list_of_lists = [[1, 2], [3, 4]]
flat = [element for sub_list in list_of_lists for element in sub_list]

You can see that the ordering of the for loops has been maintained. Using comprehensions for nested for loops in this way becomes a bit unintuitive. Past a certain level of complexity it is more difficult to understand than the same code would be using a for loop.

Just because you can achieve something in a one line comprehension using multiple for and if clauses doesn't mean that it's a good idea.

Iterating with Generators

A generator is a special type of Python function that returns a lazy iterator. You can loop over these results without the whole list of them having to be stored in memory.

For a concrete example take this function that generates the Fibonacci sequence of numbers:

def fib():
    a, b = 0, 1
    while 1:
       yield b
       a, b = b, a + b

You might then call it with code like this:

def print_fibonacci(max_size)
  for number in fib():
    if number < max_size:
      print(number)
    else:
      break

So what's going on here? From the point of view of the calling function fib() just looks like a collection.
When fib() is first invoked it executes as normal until it hits the yield b statement. It then returns control to its caller
along with the value, just like return b would do. However, fib() has only been suspended, and all its state has been saved. When the iterator it returns is queried again it resumes from where it left off instead of starting again from the beginning like a normal function.

The other way of defining generators in Python is using generator expressions. These use the same syntax as comprehensions, but are wrapped in round brackets:

cubes = (i ** 3 for i in range(20))

These return an iterator rather than a collection. If the iterator is exhausted and you ask it for another item it will return a StopIteration exception.

In fact a comprehension behaves like wrapping a generator expression in a call to list() or set():

cubes = list(i ** 3 for i in range(20))

Note that when passing a generator into a function, Python allows you to omit one of the pairs of brackets (like in the example just above - we are making a call to list()).

One tip is that functions like any, all and sum allow you to pass in a generator rather than a collection. This may lead to performance improvements due to the lazy evaluation (i.e. the function may be able to return before evaluating all of the elements from the generator).

The primary benefit of generators comes from the aforementioned lazy evaluation. This can make the memory overhead of a generator much lower than creating the whole collection at once and then iterating over it.

Take the example of reading lines from a file. If the file is large then reading the whole of it into memory may be extremely slow or impossible. The file object returned by open() is already a generator of lines, so can be used like this:

def read_lines(file):
  with open(file) as f:
    for line in f:
      print(line)

Useful types of collection from the `collections` module

An important way to improve your collections code is to make use of the available builtin types. Here are two of the handiest:

`defaultdict`

This is a subclass of dictionary that calls a factory function to supply missing values.

This lets you replace this:

fruit_sizes = {}
for fruit in fridge:
  if fruit.name in fruit_sizes:
    fruit_sizes[fruit.name].append(fruit.size)
  else:
    fruit_sizes[fruit.name] = [fruit.size]

with this:

fruit_sizes = defaultdict(list)
for fruit in fridge:
  fruit_sizes[fruit.name].append(fruit.size)

Here any missing values are initialized to empty lists. This avoids cluttering up the code with checks and initializations.

Other common arguments are to create defaultdicts of int, float, dict or set types.

To create one which sets the default to anything you like you can use lambda functions. For example if all students are assumed to have a small green hat you would use:

student_hats = defaultdict([Hat('green', 'small')])

`Counter`

Need to count occurrences of multiple different hashable objects? A Counter is your friend. It is a dictionary subclass where the elements are the keys and their counts are the values.

When creating or updating the counter you give it an iterable. It iterates through and counts the occurrences of items, matching them by their hashes.

An easy way to get started on understanding Counter is to count all of the occurrences of strings in a list.

counts = Counter(['Fred', 'Samantha', 'Jean-Claude', 'Samantha'])

counts then looks like this:

Counter({'Samantha': 2, 'Fred': 1, 'Jean-Claude': 1})

You can then update the Counter with further iterables, or other instances of Counter.

Useful functions for collection handling

Once you have a collection or a generator the next stage is to do something useful with it. Python contains many useful functions for collection handling that can improve how you write your code.

Using `any()` and `all()`

A common pattern to establish something about a collection is the following:

any_green = False
for hat in hats:
  if hat.colour == 'green':
    any_green = True
    break

This can be replaced with a call to any() using a generator:

any_green = any(hat.colour == 'green' for hat in hats)

This returns whether any item in the generator satisfies the condition, and will shortcut the evaluation just as the above for loop does with the break statement.

all() behaves similarly, but returns whether all items satisfy the condition.

Iterating over collections together with `zip()`

You often find yourself with multiple collections which you need to iterate over, collecting data together from each one.

for name, email in zip(names, emails):
  create_person(name, email)

This steps through both collections at the same time, returning a tuple consisting of elements from each one.
Here we have unpacked the tuple into separate values. zip() can take as many collections as you like, but will stop when the
shortest one is exhausted. If you'd like to deal with all values you can use itertools.zip_longest(*iterables, fillvalue=None). This will carry on until all collections are exhausted, filling in missing values with the fillvalue.

One neat use of the zip() functions is to iterate over pairs of elements in the same collection. For example:

differences = [next_elt - elt for elt, next_elt in zip(items, items[1:])]

Chaining over multiple collections

If you need to iterate over multiple collections one at a time, you can use itertools.chain().

for name in itertools.chain(first_name_list, second_name_list):
  create_person(name)

This iterates through the first collection until it is exhausted, then moves on to the next and so on.

There are lots more interesting and advanced functions in the itertools module, and we will return to them in
a further blog post.

A Field Guide to Fixing Bugs

Nick Thapen — Tue, 28 Jan 2020 00:00:00 +0000

As developers a lot of our time is spent on tracking down and fixing bugs. Most of us see it as an unwelcome chore and would rather be writing exciting new code. There's some truth in that, and bug-fixing can certainly be frustrating (and unpleasant if there's time pressure).

However tracking down and fixing software issues can also be a fascinating puzzle. Sleuthing your way to a complete understanding of an issue and then crafting a solution can make you feel like a cross between Sherlock Holmes and Scotty the Engineer.

To try and make the process quicker and more enjoyable here are some tips on the best way to identify and fix bugs.

Reproduce the bug!

The first thing you have to do when faced with a bug report is reproduce the issue. This is absolutely vital, since if you can't see the evidence that the code is broken then there's no way of checking that its been fixed.

The effort required to reproduce an issue can vary from trivial to impossible. If you've noticed the bug yourself or the bug report is detailed then there's not much to do except check that it can be reproduced reliably.
Unfortunately most users are not technical experts and so will not automatically write detailed and informative bug reports. This means that you will often need to get more details. Try and get step by step instructions on how to reproduce the issue, along with any other useful context such as software versions and the operating system used. If you're on a helpdesk it can be helpful to talk directly to the person who experienced the bug to avoid long email exchanges.

If you can't immediately reproduce the issue then make sure you are using as similar an environment to the reporting user as possible, including things like OS version, software/browser version and any configuration settings.

If the issue is sporadic or just can't be reproduced at all then you'll have to use a logging-based approach (see later on in this article).

Now reproduce it as easily as possible

To narrow down the issue it's often best to see if you can identify the crucial steps needed to reproduce it. For example if the bug relates to placing an order then steps on the bug report related to viewing account information may not be needed. This will help identify the section of the code that is at fault, as well as saving time since you might need to reproduce/debug the issue many times before you've cracked it.

If at all possible try to create a failing unit test that reproduces the issue. This will let you reproduce it instantly and capture it in the debugger, as well as verifying it is fixed once the test passes.

If it can't be reproduced yet - Log log log

Bugs that can't be reproduced can be a real nightmare. The best thing to do in this case is to gather as much information about the issue as possible. This can involve getting logs from the machines where the issue occurs, turning on enhanced logging or adding extra logging to the application and delivering it. Screenshots of the issue occurring can also be extremely helpful in identifying the circumstances in which it occurs.

An example where logging came in handy was at a previous company where users of our desktop application started reporting strange behaviour. After using the application for some time other applications would shut down, or ours would, or the whole OS would behave strangely or crash. We tried and failed to reproduce it locally so started collecting logs from users.

At first we focused on memory usage, thinking a memory leak must be to blame (the application was written in Delphi). However the logs didn't show a clear memory leak, and we were stuck on the issue for days.

Fortunately we had decided to log the OS of the users, and I noticed that all the ones having trouble were on Windows XP. This brought to mind an old blog post I had read on the limits of Windows. The post mentioned that XP had a much lower limit on memory allocated to USER objects (representing windows, icons, popups, etc) than later versions. This limit was small enough to not show much of an impact on overall memory allocation.

Armed with this hypothesis I looked at recent changes to how we handled UI elements, and discovered a change to how we created popup windows - this was the cause of the USER object handle leak. In an XP VM I was then able to reproduce the issue by right-clicking 40-50 times (something users would naturally do in the course of a day but that we didn't in our shorter testing sessions).

Identify the code which is causing it

There are several ways of finding the code which is the cause of the errant behaviour. Sometimes it is obvious, either through the nature of the error or previous experience. Where it's not try one of the following:

Check which recent changes might have introduced it.

Determine exactly when it started happening, and in which versions, and look for commits that went live at that time. Depending on where you work there may be particular files (or developers) that are more prone to issues than others.

Another pair of eyes.

If you're starting to get stuck it's often super useful to ask someone. Firstly the process of having to explain the issue to someone else can often jog something in your brain and help you to solve it. Secondly a more experienced dev, or someone who is knowledgeable about that part of the system may be able to shortcut hours or even days of work by identifying the problem straight away. As a senior developer being able to point people in the right direction was one of the most rewarding parts of the job. Don't be afraid to ask for help as long as you've had a good go at figuring out the issue and are able to explain the problem in a detailed and logical way.

Think logically

What conditions in the code must be true for the bug to occur? If you write these down, along with the assumptions that you're making, it can often help locate the issue. When you eventually find the cause of a problem it's often an obvious thing that you assumed couldn't go wrong.

A recent example came when developing a new VS Code extension for Sourcery. The Language Server Protocol API call that we were trying to use wasn't working, and we spent a long time tweaking the parameters assuming that we were doing something wrong. In the end it turned out that our dependencies weren't up to date and we were using an old version of the API that didn't contain that particular call.

It's never the OS or hardware.

When you've been tracking down a bug long enough it's easy to convince yourself that it can't be in your code and that some outlandish problem that's outside your control must be to blame. Unfortunately in 99.99% of cases it's a bug in your code. Unless the problem is a one-time or extremely sporadic issue that is fixed by rebooting, you'll just have to keep plugging away at it. If you're totally stuck it's time to:

Get some sleep

Staring at an issue long into the night is very rarely helpful. A good night's sleep will often provide the answer, and you'll come in the next morning with the solution in hand, or at least with new approaches to try.

Make the right fix, not the easy fix

Once you've found the code that's to blame it's tempting to patch it right away, but it's worth spending a little more time digging. Make sure that you really understand the root cause of the issue, and don't just paper over the cracks. For example if memory is overflowing increasing the heap size won't stop the issue from occurring again later - you need to find and fix the leak.

Verify that you've fixed it and that it won't regress.

Since you reproduced the issue at the start, you can now test it with and without your fix to make absolutely sure it goes away. If possible you should also add or extend a unit test for the issue. This makes sure that if it regresses then you'll know about it before your users do.

Thanks for reading! If you work in Python and want refactoring suggestions to avoid bugs in the first place please check out Sourcery. Also I would love to hear your best bug stories - the best place to contact me is on Twitter.

5 Refactoring Tips to Improve Your Code

Nick Thapen — Tue, 14 Jan 2020 00:00:00 +0000

Refactoring is the process of improving the quality of code without changing what it does. Taking a little bit of time to refactor your code before committing it can make it much more readable, which makes it easier to maintain. Doing this consistently turns finding bugs and adding new features from a nightmarish hellscape into a walk in the park.

Ideally your code will be fully covered with unit tests, allowing you to make changes with confidence. If it isn't then you should add tests before starting to refactor if at all possible. However, if you aren't able to add tests you can still achieve quite a bit by using automated, safe refactoring tools.

So where to start? This article contains 5 easy things you can do straight away to make your code more readable and indeed beautiful.

1. Remove commented out code.

This is the easiest possible refactoring, and it can yield excellent results for your codebase.

Commented out code is confusing. It bloats the codebase and makes it difficult to follow the true execution paths of the code. It also gets in the way of genuine comments, which can become lost in a sea of out of date code lines.

# lots of
#
# commented out lines of code
do_something_important()
#
# even more
# commented out code

In examples like this it's possible to miss vital information as you're reading the code, since you're trying to skip over the comments.

If you're worried that the commented out code might be useful some day, the correct way of storing it is in source control. If you ever need it then it will be there in the commit history. Just give the commit where you delete the comments a suitable name and they will be easy to find.

Best of all, as long as you're careful to only delete commented out lines, this refactoring can never break your code.

2. Extract magic numbers and strings

While you're developing it's often easier to write string literals and numbers directly into your code. However leaving them there is a recipe for problems down the road.

If the number or string changes later you will need to find every instance of the literal, check if it needs to change and then alter it. Making repeated changes to duplicate code in this way is one of the leading causes of bugs, since it's easy to miss out one of the usages. Replacing them with a constant means that the literal is stored in one place and only needs to change in that place.

Another problem is that a bare number like the one in this example doesn't tell you why it has the value or what it is for.

def update_text(text):
    if len(text) > 80:
        wrap(text)

Replacing it with a constant let's you give it a descriptive name, which makes the code easier to read and understand.

MAX_LINE_LENGTH = 80

def update_text(text):
    if len(text) > MAX_LINE_LENGTH:
        wrap(text)

When it comes to actually making the changes, many IDEs will help you to extract literals with an 'Extract Constant' option.

3. Remove duplication

DRY (Don't Repeat Yourself) is one of the foundational principles of software development. Duplication makes code harder to understand, and often leads to bugs when the duplicated code starts to diverge.

Code Hoisting

One easy way to remove duplication is if you notice an opportunity for code hoisting. This is where code is repeated on both branches of a conditional, so can be taken outside.


if sold > DISCOUNT_AMOUNT:
    total = sold * DISCOUNT_PRICE
    label = f'Total: {total}'
else:
    total = sold * PRICE
    label = f'Total: {total}'

By taking the assignment to label outside of the conditional we have removed a duplicate line of code, and made it clearer what the conditional is actually controlling, which is the total.

if sold > DISCOUNT_AMOUNT:
    total = sold * DISCOUNT_PRICE
else:
    total = sold * PRICE
label = f'Total: {total}'

This then allows us to further refactor if we wish, by replacing the conditional with an if expression. If you'd like a tool which can perform these sorts of refactorings automatically, try Sourcery.

total = sold * DISCOUNT_PRICE if sold > DISCOUNT_AMOUNT else sold * PRICE
label = f'Total: {total}'

Extracting functions

More commonly, code is duplicated in different parts of a function, or in two different functions.

Here you will need to extract out the code into another function, give it a meaningful name, and then call it instead of the extracted code.

class Game:

    # ...

    def was_correctly_answered(self):
        if self.not_penalised():
            print('Answer was correct!!!!')
            self.purses[self.current_player] += 1
            print(self.players[self.current_player] +
                ' now has ' +
                str(self.purses[self.current_player]) +
                ' Gold Coins.')

            winner = self._did_player_win()
            self.current_player += 1
            if self.current_player == len(self.players):
                self.current_player = 0

            return winner
        else:
            self.current_player += 1
            if self.current_player == len(self.players):
                self.current_player = 0
            return True


    def wrong_answer(self):
        print('Question was incorrectly answered')
        print(self.players[self.current_player] + " was sent to the penalty box")
        self.in_penalty_box[self.current_player] = True

        self.current_player += 1
        if self.current_player == len(self.players):
            self.current_player = 0
        return True

This code is taken (slightly modified) from the trivia kata, published under a GPLv3 license. Looking over it one clear piece of duplicated code is the following, which pops up in three places:

self.current_player += 1
if self.current_player == len(self.players):
    self.current_player = 0

This should be extracted into a function. Many IDEs will let you choose snippets of code and extract them automatically, and some such as PyCharm will also scan your code to see which parts can be replaced by a call to the new function. Let's extract this method, and call it next_player, since it moves the current_player on to the next valid value.

class Game:

    # ...

    def was_correctly_answered(self):
        if self.not_penalised():
            print('Answer was correct!!!!')
            self.purses[self.current_player] += 1
            print(self.players[self.current_player] +
                ' now has ' +
                str(self.purses[self.current_player]) +
                ' Gold Coins.')

            winner = self._did_player_win()
            self.next_player()

            return winner
        else:
            self.next_player()
            return True


    def wrong_answer(self):
        print('Question was incorrectly answered')
        print(self.players[self.current_player] + " was sent to the penalty box")
        self.in_penalty_box[self.current_player] = True

        self.next_player()
        return True


    def next_player(self):
        self.current_player += 1
        if self.current_player == len(self.players):
            self.current_player = 0

There's a lot more refactoring to be done on this code, but it has definitely improved. It reads more clearly, and if the next_player() function has to change we only need to alter the code in one place instead of three.

As you do more coding and refactoring you will learn to see more and more opportunities to remove duplication. Often you will need to massage multiple pieces of code to make them more similar, then extract out the common elements into functions.

4. Split up large functions

The longer a function is, the harder it is to read and understand. An important aspect of writing good functions is that they should do one thing, and have a name that reflects exactly what they do - a maxim known as the Single Responsibility Principle. Longer functions are more prone to doing lots of different things.

Opinions vary on how long they should be, but in general the shorter the better. A cast-iron rule is that it should definitely fit on one page of your code editor. Having to scroll up and down to see what a function does adds a lot to the cognitive load needed to understand it.

In order to split up a function you will need to extract parts of it into other functions, as described in the previous section.

Let's take a look at some code to get an idea of how to proceed:

def make_tea(kettle, tap, teapot, tea_bag, cup, milk, sugar):
    kettle.fill(tap)
    kettle.switch_on()
    kettle.wait_until_boiling()
    boiled_kettle = kettle.pick_up()
    teapot.add(tea_bag)
    teapot.add(boiled_kettle.pour())
    teapot.wait_until_brewed()
    full_teapot = teapot.pick_up()
    cup.add(full_teapot.pour())
    cup.add(milk)
    cup.add(sugar)
    cup.stir()
    return cup

Here we can see that the calls split into three groups - those that call methods on the kettle, the teapot and then the cup. From our domain knowledge (being British may help here) these also correspond to the three stages of making a cup of tea - boiling the kettle, brewing the tea and then pouring it into a cup and serving it.

Let's start at the end, and extract out the lines that relate to pouring out a cup of tea and serving it. You can do this manually or by using the 'Extract Method' functionality of your IDE.

def make_tea(kettle, tap, teapot, tea_bag, cup, milk, sugar):
    kettle.fill(tap)
    kettle.switch_on()
    kettle.wait_until_boiling()
    boiled_kettle = kettle.pick_up()
    teapot.add(tea_bag)
    teapot.add(boiled_kettle.pour())
    teapot.wait_until_brewed()
    full_teapot = teapot.pick_up()
    return pour_tea(cup, full_teapot, milk, sugar)


def pour_tea(cup, full_teapot, milk, sugar):
    cup.add(full_teapot.pour())
    cup.add(milk)
    cup.add(sugar)
    cup.stir()
    return cup

At this stage we have a mixture of levels of abstraction in our make_tea function - switching on a kettle is a simpler operation than the multiple steps involved in our pour_tea function. Each line in our functions should ideally be at a similar level, making them easier to parse as a coherent narrative.
To achieve this let's go ahead and extract out the other two functions as well.

def make_tea(kettle, tap, teapot, tea_bag, cup, milk, sugar):
    boiled_kettle = boil_water(kettle, tap)
    full_teapot = brew_tea(boiled_kettle, tea_bag, teapot)
    return pour_tea(cup, full_teapot, milk, sugar)


def boil_water(kettle, tap):
    kettle.fill(tap)
    kettle.switch_on()
    kettle.wait_until_boiling()
    return kettle.pick_up()


def brew_tea(boiled_kettle, tea_bag, teapot):
    teapot.add(tea_bag)
    teapot.add(boiled_kettle.pour())
    teapot.wait_until_brewed()
    return teapot.pick_up()


def pour_tea(cup, full_teapot, milk, sugar):
    cup.add(full_teapot.pour())
    cup.add(milk)
    cup.add(sugar)
    cup.stir()
    return cup

Looking at the top-level make_tea function you can now read the three top-level stages of making tea like a little story. If you are interested in the details of any stage you can drill down to the relevant method.

When splitting up a function it's best to identify logically consistent bits of code that hang together and do one thing. A rule of thumb is that if you can think of a good domain-relevant name for the new function then extracting it is probably a good idea.

5. Move local variable declarations close to their usages

When reading and understanding code you have to keep the variables you encounter and their state in your short-term memory. Most adults can store around 7 things at once.

It's therefore quite important to ensure that you have to deal with as few variables as possible when reading any particular piece of code. Keeping variable declarations close and in scope with their usage helps with this. Basically the closer a variable is declared to its usage, the less scanning up and down you have to do when reading the code later.

def assess_fruit(self, fruit):
    happiness = 0
    hunger = time_since_breakfast() / size_of_breakfast()
    some_other_code()
    # work out some things
    do_some_other_things()
    if is_snack_time() and isinstance(fruit, Apple):
        yumminess = fruit.size * fruit.ripeness ** 2
        happiness += hunger * yumminess
    return happiness

When reading this code you have to keep the hunger variable in mind from the start to the end of the function, since it could be used or altered anywhere.

def assess_fruit(self, fruit):
    happiness = 0
    some_other_code()
    # work out some things
    do_some_other_things()
    if is_snack_time() and isinstance(fruit, Apple):
        hunger = time_since_breakfast() / size_of_breakfast()
        yumminess = fruit.size * fruit.ripeness ** 2
        happiness += hunger * yumminess
    return happiness

Refactoring to move it into the scope where it is used resolves this issue. Now we only have to think about hunger in the appropriate scope.

(Cover image by Chris Ried on Unsplash)

How to set up a perfect Python project

Nick Thapen — Wed, 31 Jul 2019 00:00:00 +0000

When starting a new Python project, it is tempting to just dive in and start coding. Spending a tiny amount of time to setup a project with the best tools will save immense time and lead to a happier coding experience.

In an ideal world, dependencies would be identical for all developers, code
would be perfectly formatted, common errors forbidden and everything would be
covered by tests. Additionally, all of these would ensured at each commit.

In this article I'll go through how to set up a project that does exactly that. You can either follow along with the steps or jump straight to generating a new project automatically by installing pipx and pipenv then generating a new project.

Let's create a new project directory:

mkdir best_practices
cd best_practices

Python command line tools with pipx

Pipx is a handy utility that allows quick installation of python command line tools. We'll be using it to install pipenv and cookiecutter.

python3 -m pip install --user pipx
python3 -m pipx ensurepath

Dependency management with pipenv

Pipenv automatically creates and manages a
virtualenv for your projects, as well as adding/removing packages from your
Pipfile as you install/uninstall packages. It also generates the ever-important
Pipfile.lock, which is used to produce deterministic builds.

Knowing that you and your teammates are using the same library versions is a huge confidence boost. Pipenv takes care of this and has gained massive traction in the last year or so.

pipx install pipenv

Code formatting with black and isort

Black formats our code:

Black is the uncompromising Python code formatter. By using it, you agree to cede control over minutiae of hand-formatting. In return, Black gives you speed, determinism, and freedom from pycodestyle nagging about formatting. You will save time and mental energy for more important matters.

Blackened code looks the same regardless of the project you're reading. Formatting becomes transparent after a while and you can focus on the content instead.

Black makes code review faster by producing the smallest diffs possible.

while isort sorts our imports:

isort your python imports for you so you don't have to.
isort is a Python utility / library to sort imports alphabetically, and automatically separated into sections.

Let's install them using pipenv as development dependencies so they don't clutter a deployment:

pipenv install black isort --dev

Black and isort have incompatible default options so we'll override isort to follow black's lead. Create a setup.cfg file and add this config:

[isort]
multi_line_output=3
include_trailing_comma=True
force_grid_wrap=0
use_parentheses=True
line_length=88

We can run these tools with:

pipenv run black
pipenv run isort

Style enforcement with flake8

Flake8 ensures our code follows the standard python conventions as defined in PEP8. Install using pipenv:

pipenv install flake8 --dev

Just like isort it needs a little configuration to work well with black. Add this config to setup.cfg:

[flake8]
ignore = E203, E266, E501, W503
max-line-length = 88
max-complexity = 18
select = B,C,E,F,W,T4

Now we can run flake8 with pipenv run flake8.

Static types with mypy

Mypy is an optional static type checker for Python that aims to combine the benefits of dynamic (or "duck") typing and static typing. Mypy combines the expressive power and convenience of Python with a powerful type system and compile-time type checking. Mypy type checks standard Python programs; run them using any Python VM with basically no runtime overhead.

Having types in Python takes a little getting used to but the benefits are substantial. From the website:

Static typing can make programs easier to understand and maintain
Static typing can help you find bugs earlier and with less testing and debugging
Static typing can help you find difficult-to-find bugs before your code goes into production

pipenv install mypy --dev

Mypy by default will recursively check all imports for type annotations which leads to errors when libraries do not include these annotations.
We need to configure mypy to run only on our code and to ignore any errors for imports without type annotations.
We are assuming that our code lives in the best_practices package for the following config. Add this to setup.cfg:

[mypy]
files=best_practices,test
ignore_missing_imports=true

Now we can run mypy with:

pipenv run mypy

Here's a useful cheat sheet for using it.

Testing with pytest and pytest-cov

Writing tests with pytest is incredibly easy and removing any friction to writing tests means we will write more of them!

pipenv install pytest pytest-cov --dev

Here's a simple example from the pytest website:

# content of test_sample.py
def inc(x):
    return x + 1


def test_answer():
    assert inc(3) == 5

To execute it:

$ pipenv run pytest
=========================== test session starts ============================
platform linux -- Python 3.x.y, pytest-5.x.y, py-1.x.y, pluggy-0.x.y
cachedir: $PYTHON_PREFIX/.pytest_cache
rootdir: $REGENDOC_TMPDIR
collected 1 item

test_sample.py F                                                     [100%]

================================= FAILURES =================================
_______________________________ test_answer ________________________________

    def test_answer():
>       assert inc(3) == 5
E       assert 4 == 5
E        +  where 4 = inc(3)

test_sample.py:6: AssertionError
========================= 1 failed in 0.12 seconds =========================

All our tests should go in the test directory so add this config to setup.cfg:

[tool:pytest]
testpaths=test

We also want to check how much of our code is covered by tests. Create a new file .coveragerc to only return coverage statistics for our application code, again we are assuming our application code lives in the best_practices module:

[run]
source = best_practices

[report]
exclude_lines =
    # Have to re-enable the standard pragma
    pragma: no cover

    # Don't complain about missing debug-only code:
    def __repr__
    if self\.debug

    # Don't complain if tests don't hit defensive assertion code:
    raise AssertionError
    raise NotImplementedError

    # Don't complain if non-runnable code isn't run:
    if 0:
    if __name__ == .__main__.:

We can now run our tests and report coverage with

pipenv run pytest --cov --cov-fail-under=100

This will fail if our test coverage of the application code is less than 100%.

Git hooks with pre-commit

Git hooks allow you to run scripts any time you want to commit or push. This lets us run all of our linting and tests automatically every time we commit/push. pre‑commit allows easy configuration of these hooks:

Git hook scripts are useful for identifying simple issues before submission to code review. We run our hooks on every commit to automatically point out issues in code such as missing semicolons, trailing whitespace, and debug statements. By pointing these issues out before code review, this allows a code reviewer to focus on the architecture of a change while not wasting time with trivial style nitpicks.

Here we configure all of the above tools to run on any changed python files on committing, and also to run pytest coverage only when pushing as it can be slow. Create a new file .pre-commit-config.yaml:

repos:
  - repo: local
    hooks:
      - id: isort
        name: isort
        stages: [commit]
        language: system
        entry: pipenv run isort
        types: [python]

      - id: black
        name: black
        stages: [commit]
        language: system
        entry: pipenv run black
        types: [python]

      - id: flake8
        name: flake8
        stages: [commit]
        language: system
        entry: pipenv run flake8
        types: [python]
        exclude: setup.py

      - id: mypy
        name: mypy
        stages: [commit]
        language: system
        entry: pipenv run mypy
        types: [python]
        pass_filenames: false

      - id: pytest
        name: pytest
        stages: [commit]
        language: system
        entry: pipenv run pytest
        types: [python]

      - id: pytest-cov
        name: pytest
        stages: [push]
        language: system
        entry: pipenv run pytest --cov --cov-fail-under=100
        types: [python]
        pass_filenames: false

If you ever need to skip these hooks you can run git commit --no-verify or git push --no-verify

Generate a project using cookiecutter

Now we've seen what an ideal project contains, we can turn this into a template to generate a new project with a single command:

pipx run cookiecutter gh:sourcery-ai/python-best-practices-cookiecutter

Fill in the project name and repo name and your project will be generated for you.

To finish setting up, follow these steps:

# Enter project directory
cd <repo_name>

# Initialise git repo
git init

# Install dependencies
pipenv install --dev

# Setup pre-commit and pre-push hooks
pipenv run pre-commit install -t pre-commit
pipenv run pre-commit install -t pre-push

The template project contains a very simple Python file and test to try the tools out on. Once you're happy with the code you can then do your first git commit and all the hooks will be run.

Editor Integration

While it is great to know that the standard of code on our project will always be maintained at the highest level when committing code, it is somewhat frustrating to find out any issues after we think the code changes have all been done. It is much better to get the issues displayed in real-time.

Spend some time to make sure these commands are run by your code editor when saving a file. Having instant feedback means you can quickly fixup any minor issues introduced while the code is fresh in the mind.

Personally I use some excellent Vim plugins to accomplish this:

ale runs flake8 in real-time, and runs black, isort and mypy on file save
vim-test with projectionist integration runs pytest on file save

DEV Community: Nick Thapen

Python Refactorings - Part 1

1. Merge nested if conditions

2. Hoist repeated code outside conditional statement

3. Replace yield inside for loop with yield from

4. Use any() instead of for loop

5. Replace list() with []

6. Hoist statements out of for/while loops

Conclusion

How do you test code written by code?

Static analysis

Automated Tests

Open source library testing

Conclusion

Use Sourcery to review your Python code on GitHub!

Sourcery now available in VS Code!

Getting the most out of Python collections

Create and filter collections with Comprehensions

List Comprehensions

Dictionary Comprehensions

Set Comprehensions

More complex comprehensions

Iterating with Generators

Useful types of collection from the collections module

defaultdict

Counter

Useful functions for collection handling

Using any() and all()

Iterating over collections together with zip()

Chaining over multiple collections

A Field Guide to Fixing Bugs

Reproduce the bug!

Now reproduce it as easily as possible

If it can't be reproduced yet - Log log log

Identify the code which is causing it

Check which recent changes might have introduced it.

Another pair of eyes.

Think logically

It's never the OS or hardware.

Get some sleep

Make the right fix, not the easy fix

Verify that you've fixed it and that it won't regress.

5 Refactoring Tips to Improve Your Code

1. Remove commented out code.

2. Extract magic numbers and strings

3. Remove duplication

Code Hoisting

Extracting functions

4. Split up large functions

5. Move local variable declarations close to their usages

How to set up a perfect Python project

Python command line tools with pipx

Dependency management with pipenv

Code formatting with black and isort

Style enforcement with flake8

Static types with mypy

Testing with pytest and pytest-cov

Git hooks with pre-commit

Generate a project using cookiecutter

Editor Integration

Useful types of collection from the `collections` module

`defaultdict`

`Counter`

Using `any()` and `all()`

Iterating over collections together with `zip()`