DEV Community: Sanket Saurav

Positional-only arguments in Python

Sanket Saurav — Tue, 27 Aug 2019 08:02:00 +0000

The ability to specify positional-only arguments using the / marker in function definitions is among the many new improvements to the Python language coming in the upcoming 3.8 release. This addition to syntax has performance benefits and enables better API design. Let's look at the motivation behind positional-only arguments and how to use it, with examples.

Background

Keyword-only arguments have been available in Python with the * marker, and addition of / marker for positional-only arguments improves the language’s consistency. With positional-or-keyword parameters, the mix of calling conventions is not always desirable. Consider these examples:

Some function parameters already have semantic meaning: namedtuple(typenames, field_names, …)
The parameter name has no true external meaning: arg1, arg2, …, etc for min()

If the users start using a keyword argument, the library author cannot rename the parameter because it would be a breaking change. In case of min(), the name of the parameter provides no intrinsic value and forces the author to maintain its name forever since callers might pass arguments as a keywords. This problem is solved by positional-only arguments. In addition, the parsing and handling of positional-only arguments is faster.

How to use positional-only arguments

To specify arguments as positional-only, a / marker should be added after all those arguments in the function definition. Take this example:

def pow(x, y, /, mod=None):
    r = x ** y
    if mod is not None:
        r %= mod
    return r

The following would apply:

All parameters to the left of / (in this case, x and y) can only be passed positionally.
mod can be passed positionally or with a keyword.

>>> pow(2, 10)  # valid
>>> pow(2, 10, 17)  # valid
>>> pow(2, 10, mod=17)  # valid
>>> pow(x=2, y=10)  # invalid, will raise a TypeError
>>> pow(2, y=10)  # invalid, will raise a TypeError

Another example:

def table_format(items, separator=',', /, prettify=False):
    pass

Once a positional-only parameter is specified with a default, the following positional-only and positional-or-keyword parameters need to have defaults as well (in this case, prettify).
Positional-only parameters which do not have default values are required positional-only parameters (in this case, items).

The generic case

A function definition, with all flavors of argument-passing variants, would look like this:

def f(pos1, pos2, /, pos_or_kwd, *, kwd1, kwd2):
      -----------    ----------     ----------
        |             |                  |
        |        Positional or keyword   |
        |                                - Keyword only
         -- Positional only

When to use positional-only arguments

Use positional-only if names do not matter or have no meaning, and there are only a few arguments which will always be passed in the same order.
Use keyword-only when names have meaning and the function definition is more understandable by being explicit with names.

Positional-only arguments were proposed in PEP 570, and it's worth taking a look at the doc to understand the feature in more detail.

Originally posted on DeepSource Blog. Join the discussion on reddit.

What's new in Python 3.8?

Sanket Saurav — Fri, 26 Jul 2019 08:05:04 +0000

The latest, greatest version of Python is going to be out in beta soon. While there's still some time before the final stable version is available, it is worth looking into all that's new. Python 3.8 adds some new syntax to the language, a few minor changes to existing behavior, and mostly a bunch of speed improvements — maintaining the tradition from the earlier 3.7 release.

This post outlines the most significant additions and changes you should know about Python 3.8. Take a look!

1. The walrus operator

Assignment expressions have come to Python with the "walrus" operator :=. This will enable you to assign values to a variable as part of an expression. The major benefit of this is it saves you some lines of code when you want to use, say, the value of an expression in a subsequent condition.

So, something like this:

length = len(my_list)
if length > 10:
    print(f"List is too long ({length} elements, expected <= 10)")

Can now be written in short like this:

if (length := len(my_list)) > 10:
    print(f"List is too long ({length} elements, expected <= 10)")

Yay for brevity, but some might say this affects readability of code — it can be argued that the first variant here is clearer and explicit. This discussion was the center of a major controversy in the Python community.

2. Positional-only arguments

A special marker, /, can now be used when defining a method's arguments to specify that the functional only accepts positional arguments on the left of the marker. Keyword-only arguments have been available in Python with the * marker in functions, and addition of / marker for positional-only arguments improves the language's consistency and allows for a robust API design.

Take an example of this function:

def pow(x, y, z=None, /):
    r = x**y
    if z is not None:
        r %= z
    return r

The / marker here means that passing values for x, y and z can only be done positionally, and not using keyword arguments. The behavior is illustrated below:

>>> pow(2, 10)  # valid
>>> pow(2, 10, 17)  # valid
>>> pow(x=2, y=10)  # invalid, will raise a TypeError
>>> pow(2, 10, z=17)  # invalid, will raise a TypeError

A more detailed explanation on the motivation and use-cases can be found in PEP 570.

3. f-strings now support "="

Python programmers often use "printf-style" debugging. In the old days this was pretty verbose:

print "foo=", foo, "bar=", bar

f-strings make this a bit nicer:

print(f"foo={foo} bar={bar}")

But you still have to repeat yourself: you have to write out the string "foo", and then the expession "foo".

The = specifier, used as f'{expr=}' expands to the text of the expression, an equal sign, then the repr of the evaluated expression. So now, you can simply write:

print(f"{foo=} {bar=}")

A small step for the language, but a giant leap for everyone who sprinkles print() statements for debugging!

4. `reversed()` now works with `dict`

Since Python 3.7, dictionaries preserve the order of insertion of keys. The reversed() built-in can now be used to access the dictionary in the reverse order of insertion — just like OrderedDict.

>>> my_dict = dict(a=1, b=2)
>>> list(reversed(my_dict))
['b', 'a']
>>> list(reversed(my_dict.items()))
[('b', 2), ('a', 1)]

5. Simplified iterable unpacking for `return` and `yield`

This unintentional behavior has existed since Python 3.2 which disallowed unpacking iterables without parentheses in return and yield statements.

So, the following was allowed:

def foo():
    rest = (4, 5, 6)
    t = 1, 2, 3, *rest
    return t

But these resulted in a SyntaxError:

def baz():
    rest = (4, 5, 6)
    return 1, 2, 3, *rest

def baz():
    rest = (4, 5, 6)
    yield 1, 2, 3, *rest

The latest release fixes this behavior, so doing the above two approaches are now allowed.

6. New syntax warnings

The Python interpreter now throws a SyntaxWarning in some cases when a comma is missed before tuple or list. So when you accidentally do this:

data = [
    (1, 2, 3)  # oops, missing comma!
    (4, 5, 6)
]

Instead of showing TypeError: 'tuple' object is not callable which doesn't really tell you what's wrong, a helpful warning will be shown pointing out that you probably missed a comma. Pretty helpful while debugging!

The compiler now also produces a SyntaxWarning when identity checks (is and is not) are used with certain types of literals (e.g. strings, integers, etc.). You rarely want to compare identities with literals other than None, and a compiler warning can help avoid a number of elusive bugs.

7. Performance improvements

This release adds a number of performance speed-ups to the interpreter, following suit from the previous 3.7 release.

operator.itemgetter() is now 33% faster. This was made possible by optimizing argument handling and adding a fast path for the common case of a single non-negative integer index into a tuple (which is the typical use case in the standard library).
Field lookups in collections.namedtuple() are now more than two times faster, making them the fastest form of instance variable lookup in Python.
The list constructor does not over-allocate the internal item buffer if the input iterable has a known length (the input implements len). This makes the created list 12% smaller on average.
Class variable writes are now twice as fast: when a non-dunder attribute was updated, there was an unnecessary call to update slots, which is optimized.
Invocation of some simple built-ins and methods are now 20-50% faster. The overhead of converting arguments to these methods is reduced.
uuid.UUID now uses slots to reduce it's memory footprint.

Summary

The upcoming release of Python adds some great new features to the language and significantly improves the performance with fundamental speed-up fixes. There is a small number behavior changes that might require modifying existing code while upgrading to Python 3.8, but the performance gains and new syntax make it totally worth the effort. A detailed change log of all that's new can be found here.

Originally posted on DeepSource Blog.

5 common beginner mistakes in Python

Sanket Saurav — Tue, 23 Jul 2019 09:18:02 +0000

Python, being a high-level and dynamic programming language, is famously easy to use. This has contributed to it being wildly popular and growing in recent years. The ease of use also comes with ease of misuse. We've put together a list of 5 common mistakes that beginners can do when writing code in Python.

1. Unnecessary lambda expression

Functions are first-class citizens in Python. This means you can assign it to some variable, pass it as an argument in another function invocation, and so on. This can be counter-intuitive for beginners, or developers coming from other programming languages.

A common example of this pattern is:

def request(self, method, **kwargs):
    # ...
    if method not in ("get", "post"):
        req.get_method = lambda: method.upper()
    # ...

The recommended way to write this is:

def request(self, method, **kwargs):
    # ...
    if method not in ("get", "post"):
        req.get_method = method.upper
    # ...

(example)

2. Raising `NotImplemented`

This is one of those examples where similar naming can confuse developers. NotImplementedError is an exception class that should be raised when derived classes are required to override a method. NotImplemented is a constant which is used to implement binary operators. When you raise NotImplemented, a TypeError will be raised.

Incorrect:

class SitesManager(object):
    def get_image_tracking_code(self):
        raise NotImplemented

Correct:

class SitesManager(object):
    def get_image_tracking_code(self):
        raise NotImplementedError

(examples)

3. Mutable default argument

Default arguments in Python are evaluated once when the function definition is executed. This means that the expression is evaluated only once when the function is defined and that the same value is used for each subsequent call. So if you are modifying the mutable default argument — a list, dict, etc., it will be modified for all future calls.

Incorrect:

def add_item(new_item, items=[]):
    items.append(new_item)

Correct:

def add_item(new_item, items=None):
    if items is None:
        items = []
    items.append(new_item)

(example)

4. Using `assert` statement as guarding condition

Since assert provides an easy way to check some condition and fail execution, it's very common for developers to use it to check validity. But when the Python interpreter is invoked with the -O (optimize) flag, the assert statements are removed from the bytecode. So, if assert statements are used for user-facing validation in production code, the block won't be executed at all — potentially opening up a security vulnerability. It is recommended to use assert statements only in tests.

Incorrect:

assert re.match(VALID_ADDRESS_REGEXP, email) is not None

Correct:

if not re.match(VALID_ADDRESS_REGEXP, email):
    raise AssertionError

(example)

5. Using `isinstance` in place of `type`

Either type or isinstance can be used to check the type of an object in Python. But there is an important difference — isinstance takes care of inheritance while resolving the type of object, while type does not. So sometimes using isinstance might not be what you want to use. Take a look at the following example:

def which_number_type(num):
    if isinstance(num, int):
        print('Integer')
    else:
        raise TypeError('Not an integer')

which_number(False)  # prints 'Integer', which is incorrect

Here, Since bool is a subclass of int in Python, isinstance(num, int) is also evaluated as True, which is not the expected behavior. In this particular case, using type is the correct way. Read more about the difference in the behavior of these two methods here.

Originally published on DeepSource Blog. Ready to free up your code base of these anti-patterns? DeepSource flags all of these, and many others. Sign up today.

Effective code reviews: a primer

Sanket Saurav — Fri, 05 Jul 2019 22:04:52 +0000

Peer code reviews as a process have increasingly been adopted by engineering teams around the world. And for good reason — code reviews have been proven to improve software quality and save developers' time in the long run. A lot has been written about how code reviews help engineering teams by leading software engineering practitioners. My favorite is this quote by Karl Wiegers, author of the seminal paper on this topic, Humanizing Peer Reviews:

Peer review – an activity in which people other than the author of a software deliverable examine it for defects and improvement opportunities – is one of the most powerful software quality tools available. Peer review methods include inspections, walkthroughs, peer deskchecks, and other similar activities. After experiencing the benefits of peer reviews for nearly fifteen years, I would never work in a team that did not perform them.

It is worth the time and effort to put together a code review strategy and consistently follow it in the team. In essence, this has a two-pronged benefit: more pair of eyes looking at the code decreases the chances of bugs and bad design patterns entering your code base, and embracing the process fosters knowledge sharing and positive collaboration culture in the team.

Here are 6 tips to ensure effective peer reviews in your team.

1. Keep the changes small and focused

Code reviews require developers to look at someone else's code, most of which is completely new most of the times. Too many lines of code to review at once requires a huge amount of cognitive effort, and the quality of review diminishes as the size of changes increases. While there's no golden number of LOCs, it is recommended to create small pull-requests which can be managed easily. If there are a lot of changes going in a release, it is better to chunk it down into a number of small pull-requests.

2. Ensure logical coherence of changes

Code reviews are the most effective when the changes are focused and have logical coherence. When doing refactoring, refrain from making behavioral changes. Similarly, behavioral changes should not include refactoring and style violation fixes. Following this convention prevents unintended changes creeping in unnoticed in the code base.

3. Have automated tests, and track coverage

Automated tests of your preferred flavor — units, integration tests, end-to-end tests, etc. help automatically ensure correctness. Consistently ensuring that changes proposed are covered by some kind of automated frees up time for more qualitative review; allowing for a more insightful and in-depth conversation on deeper issues.

4. Self-review changes before submitting for peer review

A change can implement a new feature or fix an existing issue. It is recommended that the requester submits only those changes that are complete, and tested for correctness manually. Before creating the pull-request, a quick glance on what changes are being proposed helps ensure that no extraneous files are added in the changeset. This saves tons of time for the reviewers.

5. Automate what can be automated

Human review time is expensive, and the best use of a developer's time is reviewing qualitative aspects of code — logic, design patterns, software architecture, and so on. Linting tools can help automatically take care of style and formatting conventions. Tools like DeepSource can help catch potential bugs, anti-patterns and security issues which can be fixed by the developer before they make a change request. Most of these tools integrate well with code hosting platforms as well.

6. Be positive, polite and respectful

Finally, be cognizant of the fact that people on both sides of the review are but human. Offer positive feedback, and accept criticism humbly. Instead of beating oneself upon the literal meaning of words, it really pays off to look at reviews as people trying to achieve what's best for the team, albeit in possibly different ways. Being cognizant of this aspect can save a lot of resentment and unmitigated negativity.

References

Originally posted on DeepSource Blog.

DEV Community: Sanket Saurav

Positional-only arguments in Python

Background

How to use positional-only arguments

The generic case

When to use positional-only arguments

What's new in Python 3.8?

1. The walrus operator

2. Positional-only arguments

3. f-strings now support "="

4. reversed() now works with dict

5. Simplified iterable unpacking for return and yield

6. New syntax warnings

7. Performance improvements

Summary

5 common beginner mistakes in Python

1. Unnecessary lambda expression

2. Raising NotImplemented

3. Mutable default argument

4. Using assert statement as guarding condition

5. Using isinstance in place of type

Effective code reviews: a primer

1. Keep the changes small and focused

2. Ensure logical coherence of changes

3. Have automated tests, and track coverage

4. Self-review changes before submitting for peer review

5. Automate what can be automated

6. Be positive, polite and respectful

References

4. `reversed()` now works with `dict`

5. Simplified iterable unpacking for `return` and `yield`

2. Raising `NotImplemented`

4. Using `assert` statement as guarding condition

5. Using `isinstance` in place of `type`