DEV Community: idoazzz

Python's Type Annotations: The Best Of Both Types Worlds

idoazzz — Sat, 16 Oct 2021 09:24:46 +0000

With Great Power Comes Great Responsibility

Python is a pretty powerful language with a lot of controversial features. One of them is Typing System, I'm one of the groupies so today my mission will be to convince you it's at least not bad as people claim.
Today, we'll talk about type systems and where Python fits in.
Also, you'll see a way to reduce unwanted bugs and boost your codebase with the Type Annotations feature that was introduced in Python 3.5 (PEP484).

PEP stands for Python Enhancement Proposals, all proposals and specifications of Python's features. It's like ECMA's documents for Javascript.

This feature truly makes Python a better dynamic-typed language.

Less Runtime Errors = Efficiency

How many times have you met the following error:

AttributeError: 'X' object has no attribute 'Y'

This is one of the most annoying and time-wasting bugs that Python introduce. There are more errors that occur because of non-safety with variable types and non-existing type validations (or tests in general).
As a dynamic-typed language, all the type errors will appear in runtime.

Try to recall how frustrating it meeting this type of exception in production.
For those of you who know Javascript, you can see the huge hype over Typescript in the last few years. Why is that? Fewer bugs!
Nowadays people understand that bugs in production are way worst from adding types in their code.

But wait, let's go back and explain the fundamentals about type systems, and why I say stuff.

Type Systems and Airport Security

You arrived at the airport for taking your flight to the annual vacation.
In the entry of the airport a security guard approaches to you and ask:

Big Security Guy: "Let's say you have a choice between 2 options.
(1) Passing a security check which includes long lines and cursings.
(2) you go freely into your airplane and during the flight, you'll be checked.
What would you prefer?"

This story looks exactly like the war between dynamic and static languages.

Static-Typed Languages

Your code types will be checked in the compilation process.
Those types of languages main pros:

Types Safety Detecting programming mistakes and redundant bugs before runtime.
Documentation Better documentation for the code.

If you know about a bomb in your airport, you rather choose this choice (1).

e.g: Java, C, C++

Dynamic-Typed Languages

Your code types will be checked only during runtime.
Those types of languages main pros:

Readable Tend to be more readable and natural. Living without repeated types everywhere.
Flexibility Development flexibility with dealing with unpredictability systems.
Instantaneous Fast development cycles because of the instant launching of your code (lack of validation, even 10 seconds validation will accumulate enormous frustration).

The problem will be when there is a bomb on your flight.
For more safe flight (/run) your codebase must have tests and validations that will prevent issues in runtime.

e.g: Python, Javascript, R, Julia

Python's Place
Python is a pure dynamic-typed language.
Our goal will be to improve the downsides of dynamic-typed languages.

Main Problem: More Tests

We can agree that both types of languages codebases should have tests, but we live in an imperfect world, we don't have always the time to perform that tests, especially in large codebases in which high coverage is hard to target.
Dynamic languages, as we said, are less safe in their essence.
There are many cases that types are difficult to track. A common example is that it very hard tracking types of arguments in deep nested function calls.
So to revealing the bugs that occur because of programmer's mistakes, tests should be written.

How can we increase our safety without making any efforts and spending more time?

Type Annotations

The Best Of Both Types Worlds

Type Annotations are simply type-hints that can be attached to variables and functions declarations.
They can be used by third-party tools such as type checkers, IDEs, and linters for real-time warnings while coding.

The Python runtime does not enforce function and variable type annotations, so we didn't actually make Python more static.
Also, Python's interpreter will relate the Type Annotations as comments, so no effect on runtime.
We got statically-typed language experience with the dynamic-typed language.

Let's introduce some central features of Type Annotations.

Availiable Types

Builtin Types
It is possible hinting each one of the builtin types: int, str, float, bool, bytes, object, None.

Special Types

Notice that in Python3.9+ most of the following types are changed and won't be imported anymore from typing module anymore, e.g, List -> list.

Any Any type is possible.
List[str] List of str objects.
Tuple[int, int] Tuple of two ints.
Tuple[int, ...] Tuple of int objects.
Dict[int, int] Dict with int keys to int values.
Iterable[int] Iterable which contains int objects.
Sequence[bool] Sequence of booleans.

By the way: Sequence is an Iterable with defined length and extra functionality.

Basic Example

def greeting(name: str) -> str:
    return 'Hello ' + name

If we will run the following code greeting(3) we will get the following error:

error: Argument 1 to "greeting" has incompatible type "int"; expected "str"

Type Aliases

It is possible to define aliases to types for more readable code.

Url = str

def retry(url: Url, retry_count: int) -> None:
    pass

Generics

TypeVar is a factory that can generate parameterized types. It can be very useful in classes that can hold different data types.

from typing import TypeVar, Generic

T = TypeVar('T')

class Stack(Generic[T]):
    def __init__(self) -> None:
        # Create an empty list with items of type T
        self.items: List[T] = []

    def push(self, item: T) -> None:
        self.items.append(item)

    def pop(self) -> T:
        return self.items.pop()

    def empty(self) -> bool:
        return not self.items

# Construct an empty Stack[int] instance
stack = Stack[int]()
stack.push(2)
stack.pop()
stack.push('x')        # Type error

It is possible to create an object without specifying the type, the first object that will be inserted into the data structure will determine the class parameterized type (type inferring).

Flexibility

Sometimes you will accept several types for your variable.

Union
Union[T1,...,Tn]
Specifying a set of possible types, each one of them will be accepted.

def f(x: Union[int, str]) -> None:
    if isinstance(x, int):
        x + 1      # OK
    else:
        x + 'a'    # OK

f(1)    # OK
f('x')  # OK
f(1.1)  # Error

Optional
Optional[T]
Optional will indicates the variable holds a specified type or None.
It actually equals to Union[T, None].

def strlen(s: str) -> Optional[int]:
    if not s:
        return None  # OK
    return len(s)

For many usefull annotations I recommend exploring mypy docs.

Why Using Type Annotations?

IDE - Enforcing Like Compiler

Better than a compiler. You got your potential errors while developing!

Here few examples from Pycharm docs to understanding how awesome this thing is.
In the next example, we can see that we perform an invalid assignment.

Here we use the Final type to mark a variable as a constant. Assignment to this constant will lead to an error.

Here the IDE enforces Enum values for preventing any programmer mistakes, unified format.
Literal type makes it possible to supply specific primitive values for the variable.

Here we use TypedDict as a schema for function argument.

Those examples are a drop in the ocean, I hope it's clear now how the IDE enforces bugs and acting as a real-time compiler enforcer.

Refactoring Better

From my experience, refactoring with type validations can save a lot of time and unwanted bugs in production.
During refactoring, you usually change parts of code that can affect many other places. As we said, for catching unwanted errors in many places we need good tests coverage, without it, there is a good potential for programmer mistakes that won't be detected.
So, if your coverage is not good as you think, type annotations will help.

Type Annotations as Documentation

Docstrings are usually the solution for documentation in Python, there are many different formats such as Epytext, reST, Google (my favorite), and more.
There is no one standard strict format for docstrings, so it can be pretty difficult to enforce and check type problems in the codebase.

*Docstrings for humans, Type Annotations for linters. *
For reaching maximal experience of documentation, use type hints for enforcing any errors, and document only the non-trivial stuff.

Improve IDE Suggestions

As we know, sometimes our IDEs are pretty annoying. especially when you have int variable and string suggestions are made.
Type Annotations make suggestions feature more accurate and easier. Typing your variable with a specific type will autocomplete you with these specific type suggestions.

New Awesome Tools

Type Annotations are not only a way of enforcing types in your code. Like every feature in Python, it is possible to access the annotations of specific code.
PEP3107 (Function Annotations) specify how can we retrieve metadata from our code, using Type Annotations with __annotations__ property.

def foo(a: 'x', b: 5 + 6, c: list) -> max(2, 9):
    ...

> foo.__annotations__
{'a': 'x',
 'b': 11,
 'c': list,
 'return': 9}

Why is it good?
There are several powerful usages for this feature, a few of them are Database query mapping, Foreign-language bridges, RPC parameters encoding, and Schemas validations.
A really good example of this feature usage is FastAPI.
FastAPI is a fast and modern web framework based on Python Type Annotations. Why is that? Why shouldn't we use Flask and Django and forget about those types? Value.
Additionally, to speed, FastAPI gives few powerful features based on type hints, for example:

OpenAPI Generation
Whenever you run your server an OpenAPI (Swagger) specification file will be generated automatically, you don't have to document your endpoints anymore!

Schemas and Data Models
Another one is data models, Using the module Pydantic, with the simple specifications of your REST requests and arguments, validation and conversion will be performed so you can hold your objects without messing around with Jsons.

class Item(BaseModel):
    name: str
    description: Optional[str] = None
    price: float
    tax: Optional[float] = None


app = FastAPI()


@app.post("/items/")
async def create_item(item: Item):
    return item

This is also a drop in the ocean. As a Backend Engineer who worked with few web frameworks, I can surely say that these features are truly amazing and powerful.

MyPy

MyPy is a static type check for Python 2 and 3.
We can treat MyPy as our compiler which does not output any executable, only checks and find our bugs in the input codebase.
MyPy can type check your code and prevent bugs, it acts like a linter that runs as a static analyzer.
It is possible to create a configuration file for a more optimized and customized experience.

Example

# main.py
def print_hi(name: str) -> None:  
    print(f'Hi, {name}')  

 > mypy main.py
 > main.py:4: error: Argument 1 to "print_hi" has incompatible type "int"; expected "str"

MyPy and IDE that support Type Annotations are a powerful combination.
IDE for real-time warning and MyPy for final checkings (Possible in CI).

Few Words About Type Annotation Cons

Recalling: My goal is to convince the haters that those type annotations are not bad as they claim.
Type Annotations like every controversial feature have few drawbacks.

I'll talk about the most popular.

Adding Types to Large Codebases Without Any Types Hints

Adapting Type Annotations with the existing codebase is a challenging topic, small repos are not such an issue because adding types will be pretty fast. Large codebases cannot add Type Annotations quickly, it can be challenging.
You can try those two approaches combined for achieving types codebase:

There are few tools that are meant to solve this issue or at least help. MonkeyType and PyAnnotate tools can infer the types of your code during runtime.
It is recommended to add type hints in a graded manner, pick a subset of your code and run MyPy on this subset. Fix the errors or ignore with # type: ignore and move on. Subset by subset, Part by part. There is a big advantage having a big codebase with Type Annotations, refactoring will be easier and bugs can be detected fast.

It takes more time to write the types during development

Well-documented code will have types and documentation already, so what is the cost of moving the types declaration into type annotation?
If you don't document your code, still, insert a word per variable or method declaration doesn't seem very time-consuming.

Reducing Readability

Agree. In the end, it's a matter of taste, I met people that claim that type annotation contributes to the readability of their code.
Like everything it's a cost versus benefits game. In the beginning, it can be uglier than working without any annotation but I really think that after a while you are getting used to it.

Summary

As PEP484 said:

Python will remain a dynamically typed language, and the authors have no desire to ever make type hints mandatory, even by convention.

To conclude, type annotation can contribute a lot to your codebases.
Python rules.

Become a Better Engineer with Spaced Repetition

idoazzz — Wed, 22 Sep 2021 10:00:27 +0000

Improving Experience by Improving Memory

A few years ago I thought about what makes a person a good engineer?
Think about it, it's interesting to question.
Your main answer will probably be experience or maybe motivation.
Motivation is an interesting topic, but today I will focus on the experience.

If a good engineer must have a good experience, how can we boost our experience?

Can we improve the experience?

Think about all the problems/solutions/challenges/subjects/bugs that you faced with in the past, how much of this experience is lost during the years, and more important how can we save those experiences? Insert those experiences into the long-term memory.

Think about an interesting article that you read during the implementation of an arbitrary issue that you had. The chances are that few years from this moment you won't know that this article even exists!

I met an engineer called X (Not the real name) with perfect memory.
Everything he sees hears, and do will be remembered for a long long long time. From my point of view, it's experience worth a lot.
He was the best engineer I saw.

What can make us act like X?

As a person that read a few memory books, I found it very hard using the traditional memory tricks with coding or software subjects.

What doctor will do?

Medicine is my ideal test case! Those doctors must remember their experiences, or someone will die.
So I wondered which technique I can use for improving my experience.

After a while, I found on Youtube a few videos of MD students that use a technique called Spaced Repetition. They use this technique for remembering subjects they studied for a long time.

This method is very simple, They summarize complex subjects with few simple points, and test themselves about those points once a while, depend on how well they remember the subject.

Spaced Repetition

Spaced repetition is an evidence-based learning technique that is usually performed with flashcards.
Newly introduced and more difficult flashcards are shown more frequently, while older and less difficult flashcards are shown less frequently in order to exploit the psychological spacing effect.

Researches showed that everyone has a forgetting curve which means everybody will forget subjects that they read on someday.
Using Recalling we can reset the forgetting curve and have those memories for more time.

Intuitive Example

Let's say you read about a subject. If you will remind yourself of the key points of this subject in the week, two weeks, month, 2 months, 4 months - Trust me you'll remember this topic for a long time compared to the scenario that not contains many repetitions.
The things we remember the best are the things we saw frequently.
Researches found that recalled subject will remain in your memory for significantly more time.

This graph describes the process at a high level.

Recall is the process of reminding ourselves of a subject. With Recall we can move things from short-term memory to long-term memory easily, all you gotta do is re-read stuff!

Re-reading can be very exhausting. Personally, I hate read something again.

Spaced Repetition with Remnote

So we understand that re-read stuff is not feasible, so I search for a platform that can help me perform the Recall process and store all the knowledge that I want to store without reading again my points.
I'm using in the last year a tool called Remnote for:

Breaking down complex topics into simple intuitive bullets.
Recall those bullets with Spaced Repetition. After you broke the complex subject into key points, you can define which points will be recalled on the future. Those points will transform to Flashcards!

Flashcards Queue

Remnote have a Flashcards Queue with all the flashcards that you stored.
When you enter the queue Remnote will show you the points you summarized.
Part of the text will be invisible (it depends on how you defined your REM, won't be discussed in this article).
After you recalled the question/text you will specify how well you remembered this subject.

Here you can see a question about Django framework.
After I revealed the answer Remnote will ask me how good I was remembering that answer.
According to my feedback, it will ask me the same questions with the new compatible interval.

As you see it is possible to insert Code, Images, and more media types into a flashcard.

Real Life Scenario

Reading about new Python feature.
Deciding to remember this feature.
Summarize with few points the feature in Remnote.
Remnote will remind me of the subject once in a while ("Magic Intervals") when practicing Queue.

Self Discipline

Remnote requires self-discipline. Effective usage will require getting into Remnote queue every day and practice few flashcards (as much as you can).

Make this task a habit.
I for example enter each day in the morning for 20 cards.
The intervals that Remnote will remind you subjects depend on your answers in the flashcards, if it is difficult for you remembering a complex subject the flashcard will appear more frequently if it is easy to remember will be appeared less frequently - as I called the "Magic Interval".
It's important to say that there are many types of Spaced Repetition algorithms that support different intervals approaches.

Knowledge DB

Think about it, it's crazy! You have documentation of all the interesting stuff you read on. It's power and I'm sure it will help you in the future.

Summary

This article mentioned very few features that Remnote has. Please read more on Remnote homepage and watch Youtube videos of this concept.

For me, it helped a lot making my experience more valuable.
Today, I summarize every interesting subject that I met.
Read more about this concept, it will help you.