DEV Community: Itai Steinherz

Uncommon Python Type Hints for Special Cases

Itai Steinherz — Fri, 25 Aug 2023 11:37:52 +0000

Introduction

Python is great at being a dynamically and strongly-typed programming language. However, once you get enough mileage working with large Python codebases, you’ll probably realize that dyanmic types can be confusing, cause bugs, and result in unexpected behavior. The folks maintaining Python realized this as well, and decided to implement a built-in way to add static type annotations in Python (see PEP 484).

7 years have passed since then, and type hinting in Python has matured and evolved, with new features being implemented in every new Python version. In this post, I wanted to go over some of the more advanced and newly added features, with the hopes that you will find them useful and integrate them into your codebase.

A basic knowldge of type hinting in Python is assumed. If you need a refresher, I recommend this guide from Real Python.

Parameter specification variables

Let’s imagine you’d like to write a simple decorator, which will print a message every time a method is called:

from typing import Callable, TypeVar

R = TypeVar('R')

def log(func: Callable[..., R]) -> Callable[..., R]:
    def wrapper(*args, **kwargs):
        print('Method has been called!')
        return func(*args, **kwargs)

    return wrapper

@log
def add(a: int, b: int) -> int:
    return a + b

In case you aren’t familiar with TypeVar, a type variable lets you refer to the same type in multiple places without specifying the exact type.

In the code snippet above, a type variable was used in the log method type signature to annotate that the decorated method’s return type is the same as that of func.

The following call to add is invalid, and one would expect type checkers to report it as an error, however it will only fail at runtime:

add(1, 'a') # Fails at runtime

This is because the the decorated add has the typing Callable[..., int] (with ... meaning arguments aren’t validated). PEP 612 addresses this issue by introducing ParamSpec. ParamSpec variables are a new type of type variables, used for defining the dependencies between different callables (such as decorators and the decorated method).

Using ParamSpec, we can modify the example above to enforce the parameters types of the decorated method, while maintaining the decorator’s flexibility:

from typing import Callable, ParamSpec, TypeVar
P = ParamSpec('P')R = TypeVar('R')

def log(func: Callable[P, R]) -> Callable[P, R]:    def wrapper(*args: P.args, **kwargs: P.kwargs): print('Method has been called!')
        return func(*args, **kwargs)

    return wrapper

@log
def add(a: int, b: int) -> int:
    return a + b

The following will now be reported as an error by the type checker:

add(1, 'a') # Rejected by the type checker

This feature was shipped in Python 3.10.

User-defined type guards

Let’s assume you’re writing a utility method, validating that an object is of a desired type (used for type narrrowing):

from typing import Union

RealNumber = Union[int, float]

def is_real_number(value: object) -> bool:
    return isinstance(value, int) or isinstance(value, float)

def print_value_type(value: object):
    if is_real_number(value):
        assert value == value.real # Error: invalid type

The above code is valid, however static type checkers will report an error. This is because a type checker doesn’t have enough information to verify that value is a number. Using TypeGuard, introduced in PEP 647, we can enable type narrowing by changing the return type hint of is_real_numeber to TypeGuard[RealNumber]. This will signal to type checkers that if the method returns True, value is of type RealNumber (and if it returns False, it isn’t). Now, type checkers won’t report any errors:

from typing import Union, TypeGuard
RealNumber = Union[int, float]

def is_real_number(value: object) -> TypeGuard[RealNumber]: return isinstance(value, int) or isinstance(value, float)

def print_value_type(value: object):
    if is_real_number(value):
        assert value == value.real

This feature was shipped in Python 3.10.

Type hints for dictionaries with a fixed set of keys

Let’s assume we’d like to describe a person using a dictionary:

person = {
    'name': 'Bob',
    'age': 32
}

The dictionary’s keys should be strings, but we don’t want to constrain the types its values. The type hint Dict[str, Any] (or dict[str, Any] in Python 3.9 and later - see GenericAlias). This is better than nothing, but doesn’t really enforce our requirements for what the person dictionary should look like.

Using TypedDict, introduced in PEP 589, we can now enforce the specific key name and value types in the dictionary:

from typing import TypedDict

class Person(TypedDict):
    name: str
    age: int

Now, a type checker will recognize and enforce the typing of the values (and keys) of Person dictionaries:

person: Person = {
    'name': 'Bob',
    'age': 32
}
# Or even:
person = Person(name='Bob', age=32)

This feature was shipped in Python 3.8.

Self type hint

Let’s assume we’re developing a program which keeps track of blockchain transactions. A transaction might be represented using this simple interface:

from abc import ABC
from typing import List

class Transaction(ABC):
    def __init__ (self, parents: List["Transaction"]):
        self.parents = parents

For Bitcoin transactions, for example, we could implement BitcoinTransaction:

class BitcoinTransaction(Transaction):
    def __init__ (self, sender: str, recipient: str, parents: List["Transaction"]):
        super(). __init__ (parents)
        self.sender = sender
        self.recipient = recipient

Usage of BitcoinTransaction would look like so:

parent = BitcoinTransaction(sender="Alice", recipient="Bob", parents=[])
transaction = BitcoinTransaction(sender="Bob", recipient="Alice", parents=[parent])
# Rejected by the type checker:
print(f"The parent sender is {transaction.parents[0].sender}")

We received a type checking error because as far as the type checker is concerned, the type of transaction.parents[0] is Transaction, not BitcoinTransaction. To fix this, we can use Self:

from abc import ABC
from typing import List, Self

class Transaction(ABC):
    def __init__ (self, parents: List[Self]):
        self.parents = parents

class BitcoinTransaction(Transaction):
    def __init__ (self, sender: str, recipient: str, parents: List[Self]):
        super(). __init__ (parents)
        self.sender = sender
        self.recipient = recipient

parent = BitcoinTransaction(sender="Alice", recipient="Bob", parents=[])
transaction = BitcoinTransaction(sender="Bob", recipient="Alice", parents=[parent])
print(f"The parent sender is {transaction.parents[0].sender}")

This feature was shipped in Python 3.11.

Conclusion

The type hinting features we’ve explored should enable you to apply type checking to new categories of Python code, which previously couldn’t benefit from automated type checking. I hope this post will inspire you to write code that is more declarative, documented and reliable.

Making Meaningful Hacktoberfest Contributions

Itai Steinherz — Fri, 09 Oct 2020 16:35:18 +0000

Motivation

Following core issues with Hacktoberfest that maintainers surfaced during the beginning of this year’s Hacktoberfest, DigitalOcean made Hacktoberfest 2020 opt-in. However, this only improves one aspect of the real problem in my opinion.

While maintainers are no longer forced to deal with swag-seeking random contributors, many people simply don’t know how to contribute to OSS in a way that’s meaningful and helpful. Because of this, I wanted to write a post on my own OSS contributions, and how you can make a positive impact on OSS, instead of a negative one.

What is Hacktoberfest

In case you’re unfamiliar with Hacktoberfest, Hacktoberfest is an online event happening worldwide every October, where contributors are awarded some swag (T-shirts and stickers) for making contributions to open-source software (OSS).

My contributions to OSS

I actually first started contributing to OSS through Hacktoberfest 2017, as I like free stuff and coding. My first contributions were small documentation improvements, bug fixes, and such. I liked interacting with nice developers from all over the world, and found OSS to be a nice way to explore cool projects.

After my first Hacktoberfest, I wanted to make more substantial contributions to OSS projects, and so I looked for mature projects where I could step up, commit to working on more complex tasks, and work with the maintainers to get my work merged.

What Hacktoberfest means for me

To me, Hacktoberfest isn’t just about quick typo fixes or adding superlatives to READMEs (though I must admit I did do these types of changes a few times). Rather, it’s about familiarizing the global developer community with OSS and how it works, encouraging developers to contribute to OSS, as well as shining a light on these projects, and helping their maintainers with the burden of maintaining software for free.

How you can make a positive impact on OSS

If you’re new to open-source, I’d recommend going over a first-timers guide such as this one by GitHub.

Done with that, maybe watched a few videos on the topic, and know what you want to do? I’d pay special attention to the following points:

Be proactive

OSS is already out there, waiting for you to become involved in it. The issues are there, but usually people won’t just tell you what to do. You’ll need to take initiative, look for the issues that deserve attention, and work on them.

Make an effort

While the simplest way to contribute to OSS is to work on the issues anyone can observe just by taking a look around the project’s repo, you should aim to do more.

If you can afford to invest a bit more time into OSS — look for the harder problems that people usually shy away from. These usually come in the form of issues that have been around for months and even years, core usability issues, missing documentation, and more. Working on them will leave a huge impact on the project, make you a meaningful part of it, and give you a sense of achievement and accomplishment.

Be as communicative as possible

If you have an idea for a cool feature — that’s great! Just be sure to first open an issue for it, and discuss the details with the maintainers. Want to work on an existing issue? Also cool, but leave a comment on it first, saying you’d like to work on it. Many times I’ve encountered multiple people working on the same issue, and it’s just a shame, caused by a simple lack of communication.

Note that I’m not saying you have to get prior permission for everything. For example, if you notice some logic in a library’s source code could be optimized, that’s awesome! You should totally open a PR with your suggested improvements. However, you shouldn’t do a major refactor without discussing it first with the project’s maintainers.

Be patient

Contributing OSS is a marathon, not a sprint. Some PRs take many months and even years to get merged. It’s not that people don’t respect the time you invested into their project, but that they have other commitments as well. Doing OSS is hard, time-consuming, and not very rewarding in most cases. Keep in mind that the circumstances are different for everyone.

Conclusion

I truly appreciate people putting their time into OSS, and think that everyone can be a meaningful part of the community. I hope you’ll be able to take a thing or two from this post, and wish you luck in your OSS endeavors.

Thanks to Ben Greenberg for reading drafts of this post and giving me valuable feedback.

Getting Started with Cost Optimization on AWS

Itai Steinherz — Tue, 10 Dec 2019 16:25:10 +0000

Prologue

I’m a long-time user of AWS, and have been a cloud computing advocate for quite some time. However, it seems to me like cost optimization is a part of cloud computing which doesn’t get the attention it deserves. As I’m going to speak at AWS Community Day Tel Aviv about it, I also wanted to write a post on the subject.

This article will be an introduction to cost optimization (focused on AWS, as that’s the cloud provider I’m most familiar with), with the end goal being to give you some inspiration on how you can look differently at your own cloud bills and cut down on costs.

The Five Pillars

According to the AWS documentation, there are five general cost optimization pillars:

Right size
Increase elasticity
Leverage the right pricing model
Optimize storage
Measure, monitor, and improve

Here, I chose to focus on the three pillars which I think are the most important to be familiar with.

Right size

I’d recommend you to read Corey Quinn’s blogpost on the topic, “Right Sizing Your Instances Is Nonsense”. Basically, your future needs when it comes to instance types are often unpredictable, and so you should choose a configuration that works for now and change it as necessary instead of opting for a specific instance type and commit to that. While this approach may sound a bit contradictory to AWS’ suggestion, it will enable you to be more flexible when working with the cloud, and react quickly to changes in your computing needs.

Leverage the right pricing model

Using spot instances and Savings Plans, you can significantly cut down on EC2 costs for predictable as well as spontaneous compute needs.

If you need as much flexibility as possible, the Compute Savings Plans are the best option available, and they to apply Fargate in addition to EC2. See the pricing page for more info.

	Compute Savings Plan	EC2 Instance Savings Plan
Term	1 Year or 3 Year
Platform/OS Flexibility	Yes
Tenancy Flexibility (Shared vs Dedicated)	Yes
Regional availability	All, except mainland China. Yes, that means GovCloud too!
Overage charges	Spend above your commit is charged at on-demand rates
Region-specific	No	Yes
Instance-specific	No	Yes
Purchase Options	No upfront, partial upfront, and all upfront
Discount	Up to 66%	Up to 72%
Supports Fargate	Yes	No

Source: Corey Quinn’s post blogpost on Savings Plans.

Another alternative for EC2 is using reserved instances, which are less flexible than the Savings Plans AWS offers, and aren’t much cheaper either (this article contains a nice comparison of the two options).

Measure, monitor, and improve

This means different things to different companies and organizations, but in general, you want to regularly go over your cloud bills and be aware of what you’re spending money on. In countless times, the bill will show that by simply turning off unused EC2 instances, you can save a large percentage of the bill.

One useful monitoring tool is AWS Budgets, which is built into the AWS Billing Dashboard and allows you to easily set your budget and be alerted when the current bill is exceeding your budget, or when it’s predicted to do so.

Note that I’m mainly focused on compute costs in this post since those are usually the most significant expenses users face.

Updates from re:Invent 2019

Compute Optimizer

One of the most exciting releases from re:Invent 2019 (in my opinion) is the AWS Compute Optimizer. I’ve already talked about right-sizing instances and impractical it is, however Compute Optimizer changes a few things.

Compute Optimizer analyzes your compute workloads, and uses machine learning to process historical utilization metrics and offer recommendations which are predicted to reduce costs and improve performance.

Corey Quinn already wrote a whole blogpost on this, which I definitely recommend checking out. You can also read more about Compute Optimizer on the AWS News Blog.

Fargate Spot

Another important announcement is Fargate Spot, which basically means that you can now configure your Fargate cluster to use Spot instances when those are available, or use regular EC2 containers otherwise. Using Fargate Spot will enable users to save up to 70% (when compared to using a regular Fargate cluster). Combined with Compute Savings Plans, you can drastically cut down on Fargate costs, without compromising flexibility or elasticity.

Conclusion

I personally think that cost optimization is less about cutting down on costs wherever possible, and more about using the right tool for the right job (while remembering to shut down unused EC2 instances). Plan for the future, leave room for as much flexibility as possible, and continuously adapt your cloud usage to your needs as you go.

Big thanks to Corey Quinn for being a great resource on cloud cost optimization and inspiring me to dive into the topic.