DEV Community: Seth Michael Larson

The problem with Flask async views and async globals

Seth Michael Larson — Mon, 02 Aug 2021 18:11:41 +0000

This is a cross-post from my blog. If you enjoy my content you can subscribe via Email or RSS.

Starting in v2.0 Flask has added async views which allow using async and await within a view function. This allows you to use other async APIs when building a web application with Flask.

If you're planning on using Flask's async views there's a consideration to be aware of for using globally defined API clients or fixtures that are async.

Creating an async Flask application

In this example we're using the async Elasticsearch Python client as our global fixture. We initialize a simple Flask application with a single async view that makes a request with the async Elasticsearch client:

# app.py
from flask import Flask, jsonify
from elasticsearch import AsyncElasticsearch

app = Flask(__name__)

# Create the AsyncElasticsearch instance in the global scope.
es = AsyncElasticsearch(
    "https://localhost:9200",
    api_key="..."
)

@app.route("/", methods=["GET"])
async def async_view():
    return jsonify(**(await es.info()))

Running with gunicorn via $gunicorn app:app and then visiting the app via http://localhost:8000. After the first request everything looks fine:

{
  "cluster_name": "d31d9d6abb334a398210484d7ac8567b",
  "cluster_uuid": "K5uyniiMT9u2grNBmsSt_Q",
  "name": "instance-0000000001",
  "tagline": "You Know, for Search",
  "version": {
    "build_date": "2021-04-20T20:56:39.040728659Z",
    "build_flavor": "default",
    "build_hash": "3186837139b9c6b6d23c3200870651f10d3343b7",
    "build_snapshot": false,
    "build_type": "docker",
    "lucene_version": "8.8.0",
    "minimum_index_compatibility_version": "6.0.0-beta1",
    "minimum_wire_compatibility_version": "6.8.0",
    "number": "7.13.1"
  }
}

However when you refresh the page to send a second request you receive an InternalError and the following traceback:

Traceback (most recent call last):
  ...
  File "/app/app.py", line 13, in async_route
    return jsonify(**(await es.info()))
  File "/app/venv/lib/...", line 288, in info
    return await self.transport.perform_request(
  File "/app/venv/lib/...", line 327, in perform_request
    raise e
  File "/app/venv/lib/...", line 296, in perform_request
    status, headers, data = await connection.perform_request(
  File "/app/venv/lib/...", line 312, in perform_request
    raise ConnectionError("N/A", str(e), e)

ConnectionError(Event loop is closed) caused by:
  RuntimeError(Event loop is closed)

Why is this happening?

The error message mentions the event loop is closed, huh? To understand why this is happening you need to know how AsyncElasticsearch is implemented and how async views in Flask work.

No global event loops

Async code relies on something called an event loop. So any code using async or await can't execute without an event loop that is "running". The unfortunate thing is that there's no running event loop right when you start Python (ie, the global scope).

This is why you can't have code that looks like this:

async def f():
    print("I'm async!")

# Can't do this!
await f()

instead you have to use asyncio.run() and typically an async main/entrypoint function to use await like so:

import asyncio

async def f():
    print("I'm async!")

async def main():
    await f()

# asyncio starts an event loop here:
asyncio.run(main())

(There's an exception to this via python -m asyncio / IPython, but really this is running the REPL after starting an event loop)

So if you need an event loop to run any async code, how can you define an AsyncElasticsearch instance in the global scope?

How AsyncElasticsearch allows global definitions

The magic of global definitions for AsyncElasticsearch is delaying the full initialization of calling asyncio.get_running_loop(), creating aiohttp.Session, sniffing, etc until after we've received our first async call. Once an async call is made we can almost guarantee that there's a running event loop, because if there wasn't a running event loop the request wouldn't work out anyways.

This is great for async programs especially, as typically a single event loop gets used throughout a single execution of the program and means you can create your AsyncElasticsearch instance in the global scope how users create their synchronous Elasticsearch client in the global scope.

Using multiple event loops is tricky and would likely break many other libraries like aiohttp in the process for no (?) benefit, so we don't support this configuration. Now how does this break when used with Flask's new async views?

New event loop per async request

The simple explanation is that Flask uses WSGI to service HTTP requests and responses which doesn't support asynchronous I/O. Asynchronous code requires a running event loop to execute, so Flask needs to get a running event loop from somewhere in order to execute an async view.

To do so, Flask will create a new event loop and start running the view within this new event loop for every execution of the async view. This means all the async and await calls within the view will see the same event loop, but any other request before or after this view will see a different event loop.

The trouble comes when you want to use async fixtures that are in the global scope, which in my experience is common in small to medium Flask applications. Very unfortunate situation! So what can we do?

Fixing the problem

The problem isn't with Flask or the Python Elasticsearch client, the problem is the incompatibility between WSGI
and async globals. There are a couple of solutions, both of which involve Async Server Gateway Interface (ASGI), WSGI's async-flavored cousin which was designed with async programs in mind.

Use an ASGI framework and server

One way to avoid the problem with WSGI completely is to simply use a native ASGI web application framework instead.
There are a handful of popular and widely used ASGI frameworks you can choose from:

If you're looking for an experience that's very similar to Flask you can use Quart which is inspired by Flask. Quart even has a guide about how to migrate from a Flask application to using Quart! Flask's own documentation for async views actually recommends using Quart in some cases due to the performance hit from using a new event loop per request.

If you're looking to learn something new you can check out FastAPI which includes a bunch of builtin functionality for documenting APIs, strict model declarations, and data validation.

Something to keep in mind when developing an ASGI application is you need an ASGI-compatible server. Common choices include Uvicorn, Hypercorn, and Daphne. Another option is to use the Gunicorn with Uvicorn workers.

All the options mentioned above function pretty similarly so pick whichever one you like. My personal choice has historically been Gunicorn with Uvicorn workers because of how widely used and mature Gunicorn is relative to how new the other libraries are.

You can do so like this:

$ gunicorn app:app -k uvicorn.workers.UvicornWorker

Use WsgiToAsgi from asgiref

If you really love Flask and want to continue using it you can also use the asgiref package provides an easy wrapper called WsgiToAsgi that converts a WSGI application to an ASGI application.

from flask import Flask, jsonify
from elasticsearch import AsyncElasticsearch

# Same definition as above...
wsgi_app = Flask(__name__)
es = AsyncElasticsearch(
    "https://localhost:9200",
    api_key="..."
)

@wsgi_app.route("/", methods=["GET"])
async def async_view():
    return jsonify(**(await es.info()))

# Convert the WSGI application to ASGI
from asgiref.wsgi import WsgiToAsgi

asgi_app = WsgiToAsgi(wsgi_app)

In this example we're converting the WSGI application wsgi_app into an ASGI application asgi_app which means when we run the application a single event loop will be used for every request instead of a new event loop per request.

This approach will still require you to use an ASGI-compatible server.

Everything to know about Requests v2.26.0

Seth Michael Larson — Tue, 13 Jul 2021 18:13:46 +0000

This is a cross-post from my blog. If you enjoy my content you can subscribe via Email or RSS.

Requests v2.26.0 is a large release which changes and removes many features and dependencies that you should know about when upgrading.

Summary of the release

What changes are important?

Changed the requests[security] extra into a no-op. Can be safely removed for v2.24.0+ for almost all users (OpenSSL 1.1.1+ and not relying on specific features in pyOpenSSL)
Dropped support for Python 3.5
Changed encoding detection library for Python 3.6+ from chardet to charset_normalizer
Added support for Brotli compressed response bodies

What should you do now?

Upgrade to Requests v2.26.0 if you're not using Python 3.5 - Stop using requests[security] and instead install just requests
Regenerate your lock files and pinned dependencies if you're using pip-tools, poetry, or pipenv
Read the full set of changes for v2.26.0

Encoding detection with charset_normalizer

The following section has a brief discussion of licensing issues. Please remember that I am not a lawyer and don't claim to understand anything about open source licenses.

Requests uses character detection of response bodies in order to reliably decode bytes to str when responses don't define what encoding to use via Content-Type. This feature only gets used when you call the Response.text() API.

The library that Requests uses for content encoding detection has for the past 10 years been chardet which is licensed LGPL-2.1.

The LGPL-2.1 license is not a problem for almost all users, but an issue arises with statically linked Python applications which are pretty rare but becoming more common. When Requests is bundled with a static application users can no longer "switch out" chardet for a different library which causes a problem with LGPL.

Starting in v2.26.0 for Python 3 the new default library for encoding detection will be charset_normalizer which is MIT licensed. The library itself is relatively young so a lot of work has gone into making sure users aren't broken with this change including extensive tests against real-life websites and comparing the results against chardet to ensure better performance and accuracy in every case.

Requests will continue to use chardet if the library is installed in your environment. To take advantage of charset_normalizer you must uninstall chardet from your environment. If you want to continue using chardet with Requests on Python 3 you can install chardet or install Requests using requests[use_chardet_on_py3]:

$ python -m pip install requests chardet

- OR -

$ python -m pip install requests[use_chardet_on_py3]

Removed the deprecated [security] extra

Before Requests v2.24.0 the pyOpenSSL implementation of TLS was used by default if it was available. This pyOpenSSL code is packaged along with urllib3 as a way to use Subject Name Identification (SNI) when Python was compiled against super-old OpenSSL versions that didn't support it yet.

Thankfully these super-old versions of OpenSSL aren't common at all anymore! So now that pyOpenSSL code that urllib3 provides is a lot less useful and now a maintenance burden for our team, so we now have a long-term plan to eventually remove this code. The biggest dependency using this code was Requests, a logical first place to start the journey.

Starting in Requests v2.24.0 pyOpenSSL wouldn't be used unless Python wasn't compiled with TLS support (ie, no ssl module) or if the OpenSSL version that Python was compiled against didn't support SNI. Basically the two rare scenarios where pyOpenSSL was actually useful!

The release of v2.24.0 came and went quietly which signaled to our team that our long-term plan of actually removing pyOpenSSL will likely go smoothly. So in Requests v2.25.0 we officially deprecated the requests[security] extra and in v2.26.0 the [security] extra will be a no-op. Instead of installing pyOpenSSL and cryptography no dependencies will be installed.

What this means for you is if you've got a list of dependencies that previously used requests[security] you can remove the [security] and only install requests. If you have a lock file via a tool like pip-tools or poetry you can regenerate the lock file and potentially see pyOpenSSL and cryptography removed from your lock file. Woo!

Dropped support for Python 3.5

Starting in Requests v2.25.0 there was a notice for Python 3.5's deprecation in the changelog. Now that 2.26.0 has arrived Requests will only be supported with Python 2.7.x and 3.6+.

This is a big win for Requests maintainers as it progressively becomes more and more difficult to maintain a codebase that supports a wide range of Python versions.

Brotli support via urllib3

Since v1.25 urllib3 has supported automatically decoding Brotli-compressed HTTP response bodies using either Google's brotli library or the brotlicffi library (previously named brotlipy).

Before v2.26.0 Requests would never emit an Accept-Encoding header with br signaling Brotli support even if urllib3 would have been able to decode the response. Now Requests will use urllib3's feature detection for Brotli and emit Accept-Encoding: gzip, deflate, br. This is great news for servers that support Brotli on pre-compressed static resources like fonts, CSS, and JavaScript. Woo!

To take advantage of Brotli decoding you need to install one of the Brotli libraries mentioned above. You can ensure you're getting the right library for your Python implementation by installing like so:

$ python -m pip install urllib3[brotli]

API Design for Optional Async Context Managed Resources

Seth Michael Larson — Tue, 11 Aug 2020 21:11:20 +0000

This is a cross-post from my blog Python ♥ HTTP. If you enjoy my content and want it sooner you can follow me via RSS.

I had a discussion on Twitter with Mike Bayer, author of SQLAlchemy, about API design regarding async functions that return a resource to optionally be used as a async context manager. I wanted to do a quick write-up so the results of the discussion wouldn't be lost to the Twitter aether. Here we go:

Take the Python function open(). The function can be used in multiple ways:

# Without a context manager,
# requires an explicit close():
f = open("file.txt")
f.read()
f.close()

# As a context manager, file is
# closed in __exit__():
with open("file.txt") as f:
    f.read()

Now what if we wanted the above but to be asynchronous? What if we wanted to write an async function (named async_open()) which has the same behavior as open() above but is all async?

First let's start with the "without async context manager" case:

f = await async_open("file.txt")

Seems straightforward enough. All the async magic would be within the function itself. Now what if we wanted to add the async context manager case?

An API which returns an async context manager is typically a synchronous function itself so you don't have to write the following:

async with (await async_open("file.txt")) as f:
    ...

The above API in my opinion looks clunky to use. You may already see the problem between the two code samples above. We want async_open to be both asynchronous and synchronous to fit both usages. Oh no!

The big question is where do we put the async code that happens either when await is used or when async with is used without the user signalling which will be used.

After a little bit of thinking I landed on the following:

class AsyncFile:
    """The object that has all the file object
    methods like read() and close() but async!
    """
    async def read(self):
        print("read()")

    async def close(self):
        print("close()")


class AwaitOrAenter:
    """The special object that handles either
    an __await__() call for the non-context
    managed case or an __aenter__() call for
    the context managed case.
    """
    def __init__(self, filepath):
        self._filepath = filepath
        self._file = None

    async def _async_open(self):
        # ( Async magic and open an AsyncFile )
        self._file = AsyncFile()
        return self._file

    def __await__(self):
        return self._async_open().__await__()

    async def __aenter__(self):
        return await self._async_open()

    async def __aexit__(self, *_):
        await self._file.close()


def async_open(filepath):
    return AwaitOrAenter(filepath)

The above API can now be used in both of these cases:

f = await async_open("file.txt")
await f.read()
await f.close()

async with async_open("file.txt") as f:
    await f.read()

Looks like the API works as intended! 🥳

Why URLs are Hard: Path Parameters and urlparse

Seth Michael Larson — Sat, 11 Apr 2020 19:18:48 +0000

This is a cross-post from my blog Python ♥ HTTP. If you enjoy my content and want it sooner you can follow me via RSS.

Welcome to the first installment of "Why URLs are Hard": a series of stories that I've accumulated from reading a lot about URLs.

We take URLs for granted and mostly think of them as very simple things because of how often we interact with clean and simple URLs like https://example.com. Little do you know there are decades of ancient dark magic that occurred before we ended up with URLs we know and love today.

This story is about finding a mysterious API in Python's urlparse function and discovering a now almost entirely unused URL feature. Come along with me! :)

Comparing urlparse to RFC 3986

I was evaluating urlparse from the urllib.parse module and how it performed compared to other URL parser libraries.

Within the documentation it's mentioned that URLs are parsed according to RFC 3986 which is a set of rules that describe how to segment a URL into different components. Let's take a quick look at that standard to see what parts of a URL we see.

There's a cute little ASCII diagram showing off all the parts of a URL:

 foo://example.com:8042/over/there?name=ferret#nose
 \_/   \______________/\_________/ \_________/ \__/
  |           |            |            |        |
scheme     authority       path        query   fragment

... and then the authority section is further decomposed into:

authority = [ userinfo "@" ] host [ ":" port ]

One of the best parts of reading RFCs is thinking about how much effort people put into the adorable ASCII art :)

Okay, now that we know what to expect let's try out urlparse with the URL from the RFC:

>>> from urllib.parse import urlparse
>>> url = (
... "foo://user:pass@example.com:8042"
... "/over/there?name=ferret#nose"
)
>>> parts = urlparse(url)
>>> parts
ParseResult(
    scheme='foo',
    netloc='user:pass@example.com:8042',
    path='/over/there',
    params='',
    query='name=ferret',
    fragment='nose'
)
>>> parts.hostname
'example.com'
>>> parts.port
8042
>>> parts.username
'user'
>>> parts.password
'pass'

Okay so looks like we have this as a mapping from ParseResult to RFC 3986:

parts.scheme -> scheme
parts.netloc -> authority
- parts.username:password -> userinfo
- parts.hostname -> host
- parts.port -> port
parts.path -> path
parts.params -> ???
parts.query -> query
parts.fragment -> fragment

Notice the ??? in the list? I was confused too. No matter what I put into my URL I couldn't get anything to show up in ParseResult.params.

The documentation for ParseResult.params is "Parameters for last path element" and then isn't mentioned much anywhere else. Googling around is tough too because "params" is Requests way of adding to the query string for the requested URL so most results are about that.

When googling "Path parameters" I found this article from 2008 which pointed to the last paragraph of RFC 3986 Section 3.3 which explains path parameters:

Aside from dot-segments in hierarchical paths,
a path segment is considered opaque by the
generic syntax.  URI producing applications
often use the reserved characters allowed in a
segment to delimit scheme-specific or dereference-
handler-specific subcomponents.  For example,
the semicolon (";") and equals ("=") reserved
characters are often used to delimit parameters
and parameter values applicable to that segment.

So ; and = have special meaning within the path, let's throw those into urlparse and see what happens:

>>> urlparse("http://example.com/a;z=y;x/b;c;d=e")
ParseResult(
    scheme='http',
    netloc='example.com',
    path='/a;z=y;x/b',
    params='c;d=e',
    query='',
    fragment=''
)

Huh, I didn't expect it to pull the values actually outside of the path component. And it looks like it only pulled the params from the last segment, /a;z=y;x/ is untouched. Wonder how many bugs are lurking out there because of this quirk. :)

So if you're relying on URL parsing and directly inspecting the path component make sure you check your implementation and amend it to add f";{result.params}" if params is non-empty. Either that or use a URL parser that doesn't have this quirk like rfc3986

I especially recommend using another library if you're making security decisions based on the URL. A write-up from 2011 details a security issue related to path parameters which an application using ParseResult.path alone would likely also be vulnerable to.

Hope you learned something and stay safe!

urllib3 in 2020

Seth Michael Larson — Sat, 14 Mar 2020 17:40:21 +0000

This is a cross-post from my blog Python ♥ HTTP. If you enjoy my content and want it sooner you can follow me via RSS.

2019 was a transformative year for urllib3 and I'm hoping to keep up the pace with Python's most used HTTP client. Here are a list of ideas, some easier achieved than others, that would be great to ship this year!

Acquire Funding for Larger Projects

We're excited to be working with Tidelift for another year as one of the original lifted Python projects.
They've enabled us to provide a comprehensive security disclosure workflow and help me and another maintainer to "keep the lights on" so to speak for the project.

However for some of the larger goals mentioned in this post we'll likely require dedicated developer-hours to achieve. In 2019 ~78% of urllib3's $23,580 in funding came from grants. What urllib3 was able to achieve with the grants has increased the security of the project and benefited millions of people, your generous organization could help us do the same in 2020!

If your organization benefits from urllib3 being secure, maintained, and performant, and maintained please consider contacting me to discuss a grant opportunity or subscribing to Tidelift to keep urllib3 and many more Python projects secure and maintained.

Mentor New Contributors

Python's HTTP stack has seen a surge of activity and will need even more helping hands to accomplish our goals and keep Python on the bleeding edge in the HTTP space and keep our dependable existing libraries up-to-date.

If you're looking to get started in Python Open Source and have an interest in making high-impact contributions, learn a lot about networking, or bolster your resume reach out to me.

Use Operating System Trust Stores

Python's HTTP clients all rely on a trust store named certifi which is the Mozilla CA Certificate Store, packaged and released on PyPI. This works well for the Internet at large as it's the same CA certificate store that powers Firefox and curl. However many organizations and IT machine configurations only ensure that the operating system CA bundle is up to date and having certifi be on separate means organizations can't control the safety of their users the same way that they control safety of browsers and other tools.

We'd also want this work to be usable by other projects in the Python ecosystem as this is an important feature for any library that uses TLS.

oscrypto.trust_list in particular looks like a promising start.

TLS 1.2+ by Default

Thanks to the security trail-blazing of many organizations TLS 1.2 is becoming the new minimum security level for the Internet. CDNs and cloud services mean almost all services deployed there use TLS 1.2+ whether their users know it or not by taking configuration of TLS termination away from users.

Browsers are also doing their part by deprecating and eventually removing TLS 1.0 and 1.1 from future versions. Firefox, Edge, Safari, and Chrome have all committed to removing TLS <1.2 in 2020.

This means 2020 is a great time for us to also push the better-security-by-force envelope. :)

Setting TLS 1.2+ means that we'll be secure against TLS downgrade attacks or if a vulnerability is found in TLS 1.0 or 1.1 our users will already be protected. It also means that we can restrict our default cipher suites to rock-solid secure ones instead of ones that are there only to be compatible with older TLS versions.

Even with all you'll still be able to use TLS 1.0 or 1.1 if you specify those versions manually via ssl_version in the case you have to connect to an older server.

Drop Support for Python <2.7.9

Python 2.7.9 is the first Python version that supported ssl.SSLContext() objects. Currently urllib3 supports all Python 2.7 versions but maintains a large chunk of code that allows us to work with Python versions that don't have proper SSLContext objects. This means that we're forced to do hostname verification ourselves rather than rely on OpenSSL's own implementation of certificate verification.

We'll continue to support Python interpreters that don't have a compiled ssl module, however HTTPS connections won't be possible as is the case today.

We will also continue to in general support Python 2.7 for the forseeable future and won't consider removing support via a breaking change until Pip and other upstream projects also drop support.

For some data on what percentage of downloads would be effected I've created a small data table. The number of downloads effected by the change would be ~4%.

Add Support for Zstandard

Zstandard defined in RFC 8478 is looking to be a great replacement for the long-dominant gzip as a content encoding. With both great compression ratios and quick parallelizable CPU usage I'm guessing this technology will soon take the Internet by storm, especially at the edge / CDN layer.

When Zstandard is standardized and accepted it'd be great to already have the feature in place to immediately start giving our users the new performance benefits.

Switch from Travis + AppVeyor to GitHub Actions

The number of Python versions and configurations that urllib3 supports is quite large, testing against Linux, macOS, and Windows to ensure we're compatible with all major OS flavors. With Travis dropping the default concurrency level to 3 for OSS projects and AppVeyor continuing to support only 1 concurrent job we're feeling cornered into long CI durations.

Fortunately we've been experiencing success with GitHub Actions as a CI platform in other projects with the 20 concurrent jobs as well as great flexibility and access to all platforms from one service and would like to switch wholesale to GA from Travis and AppVeyor.

urllib3 Sustainability and Achievements in 2019

Seth Michael Larson — Sat, 28 Dec 2019 16:29:10 +0000

This is a cross-post from my blog Python ♥ HTTP. If you enjoy my content and want it sooner you can follow me via RSS.

urllib3 has had probably one of it's most eventful years in recent times,
especially with regards to sustainability of the project thanks to sponsors and grants.

I'm looking forward to 2020 and have many ideas for where the project is headed that I'll
be sharing in a future post. For now let's review what was accomplished in 2019:

Grants and Sponsorships

urllib3 received $23,580 USD throughout the year of 2019.
We're very grateful for our donators and sponsors, this year
would not have been as productive without you. Thank you!

Here's the breakdown on where that money came from:

Grant from CERT Governmental Luxembourg for $10,944
Grant from GitCoin Grants for $7,636 (1 DAI ~ $1)
Sponsorship from Tidelift for $5,000

The breakdown above shows that most of our funding for this year came from grants.
Hopefully we can continue this into 2020 as the major accomplishments for the project
were completed as a result of dedicated developer(s) spending extended periods of time
working on features.

If you or your organization rely on urllib3 and would like to sponsor urllib3's development
send an email to sethmichaellarson@gmail.com and andrey.petrov@shazow.net.

Releases and Changes

urllib3 made 10 releases during 2019, up from only 3 releases during 2018.
The highlights of those releases include:

Strict compliance to RFC 3986 for URL parsing.
This functionality was implemented as a part of the two grants
listed above and helped protect users from the new class of
attacks related to URL parsers. See CVE-2019-9740, CVE-2019-9636, CVE-2019-10160.
Added support for TLSv1.3 for OpenSSL 1.1.1+. This functionality was implemented
as a part of the grant from GOVCERT LU. TLS 1.3 adds additional security and
performance benefits for HTTPS connections.
Added automatic downstream integration testing for Requests and Botocore
and automated deploys to PyPI from CI. This means we can ship releases more frequently
and also be more confident that the changes being made won't break the universe.
Our CI was also augmented to be less flaky resulting in smoother merges for Pull Requests.
This work was done as a part of both above grants.
Added support for Brotli as a Content-Encoding. This means that if the requested website
also supports Brotli your response bodies will be even smaller than gzip and save bandwidth.
Added support for Python 3.8. Python 3.9 alphas have just started coming out and there are
already issues on the horizon.

Achievements

These achievements aren't related to library features but are still super-fun to celebrate!

We eclipsed 1 billion (1,000,000,000) total downloads on PyPI,
something that only ~10 projects have done. This number is unimaginably large and shows
how essential a secure HTTP client library is to the Python ecosystem.
We receive a majority of our downloads from Python 3.X instead of Python 2.X
for the first time. About ~50% of all downloads still come from Python 2.7 but that number is very slowly decreasing
over time.
We have a logo now thanks to Ryan Feeley and Jess Shapiro! ♥

Thank You

Thanks to everyone who contributed to urllib3, your contributions are making a huge difference.
If you'd like to join our little team and start contributing
we have a guide on how to get started.

Optimize Web App Performance with HTTP Header Compression

Seth Michael Larson — Mon, 09 Dec 2019 16:57:32 +0000

This is a cross-post from my blog Python ♥ HTTP. If you enjoy my content and want it sooner you can follow me via RSS.

HTTP header compression is a new feature within HTTP/2 and HTTP/3 that drastically reduces the amount of data that needs to be transported over the wire by building both a static and dynamic encoding of header keys and values into tiny representations.

Header compression is defined in these two documents:

Cloudflare wrote an article about the technology itself, which I won't discuss a lot here. If you don't know about header compression in HTTP/2 and HTTP/3 you'd benefit a lot from reading the article before continuing with this post! Instead I'm going to talk about all the fun edge-cases and gotchas and finally some theory-crafting! Woohoo!

HTTP headers are by far the smallest component of an HTTP request / response cycle. The largest component are typically the HTTP request and response bodies. So prefix this entire post by remembering we're optimizing probably less than 15% of total data transfer over a typical HTTP request / response cycle. Still matters, but it's definitely a "micro" optimization!

HTTP bodies are already made more efficient by compression techniques available in HTTP/1.x like gzip and brotli (and hopefully zstd soon!). Changes to header representation are one of the real big changes with HTTP/2 and onwards.

Edge-cases for Optimizing Header Compression

If your website is fronted by Cloudflare or another CDN, chances are your application is being served over HTTP/2 and (if not already) HTTP/3 soon.

Here are some tips on how to take advantage of header compression in your applications.

Lowercase Characters Encode Smaller

Some time ago I tweeted out this fun-fact about HTTP/2 huffman encoding which inspired this blog post:

When using HTTP/2 you can save ~half a bit for every lowercase character you send in a header instead of an uppercase character due to the Huffman encoding weights in HPACK.

A-Z: 6.5 bits per character

a-z: 6.038 bits per character

Be responsible, lowercase your headers. 🌈

Header names already must be all lowercase for HTTP/2, but header values commonly have case-insensitive components. (Check the RFC for that header type if you're unsure!)

Fold Duplicate Headers Before Sending

Duplicate header entries are allowed by HTTP but they should be joined with ; to not take up more space than needed. Not a lot of services do this and this is already bad form in HTTP/1.X, just wanted to note it down.

Strict-Transport-Security: max-age=31536000
Strict-Transport-Security: includesubdomains
Strict-Transport-Security: preload

should instead be:

Strict-Transport-Security: max-age=31536000; includesubdomains; preload

The exception to this rule is Set-Cookie which cannot be folded this way without breaking its semantics.

Know the Headers and Values in the Static Table

Knowing the names and values in the static table can help send smaller requests and responses for applications. The static table works via exact matches so any one character being different means that the header can't be optimized.

Even when semantically the order of values doesn't matter in any of these cases, they need to be exactly as they are below otherwise HPACK and QPACK can't kick in and replace the header with a reference to the static table:

Accept-Encoding: gzip, deflate, br
Content-Security-Policy: script-src 'none'; object-src 'none'; base-uri 'none'
Access-Control-Allow-Methods: get, post, options
Strict-Transport-Security: max-age=31536000; includesubdomains; preload

Spacing Around Delimiters

Put a space ' ' after every delimiter (e.g. ; and ,) unless you're specifically encoding Content-Type: text/plain;charset=utf-8. That's the only entry in the static table that doesn't have a space after the ; delimiter.

QCRAM → QPACK

Just wanted to note that the original name for QPACK was QCRAM. The urgency associated with the word "cram" makes me smile whenever I think about it.

The name was changed within this pull request.

Header Compression Theory-crafting

Time to delve into the land of "what-if", and no better time than when HTTP/3 is being finalized. ;)

My thought on keeping header compression and HTTP/2 + HTTP/3 "simple" is that these protocols are both hard to implement and once implemented probably won't change in any significant way until HTTP/N+1, so trying to get as much right as possible benefits everyone for many years.

Separate Huffman Codes for Headers and Values

Header names is a much more constrained charset than header values which have to represent all possible bytes.

Below is the ABNF grammar for a header name taken from RFC 7230.
ABNF grammars are very common in RFCs for describing how protocols look on the wire. If you want to get more into HTTP I recommend learning more!

field-name     = token
token          = 1*tchar
tchar          = "!" / "#" / "$" / "%"
                 "&" / "'" / "*" / "+"
                 "-" / "." / "^" / "_"
                 "`" / "|" / "~" /
                 DIGIT / ALPHA

(Basically means one or more alpha-numerics with all the symbols listed above)

Header names also must be lowercase per the HTTP/2 RFC.

Given these two data-points you can boil down the total possible number of bytes in a valid HTTP header name to be:

15 for symbols
10 for digits
26 for lowercase characters
Total: 51 bytes instead of 256 for full-coverage!

This means that creating a Huffman encoding for these 51 bytes can be more compact than having to cover all 256 bytes. Free bandwidth savings!

If you have to break the RFC and create an invalid HTTP header with bytes outside of the 51 then that header name can be encoded in raw form. Huffman encoding is optional in HTTP/2.

Having header names and values with separate huffman encodings also allows for different weighting of value huffman codes. Digits are very rare within header names but are very common within header values! That means both shorter header names and shorter header values.

More Headers in the Static Table

Headers in the static table are basically free when it comes to size.
HTTP/2 → HTTP/3 increased the size of the static table dramatically. Especially when it comes to having "values" in the static table.

HTTP/2 HPACK: 61 entries, 14 with values (~23% with values)
HTTP/3 QPACK (draft 11): 98 entries, 78 with values (~80% with values)

Obviously there's some diminishing returns here, but the biggest argument I see against adding almost every HTTP header in common use to the static table is storage size in memory (and the current table isn't that large). See the next section on a way to mitigate this issue!

Extensible Static Table

Have "maximum known static table index" be a negotiation parameter. This allows for future expansion of the static table as HTTP grows and new headers are standardized. Also allows for smaller / constrained devices from having to have a large static table in memory.

Allow Origins to Manage their own "Static Table"

Vendor-specific headers are excluded from the static table despite their high usage by specific services. (youtube-client-id, etc). Vendor-specific headers would begin immediately after the largest known static table index. When receiving a new "maximum known static table index" the cached static table would be discarded and started anew. Would have to be some method to manage this static table.

Really services would probably only need a handful of headers, as almost all services only have a few headers that are their own and then rely on HTTP's standard headers for most functionality.

Services would need a way to confirm that a client still had the custom static table for their service, maybe this can be a handshake parameter or something more involved?

This opens up a way for individual clients to be fingerprinted, but it's no worse than caches / cookies (?) and is by definition optional.

Designing for Real-World HTTPS

Seth Michael Larson — Fri, 06 Dec 2019 16:16:29 +0000

This is a cross-post from my blog Python ♥ HTTP. If you enjoy my content and want it sooner you can follow me via RSS.

Users care about security, privacy, and above all, for their HTTP client library to work.
When TLS or certificate issues get in the way of making requests to a web server
oftentimes users will insecurely configure their HTTP client.

And who can blame them when they don't have the documentation or tools to make proper
security decisions in a world full of legacy software and mis-configured servers?

This post discusses how to design an HTTP client for better security and user-experience.
There's a lot of additional / optional readings linked in this post. I encourage you if you
want to learn a whole lot more to take a look at those resources too.

What is HTTPS?

HTTPS is defined in RFC 2818 as "HTTP over TLS".
If you're unsure what TLS is, you can read
this explanation by Cloudflare.

How do users typically solve TLS issues?

If you type "requests certificate verify failed"
into Google the top result you'll receive is this StackOverflow answer
which (among some good advice) mentions verify=False which disables certificate verification.

This will certainly allow your request to "succeed" but will also severely degrade the
security of the HTTPS requests being made.

The warnings against using verify=False within the StackOverflow answer may persuade some,
but searching "verify=False" in Python code on GitHub
yields >150,000 results.

Our goal as an interface designer is to keep verification on!

What issues do users experience?

Imagine you're trying to interact with an external service as a part of
your job via HTTP and of course, you want to use HTTPS because that's the proper thing to do!

The website's hostname is tribunnews.com which just so happens to
be at the time of this writing #41 most visited website in the world.
You'd expect them to have a proper TLS config, right?

When you make an HTTPS request to the website you receive this error:

[SSL: UNSUPPORTED_PROTOCOL] unsupported protocol (_ssl.c:1108)

Now without using Google, can you figure out what this error means
and what you should do to fix it?

If you guessed that the server only supports TLSv1.0, you'd be right!
The client library you're using needs to be configured to use TLSv1.0
which is considered an insecure protocol to use nowadays.

After configuring your client properly (if you're able to!) you make
another request, this time receiving this:

[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1108)

At this point unfortunately you're out of luck, because the website presents a certificate
with two subjectAltName entries:

*.maintenis.com
maintenis.com

... neither of which match our intended target website tribunnews.com.

When a certificate error occurs we can't be certain that the web server we're talking to is
the one we intend to be talking to, so now what? What would you do in this situation?

If you guessed using www.tribunnews.com (???) you'd be correct! Using the www sub-domain gets you the
website you want with all proper TLS configuration and certificates.

Browsers are smart and try www if the host is a privately-registrable domain name and the connection fails
for any reason, so the website works fine for browsers but doesn't work for our HTTP clients... What a nightmare!

Now try solving these problems under a time crunch and without a deep knowledge
of TLS and Public Key Infrastructure. You can now hopefully see why users disable
security features to mitigate TLS issues.

More Examples of HTTPS Issues

Issues when using HTTPS can be categorized into two different groups,
TLS issues and certificate issues. TLS issues are usually to do
with the TLS protocol itself, things like not supporting the same TLS version,
having no ciphers shared between the client and server,
or sending incorrect data during the TLS handshake.

I've found that certificates issues tend to occur more frequently compared to
TLS protocol issues so I've focused on them below.

Here is the list of sources that I've looked through to find examples of these issues:

I've also ran TLS handshakes on the top 10,000 domains in the Alexa 1 Million and
kept the results of all domains where the TLS handshake failed when configured
with certifi (a common choice for Python HTTP libraries
as a default CA certificate bundle) and with all TLS 1.2+ and all ciphers.

All of these below issues can be found at least once in the top 10,000 domains.

Certificate Doesn't Have a `subjectAltName`, only `commonName`

commonName was deprecated over 19 years ago in
RFC 2818. Here's the
exact language:

If a subjectAltName extension of type dNSName is 
present, that MUST be used as the identity. Otherwise,
the (most specific) Common Name field in the Subject
field of the certificate MUST be used. Although the
use of the Common Name is existing practice, it is
deprecated and Certification Authorities are encouraged
to use the dNSName instead.

Which unfortunately says that commonName must be used if there
are no subjectAltName dNSName entries. Meaning it's totally
valid for CAs to keep minting certificates with common names.

This means we probably need to keep supporting a way to verify
certificates via commonName for the time being.

Thankfully browsers and macOS are driving the world of standards
forward by removing support for common name.
macOS dropped common name support completely in iOS 13 / macOS 10.15:

TLS server certificates must present the DNS name of
the server in the Subject Alternative Name extension
of the certificate. DNS names in the CommonName of a
certificate are no longer trusted.

Nothing like your website not working on Chrome, Firefox, macOS and iPhones
to make a business want to upgrade their certificate.

Certificate is Self-Signed

This can be the case for a real web server, a certificate created for a
single peer-to-peer connection like two devices being manufactured together,
or most likely: a user-generated certificate for integration testing.

In this case, especially when the certificate is available to the
user and can be verified locally, Certificate Pinning
solves this problem! Certificate pinning is where you hard-code the signature
or fingerprint of the certificate you're expecting to be presented
in the handshake and if you receive a certificate with the same
fingerprint then the certificate is automatically trusted.

Using certificate pinning is a form of
"Trust on First Use"
which is nearly as secure as verifying a certificate using PKI and CA certs
except for the very first TLS handshake where you make the first "trust" decision yourself.

(NOTE: Certificate Signatures / Fingerprints are different from Certificate Thumbprints.
The former is used for cryptographic reasons, the second is for referencing only.)

The tough part about this feature is that error messages and documentation
don't usually recommend using the feature as an alternative to
disabling certificate verification. It's also non-trivial to calculate
the certificate fingerprint by hand.

Certificate pinning is currently supported by
urllib3 and
aiohttp.
Requests doesn't provide a way to natively verify a certificate via a fingerprint.

Client libraries should use only SHA256 or stronger because MD5 is insecure and SHA1 is on the edge of insecure.

Client libraries should also ensure users are aware of the pitfalls and dangers associated with
updating the certificate fingerprint or using certificate pinning with certs that are liable to change
frequently like certificates that are short-lived (~less than 6 months to expiry).

Certificate `subjectAltName` or `commonName` Doesn't Match Host

In this case the web server is presenting a certificate that we'd
be able to verify with our cert trust store but the fields for
verifying the hostname aren't the same as the host we're connecting to.

For this situation the most useful information for the user is
to display what the fields are on the certificate along with the
host that they're connecting to. Many libraries won't show the
certificates subjectAltName and commonName entries in the error.

I've seen this error come up where the service had a unicode hostname
encoded in the certificate with IDNA 2003 because it contained an emoji
(which is invalid in IDNA 2008), but urllib3 no longer supported IDNA 2003
due to security issues with URL parsing.

WHATWG-URL recommends falling back on IDNA 2003, but this requires a lot of care
to mitigate all security issues related to using the encoding within URLs.

Recommendations for HTTP Client Libraries

Above all, think about users when designing an interface to configure security.
An interface where it's easy to create secure configurations will result in
better security for every use-case.

Here's a list of elements I think result in such an interface:

Provide a State-of-the-Art TLS Configuration by Default

90%+ users will be using these defaults so you want them to be the best they can reasonably be.

At the time of writing this means:

TLSv1.2+ only (Over 95% of websites support TLSv1.2+)
AEAD Ciphers (AES-GCM, CHACHA20)
Key Exchanges that support Forward Secrecy (ECDHE, DHE)
System Certificate Trust Store (This is still an open issue for Python, for now certifi is a crutch.)

Here's how to configure the above in Python via ssl.SSLContext:

import ssl
import certifi

ctx = ssl.SSLContext(ssl.PROTOCOL_TLS)
ctx.verify_mode = ssl.CERT_REQUIRED
ctx.load_verify_locations(certifi.where())
ctx.options |= (
    ssl.OP_NO_SSLv2 |
    ssl.OP_NO_SSLv3 |
    ssl.OP_NO_TLSv1 |
    ssl.OP_NO_TLSv1_1 |
    ssl.OP_NO_COMPRESSION
)
ctx.set_ciphers(":".join([
    "ECDHE+AESGCM",
    "ECDHE+CHACHA20",
    "DHE+AESGCM",
    "DHE+CHACHA20"
]))

Don't Allow Users to Disable Certificate Verification

Especially for HSTS domains, where this is a requirement to be compliant
with RFC 6797.
Being able to turn off certificate verification results in insecure
software. Using cert pinning is a more secure alternative to disabling verification.

Provide these Features for Configuring TLS

Minimum and maximum supported version of TLS.
Recommend strongly that users don't set a maximum version
unless they're targeting one specific version on a single service.
TLS ciphers should be added / removed depending on
the minimum and maximum TLS version. I don't think direct
access to configuring ciphers should be necessary but make
sure you cover all the common ciphers and key-exchanges.
Look at what Firefox uses as their default for an example.
An interface to provide a pre-configured SSLContext is optional,
in my opinion if an interface wrapping TLS is good enough providing
an interface for SSLContext only allows for disabling security features.
Ability to specify a non-default certificate bundle
or directory of certificates to verify against.
Unfortunately it's optimal to support both an
"additive" method and a "replace" method.
Additive is useful when the rest of your trust store
is still needed like in the case of a proxy certificate: you
still want to use your actual trust store to verify
external services.
Replace is useful when you want to use a singular certificate
for the requests you're making and don't want to accidentally
trust a service outside that trust store.

I'd say that the 'additive' use-case is more common with typical
users but supporting both use-cases is important. I need to do
more thinking on this about what a proper interface would look like.

Support Certificate Pinning as a means of verification

Provide documentation on how to use it, how to update the fingerprint,
and what using certificate pinning means for security.

Recommend Solutions for Local Testing

For local testing recommend users use trustme
or mkcert to generate real
certificates so that HTTPS and TLS function the same locally
as they would in the real world. This removes the need for disabling verification
during testing or using pinning on a self-signed cert.

Provide Better Error Messages and Documentation

Error messages should contain all information
they need to make proper security decisions. This includes information
from the certificate. This may require you to use getpeercert(binary_form=True)
and parse with a library like asn1crypto
because Python doesn't give you nice parsed certificates unless cert
verification is enabled.

Document common TLS and certificate issues and the recommended
way for mitigating those issues. If possible, provide custom error
types for common issues along with pointers to official documentation
to prevent StackOverflow from recommending insecure configurations
to your users.

Allow Configuring TLS via Environment Variables

Usually HTTP clients are buried a layer or more deep within an library
or tool being used and may sometimes not provide the means to configure
the HTTP client library properly. Allowing environment variables to
configure the library fixes this issue for users.

Did you have a scenario where the above doesn't work well or you want
to discuss this subject more? I'd love to hear about it, please let me know
by tagging me Twitter.

Python Data Streaming to Google Cloud Storage with Resumable Uploads

Seth Michael Larson — Sun, 18 Mar 2018 13:44:58 +0000

A few days ago I spent a large chunk of my afternoon working on implementing memory-efficient data streaming to Google Cloud Storage (GCS) from a Python runtime.

There were several roadblocks along the way and I'd like to create the documentation that I wish I could find while working on the issue.

This article uses Python 3.6.4 but can be adapted for other Python versions.

GCS support in `google-cloud` Module

The google-cloud package is a giant collection of modules that can be used to interface with all of the Google Cloud Platform services so it's a great place to start.

python -m pip install -U google-cloud

Within the google-cloud package is a module called google.cloud.storage which deals with all things GCS.

I downloaded and setup my GOOGLE_APPLICATION_CREDENTIALS locally and opened up a Python console to test out some of the functionality. I was able to quickly connect to GCS, create a Bucket, create a Blob, and upload binary data to the Blob.

from google.cloud import storage

client = storage.Client()
bucket = client.create_bucket('test-bucket')
blob = client.blob('test-blob')

blob.upload_from_string(
    data=b'x' * 1024,
    content_type='application/octet-stream',
    client=client
)

One thing I immediately noticed was that for building Armonaut my use-case would be progressively streaming output to GCS without saving the output to the file-system of the compute instance. There had to be a way to stream data rather than upload it all in one go.

Resumable Uploads to the Rescue!

The initial research I did uncovered Resumable Uploads as an option for Google Cloud Storage. From their description it says that they have the following use-cases:

You are uploading a large file.
The chances of network failure are high.
You don't know the size of the file when the upload starts.

Reasons #1 and #3 both applied to my use-case so I started investigating further.

I searched the google-cloud documentation for a mention of resumable uploads which yielded the Blob.create_resumable_upload_session() method. This method starts a Resumable Upload and returns a URL.

Resumable Media Package

The set of interactions that must occur for a Resumable Upload to complete successfully were quite complex and I suspected there was already a package that handles this exchange. I found the google-resumable-media package with a bit of Googling. ;-)

python -m pip install -U google-resumable-media

The key part of this package I was interested in is the google.resumable_media.requests.ResumableUpload class which takes an authorized transport and then allows you to upload data in chunks and recover when errors are detected.

So far this was the code I was working with:

import io
from google.auth.transport.requests import AuthorizedSession
from google.cloud import storage
from google.resumable_media.requests import ResumableUpload

chunk_size = 256 * 1024  # Minimum chunk-size supported by GCS
stream = io.BytesIO(b'x' * (1024 * 1024))  # Fake data stream

client = storage.Client()
bucket = client.bucket('test-bucket')
blob = client.blob('test-blob')

# Create a Resumable Upload
url = blob.create_resumable_upload_session(
    content_type='application/octet-stream',
    client=client
)

# Pass the URL off to the ResumableUpload object
upload = ResumableUpload(
    upload_url=url,
    chunk_size=chunk_size
)
transport = AuthorizedSession(credentials=client._credentials)

# Start using the Resumable Upload
upload.initiate(
    transport=transport,
    content_type='application/octet-stream',
    stream=stream,
    metadata={'name': blob.name}
)

Problem was I was getting an error on upload.initiate(). It was complaining that there was no Location header on the response. I investigated this issue and found that create_resumable_upload_session() was doing the work of upload.initiate()! I removed that step and instead used the API endpoint provided in the Resumable Upload documentation.

# Create a Resumable Upload
url = (
    f'https://www.googleapis.com/upload/storage/v1/b/'
    f'{bucket.name}/o?uploadType=resumable'
)
upload = ResumableUpload(
    upload_url=url,
    chunk_size=chunk_size
)
transport = AuthorizedSession(credentials=client._credentials)

# Start using the Resumable Upload
upload.initiate(
    transport=transport,
    content_type='application/octet-stream',
    stream=stream,
    metadata={'name': blob.name}
)

This snippet worked to start a Resumable Upload! Now to stream the data.

Streaming Data and `stream_last=False`

The ResumableUpload object has a method called transmit_next_chunk which tells the upload that the next chunk may be uploaded. While reading the documentation about this method I found stream_final which was a parameter of the ResumableUpload.initiate method.

I found that if stream_final is set to False then the ResumableUpload will detect the "end" of the stream when a chunk is transmitted that is less than the chunk_size parameter set in its constructor. This meant that to stream an unknown amount of data that each chunk would have to be >256KiB and would have to buffer output until that size was reached to be trasmitted.

Enjoying this post? Check out my Dev Blog for more.

Putting it All Together

After getting a simple example working I created a class that handles a single stream of unknown length data being uploaded to a blob progressively and recovers from network errors if detected.

To accomplish this I implemented an object that both buffered data and had a file-like interface in order for it to be used by ResumableUpload as a stream and be passed into other functions that require file-like objects for writing data.

Here is my final implementation:

from google.auth.transport.requests import AuthorizedSession
from google.resumable_media import requests, common
from google.cloud import storage

class GCSObjectStreamUpload(object):
    def __init__(
            self, 
            client: storage.Client,
            bucket_name: str,
            blob_name: str,
            chunk_size: int=256 * 1024
        ):
        self._client = client
        self._bucket = self._client.bucket(bucket_name)
        self._blob = self._bucket.blob(blob_name)

        self._buffer = b''
        self._buffer_size = 0
        self._chunk_size = chunk_size
        self._read = 0

        self._transport = AuthorizedSession(
            credentials=self._client._credentials
        )
        self._request = None  # type: requests.ResumableUpload

    def __enter__(self):
        self.start()
        return self

    def __exit__(self, exc_type, *_):
        if exc_type is None:
            self.stop()

    def start(self):
        url = (
            f'https://www.googleapis.com/upload/storage/v1/b/'
            f'{self._bucket.name}/o?uploadType=resumable'
        )
        self._request = requests.ResumableUpload(
            upload_url=url, chunk_size=self._chunk_size
        )
        self._request.initiate(
            transport=self._transport,
            content_type='application/octet-stream',
            stream=self,
            stream_final=False,
            metadata={'name': self._blob.name},
        )

    def stop(self):
        self._request.transmit_next_chunk(self._transport)

    def write(self, data: bytes) -> int:
        data_len = len(data)
        self._buffer_size += data_len
        self._buffer += data
        del data
        while self._buffer_size >= self._chunk_size:
            try:
                self._request.transmit_next_chunk(self._transport)
            except common.InvalidResponse:
                self._request.recover(self._transport)
        return data_len

    def read(self, chunk_size: int) -> bytes:
        # I'm not good with efficient no-copy buffering so if this is
        # wrong or there's a better way to do this let me know! :-)
        to_read = min(chunk_size, self._buffer_size)
        memview = memoryview(self._buffer)
        self._buffer = memview[to_read:].tobytes()
        self._read += to_read
        self._buffer_size -= to_read
        return memview[:to_read].tobytes()

    def tell(self) -> int:
        return self._read

The class can be used like so:

client = storage.Client()

with GCSObjectStreamUpload(client=client, bucket='test-bucket', blob='test-blob') as s:
    for _ in range(1024):
        s.write(b'x' * 1024)

Thanks for reading!

Mashpack - Beating Messagepack at its own Game

Seth Michael Larson — Tue, 23 Jan 2018 15:01:10 +0000

Mashpack is a JSON-object serialization format that is a 'dialect' of the popular format Messagepack. I read Messagepacks specification and noticed a few details that I thought could be improved so I challenged myself to 'beat' Messagepack. Here are my results:

Key Design Differences

Single-Byte Prefix Reordering
Typed Arrays
8-bit Array Type
Extensions

Single-byte Header Data Types

Both Mashpack and Messagepack use bit-prefix tricks to pack certain small data types into a single byte while still allowing enough prefix space to support all the other data types we need to ensure an efficient use of space.

Both specifications use of single-byte data types matters quite a bit as it is the hope that most serialized objects will end up as one of the single-byte header data types due to their small header footprint so having a large and common range of values is invaluable.

Spec	Single-byte Header Data Types
Mashpack	`MAPP`, `STRP`, `MARRAYP`, `INTP`, `NINTP`
Messagepack	`positive fixint`, `fixstr`, `negative fixint`, `fixmap`, `fixarray`

In Mashpack the string and map data types are prioritized with
the largest amount of space. Compare this to Messagepack which prioritizes
positive integers quite heavily and all other data types have smaller ranges as a result.

See the tables below comparing the two specs:

Data Type	Range in Mashpack	Range in Messagepack
`MAPP`/`fixmap`	0 to 63 key-value pairs	0 to 15 key-value pairs
`STRP`/`fixstr`	0 to 63 encoded bytes	0 to 31 encoded bytes
`MARRAYP`/`fixarray`	0 to 31 elements	0 to 15 elements
`NINTP`/`negative fixint`	-1 to -32	-1 to -32
`INTP`/`positive fixint`	0 to 31	0 to 127

See all Mashpack's data types in one table for more information.

Typed and Mixed Arrays

Mashpack adds a new data type that isn't present in Messagepack called the 'typed array' or just ARRAY*. This in contrast to a 'mixed array' (MARRAY*) requires that all objects within the array be the same data type. Objects which have different headers but are in the same family can be 'upgraded' to a larger data type long as the resulting typed array is smaller than having a mixed array.

An example is a list of integers, one of which would be encoded into an INTP type and all others being converted into INT8 would be converted into an ARRAY8[INT8]. See below:

object = [1, 255, 255]
could be naively encoded into:
MARRAYP([INTP(1), INT8(255), INT8(255)]) = 7 bytes

but converting the INTP(1) into INT8(1) we can take
advantage of typed arrays to end up with this:

ARRAY8[INT8]([1, 255, 255]) = 6 bytes

Adding an 8-bit Array Type

Despite having an unused data type id (0xC1) Messagepack does not define
an 8-bit array type. This in addition to the smaller range of fixarray
being only 15 elements means that arrays with 16 elements in Messagepack
use 3 bytes of header rather than 1 in Mashpack using MARRAYP.

Larger Strings means Unicode Friendly

Unicode brought a lot of harmony to what used to be a really hard problem with universally rendering bytes into strings. One of the necessary evils of unicode is that not all characters are 8 bits meaning that every string that's not ASCII-only requires additional space to store. Mashpack being able to store over double the number of encoded bytes within a one-byte string helps with this problem.

Messagepack's Python implementation also seems to have had a strange relationship with its string data types due to Python 2. The current Mashpack Python implementation supports Python 3+ only and only allows UTF-8 encoding for its string type.

Larger Strings means URLs are Mostly Short Too!

A lot of APIs nowadays have helpful 'explorative' URLS to hint API users about additional content. This will almost always take the form of a complete URL which are typically quite long! Almost always longer than 31 bytes. Having a larger one-byte string helps solve this problem. See the compression performance comparison section.

Extensions

Mashpack defines three extension data type sizes (EXT8, EXT16, and EXT32) while Messagepack defines 8 (fixint 1, fixint 2, fixint 4, fixint 8, fixint 16, ext8, ext16, and ext32)

Extensions work very similarly in Messagepack and Mashpack in that they have an extension code and data section. The extension code is an unsigned integer between 0 and 255 that denotes what extension is being used. The difference between Messagepack and Mashpack's extensions is that the fixed extension type for Messagepack defines 'fixed' extensions that are exactly 1, 2, 4, 8, or 16 bytes long. This allows for 2-bytes of header for extensions with smaller data sections.

Mashpack only allows a 3-byte header minimum, so small data extensions suffer slightly storage-wise under Mashpack. When using a data section 17 bytes or longer extensions pack the same under Mashpack and Messagepack.

Fewer Prefix Tiers

Mashpack's prefix tiering (2x2->3x3->31x8) having only three layers compared to Messagepack's prefix tiering (1x1->2x3->2x4->31x8) means Mashpack will only use a maximum of 3 comparisons to determine a packed objects type compared to 4 for Messagepack.

Fast Unpacking of Positive Fixint

Messagepack's positive fixint type being prefixed as a prefix of 0xxxxxxx allows the integer to be read directly from the byte it resides in without masking.

Mashpack doesn't have this property but only requires one bit-mask operation to make up for it. This difference was a necessary casualty in ensuring better bit prefixes and ranges for other objects.

Both Mashpack and Messagepack's NINTP and negative fixint have this property for negative numbers with a prefix of 111xxxxx.

Performance

Compression Differences

Here's a few small, medium, and large objects for comparing compression between Mashpack and Messagepack. Lower is better.

Object	Mashpack Size	Messagepack Size	Difference
127	2	1	+1 (+50%)
[1] * 16	17	19	-2 (-10.5%)
URL	46	48	-2 (-4.1%)
[127] * 100	103	101	+2 (+1.9%)
[255] * 100	103	203	-100 (-49.2%)
GitHub Push Event	6091	6192	-101 (-1.6%)

Speed Differences

Here's a few small and large objects that were processed 10,000 times to test differences in packing and unpacking speed between Mashpack and Messagepack. Lower is better.

NOTE: These speed comparisons were done on an Intel i5-6400 4x2.7GHz with 16 GB RAM and using the Python 'fallback' version of Messagepack as there is no Cython implementation of Mashpack yet. More benchmarks will be coming to compare Cython implementations later.

Packing Objects

Object	Mashpack Time	Messagepack Time	Difference
[1, 1, 1, 1, 1]	0.22 seconds	0.31 seconds	-0.09 seconds
GitHub Push Event	42.9 seconds	59.4 seconds	-16.5 seconds

Unpacking Objects

Object	Mashpack Time	Messagepack Time	Difference
[1, 1, 1, 1, 1]	0.15 seconds	0.15 seconds	0 seconds
GitHub Push Event	8.39 seconds	9.72 seconds	-1.33 seconds

How can I Help or Contribute?

Mashpack is a brand new specification! Its got plenty left to do before it's cemented into a standard. This is always the most fun part of projects, so I want to share that with everyone who's interested. :)

Everyone that contributes to the Mashpack repository will receive Collaborator for the repository as I would like this to be a project for the community. All changes require code review by one other Collaborator and any Code Owners.

New to Open Source or Python?

There are some issues that I've marked with good first issue that I'm willing to put extra time, effort, and resources into helping you through. If you want to tackle one of these issues but feel the need for additional support you can message me on Twitter to ask questions. :-)

Ready to Tackle Tougher Issues?

There are a lot of issues to contribute to in this category all marked with the help wanted label. I'll get around to these eventually but if you'd like to contribute and get your name out there this is a great place to start.

Are you familiar or interested in learning Cython?

As stated above there is no Cython implementation of the Mashpack Packer or Unpacker objects. Having a Cython implementation would increase the speed of the operations ten-fold and are necessary to be a good competitor to Messagepack. I've marked all issues related to Cython with a label.

Check Name Availability for Multiple Services

Seth Michael Larson — Fri, 22 Sep 2017 20:02:22 +0000

Hello everyone! This is going to be a little bit different than most of my posts in that it's quite short. I was searching for names for an Open Source project that I have been working on and found it very frustrating that the names I was searching for would be available on one platform and unavailable on another. This led to a lot of time being sunk into searching for a good name which could have been spent programming.

To aid me in this quest I whipped up a tool called Avail which is run from the command line and outputs whether the name you give it is available on a variety of platforms including GitHub, Dockerhub, npm, Twitter, .com, URL shortener available, and more!

You can install it via pip like so:

python -m pip install avail

And then use it like this:

avail [NAME] or like this python -m avail [NAME]

The tool's source code is hosted on GitHub and is licensed under Apache-2.0. Pull requests and suggestions for more services are welcome!

I hope this tool is able to help you out, if only to save a few minutes of searching.

DEV Community: Seth Michael Larson

The problem with Flask async views and async globals

Creating an async Flask application

Why is this happening?

No global event loops

How AsyncElasticsearch allows global definitions

New event loop per async request

Fixing the problem

Use an ASGI framework and server

Use WsgiToAsgi from asgiref

Everything to know about Requests v2.26.0

Summary of the release

What changes are important?

What should you do now?

Encoding detection with charset_normalizer

Removed the deprecated [security] extra

Dropped support for Python 3.5

Brotli support via urllib3

API Design for Optional Async Context Managed Resources

Why URLs are Hard: Path Parameters and urlparse

Comparing urlparse to RFC 3986

urllib3 in 2020

Acquire Funding for Larger Projects

Mentor New Contributors

Use Operating System Trust Stores

TLS 1.2+ by Default

Drop Support for Python <2.7.9

Add Support for Zstandard

Switch from Travis + AppVeyor to GitHub Actions

urllib3 Sustainability and Achievements in 2019

Grants and Sponsorships

Releases and Changes

Achievements

Thank You

Optimize Web App Performance with HTTP Header Compression

Edge-cases for Optimizing Header Compression

Lowercase Characters Encode Smaller

Fold Duplicate Headers Before Sending

Know the Headers and Values in the Static Table

Spacing Around Delimiters

QCRAM → QPACK

Header Compression Theory-crafting

Separate Huffman Codes for Headers and Values

More Headers in the Static Table

Extensible Static Table

Allow Origins to Manage their own "Static Table"

Designing for Real-World HTTPS

What is HTTPS?

How do users typically solve TLS issues?

What issues do users experience?

More Examples of HTTPS Issues

Certificate Doesn't Have a subjectAltName, only commonName

Certificate is Self-Signed

Certificate subjectAltName or commonName Doesn't Match Host

Recommendations for HTTP Client Libraries

Provide a State-of-the-Art TLS Configuration by Default

Don't Allow Users to Disable Certificate Verification

Provide these Features for Configuring TLS

Support Certificate Pinning as a means of verification

Recommend Solutions for Local Testing

Provide Better Error Messages and Documentation

Allow Configuring TLS via Environment Variables

Sponsored Work on urllib3

Python Data Streaming to Google Cloud Storage with Resumable Uploads

GCS support in google-cloud Module

Resumable Uploads to the Rescue!

Resumable Media Package

Streaming Data and stream_last=False

Enjoying this post? Check out my Dev Blog for more.

Putting it All Together

Mashpack - Beating Messagepack at its own Game

Key Design Differences

Single-byte Header Data Types

Typed and Mixed Arrays

Adding an 8-bit Array Type

Larger Strings means Unicode Friendly

Larger Strings means URLs are Mostly Short Too!

Extensions

Fewer Prefix Tiers

Fast Unpacking of Positive Fixint

Certificate Doesn't Have a `subjectAltName`, only `commonName`

Certificate `subjectAltName` or `commonName` Doesn't Match Host

GCS support in `google-cloud` Module

Streaming Data and `stream_last=False`