Requests is undeniably unmissable for anyone willing to start in the Python ecosystem. Yet the community is broadly unaware of how sad the situation has become.
The incident
Due to dramatics circumstances, it was partially locked and put in "maintenance mode" only, meaning
that it would no longer evolve (features speaking).
Some of you may have eared, more than six years ago that Requests was working toward a major version, namely the V3.
There was even a completed fundraiser with a substantial amount, around $30k.
At the time, there was a plan for bringing Async, and HTTP/2 support (most notably). You'll guess that none of exposed promises ever landed.
Impacts
This incident led to several impacts.
- Maintainers felt too much pressure on making steps toward a major, some of them said: "We just can't deliver the promises, and it's going hard to explain why".
- All the back pressure landed at urllib3.
- Abruptly started to refuse any changes, saying: "Requests is feature complete, and it's popularity makes it nearly impossible to fix bugs that may be considered feature to this point".
- Newer project started ditching HTTP client Requests by learning to use a new one.
- Some large scale projects bound to Requests did not migrate because of the potential high cost doing it.
Let's be clear, this article is not about saying "Requests maintainers are bad" but rather saying we don't agree, and everyone is entitled to his choices.
For many years now, Requests has been frozen. Being left in a vegetative state and not evolving, this blocked millions of developers from using more advanced features.
The Journey into Waking Up
Initially, like many other reasonable person, I said to myself "Let's wait, Requests will eventually unblock the situation!"
Yet the years are still passing and nothing.
Let's realize something, Internet Explorer 11 just celebrated its tenth anniversary last year, and you’ll never guess that it
has support for HTTP/2 integrated! Yet Requests is unable to do so.
I said to myself: "Why are we waiting for this indefinitely? They don't owe us anything! There's a reasonable chance we are waiting... trapped in a circular reference.. Let's act."
Then, I tried to get a firm grip on urllib3 base code, contributing this and there until I was ready to kick things up with a
proof of concept that would have put urllib3 far ahead. Without any breaking changes.
I was delusional. This was a bit of a shock, but six months passed between my initial kick off and my formal give up, and here's why in a nutshell:
- Every bits of idea brought to the table was dismissed, and never seriously analysed.
This felt as good as taking the coldest wind on your face while exiting your warm home. It's the hard truth of the OSS,
it's not a democracy and it's not ruled by pure logic, but humans.
No far latter on, urllib3 called out a fundraiser to help with HTTP/2 support (only) and claimed around $40k to be able to deliver. They probably meant well, but[...]
This was the last straw for me, knowing the past mistakes, and actually the best incentive to break the cycle. Everything point to another year before having the slightest sign of a PRI * HTTP/2.0\r\n\r\nSM\r\n\r\n into our production pipes. They clearly lack the time to move forward (or incentives? or both?). No worries, as developers, we should share this burden for the greater good.
From my perspective (my opinion only), the OSS ecosystem should not fund raise money for a decade old protocol, and most of time should deliver prior to asking. Entities raising money with no solid proof of concept, that's rare!
This felt like a bit of a hostage situation, but it could only be me seeing it this way.
The Solution
So many years and various expertise have been brought to Requests and urllib3.. Do we throw all of it into the garbage disposal? Engineers should be reasonable scientists and not zeled revisionists.
We don't have to reinvent the wheel all over again, HTTP client Requests is well established and really pleasant in its usage.
We believe that Requests has the most inclusive and developer friendly interfaces.
So how to propose an attractive continuity for Requests? How hard could this be to (i) bring a true upgrade experience and (ii) keeping the compatibility as high as possible?
Sometime the OSS ecosystem act like it is trying to land a rover on the Moon, let's be more realistic and bring our vision down to firm grounds.
There's a chance offered to us to rehabilitate Requests (and urllib3) and give a fresh breath of air to us, the developers. Without having to learn something as basic as handling a HTTP client.
So, the work started, without looking back.
Just around my intent to give up on urllib3 evolution, we proposed a fork of both urllib3 and Requests, namely urllib3-future and Niquests.
They are (truly) drop-in replacements, at the time of writing, those forks are six-months old! And the whole work around this is roughly a year old. We listened to what the community needed by... triaging all the issues on major repositories (Requests, urllib3, and dependants of them) so that the upgrade would be truly useful.
Are you seeing it coming? Great! Let's look at the feature table comparison against requests, httpx and aiohttp!
| Feature | niquests | requests | httpx | aiohttp |
|---|---|---|---|---|
HTTP/1.1 |
✅ | ✅ | ✅ | ✅ |
HTTP/2 |
✅ | ❌ | ✅ | ❌ |
HTTP/3 over QUIC |
✅ | ❌ | ❌ | ❌ |
Synchronous |
✅ | ✅ | ✅ | ❌ |
Asynchronous |
✅ | ❌ | ✅ | ✅ |
Thread Safe |
✅ | ✅ | ❌ | N/A |
Task Safe |
✅ | N/A | ✅ | ✅ |
OS Trust Store |
✅ | ❌ | ❌ | ❌ |
Multiplexing |
✅ | ❌ | Limited | ❌ |
DNSSEC |
✅ | ❌ | ❌ | ❌ |
Customizable DNS Resolution |
✅ | ❌ | ❌ | ✅ |
DNS over HTTPS |
✅ | ❌ | ❌ | ❌ |
DNS over QUIC |
✅ | ❌ | ❌ | ❌ |
DNS over TLS |
✅ | ❌ | ❌ | ❌ |
Multiple DNS Resolver |
✅ | ❌ | ❌ | ❌ |
Network Fine Tuning & Inspect |
✅ | ❌ | Limited | Limited |
Certificate Revocation Protection |
✅ | ❌ | ❌ | ❌ |
Session Persistence |
✅ | ✅ | ✅ | ✅ |
In-memory Certificate CA & mTLS |
✅ | ❌ | Limited | Limited |
SOCKS 4/5 Proxies |
✅ | ✅ | ✅ | ❌ |
HTTP/HTTPS Proxies |
✅ | ✅ | ✅ | ✅ |
TLS-in-TLS Support |
✅ | ✅ | ✅ | ✅ |
Direct HTTP/3 Negotiation |
✅ | N/A | N/A | N/A |
Happy Eyeballs |
✅ | ❌ | ❌ | ✅ |
Package / SLSA Signed |
✅ | ❌ | ❌ | ✅ |
And you would think... performance..? are bad?
Look at the performance comparison against them!
Scenario: Fetch a thousand requests using 10 tasks or threads, each with a hundred requests using a single pool of 10 connection.
High-Level APIs
| Client | Average Delay to Complete | Notes |
|---|---|---|
| requests | 987 ms | ThreadPoolExecutor. HTTP/1.1 |
| httpx | 735 ms | Asyncio. HTTP/2 |
| niquests | 470 ms | Asyncio. HTTP/2 |
Simplified APIs
| Client | Average Delay to Complete | Notes |
|---|---|---|
| requests core | 643 ms | ThreadPoolExecutor. HTTP/1.1 |
| httpx core | 550 ms | Asyncio. HTTP/2 |
| aiohttp | 220 ms | Asyncio. HTTP/1.1 |
| niquests core | 210 ms | Asyncio. HTTP/2 |
Did you give up on HTTP/2 due to performance concerns? Think again! Multiplexing and response laziness open up a wide range
of possibilities!
And everything you are witnessing is in fact "Requests"-extended and you have nothing to learn again to get started. We kept 98% of the test cases from both Requests and urllib3, then extended them to cover newer features.
We are looking forward to see you at the Niquests repository! Let's say it's a promotional offer, get everything and more for $̶4̶0̶k̶ $0!
- #1 You depends on Requests? Discover Niquests instead.
- Drop-in replacement, simple patch needed, tiny effort required.
- #2 You depends on urllib3? Discover urllib3.future instead.
- Drop-in replacement, no patch required, atom-like sized effort required.
By visiting the GitHub repository and documentation you will be able to learn more about this, and hopefully make a decisive switch!
Top comments (0)