loading...
Cover image for HTTPX: Help Build The Future Of Python HTTP

HTTPX: Help Build The Future Of Python HTTP

florimondmanca profile image Florimond Manca ・4 min read

GitHub logo encode / httpx

A next generation HTTP client for Python. 🦋

HTTPX

HTTPX - A next-generation HTTP client for Python.

Test Suite Package version

HTTPX is a fully featured HTTP client for Python 3, which provides sync and async APIs, and support for both HTTP/1.1 and HTTP/2.

Note: HTTPX should be considered in beta. We believe we've got the public API to a stable point now, but would strongly recommend pinning your dependencies to the 0.13.* release, so that you're able to properly review API changes between package updates. A 1.0 release is expected to be issued sometime around mid-2020.


Let's get started...

>>> import httpx
>>> r = httpx.get('https://www.example.org/')
>>> r
<Response [200 OK]>
>>> r.status_code
200
>>> r.headers['content-type']
'text/html; charset=UTF-8'
>>> r.text
'<!doctype html>\n<html>\n<head>\n<title>Example Domain</title>...'

Or, using the async API...

Use IPython

HTTPX 🦋 is a next generation HTTP client for Python. It is Requests-compatible, supports HTTP/2 and HTTP/1.1, has async support, and an ever-expanding community of 35+ awesome contributors. ❤

HTTPX is part of a growing ecosystem of async-capable Python projects lead by the Encode organization (which backs the Django REST Framework). Some of the most notable projects include Uvicorn, Starlette, and Databases.

Hacktoberfest started yesterday, and on behalf of the HTTPX maintenance team I can say we're very excited to be part of it this year!

Before getting to the contributing guide, a short preview of what HTTPX can do…

Your first request to dev.to using HTTPX:

>>> import httpx
>>> r = httpx.get('https://dev.to')
>>> r.status_code
200
>>> r.headers['content-type']
'text/html; charset=utf-8'
>>> r.text[:50]
'<!DOCTYPE html>\n<html lang="en">\n  <head>\n    <met'
>>> r.http_version
'HTTP/1.1'

Does DEV support HTTP/2?

>>> import httpx
>>> # The top-level API only supports HTTP/1.1 for efficiency reasons,
>>> # but 'Client' uses HTTP/2 by default.
>>> with httpx.Client() as client:
...    r = client.get('https://dev.to')
...
>>> r.http_version
'HTTP/2'  # YAY! 🎉

HTTPX can even call into Python web apps (WSGI + ASGI). A Flask example…

>>> from flask import Flask
>>> import httpx
>>> app = Flask(__name__)
>>> @app.route("/")
... def hello():
...     return "Hello, world"
...
>>> with httpx.Client(app=app, base_url='http://testserver') as client:
...     r = client.get('/')
...
>>> r.status_code
200
>>> r.text
'Hello, world'  # Tada! 🦋
>>>

And it supports async/await! Let's fire an IPython REPL to try this out:

In [1]: import httpx
In [2]: import asyncio
In [3]: # We're going to fetch tag pages concurrently...
In [4]: async def fetch(tag, client):
   ...:     return await client.get(f'https://dev.to/t/{tag}')
   ...:

In [5]: async with httpx.AsyncClient() as client:
   ...:     responses = await asyncio.gather(
   ...:         fetch('hacktoberfest', client),
   ...:         fetch('python', client),
   ...:         fetch('opensource' client),
   ...:     )

In [6]: responses
Out[6]: [<Response [200 OK]>, <Response [200 OK]>, <Response [200 OK]>]
In [7]: [r.url for r in responses]
Out[7]:
[URL('https://dev.to/t/hacktoberfest'),
 URL('https://dev.to/t/python'),
 URL('https://dev.to/t/opensource')]

(This example uses asyncio, but trio is also supported!)

HTTPX already has a lot of features built-in, but it's still in an alpha stage as some of the areas still need more work.

So enough of an intro, here's how YOU can help us build the future of Python HTTP.

How you can help

Try it out!

The most straight-forward way to help move HTTPX forward is by trying it out for yourself.

For example:

  • Run the sample code in the introduction to this post to see HTTPX in action!
  • Build a simple application that calls a public API using HTTPX.
  • Migrate an application that uses Requests to make HTTP calls over to HTTPX.
  • Use HTTPX as a test client for your web project powered by Flask, Starlette, FastAPI, etc. More on this in Calling into Python web apps.
  • Build a web spider that fetches web pages in parallel using HTTPX.
  • ... And any other idea you feel like trying!

However you decide to try HTTPX out, we'd love to hear about it! Send us a link to a repo, blog post, or anything else you'd like to share via the comments section. :-)

Contribute!

As of writing this post, there's a whopping list of 10 hacktoberfest-labelled issues, all up for grabs! Expect more to come!

Below are some highlights of what's available at the moment.

Documentation

Documentation is critical to any software project. It helps new users get started quickly, and existing users find the information they need on how to use a particular feature or piece of API.

We know that the HTTPX documentation needs refining in several places. Feel free to walk through it and submit PRs for anything you think can be improved — be it typos, copy, or missing information. :-)

.post(), .put(), and .patch() functions in API docs missing 'files' parameter #409

Our API docs are missing the files parameter on .post(), .patch(), and .put() for Client and AsyncClient

Environment Variables docs should justify each usage case. #404

Our environment variable documentation should include some justification or context for each case, by doing one of the following for each envvar:

  • Make it clear that an environment variable is specific to HTTPX. (Probably only relevant to anything prefixed by HTTPX_)
  • Make it clear that an environment variable is used as a convention by various tooling, with at least one relevant hyperlink.

Eg.

SSLKEYLOGFILE could give some more context about usage - "The keylog file is designed for debugging purposes only." doesn't make it clear how or when to use it, or which debug tooling it's usful with.

SSL_CERT_FILE and SSL_CERT_DIR can include a link to some docs about their convention. Perhaps give an example of other tooling that supports them?

HTTP_PROXY, HTTPS_PROXY, ALL_PROXY can include a link to some docs about their convention. Perhaps give an example of other tooling that supports them?

More explicit documentation that `httpx.Client` is equivelent to `requests.Session`. #403

This isn't an issue (sorry if i posted in a wrong place), but i read the httpx doc and didn't find yet a solution. In requests i use s = requests.session(), how can i do the same thing with httpx? it will be possible to do something like s = httpx.session()? Thanks and sorry again.

Bug fixes

Fact: all software has bugs 🐛, and surely HTTPX is no different!

Some of the bugs have already been tracked…

Host header contains userinfo component #391

BaseRequest.prepare() incorrectly sets the default Host header to userinfo@example.com:123 if given the following URL: http://userinfo@example.com:123.

We should be removing the userinfo component from the authority before assigning to Host.

…But most of them still need to be discovered, tracked down, and fixed!

This is also why it's super important that you give HTTPX a try. Your use case may not be one we have thought about yet, or it might be in conditions that haven't be accounted for yet.

If you find about something that doesn't look right, please open an issue and we'll look into it together!

New features

HTTPX already has a lot of features built-in, but we've got ideas on how to push things even further.

Below are some features you can tackle. If you try out HTTPX and feel like something's missing, please open an issue! We'll discuss it together and will gladly help you get started if you'd like to implement the feature.

Add support for the NO_PROXY environment variable #360

The NO_PROXY environment variable is described here:

In order to accomplish this task we'd need:

  • Implementing NO_PROXY, a list of hosts that will bypass the proxy if found within the list.
  • Only enabled if trust_env=True
  • Documentation updates for the new environment variable.
  • Test cases for all code branches.

[Feature Request]: raise httpx exception instead of buildin ConnectionResetError and ConnectionError #349

This is a feature requests instead of bug report.

I'm using requests in my web server to request upstream api. So I can handler all requests.RequestException in my middleware and return a 502 upstream error.

But after I migrated to httpx, httpx doesn't have such exception when issuing a http(s) request, it will raise httpx.ConnectTimeout, buildin ConnectionError and ConnectResetError. And last 2 exception are not only raised by httpx.

Maybe httpx should also wrap ConnectionError and other buildin connection exception?

Closing thoughts

There are many ways to contribute to HTTPX, and all contributions are welcome! Once you've selected an issue, documentation improvement, feature request or bug report you'd like to work on, read the contributing guide to get started, and hack away!

Thank you so much for reading through this post, and we look forward to seeing the amazing contributions you'll make. 💖🦋

— The HTTPX maintenance team

Discussion

pic
Editor guide