DEV Community: Hugo van Kemenade

Moved to hugovk.dev

Hugo van Kemenade — Thu, 02 Jan 2025 19:51:13 +0000

🚀 See you over at hugovk.dev!

A surprising thing about PyPI's BigQuery data

Hugo van Kemenade — Sun, 24 Nov 2024 20:45:31 +0000

Moved to https://hugovk.dev/blog/2024/a-surprising-thing-about-pypis-bigquery-data/

You can get download numbers for PyPI packages (or projects) from a Google BigQuery dataset. You need a Google account and credentials, and Google gives 1 TiB of free quota per month.

Each month, I have automation to fetch the download numbers for the 8,000 most popular packages over the past 30 days, and make it available as more accessible JSON and CSV files at Top PyPI Packages. This data is widely used for research in academia and industry.

However, as more packages and releases are uploaded to PyPI, and there are more and more downloads logged, the amount of billed data increases too.

This chart shows the amount of data billed per month.

At first, I was only collecting downloads data for 4,000 packages, and it was fetched for two queries: downloads over 365 days and over 30 days. But as time passed, it started using up too much quota to download data for 365 days.

So I ditched the 365-day data, and increased the 30-day data from 4,000 to 5,000 packages. Later, I checked how much quota was being used and increased from 5,000 packages to 8,000 packages.

But then I exceeded the BigQuery monthly quota of 1 TiB fetching data for July 2024.

To fetch the missing data and investigate what's going in, I started Google Cloud's 90-day, $300 (€277.46) free-trial 💸

Here's what I found!

Finding: it costs more to get data for downloads from only pip than from all installers

I use the pypinfo client to help query BigQuery. By default, it only fetches downloads for pip.

Only pip

This command gets one day's download data for the top 10 packages, for pip only:

$ pypinfo --limit 10 --days 1 "" project
Served from cache: False
Data processed: 58.21 GiB
Data billed: 58.21 GiB
Estimated cost: $0.29

Results:

project	download count
boto3	37,251,744
aiobotocore	16,252,824
urllib3	16,243,278
botocore	15,687,125
requests	13,271,314
s3fs	12,865,055
s3transfer	12,014,278
fsspec	11,982,305
charset-normalizer	11,684,740
certifi	11,639,584
Total	158,892,247

All installers

Adding the --all flag gets one day's download data for the top 10 packages, for all installers:

$ pypinfo --all --limit 10 --days 1 "" project
Served from cache: False
Data processed: 46.63 GiB
Data billed: 46.63 GiB
Estimated cost: $0.23

project	download count
boto3	39,495,624
botocore	17,281,187
urllib3	17,225,121
aiobotocore	16,430,826
requests	14,287,965
s3fs	12,958,516
charset-normalizer	12,781,405
certifi	12,647,098
setuptools	12,608,120
idna	12,510,335
Total	168,226,197

So we can see the default pip-only costs an extra 25% data processed and data billed, and costs an extra 25% in dollars.

Unsurprisingly, the actual download counts are higher for all installers. The ranking has changed a bit, but I expect we're still getting more-or-less the same packages in the top thousands of results.

Queries

It sends a query like this to BigQuery for only pip:

SELECT
  file.project as project,
  COUNT(*) as download_count,
FROM `bigquery-public-data.pypi.file_downloads`
WHERE timestamp BETWEEN TIMESTAMP_ADD(CURRENT_TIMESTAMP(), INTERVAL -2 DAY) AND TIMESTAMP_ADD(CURRENT_TIMESTAMP(), INTERVAL -1 DAY)
  AND details.installer.name = "pip"
GROUP BY
  project
ORDER BY
  download_count DESC
LIMIT 10

And for all installers:

SELECT
  file.project as project,
  COUNT(*) as download_count,
FROM `bigquery-public-data.pypi.file_downloads`
WHERE timestamp BETWEEN TIMESTAMP_ADD(CURRENT_TIMESTAMP(), INTERVAL -2 DAY) AND TIMESTAMP_ADD(CURRENT_TIMESTAMP(), INTERVAL -1 DAY)
GROUP BY
  project
ORDER BY
  download_count DESC
LIMIT 10

These queries are the same, except the default has an extra AND details.installer.name = "pip" condition. It seems reasonable it would cost more to do extra filtering work.

Installers

Let's look at the installers:

$ pypinfo --all --limit 100 --days 1 "" installer
Served from cache: False
Data processed: 29.49 GiB
Data billed: 29.49 GiB
Estimated cost: $0.15

installer name	download count
pip	1,121,198,711
uv	117,194,833
requests	29,828,272
poetry	23,009,454
None	8,916,745
bandersnatch	6,171,555
setuptools	1,362,797
Bazel	1,280,271
Browser	1,096,328
Nexus	593,230
Homebrew	510,247
Artifactory	69,063
pdm	62,904
OS	13,108
devpi	9,530
conda	2,272
pex	194
Total	1,311,319,514

pip still by far the most popular, and unsurprising uv is up there too, with about 10% of pip's downloads.

The others are about 25% or less of uv. A lot of them are mirroring services that we wanted to exclude before.

I think given uv's importance, and my expectation that it will continue to take a bigger share of the pie, plus especially the extra cost for filtering by just pip, means that we should switch to fetching data for all downloaders. Plus the others don't account for that much of the pie.

Finding: the number of packages doesn't affect the cost

This was the biggest surprise. Earlier I'd been increasing or decreasing the number to try and remain under quota. But it turns out it makes no difference how many packages you query!

I fetched data for just one day and all installers for different package limits: 1000, 2000, 3000, 4000, 5000, 6000, 7000, 8000. Sample query:

SELECT
  file.project as project,
  COUNT(*) as download_count,
FROM `bigquery-public-data.pypi.file_downloads`
WHERE timestamp BETWEEN TIMESTAMP_ADD(CURRENT_TIMESTAMP(), INTERVAL -2 DAY) AND TIMESTAMP_ADD(CURRENT_TIMESTAMP(), INTERVAL -1 DAY)
GROUP BY
  project
ORDER BY
  download_count DESC
LIMIT 8000

Result: Interestingly, the cost is the same for all limits (1000-8000): $0.31.

Repeating with one day but filtering for pip only:

Result: Cost increased to $0.39 but again the same for all limits.

Let's repeat with all installers, but for 30 days, and this time query in decreasing limits, in case we were only paying for incremental changes: 8000, 7000, 6000, 5000, 4000, 3000, 2000, 1000:

Result: Again, the cost is the same regardless of package limit: $4.89 per query.

Well then, let's repeat with the limit increasing by powers of ten, up to 1,000,000! This last one fetches data for all 531,022 packages on PyPI:

limit	projects count	estimated cost	bytes billed	bytes processed
1	1	0.20	43,447,746,560	43,447,720,943
10	10	0.20	43,447,746,560	43,447,720,943
100	100	0.20	43,447,746,560	43,447,720,943
1000	1,000	0.20	43,447,746,560	43,447,720,943
8000	8,000	0.20	43,447,746,560	43,447,720,943
10000	10,000	0.20	43,447,746,560	43,447,720,943
100000	100,000	0.20	43,447,746,560	43,447,720,943
1000000	531,022	0.20	43,447,746,560	43,447,720,943

Result: Again, same cost, whether for 1 package or 531,022 packages!

Finding: the number of days affects the cost

No surprise. I'd earlier noticed 365 days too took much quota, and I could continue with 30 days.

Here's the estimated cost and bytes billed (for one package, all installers) between one and 30 days (f"pypinfo --all --json --indent 0 --days {days} --limit 1 '' project"), showing a roughly linear increase:

Conclusion

It doesn't matter how many packages I fetch data for, I might as well fetch all and make it available to everyone, depending on the size of the data file. It will make sense to still offer a smaller file with 8,000 or so packages: often you just need a large-ish yet manageable number.
It costs more to filter for only downloads from pip, so I've switched to fetching data for all installers.
The number of days affects the cost, so I will need to decrease this in the future to stay within quota. For example, at some point I may need to switch from 30 to 25 days, and later from 25 to 20 days.

More details from the investigation, the scripts and data files can be found at
hugovk/top-pypi-packages#36.

And let me know if you know any tricks to reduce costs!

Header photo: "The Balancing Rock, Stonehenge, Near Glen Innes, NSW" by the Royal Australian Historical Society, with no known copyright restrictions.

Speed up CI with uv ⚡

Hugo van Kemenade — Sat, 02 Nov 2024 13:11:22 +0000

Moved to https://hugovk.dev/blog/2024/speed-up-ci-with-uv/

We can use uv to make linting and testing on GitHub Actions around 1.5 times as fast.

Linting

When using pre-commit for linting:

name: Lint

on: [push, pull_request, workflow_dispatch]

env:
  FORCE_COLOR: 1

permissions:
  contents: read

jobs:
  lint:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
        with:
          persist-credentials: false
      - uses: actions/setup-python@v5
        with:
          python-version: "3.x"
          cache: pip
      - uses: pre-commit/action@v3.0.1

We can replace pre-commit/action with tox-dev/action-pre-commit-uv:

       - uses: actions/setup-python@v5
         with:
           python-version: "3.x"
-          cache: pip
-      - uses: pre-commit/action@v3.0.1
+      - uses: tox-dev/action-pre-commit-uv@v1

name: Lint

on: [push, pull_request, workflow_dispatch]

env:
  FORCE_COLOR: 1

permissions:
  contents: read

jobs:
  lint:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
        with:
          persist-credentials: false
      - uses: actions/setup-python@v5
        with:
          python-version: "3.x"
      - uses: tox-dev/action-pre-commit-uv@v1

This means uv will create virtual environments and install packages for pre-commit, which is faster for the initial seed operation when there's no cache.

Lint comparison

For example: python/blurb#32

	Before	After	Times faster
No cache	60s	37s	1.62
With cache	11s	11s	1.00

Testing

When testing with tox:

name: Test

on: [push, pull_request, workflow_dispatch]

permissions:
  contents: read

env:
  FORCE_COLOR: 1

jobs:
  test:
    runs-on: ubuntu-latest
    strategy:
      fail-fast: false
      matrix:
        python-version: ["3.9", "3.10", "3.11", "3.12", "3.13", "3.14"]

    steps:
      - uses: actions/checkout@v4
        with:
          persist-credentials: false

      - name: Set up Python ${{ matrix.python-version }}
        uses: actions/setup-python@v5
        with:
          python-version: ${{ matrix.python-version }}
          allow-prereleases: true
          cache: pip

      - name: Install dependencies
        run: |
          python --version
          python -m pip install -U pip
          python -m pip install -U tox

      - name: Tox tests
        run: |
          tox -e py

We can replace tox with tox-uv:

       - name: Set up Python ${{ matrix.python-version }}
         uses: actions/setup-python@v5
         with:
           python-version: ${{ matrix.python-version }}
           allow-prereleases: true
-          cache: pip

-      - name: Install dependencies
-        run: |
-          python --version
-          python -m pip install -U pip
-          python -m pip install -U tox
+      - name: Install uv
+        uses: hynek/setup-cached-uv@v2

       - name: Tox tests
         run: |
-          tox -e py
+          uvx --with tox-uv tox -e py

name: Test

on: [push, pull_request, workflow_dispatch]

permissions:
  contents: read

env:
  FORCE_COLOR: 1

jobs:
  test:
    runs-on: ubuntu-latest
    strategy:
      fail-fast: false
      matrix:
        python-version: ["3.9", "3.10", "3.11", "3.12", "3.13"]

    steps:
      - uses: actions/checkout@v4
        with:
          persist-credentials: false

      - name: Set up Python ${{ matrix.python-version }}
        uses: actions/setup-python@v5
        with:
          python-version: ${{ matrix.python-version }}
          allow-prereleases: true

      - name: Install uv
        uses: hynek/setup-cached-uv@v2

      - name: Tox tests
        run: |
          uvx --with tox-uv tox -e py

tox-uv is tox plugin to replace virtualenv and pip with uv in your tox environments. We only need to install uv, and use uvx to both install tox-uv and run tox, for faster installs of tox, the virtual environment, and the dependencies within it.

Test comparison

For example: python/blurb#32

	Before	After	Times faster
No cache	2m 0s	1m 26s	1.40
With cache	1m 58s	1m 22s	1.44

No pip with tox-uv

One difference with uv environments compared to regular venv/virtualenv ones is that they do not have pip. This means calls to pip need replacing, for example in tox.ini:

 [testenv]
-commands_pre =
-    {envpython} -m pip install -U -r requirements.txt
+deps =
+    -r requirements.txt
 pass_env =
     FORCE_COLOR
 commands =
     {envpython} -m pytest {posargs}

If you still need pip (or setuptools or wheel), add uv_seed = True to your [testenv] to inject them.

Bonus tip

Run the new tool zizmor to find security issues in GitHub Actions.

Header photo: "Road cycling at the 1952 Helsinki Olympics" by Olympia-Kuva Oy & Helsinki City Museum, Public Domain.

Python Core Developer Sprint 2024

Hugo van Kemenade — Sat, 05 Oct 2024 19:39:48 +0000

Moved to https://hugovk.dev/blog/2024/python-core-developer-sprint-2024/

🐍🏃The week before last was the annual Python Core Dev Sprint, graciously hosted by Meta in Bellevue, WA!

The idea: bring a bunch of Python core team members, triagers, and special guests to the same room for a week. It's hugely beneficial and productive, we held many in-depth discussions that just don't happen when we're all remote and async, and got to work on many different things together.

The sprint room

During the week, I reviewed 39 PRs, created 15, merged 10, updated 4, and closed 2 issues.

Monday highlights

As release manager for Python 3.14, I discussed with Brett Cannon one of his project ideas which will come after lock files, and after the next big one.

Also as RM, discussed with Russell Keith-Magee, Ned Deily, Łukasz Langa and Thomas Wouters about including official binaries for iOS and Android, which wandered into ideas about security releases.

I did some maintenance of our PyPI projects, adding PEP 740 attestations, support for the new Python 3.13 and dropping support for the very-nearly-EOL 3.8.

Tuesday highlights

Started investigating slow doctest on 3.13+ with Alex Waygood, who on Wednesday narrowed it down to a problem with the new incremental garbage collector, which would go on to be reverted by Friday and result in Python 3.13's Monday release to be postponed and replaced with an extra release candidate. Not ideal, but much better to discover these things before the big release.

We had a Q&A session with the Steering Council: Barry Warsaw, Emily Morehouse, Gregory P. Smith, Pablo Galindo Salgado and Thomas.

The Python Steering Council

Proofread Guido van Rossum's STAR voting proposal for electing future steering councils.

Discussed with Eric Snow his novel method for displaying many code samples in a table, using <details> disclosures to prevent the table being too wide. Looks like a good solution!

Wednesday highlights

I applied the finishing touches to PEP 2026 (Calendar versioning for Python) and Barry gave it a final review. Ready for submission!

Seth Larson, the PSF Security Developer-in-Residence, wasn't at the sprint but we discussed our plan to stop providing GPG signatures for CPython and rely on SigStore instead. Expect a PEP soon!

Also not at the sprint, I recommended PSF Infrastructure Engineer Jacob Coffee as a CPython triager. Welcome aboard!

The whole room discussed including static type annotations in CPython.

We had a Q&A session with two of the three Developers-in-Residence, Łukasz and Petr Viktorin.

Q&A with Łukasz and Petr

Discussed expanding the voter pool for Steering Council elections with Mariatta, Greg and Thomas.

Larry Hastings handed out, in return for oohs and aahs, some nice P.C.D.S. 2024 stickers he generously designed and printed up for us. Thanks!

PCDS 2024 stickers by Larry

Thursday highlights

On the 26th September, at 10:26 Bellevue time (20:26 Helsinki time), I submitted PEP 2026 to the Steering Council!🤞

Brett discussed whether we should update PEP 387 to prefer 5 year deprecations instead of 2 years.

Brandt Bucher gave us all an update on the progress of the Just-in-Time (JIT) compiler ("we went from 0% slower to 0% faster!") and we discussed plans for Python 3.14.

Because I couldn't attend Thursday's Helsinki Python meetup due to being at another kind of Python meetup on the other side of the world, I gave the famous HelPy quiz to the assembled core devs. Unsurprisingly they did pretty well, but the most incorrect answer was a pleasant surprise: we've had ~400 not ~80 new contributors to Python 3.13!

Pablo performed card tricks!

Magic from Pablo

Meta took us out for a delicious dinner at a local fish restaurant. Thank you!

Friday highlights

Mariatta presented ideas to Jelle Zijlstra, Petr, Russell and me about to use modern tools to create a modern, interactive tutorial.

Also during the week, continued work with Adam Turner on improving the docs.python.org build. Adam wasn't at the sprint, so tag-teamed PR reviews overnight. After much work straddling many teams, projects and repos, we've got the full HTML build loop for 13 languages × 3 versions down from over 40 hours to just under 9 hours, with more improvements coming.

Made a demo of the CPython docs using the PyData Sphinx Theme.

Along with around 25 others, I was on Łukasz and Pablo's core.py podcast.

Łukasz and Pablo in their ad-hoc podcast studio in a Meta meeting room

Itamar gave us cake for the podcast's first birthday!

cake.py. Photo by Itamar Oren.

Thank you

It was a hugely productive week, big thanks to Itamar Oren and Meta for organising and hosting!

See also Mariatta's excellent blog posts, and I recommend the core.py podcast with short interviews with some 25 attendees! Łukasz and Pablo were also guests on the Changelog podcast during the sprint.

Header photo by Itamar Oren

PyCon US 2024: A roundup of writeups

Hugo van Kemenade — Thu, 13 Jun 2024 14:41:55 +0000

Moved to https://hugovk.dev/blog/2024/pycon-us-2024-a-roundup-of-writeups/

If you read just one, check Kati's thorough recap!

22nd May 2024

Echos of the People API user guide by Ned Batchelder

24th May 2024

Wagtailers spread their wings at PyCon 2024 by Meagen Voss (@vossisboss)
Flet at PyCon US 2024 by Feodor Fitsner

27th May 2024

PyCon US 2024: My First PyCon in US 🫶🏻 by Tomy Hsieh

28th May 2024

PyCon 2024 Reflection by Trey Hunner
Weeknotes: PyCon US 2024 by Simon Willison
3 key takeaways from PyCon US 2024 by Luis Juncal & Yolanda Robla

30th May 2024

1st June 2024

PyCon US 2024 Recap by Kati Michel (@katherinemichel)

4th June 2024

PyCon US 2024 by Yilei Yang

13th June 2024

PyCon US 2024 as Security Developer-in-Residence by Seth Michael Larson (@sethmlarson)

14th June 2024

The Python Language Summit 2024 by Seth Michael Larson (@sethmlarson)

16th June 2024

PyCon US 2024 Highlights by David Foster

19th June 2024

From Pittsburgh to New York: A PyCon US 2024 Adventure by Monica Oyugi (@monicaoyugi)

Header photo: Downtown Pittsburgh seen between the Andy Warhol Bridge and Roberto Clemente Bridge

Help test Python 3.13!

Hugo van Kemenade — Tue, 04 Jun 2024 13:04:03 +0000

Moved to https://hugovk.dev/blog/2024/help-test-python-313/

Calling all Python library maintainers! 🐍

The Python 3.13 beta is out! 🎉

PEP 719 defines the release schedule for Python 3.13.0:

The first beta candidate came out on 8th May 2024
The first release candidate is set for 30th July 2024
And the full release is set for 1st October 2024

In his announcement, Thomas Wouters, release manager for Python 3.12 and 3.13, said:

We strongly encourage maintainers of third-party Python projects to test with 3.13 during the beta phase and report issues found to the Python bug tracker as soon as possible. While the release is planned to be feature complete entering the beta phase, it is possible that features may be modified or, in rare cases, deleted up until the start of the release candidate phase (Tuesday 2024-07-30). Our goal is to have no ABI changes after beta 4 and as few code changes as possible after 3.13.0rc1, the first release candidate. To achieve that, it will be extremely important to get as much exposure for 3.13 as possible during the beta phase.

Test with 3.13

It's now time for us library maintainers to start testing our projects with 3.13. There's two big benefits:

There have been removals and changes in Python 3.13. Testing now helps us make our code compatible and avoid any big surprises (for us and our users) at the big launch in October.
We might find bugs in Python itself! Reporting those will help get them fixed and help everyone.

How

GitHub Actions: setup-python

To test the latest alpha, beta or release candidate with actions/setup-python, add 3.13 and allow-prereleases: true to your workflow matrix.

For example:

jobs:
  test:
    runs-on: ubuntu-latest
    strategy:
      fail-fast: false
      matrix:
        python-version: ["3.10", "3.11", "3.12", "3.13"]

    steps:
      - uses: actions/checkout@v4

      - name: Set up Python ${{ matrix.python-version }}
        uses: actions/setup-python@v5
        with:
          python-version: ${{ matrix.python-version }}
          allow-prereleases: true

(We can instead use 3.13-dev and omit allow-prereleases: true, but I find the above a bit neater, and when 3.13.0 final is released in October, it will continue testing with full release versions.)

GitHub Actions: deadsnakes

For the bleeding edge, we can use deadsnakes/action to test the latest nightly build:

jobs:
  test:
    runs-on: ubuntu-latest
    strategy:
      fail-fast: false
      matrix:
        python-version: ["3.10", "3.11", "3.12", "3.13-dev"]

    steps:
      - uses: actions/checkout@v4

      - name: Set up Python ${{ matrix.python-version }}
        if: "!endsWith(matrix.python-version, '-dev')"
        uses: actions/setup-python@v5
        with:
          python-version: ${{ matrix.python-version }}

      - uses: deadsnakes/action@v3.1.0
        name: Set up Python ${{ matrix.python-version }} (deadsnakes)
        if: endsWith(matrix.python-version, '-dev')
        with:
          python-version: ${{ matrix.python-version }}

When to support 3.13?

When should you declare support and add the Programming Language :: Python :: 3.13 Trove classifier? Some projects already have!

If you have a pure Python project, you can release now.

If you have C extensions and other projects depend on yours, a preview release with wheels will help them test and prepare. I've already started releasing these.

ABI breaks?

ABI breaks during the beta are infrequent and unintentional. If they happen, you can rebuild your wheels and upload them to an existing PyPI release by adding an optional build tag to the filename:

The wheel filename is {distribution}-{version}(-{build tag})?-{python tag}-{abi tag}-{platform tag}.whl.

build tag: [...] Acts as a tie-breaker if two wheel file names are the same in all other respects (i.e. name, version, and other tags).

For example, this updates the filename and metadata with build number 1, and removes the original file:

python -m pip install "wheel>=0.4.0"
wheel tags --build=1 --remove sampleproject-5.0.0-cp313-cp313-macosx_10_10_x86_64.whl
# this creates a file named sampleproject-5.0.0-1-cp313-cp313-macosx_10_10_x86_64.whl

Upload it, and the new file will be used instead of the old one. See also Brett Cannon's advice on making new wheels.

In any case, let's start testing 3.13 now! 🚀

Printable PyCon 2024 schedule

Hugo van Kemenade — Sat, 11 May 2024 11:01:33 +0000

Moved to https://hugovk.dev/blog/2024/printable-pycon-2024-schedule/

Want to print out the PyCon US schedule? Paper doesn't run out of batteries, is easy to scribble on, and stuff into a pocket (technical term: the affordances of paper).

Here's some custom CSS and JavaScript to make it nicely printable.

Install the Styler browser extension
View a PyCon schedule page such as https://us.pycon.org/2024/schedule/talks/ and click the Styler extension's S icon
Paste this CSS into the upper box:

body.pycon-schedule div.internal-page-header,
body.pycon-schedule div.panel-heading,
body.pycon-schedule footer,
body.pycon-schedule div.badges {
  display: none;
}

body.pycon-schedule .container {
  margin: 0;
  max-width: fit-content;
}
body.pycon-schedule main.content {
  margin: 0;
  width: 100% !important;
}
body.pycon-schedule .calendar {
  left: auto !important;
  width: 100% !important;
}
body.pycon-schedule .slot {
  padding: 5px;
}
@media print {
  a:after { content:''; }
  a[href]:after { content: none !important; }
}

Paste this JavaScript into the lower box:

$(document).ready(function() {
    if (window.location.pathname.match('2024/schedule/') && 
       !window.location.pathname.match('2024/schedule/presentation/')) {
    $("body").addClass("pycon-schedule");
  }
});

Print!

It'll run on any of the https://us.pycon.org/2024/schedule/* pages, but not the individual presentation pages such as https://us.pycon.org/2024/schedule/presentation/61/

It's not perfect, the right edge is slightly cut off, but it's more printable than the original.

Once printed, you can click the Styler icon and the x button to disable Styler for the site so you can browse the original version.

Help us test free-threaded Python without the GIL

Hugo van Kemenade — Fri, 10 May 2024 12:18:27 +0000

Moved to https://hugovk.dev/blog/2023/help-us-test-free-threaded-python-without-the-gil/

Python 3.13 is due out in October 2024 and work is underway to implement experimental support for PEP 703 - Making the Global Interpreter Lock Optional in CPython.

See also Free-threaded CPython in "What's New in Python 3.13?"

As the Steering Council noted in their acceptance of the PEP, to succeed it's important to have community support.

Projects will need to test their code with free-threaded (aka "nogil" but don't call it that!) Python builds to help us find bugs in CPython, and to check their code is compatible.

Here's some ways to test.

Official installers
Build it yourself
deadsnakes
GitHub Actions?
Fedora

Official installers

The official macOS (starting in beta 2) and Windows installers has an option to install the free-threading binaries, which also installs python3.13t alongside the regular python3.13:

Build it yourself

Check out the CPython Git repo and build yourself using the --disable-gil configuration flag.

For example, on macOS I run:

$ GDBM_CFLAGS="-I$(brew --prefix gdbm)/include" \
  GDBM_LIBS="-L$(brew --prefix gdbm)/lib -lgdbm" \
  ./configure --config-cache \
              --with-system-libmpdec \
              --with-openssl="$(brew --prefix openssl@3.0)" \
              --disable-gil && make -s -j10
$ ./python.exe -c "import sysconfig; print(sysconfig.get_config_var('Py_GIL_DISABLED'))"
1
$ ./python.exe -c "import sys; print(sys._is_gil_enabled())"
False

More details in the devguide:

https://devguide.python.org/getting-started/setup-building/

GitHub Actions: deadsnakes/action

We can use the deadsnakes/action to test Ubuntu.

For example:

on: [push, pull_request, workflow_dispatch]

jobs:
  build:
    runs-on: ubuntu-latest
    strategy:
      matrix:
        python-version: [3.12, 3.13-dev]
    name: main
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        if: "!endsWith(matrix.python-version, '-dev')"
        with:
          python-version: ${{ matrix.python-version }}
      - uses: deadsnakes/action@v3.1.0
        if: endsWith(matrix.python-version, '-dev')
        with:
          python-version: ${{ matrix.python-version }}
          nogil: true
      - run: |
          python --version --version
          which python
          python -c "import sysconfig; print(sysconfig.get_config_var('Py_GIL_DISABLED'))"
          python -c "import sys; print(sys._is_gil_enabled())"

GitHub Actions: actions/setup-python

I've asked for support at actions/setup-python#771, they're looking into it 🤞

In the meantime, give it an upvote, and use deadsnakes/action ⤴️

deadsnakes: PPA

The deadsnakes project provides Personal Package Archives of Python packaged for Ubuntu, included free-threaded builds.

For example, python3.13-nogil under "python3.13 - 3.13.0~a2-1+focal1" and "python3.13 - 3.13.0~a2-1+jammy1" at https://launchpad.net/~deadsnakes/+archive/ubuntu/ppa/+packages

Fedora

Fedora uses the python3.13t executable name as decided by the Steering Council:

$ # Install
$ sudo dnf install python3.13-freethreading
$ # Run
$ /usr/bin/python3.13t -c "import sysconfig; print(sysconfig.get_config_var('Py_GIL_DISABLED'))"
1
$ /usr/bin/python3.13t -c "import sys; print(sys._is_gil_enabled())"
False

How to check your build

To confirm if you're using a free-threaded build, check the double --version option (starting in beta 2):

$ # Regular build
$ python3.13 --version --version
Python 3.13.0b2 (v3.13.0b2:3a83b172af, Jun  5 2024, 12:50:24) [Clang 15.0.0 (clang-1500.3.9.4)]
$ # Free-threaded build
$ python3.13t --version --version
Python 3.13.0b2 experimental free-threading build (v3.13.0b2:3a83b172af, Jun  5 2024, 12:57:31) [Clang 15.0.0 (clang-1500.3.9.4)]

Or the Py_GIL_DISABLED macro:

$ # Regular build
$ python3.13 -c "import sysconfig; print(sysconfig.get_config_var('Py_GIL_DISABLED'))"
0
$ # Free-threaded build
$ python3.13t -c "import sysconfig; print(sysconfig.get_config_var('Py_GIL_DISABLED'))"
1

By default, the GIL is disabled for free-threaded builds, and can be re-enabled by setting the PYTHON_GIL environment variable to 1 or running Python with -X gil 1. You can check with sys._is_gil_enabled():

$ # Regular build: GIL is always enabled
$ python3.13 -c "import sys; print(sys._is_gil_enabled())"
True
$ # Free-threaded build: GIL is disabled by default
$ python3.13t -c "import sys; print(sys._is_gil_enabled())"
False
$ # Free-threaded build: re-enable with -X
$ python3.13t -X gil=1 -c "import sys; print(sys._is_gil_enabled())"
True
$ # Free-threaded build: re-enable with env var
$ PYTHON_GIL=1 python3.13t -c "import sys; print(sys._is_gil_enabled())"
True

References:

Sphinx docs: How to activate tabs for your OS

Hugo van Kemenade — Tue, 02 Apr 2024 18:41:30 +0000

Moved to https://hugovk.dev/blog/2024/sphinx-docs-how-to-activate-tabs-for-your-os/

On the Python Developer's Guide and Pillow documentation we have some pages with tabs for different operating systems:

It's possible to add some JavaScript so that the matching tab is activated based on the visitor's operating system.

Here's how!

Sphinx Inline Tabs

First add the Sphinx Inline Tabs extension to your docs' requirements.txt:

# requirements.txt
sphinx-inline-tabs>=2023.4.21

JavaScript

Next, add activate_tab.js to your _static/ directory:

// activate_tab.js
// Based on https://stackoverflow.com/a/38241481/724176
function getOS() {
  const userAgent = window.navigator.userAgent,
    platform =
      window.navigator?.userAgentData?.platform || window.navigator.platform,
    macosPlatforms = ["macOS", "Macintosh", "MacIntel", "MacPPC", "Mac68K"],
    windowsPlatforms = ["Win32", "Win64", "Windows", "WinCE"],
    iosPlatforms = ["iPhone", "iPad", "iPod"];

  if (macosPlatforms.includes(platform)) {
    return "macOS";
  } else if (iosPlatforms.includes(platform)) {
    return "iOS";
  } else if (windowsPlatforms.includes(platform)) {
    return "Windows";
  } else if (/Android/.test(userAgent)) {
    return "Android";
  } else if (/Linux/.test(platform)) {
    return "Unix";
  }

  return "unknown";
}

function activateTab(tabName) {
  // Find all label elements containing the specified tab name
  const labels = document.querySelectorAll(".tab-label");

  labels.forEach((label) => {
    if (label.textContent.includes(tabName)) {
      // Find the associated input element using the 'for' attribute
      const tabInputId = label.getAttribute("for");
      const tabInput = document.getElementById(tabInputId);

      // Check if the input element exists before attempting to set the "checked" attribute
      if (tabInput) {
        // Activate the tab by setting its "checked" attribute to true
        tabInput.checked = true;
      }
    }
  });
}

`conf.py`

Add the extension and JavaScript to your conf.py:

# conf.py
extensions = [
    "sphinx_inline_tabs",
]

html_js_files = [
    "activate_tab.js",
]

reStructuredText

Almost there!

Add tabs to the reStructuredText files.

For example, here we have three different commands; one for Unix, one for macOS, and one for Windows:

.. tab:: Unix

    .. code-block:: shell

        ./python -m test -h

.. tab:: macOS

    .. code-block:: shell

        ./python.exe -m test -h

.. tab:: Windows

    .. code-block:: dosbatch

        .\python.bat -m test -h

Finally, add the JavaScript to the same reST page to activate the correct tab:

.. raw:: html

    <script>
    document.addEventListener('DOMContentLoaded', function() {
      activateTab(getOS());
    });
    </script>

You can see the results here. When the page loads, the browser finds the browser name, and activates the tabs with a matching name.

If you have many sets of tabs on a page, the corresponding OS tab will be activated for all. And if you click on another OS tab, all the others with the same name are activated.

Happy tabbing!

Header photo: "The Great Pyramid and the Sphinx" by National Galleries of Scotland, with no known copyright restrictions.

Tech style guides

Hugo van Kemenade — Sat, 02 Mar 2024 15:25:54 +0000

Moved to https://hugovk.dev/blog/2023/tech-style-guides/

Here's some tech style guides:

And some other style guides:

TIL: exclude_also with coverage.py

Hugo van Kemenade — Mon, 06 Nov 2023 20:05:55 +0000

Moved to https://hugovk.dev/blog/2023/til-excludealso-with-coveragepy/

Sometimes you have code you want to exclude from the test coverage report, because it doesn't really make sense to test it.

For example, maybe you want to exclude:

if __name__ == "__main__":
    main()

The old advice was to add something like this to .coveragerc:

[report]
# Regexes for lines to exclude from consideration
exclude_lines =
    # Have to re-enable the standard pragma:
    pragma: no cover

    # Don't complain if non-runnable code isn't run:
    if __name__ == .__main__.:

But since coverage.py 7.2.0 (2023-02-22) you can use exclude_also instead and skip that pragma:

[report]
# Regexes for lines to exclude from consideration
exclude_also =
    # Don't complain if non-runnable code isn't run:
    if __name__ == .__main__.:

Which is:

 [report]
 # Regexes for lines to exclude from consideration
-exclude_lines =
-    # Have to re-enable the standard pragma:
-    pragma: no cover
-
+exclude_also =
     # Don't complain if non-runnable code isn't run:
     if __name__ == .__main__.:

Thanks

To Brian Okken for the tip.

To Ned Batchelder for maintaining Coverage.py.

To the Library of Congress and Flickr Commons for the photo of a covered wagon.

Why are there still so many downloads for EOL Python 3.7?

Hugo van Kemenade — Tue, 22 Aug 2023 19:25:29 +0000

Moved to https://hugovk.dev/blog/2023/why-are-there-still-so-many-downloads-for-eol-python-37/

Python 3.7 was first released on 2018-06-27 and recently reached end-of-life on 2023-06-27 (PEP 537).

This means it is no longer receiving security updates and you should upgrade to a newer version (at least 3.8, but preferably 3.11):

Source: Python Developer's Guide

However, if you look at download numbers from PyPI, 3.7 still accounts for a large share. 3.7 accounted for 25% of all downloads from PyPI in July 2023, compared with 27% for 3.8:

Source: pypi-tools

But why does such an old Python version have so many downloads?

All downloads

Let's dig into the numbers using a handy tool called pypinfo, which helps us analyse the PyPI data from Google BigQuery.

This command fetches all of yesterday's downloads per Python version:

pypinfo --days 1 --percent --markdown "" pyversion

Python version	percent	download count
3.8	25.00%	189,678,872
3.7	23.35%	177,150,010
3.9	20.25%	153,663,903
3.10	15.52%	117,751,108
3.11	6.90%	52,381,884
3.6	6.29%	47,749,879
2.7	2.32%	17,602,650
3.5	0.23%	1,778,388
3.4	0.10%	770,135
3.12	0.03%	224,223
3.13	0.00%	3,920
3.3	0.00%	1,165
2.8	0.00%	57
3.2	0.00%	35
None	0.00%	3
Total		758,756,232

All downloads by OS

But what happens if we check which distros are responsible for those downloads?

This command gives us the top 20:

pypinfo --days 1 --limit 20 --percent --markdown "" system distro pyversion

system name	distro name	Python version	percent	download count
Linux	Ubuntu	3.8	18.87%	128,991,062
Linux	Amazon Linux	3.7	14.44%	98,738,952
Linux	Ubuntu	3.9	12.14%	83,019,828
Linux	Ubuntu	3.10	9.66%	66,019,309
Linux	Ubuntu	3.7	5.89%	40,257,060
Linux	Debian GNU/Linux	3.8	5.70%	38,958,367
Linux	Debian GNU/Linux	3.9	5.63%	38,482,436
Linux	Debian GNU/Linux	3.7	4.25%	29,035,532
Linux	Debian GNU/Linux	3.10	4.15%	28,348,346
Linux	Debian GNU/Linux	3.6	3.14%	21,441,883
Linux	Ubuntu	3.11	3.01%	20,570,619
Linux	Debian GNU/Linux	3.11	2.72%	18,588,584
Linux	Amazon Linux	3.9	2.43%	16,593,595
Linux	CentOS Linux	3.6	1.70%	11,605,142
Linux	Amazon Linux	3.8	1.47%	10,035,087
Linux	Amazon Linux	3.10	1.46%	9,969,514
Linux	Ubuntu	2.7	1.02%	6,960,697
Linux	Amazon Linux AMI	3.6	0.79%	5,390,823
Linux	Ubuntu	3.6	0.79%	5,388,370
Windows	None	3.10	0.76%	5,227,519
Total				683,622,725

We can see Ubuntu with 3.8 is responsible for the largest share of 17%. (That's fine, 3.8 is supported until 2024-10-14.)

The next is Amazon Linux with 3.7, responsible for a whopping 15% of all downloads!

The others responsible for 3.7 have a much lower share: Ubuntu (6%) and Debian (4%).

Tip: replace "" in the commands above with a package name to get data for just that package, for example:

pypinfo --days 1 --limit 20 --percent --markdown requests system distro pyversion

Header photo: Space shuttle Discovery landing at Edwards Air Force Base, California, 9th December, 1992 (source: NASA on The Commons