DEV Community: Lucy Linder

Understanding Cloudflare Caching: What Gets Cached and How to Control It

Lucy Linder — Mon, 17 Mar 2025 13:10:00 +0000

I used to think that Cloudflare caching just worked out of the box - set your website behind Cloudflare, and boom! Your HTML and media pages are cached, performance increases, and your boss is happy.

Well… not quite. Cloudflare is a great tool, but to truly take advantage of it, you must take the time to understand how it works. Especially, you may have to set up a somewhat counter-intuitive rule. Curious? Let’s break it down!

ℹ BEFORE WE START ℹ This article is far from complete, I couldn’t cover all of Cloudflare’s cache nuances without losing my audience (and myself). Instead, I will focus on key concepts and configurations, assuming you already know how to set up an application behind Cloudflare.

In this article:

Cloudflare Cache Statuses
What is a “resource”
Which resources are eligible for cache
Which eligible resources are cached
How to extend cached resources to HTML (and others)

Cloudflare Cache Statuses

First, let’s understand the headers and tools Cloudflare gives us to understand how our pages are cached (or not). When a page is served through Cloudflare, it adds some Cf-* cache headers.

The most important one is Cf-Cache-Status, which can take the following values:

HIT → cacheable resource, served from the cache.
MISS → cacheable resource, never cached (or at least not in the cache at the time of the request). Served from the origin.
EXPIRED → cacheable resource, was in the cache but has an expired TTL. Served from the origin.
BYPASS → cacheable resource, not cached due to directive, cookie, etc …
DYNAMIC → uncacheable resource

So, resources (or URLs) are organized into cacheable and uncacheable assets. Cacheable resources have more nuanced statuses, depending on the state of the Cloudflare cache and other settings such as the TTL, the Cookies, etc.

(Another useful header is the Cf-Ray, which is a unique identifier of this request. You can use this ID in the CloudFlare console to find out more about the request)

What is a “resource”

👉 By default, Cloudflare considers URLs with different query parameters as different pages.

In the previous section, we talked about resources and hinted they mapped to URLs. To clarify, the definition of a resource depends on the Cache Level of your Cloudflare zone. The Cache Level can be:

Basic, no query string → only cache when the URL has no query string
- derlin.ch/main is cached, derlin.ch/main?foo=bar isn’t
Simple, ignore query strings → deliver the same resource regardless of query strings
- derlin.ch/main?ignore=this is cached and the same resource as derlin.ch/main
Standard, aggressive → deliver a different resource with each unique query string
- derlin.ch/main?foo=bar, derlin.ch/main and derlin.ch/main?foo=bar&x=1 are all separate resources cached separately

The Cache Level is Standard by default, meaning a CloudFlare resource maps 1-1 to a URL.

Which resources are eligible for cache

👉 Resources eligible for cache must end with a whitelisted extension; JSON and HTML are not cached.

What makes a resource eligible for cache?

Let’s say you have your portfolio website behind Cloudflare. You would expect your public home page, e.g. https://derlin.ch, to be cached automatically, right? Nope!

Cloudflare only caches based on file extension and not by MIME type.

The Cloudflare CDN does not cache HTML or JSON by default.

When we say extension, we talk about the last bit of the URL (think file extensions). The Content-Type header (MIME type) is never taken into account. Furthermore, Cloudflare has a list of default cached file extensions (.jpg, .css, .js, …), and URLs not part of this list are de facto uncacheable.

To better understand:

derlin.ch/data/cv.pdf is cacheable (.pdf is part of the list)
derlin.ch/github-logo.png is cacheable (.png is part of the list)
derlin.ch/index.html is not cacheable (.html is not part of the list)
derlin.ch or derlin.ch/main is not cacheable (no extension)

Which eligible resources are cached

👉 Cloudflare strictly respects cache control headers returned by the origin.

Once a resource is eligible for cache (cacheable), Cloudflare still needs to decide whether or not to cache it. For this, it strictly respects cache-control headers. In other words, the cache / not cache decision is based on the following information:

the Cache-Control and Expiresheaders
the origin status code

From the Mozilla documentation:

The HTTP Cache-Control holds directives (instructions) in responses (and requests!) that control caching in browsers and shared caches (e.g., Proxies, CDNs).

The HTTP Expires response header contains the date/time after which the response is considered expired in the context of HTTP caching.

Most web frameworks these days come with pre-configured cache-control directives. If your application doesn’t have them, you have work to do!

QUICK DIGRESSION

I have to cite the docs on this, just because I love the “its an option, but you can’t do anything about it on basically all subscription levels” 😂:

When enabled on an Enterprise customer's website, it indicates that Cloudflare should strictly respect Cache-Control directives received from the origin server. Free, Pro, and Business customers have this option enabled by default and cannot disable it.

Cloudflare's Cache Rules allows users to either augment or override an origin server's Cache-Control headers or default policies set by Cloudflare.

When fetching a resource from the origin, Cloudflare looks at the response and applies the following default cache behavior:

Cloudflare does cache the resource if one of the following is true:
- The Cache-Control header is set to public and max-age is greater than 0
- The Expires header is set to a future date
  
  (If both max-age and an Expires header are present, Cloudflare uses max-age)
Cloudflare does not cache the resource if one of the following is true:
- The Cache-Control header is set to private, no-store, no-cache, or max-age=0
- The Set-Cookie header exists.
- The HTTP request method is anything other than a GET.

If there is no Cache-Control or Expires, CloudFlare caches responses (with varied TTL) only when the origin status code is one of 200, 206, 301-303, 404 and 410.

In other words:

all resources with cacheable extensions will be cached unless you have headers that tell Cloudflare otherwise
redirects and 404s will be cached
errors such as 500 won’t be cached

(NOTE: Your zone settings may override this behavior, and there are additional specifics I haven’t covered - e.g. cacheable size limits, Set-Cookie handling, and more. When in doubt, always refer to the official documentation.)

How to extend cached resources to HTML (and others)

You have understood by now that most of the pages making up your website (HTML, JSON, …) are not even considered for caching - even though they would likely benefit from it!

You may believe changing this behavior is tricky and requires a deep knowledge of your pages and Cloudflare. What if I tell you it only requires a teeny-tiny generic rule at the right place?

Indeed, assuming your application is configured to return relevant cache-control headers, the magic comes in one of two forms:

either create a new Page Rule matching * and setting Cache-Level: Cache Everything

or use a Cache Rule with a similar yield:

(Cache Rules are now preferred over Page Rules).

It may sound weird (especially the page rule), but this will promote all resources to cacheable (a.k.a. eligible for cache). If you followed along, you know that it doesn’t impact the next step of the process: Cloudflare will still use the Cache-Control headers and status code to make the final cache / no cache decision.

In other words, your admin pages (with a proper Cache-Control: private, no-store, no-cache) and pages with a session cookie won’t be cached, but now your HTML home page (with a proper Cache-Control: public, max-age=3600) will!

Once this rule is on, the number of DYNAMIC pages (uncacheable) will reduce drastically. You may still need to fine-tune the behaviors, but at least it will provide a decent baseline to work with.

I was initially confused by Cloudflare’s caching behavior, especially when HTML pages returned a DYNAMIC cache status. The documentation is extensive but overwhelming and I spent quite a few hours on it, trying to figure it out. The redaction of this article helped clarify everything in my head.

The rule shared in the last section is something we use in production. Without it, the cache gains would be far less tangible.

I can only hope it helps you too. Don’t hesitate to drop a like or a comment, your reactions always make me smile 🤗.

How to connect to AWS OpenSearch or Elasticsearch clusters using python

Lucy Linder — Wed, 18 Dec 2024 13:01:00 +0000

Connecting to an OpenSearch (ES) service running in AWS using Python is painful. Most examples I find online either don't work or are outdated, leaving me constantly fixing the same issues. To save time and frustration, here’s a collection of working code snippets, up-to-date as of December 2024.

Connect using the opensearch-py library (OpenSearch + ElasticSearch)
Connect using the elasticsearch library (ElasticSearch only)
- elasticsearch >= 8
- elasticsearch < 8

Connect using the opensearch-py library (OpenSearch + ElasticSearch)

This is my preferred way of connecting to an ES instance managed by AWS. It works for both ElasticSearch and OpenSearch clusters, and the authentication can take advantage of AWS profiles.

Install opensearch-py and boto3 (for authentication):

pip install opensearch-py boto3

At the time of writing, this installs opensearch-py==2.8.0 and boto3==1.35.81.

Now, you can create a client using the following:

import boto3

from opensearchpy import (
    AWSV4SignerAuth,
    OpenSearch,
    RequestsHttpConnection,
)

es_host = "search-my-aws-esdomain-5k2baneoyj4vywjseocultv2au.eu-central-1.es.amazonaws.com"
aws_access_key = "AKIAXCUEGTAF3CV7GYKA"
aws_secret_key = "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"
region = "eu-central-1"

# Note: you can also use boto3.Session(profile_name="my-profile") or other ways
session = boto3.Session(
    aws_access_key_id=aws_access_key,
    aws_secret_access_key=aws_secret_key,
    region_name=region,
)

client = OpenSearch(
    hosts=[{"host": es_host, "port": 443}],
    http_auth=AWSV4SignerAuth(session.get_credentials(), region, "es"),
    connection_class=RequestsHttpConnection,
    use_ssl=True,
)

Note that boto3.Session supports various ways of creating a session: using a profile, environment variables, and more. I will let you check it out!

Once you have it, check the connection using:

client.ping() # should return True
client.info() # use this to get a proper error message if ping fails

To check indices:

# List all indices
client.cat.indices()
client.indices.get("*")

# Check the existence of an indice
client.indices.exists("my-index")

Connect using the elasticsearch library (ElasticSearch only)

🔥 This only works for ElasticSearch clusters! Connecting to an OpenSearch cluster raises

UnsupportedProductError: The client noticed that the server is not Elasticsearch and we do not support this unknown product

elasticsearch >= 8

Most snippets are still referencing RequestsHttpConnection, a class that was removed in elasticsearch 8.X. If you were googling for the error cannot import name 'RequestsHttpConnection' from 'elasticsearch’, you are at the right place!

Install elasticsearch (this should install elastic-transport as well), and requests_aws4auth . The latter, based on requests, is required to handle authentication to AWS:

pip install elasticsearch requests-aws4auth

At the time of writing, this installs elastic-transport==8.15.1, elasticsearch==8.17.0 and requests-aws4auth==1.3.1.

Now, you can create a client using the following:

from elastic_transport import RequestsHttpNode
from elasticsearch import Elasticsearch
from requests_aws4auth import AWS4Auth

es_endpoint = "search-my-aws-esdomain-5k2baneoyj4vywjseocultv2au.eu-central-1.es.amazonaws.com"
aws_access_key = "AKIAXCUEGTAF3CV7GYKA"
aws_secret_key = "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"
region = "eu-central-1"

es = Elasticsearch(
    f"https://{es_host}",
    http_auth=AWS4Auth(
        aws_access_key, 
        aws_secret_key, 
        region,
        "es",
    ),
    verify_certs=True,
    node_class=RequestsHttpNode,
)

Once you have it, check the connection using:

es.ping() # should return True
es.info() # use this to get a proper error message if ping fails

elasticsearch < 8

If you are still on an old version of elasticsearch:

pip install "elasticsearch<8" requests-aws4auth

Currently elasticsearch==7.17.12, requests-aws4auth==1.3.1.

Now, you can create a client using the following:

from elasticsearch import Elasticsearch, RequestsHttpConnection
from requests_aws4auth import AWS4Auth

es_endpoint = "search-my-aws-esdomain-5k2baneoyj4vywjseocultv2au.eu-central-1.es.amazonaws.com"
aws_access_key = "AKIAXCUEGTAF3CV7GYKA"
aws_secret_key = "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"
region = "eu-central-1"

es = Elasticsearch(
    host=es_endpoint,
    http_auth=AWS4Auth(
        aws_access_key, aws_secret_key, region, "es"
    ),
    use_ssl=True,
    port=443,
    verify_certs=True,
    connection_class=RequestsHttpConnection,
)

Check the connection:

es.ping() # should return True
es.info() # use this to get a proper error message if ping fails

The Twelve-Factor App: A Blueprint for Scalable, Maintainable Software

Lucy Linder — Mon, 26 Aug 2024 12:19:09 +0000

I am working for a PaaS (Platform As a Service), meaning that we not only have our internal apps to maintain, but we manage thousands of client apps that are all different and yet have to run on the same platform.

How? At Divio, we strictly follow the Twelve-Factor App and have built our platform so that an app adhering to the same standards (plus a few conventions) can be deployed in minutes.

Initially published by Heroku engineers in 2014, these principles are best practices that guide the design of "Web Apps" to boost flexibility, scalability, maintainability, security, and [... ask ChatGPT for more ...]. They make it easier to deploy anywhere, especially in the cloud.

I had trouble understanding the benefits until I practically witnessed the power of those 12 simple best practices. Some I already applied intuitively (without knowing why), and others opened a new way of designing and architecting software. Sticking to those principles is a game-changer, as they simplify and resolve many common problems and pitfalls.

I've been dying to talk about them for a long time, so without further ado, let's dive in!

(I will stay quite generic in this article, let me know in the comments if you want a follow-up with more practical examples).

The Twelve-Factor App principles:

I Codebase
II. Dependencies
III. Config
IV. Backing services
V. Build, release, run
VI. Processes
VII. Port binding
VIII. Concurrency
IX. Disposability
X. Dev/prod parity
XI. Logs
XII. Admin processes

I Codebase

one codebase tracked in revision control, many deploys

An app should only have a single codebase, and each codebase must be using a version control (git, mercurial, svn). If an app is split into multiple codebases, then it is not an app, it's a distributed system - and the factors should apply to each app composing it.

"Deploys" refer to running instances of the app. Many deploys mean the same codebase must be used in all environments - in production(s), locally, in QA, etc.

II. Dependencies

Explicitly declare and isolate dependencies

First, an app should never rely on implicit, transitive or system-wide packages, but instead declare all its dependencies (including their version and possibly an SHA) using a dependency manifest. Examples are pom.xml in Java, requirements.txt or pyproject.toml in Python, Gemfile in Ruby, package.json in Node.

Second, dependencies should be isolated from the underlying system on which an instance runs. This is typically achieved using tools such as virtual environments in Python and containerization (Docker).

This factor thus ensures consistency and portability. By making the app's environment predictable and reproducible, it suppresses the "it works on my machine" conundrum and streamlines the whole flow (onboarding new developers, boosting the stability of build & tests in CI, deploying on new environments, etc).

III. Config

Store config in the environment

Anything that can vary between deploys should be read from environment variables. Config usually entails:

connection information to backing services (database, caching layer, ...)
credentials to external services (Facebook, AWS, kubectl, ...)
other configurations: hostname, size of the thread pool, timeouts, etc.

Strictly separating config from code allows the same codebase to run in widely different environments (locally, in staging or QA, in production(s)). Environment variables are a simple construct supported everywhere that can be managed securely and are easily updated. Thus, the configuration can be adjusted dynamically at any time just by restarting the app. This makes the code flexible, portable, and well-suited for CI/CD. Moreover, it avoids the mistake of committing sensitive information!

Some additional tips on environment variables

Avoid the use of default values when reading configs from the environment. Always be explicit about config in every environments makes it clearer and prevents mistakes. If something is missing, crash early.
Prefer long, descriptive names, and clearly state the type or unit expected. For example, CACHE_READ_TIMEOUT_SECONDS=5.
Group similar configuration using a prefix, not a suffix. This way, you can easily sort through your configurations (env | sort).
Simplify local development by taking advantage of .env files (e.g. a committed base.env and an uncommitted secrets.env). If you are using docker, they are easy to source and if you are not, consider tools such as direnv.

IV. Backing services

Treat backing services as attached resources

A backing service is any service the app consumes as part of its normal operation. This may include databases, caching systems, SMTP servers, etc. Those services should be accessed solely via the network using config(s) stored in the environment (see III. Config). Here is an example config for a database (DSN stands for Data Source Name): DATABASE_DSN=postgresql://user:password@hostname/db_name).

Treating all backing services the same way, the app doesn't differentiate between local and third-party services. It doesn't matter how the resource is managed or where it is running, as long as it is accessible over the network and complies with the expected protocol. This brings loose coupling and flexibility, as backing services become interchangeable: a database may be swapped from an on-prem instance to an AWS RDS database without any code change, while locally, the app can connect to a database running on Docker.

Design for flexibility, but only if required

Ideally, your app should settle on one backing service implementation (e.g. PostgreSQL for database). However, if you start having to support multiple implementations of a service, rely as much as possible on standard protocols and connection libraries (e.g. an ORM). If this is not enough, switch drivers at runtime with the strategy and bridge design patterns. Other design patterns (dependency-injection) are also handy!

IMPORTANT: A Twelve-Factor app should be stateless (VI. Processes) with every persistent information stored in backing services. This includes the filesystem! If you use the filesystem for more than temporary operations or static assets loading, consider object storages such as MinIO and S3.

V. Build, release, run

Strictly separate build and run stages

The lifecycle of a codebase should follow three separate, non-overlapping steps: build, release, and run, each with a decreasing level of complexity. The build compiles the code into an artifact (e.g. an executable, a docker image). The release adds the configuration for a specific environment. The run starts the instance (or restarts it during a scale or after a crash).

For traceability, each release should be immutable and uniquely identified. This way you can reproduce any version of your application consistently, making debugging, deployment, and scaling easier.

How we do it at Divio

The way we separate those steps at Divio is a good example, so let me give you the gist.

At Divio, each environment tracks a specific branch of a git repo and has a settings section for you to configure environment variables. When you click the deploy button, what we do is:

build: we clone your repository and create a build image (the artifact) from your Dockerfile.
release: we build a new release image that inherits from the build image and adds all the environment variables defined in the settings at the time (that is, we build a Dockerfile containing only a FROM build-image and one or more ENV XXX=YYY directives). In other words, your current environment variable settings are frozen within the release image. Each release has a unique, incremental build number.
run: we simply start one or more containers from the release image. Those containers can be scaled and moved around easily as the release image is self-contained!

VI. Processes

Execute the app as one or more stateless processes

An app should consist of one or more Unix-like processes that can be started and stopped anywhere at will. To make this possible, each process must be disposable, strictly stateless, and share nothing.

In other words, all persistent data must be stored in backing services (IV. Backing services). Memory and filesystems may only be used for transient, temporary, and single-process operations. Processes should behave the same whether they all run on the same machine or in another part of the world.

This makes the app highly resilient and easily scalable.

VII. Port binding

Export services via port binding

The app should be entirely self-contained, and expose its functionalities by binding to a port.

Self-contained stresses an app should not rely on any pre-installed application on the running environment. A web or WSGI application should thus ship with its own server (Apache, Nginx, ...) instead of relying on an external one. This is an extension of II. Dependencies.

Binding to a port means the app is accessible via a URL (or another locator such as a web socket address) and can itself be turned into a backing service for another app.

The port is the contract with the execution environment and the app should not care about anything else - such as exposing the port to the outside. It means the app may be accessed directly during development (e.g. https://localhost:<port>/). At the same time, in production, a routing layer takes care of mapping a public domain (and handling SSL termination) to the app's exposed port. The app itself should never handle the routing directly - it is the job of the execution environment.

Note that it doesn't only apply to HTTP/HTTPS: nearly everything can run by listening to a port for incoming requests!

VIII. Concurrency

Scale out via the process model

Processes are first-class citizens. An app should be designed to handle multiple tasks simultaneously via the process model. Instead of managing multiple threads or processes from within the app, each type of work is assigned to a separate process. Those processes can in turn be moved around and scaled up and down as needed (VI. Processes).

For example, a typical web application is composed of web processes to handle the HTTP requests, worker processes to run long-running background tasks, and cron processes running scheduled tasks. Processes can even be more specialized: a set of workers for time-sensitive tasks versus low-priority workers.

This specialization of tasks means each process type can be optimized for its particular function (e.g. more CPU time for high-priority workers or running low-priority workers on cheaper hardware) and can be scaled separately to face any load scenarios.

The ability to start different kind of processes anywhere (since they are stateless, IV. Processes) makes the app highly resilient, flexible, and especially suited for cloud environments.

IX. Disposability

Maximize robustness with fast startup and graceful shutdown

From IV. Processes and VIII. Concurrency, it follows that processes are disposable resources. As such, they must be able to start quickly and handle both planned and spurious shutdowns gracefully - by leaving the system in a consistent state.

Graceful shutdown is achieved by exiting properly upon a termination signal. Handling crashes properly is eased by the fact that processes are stateless.

For example, a worker should either finish or return the current task to the queue upon a termination signal. The queuing system should re-enqueue any unfinished tasks after a given timeout so the sudden death of a worker is not an issue.

Disposability is the cornerstone that lets you scale processes at will without fear. It guarantees your system stays robust and coherent without losing flexibility.

X. Dev/prod parity

Keep development, staging, and production as similar as possible

The gaps between environments should be kept as low as possible. The goal is to minimize the differences in time (how long it takes to deploy a change in production), personnel (people involved in developing and deploying the app), and tools (software, infrastructure/backing services).

Especially for backing services, ensure you use the same service type and versions in all environments. It may come in different flavors (e.g. a simple docker container running postgres:15 locally, a fully-pledged RDS 15.7 on prod), but the version and protocols should be aligned.

This principle shares many similarities with the agile and DevOps movements, both of which have demonstrated significant benefits across all aspects of the software lifecycle.

XI. Logs

Treat logs as event streams

An app should log information to the standard output (stdout) as an ephemeral stream and delegate the log management to the execution environment.

This way, it is up to each execution environment to decide what to do with logs. In production, you may have one or more log aggregators pushing data to external data sources (ElasticSearch, LogStash, Kafka, ...) for monitoring and event detection, while locally, developers can just tail the logs.

All advanced log management tools can hook to stdout. On the other hand, an app logging to a file requires a filesystem (so not stateless) and makes it harder to access the logs. In short, do not limit the possibilities by complexifying your code. A simple print (with timestamps) will do!

XII. Admin processes

Run admin/management tasks as one-off processes

We often need to run on-off administrative or maintenance tasks such as migrating data, clearing the cache, re-indexing content, or simply executing arbitrary code to get data or understand the system.

The Twelve-Factor app recommends always running those tasks against a release, (V. Build, release, run) using the same codebase and config as the other regular processes. A containerized application makes it easier, as we can spin up a new container alongside the app with the exact same config for one-off purposes.

To avoid synchronization issues, admin code meant to run more than once should also ideally be part of the codebase.

For this purpose, frameworks and languages shipping with a REPL (Read Eval Print Loop) shell have a strong advantage. A good example is Django, which provides the ./manage.py shell utility and makes it easy to register custom management commands that run with the same ./manage.py entrypoint (I abuse its power, even in production). Ruby and NodeJS have similar mechanisms.

The Twelve-Factor, similar to Design Patterns, is here to answer common problems so we do not constantly reinvent the wheel.

These best practices may seem obtuse, but if you pay attention, you'll notice that many well-designed apps adhere to these principles, knowingly or not. Detect them in the wild and apply them until they become second nature. From my experience, you won't regret it!

Let me know if you'd like a follow-up article using a project to exemplify those principles and how to put them into use, and happy coding :).

Seriously, you need to learn git

Lucy Linder — Mon, 22 Jul 2024 07:00:00 +0000

WARNING: ranting incoming.

I have seen countless articles about git (one of the hottest subjects out here), yet I am still puzzled at how few developers actually took the time to learn git.

You do not know git

To clarify what I mean by that, let's play a game. If I ask you to create a feature branch, commit some changes, and make a pull request, I am sure everything is fine. But now, let me vary the game:

level one: you can't use any external tool such as Gitlab or your favorite IDE. No clickops, terminal only. This holds for all the other levels.
level two: you have to split your changes into 5 commits. Bonus: the files are from only two files. And please, don't use the old trick of copying all the changes somewhere, and then pasting them one by one to "prepare" each commit.
level three: you pushed your commits 1 to 5, but your reviewer asks to change something in commit 2, reorder commits 3-4, meld commit 5 into 4, change the commit message of commit 1, and rebase your branch to master.
level four: you made a mistake during the level three operation. You need to "recover" and start again, but you already pushed your changes, so a force pull from the origin won't work.
bonus: a bug was introduced into the codebase, but you have no idea when. How do you use git to find the commit that started it all?

All of this is doable when you understand the concepts and inner workings of git - (I wanted to write "when you start actually giving a f**k about the tool you use every day").

Why bother? Well, the operations I outlined in the game can:

speed up your development process,
make your codebase cleaner and your reviews easier: if all the developers in your team learn how to properly split commits and organize them, you can start thinking about conventional commits, automated changelogs, atomic commits, continuous deployment, clean review processes, etc.
make you fit for open-source: yup, you'll need all that if you want one day to contribute to the Linux kernel or other important codebases!
save your life when inevitably someone (you?) screws up a git repo
[... ask ChatGPT to fill up the list ...]

Don't get me wrong. I am not saying everyone should be a complete master of git, but when 99% of the codebases worldwide are managed through git and "gitting" is a daily activity, I believe spending some time understanding the power of it may be worth it. I myself do not know all the git commands by heart, but I know enough to be aware of the possibilities (and what to google for).

But you can start learning

Now that I (I hope!) convinced you, here are some pointers on how to start.

Start with the basics

Start by learning git the hard way. Stop using your IDE or git UI for a while, until you feel at ease with all the basic operations directly using the command line. Do not learn by heart, but ask yourself what git is doing under the hood. For example, "staging area", "branches" and "remote" should be crystal clear. You should also understand what's inside the .git/config , how .gitignore works, and what are refs.

Here are some of what I consider "basic" commands:

the obvious clone, pull, push, fetch
remote management: git remote add|remove|show origin
git status, and how to add(remove) files/folders to the staging area (e.g. git rm --cached -r foo/)
commit (with short and long messages), and especially amend the last commit (e.g. git commit --amend --reset-author). Also, ensure you know how to revert the latest commit(s) (git reset --hard/--soft).
basic branch and tag management: git checkout [-b] ..., git branch ..., git tag .... Note that checkout works not only with branches but also with files!
stashing (please, do it for yourself!): git stash / git stash pop, ...
merge and basic rebase: git merge, git rebase
understand git log, (or better, git log --graph --format=oneline) and view a specific ref (git show, git revparse)
some other handy commands: git diff, git revparse,...

Note: this is a very generic advice. When I started programming, my first Java class was not about primitives, it was about java and javac. I started with a Notepad and a terminal, and only 4 weeks into the class was I allowed to use an IDE. This was brilliant and stayed with me ever since.

Get comfortable with rebasing

git rebase is so powerful it is blinding. You could spend months just playing with it. I personally only know 2 things by heart: (a) how to rebase a branch onto another (e.g. git rebase master from a feature branch) and (b) how to do interactive rebase.

Play with interactive rebase yourself, and get familiar with the options: pick, reword, edit, squash, fixup. Learn how to handle conflicts during rebase, and how to abort.

If you fear you'll mess up, don't forget nothing is undoable in git. You can stop an interactive rebase at any point using the git rebase --abort command. If you already finished the rebase, you still have the reflog ;).

Experiment and learn how to undo anything

Trying things out is difficult when you fear about messing up. But anything you do on your local repository can be rolled back. You may have already learned about git reset and git pull --force, so the next step is to get acquainted with the reflog.

Reference logs (reflogs), keep the history of when the tips of branches and other references in the local repository are modified. For example, when you create/delete a branch, do a rebase, merge something, etc. Each log is associated with an SHA, that you can use to go back in time exactly as you do with commits, using git reset <reflog SHA>.

I'll let you figure out the rest by yourself! But now that you are aware of its existence, there is no excuse to try things: merge, rebase, delete, mess up all you want, you're covered.

Be curious

When you return to using your IDE / git GUI (tip: try lazygit, it's AMAZING), always ask yourself what command(s) are used in the background.

Do a git --help once in a while, and read about the commands you never encountered. For example, have a look at git blame, git bissect, etc.

Ask your peers how they use git, and learn from them. Share your learnings and spread the knowledge.

Up your game

Now that you're familiar with git, it is time to step up your game. Think of how it could help you streamline your processes and how it can benefit you and your team.

As a bonus, here are additional advice:

Sign your commits using GPG keys (this will annotate your commits with a nice "verified" badge on GitHub)
Improve your pull requests by following the conventional commits convention, and your reviews by following the conventional comments.
Keep a clean git history using squashing, and think about adopting atomic commits

Idempotency in 256 characters or less

Lucy Linder — Sun, 23 Jun 2024 14:52:48 +0000

This is a submission for DEV Computer Science Challenge v24.06.12: One Byte Explainer.

Explainer

Borrowed from Mathematics, an idempotent operation can run multiple times without changing the result. Writing to a file is idempotent, but appending to a file is not. A script using idempotent operations can thus fail midway and be restarted safely.

Additional Context

This idempotency concept is so important, yet too often unknown. It is the magic behind famous tools such as Terraform, Ansible, Kubernetes Manifests, etc. Fault-tolerant REST APIs are mostly idempotent REST APIs. It is everywhere, yet cleverly hidden. It is what makes operations repeatable and consistent.

There are better explanations online, but if my submission makes at least one person curious about this concept and want to know more, then I won already 🙂 !

From Jar to Brew: distribute your Java programs easily with Homebrew and GitHub Actions

Lucy Linder — Mon, 20 May 2024 12:10:00 +0000

In this article, we'll explore how to make a program packaged as a Jar (Java, Kotlin, Scala, ...) easily installable by end users using homebrew and GitHub actions.

This is inspired by my work to distribute bitdowntoc with brew and aims at answering two questions:

how to create a homebrew tap to distribute a jar with brew?
how to automatically update the tap when a new version is available (using GitHub and GitHub actions)?

Let's get started!

Prerequisites
Publish your JAR via homebrew
- Step 1: create a fat jar
- Step 2: make your jar available from a public URL
- Step 3: create a homebrew tap
- Step 4: write the formula
- Step 5: validate and test the formula
Keep the homebrew tap in sync
- Step 1: the tap workflow
- Step 2: the app workflow
Conclusion

(TOC generated with bitdowntoc)

Prerequisites

you have standalone software written in a JVM-compatible language (Java, Scala, Kotlin, ...) that you want to distribute easily.
you are hosting it on GitHub. For the second part, you use GitHub releases.
you have homebrew installed and roughly know what it is about.

In this article, I will use bitdowntoc, a command line (and online) tool to add a table of contents to your Markdown files. Written in Kotlin, it is available on the web and in the command line (using a jar or a native executable).

Publish your JAR via homebrew

Step 1: create a fat jar

The best way to ship JVM-compatible software to all architectures and OSes is through a fat jar. The advantage of a jar (Java ARchive) is it only requires a JRE (Java Runtime Environment) on the target machine to execute. The colloquial term "fat jar" designates a self-contained jar - an archive that includes all its necessary dependencies.

Depending on your language and build system, there are many ways to create a fat jar - too much to enumerate here. If you use Java + Gradle (groovy), baeldung's article "Creating a Far Jar in Gradle" is a good start.

🔥 Ensure your app follows semver (semantic versioning) and the jar is called <app>-<version>.jar, where <version> matches <major>.<minor>.<patch>. This should be the default when using common tooling.

Step 2: make your jar available from a public URL

If you are using GitHub for release management, this means attaching your jar to each release. This can be done manually, or completely automated using GitHub actions.

bitdowntoc uses release-please for release automation, and action-gh-release to attach build artifacts to the release. For more details, check out the release-please and upload-jar in the release.yml workflow (I should write an article on release automation 😄).

However you do it, ensure you can download a properly named jar from a public URL. Here is the URL for bitdowntoc version 2.1.0:

https://github.com/derlin/bitdowntoc/releases/download/v2.1.0/bitdowntoc-jvm-2.1.0.jar

Step 3: create a homebrew tap

Homebrew supports formulae and casks. Formulae build from upstream sources, while casks install macOS native applications. We could argue a JAR is a native application, but it still requires some fiddling during installation, so we'll use a formula.

taps are git repositories of formulae and casks that homebrew uses for discovery. When installing homebrew, it comes with some built-in taps such as homebrew-core.

Adding your software to built-in taps requires it to be "notable enough" (at least 30 forks, 30 watchers, and 75 stars). If it is your case, go ahead and create a pull request by following the Formula Cookbook! If like me you lack exposure, you'll have to create and maintain your own tap.

👉 The following steps are a condensed version of brew's How to Create and Maintain a Tap.

Brew is very developer-friendly. To bootstrap a new tap, run the following in a terminal:

GH_USERNAME=derlin
APP_NAME=bitdowntoc

brew tap-new $GH_USERNAME/$APP_NAME
cd $(brew --repository)/Library/Taps/$GH_USERNAME/homebrew-$APP_NAME

Replace derlin with your GitHub username, and bitdowntoc with the name of your application. This will initialize a git repository called homebrew-<appname> at the path of the taps on your system with a README.md and a Formula directory. Cd into it using:

cd $(brew --repository)/Library/Taps/derlin/homebrew-bitdowntoc

The tap is created, let's write the formula.

Step 4: write the formula

In the tap repository created above, add a ruby file in the Formula repository. Here is the bitdowntoc formula, that you can adapt to your app:

class Bitdowntoc < Formula
  # ↓↓ ①
  desc "Markdown Table Of Content (TOC) generator"
  homepage "https://derlin.github.io/bitdowntoc"
  url "https://github.com/derlin/bitdowntoc/releases/download/v2.1.0/bitdowntoc-jvm-2.1.0.jar"
  sha256 "b7a21d2a793a2ecb6c20f27aa8fffc72702daa364b2a89bed32580e302194650"
  license "Apache-2.0"

  ## ↓↓ ②
  depends_on "openjdk"

  ## ↓↓ ③
  def install
    libexec.install "bitdowntoc-jvm-#{version}.jar"
    bin.write_jar_script libexec/"bitdowntoc-jvm-#{version}.jar", "bitdowntoc"
  end

  ## ↓↓ ④
  test do
    assert_match "BitDownToc Version: #{version}", shell_output("#{bin}/bitdowntoc --version")
  end
end

There are 4 parts to this Ruby file.

Section ① defines the metadata and the "source" URL (the jar). To get the correct sha, use sha256sum on Linux or shasum -a 256 on Mac. Ensure your description is less than 80 characters.

Section ② specifies that the formula requires Java to be installed.

Section ③ copies the jar (downloaded and unpacked automatically by homebrew using the URL in ①) in the libexec folder and creates a bash executable to launch it in the bin folder. The write_jar_script method takes care of resolving to the proper JDK. In my system, for instance, the bin script looks like this after install:

cat /opt/homebrew/Cellar/bitdowntoc/2.1.0/bin/bitdowntoc
#!/bin/bash
export JAVA_HOME="${JAVA_HOME:-/opt/homebrew/opt/openjdk/libexec/openjdk.jdk/Contents/Home}"
exec "${JAVA_HOME}/bin/java"  -jar "/opt/homebrew/Cellar/bitdowntoc/2.1.0/libexec/bitdowntoc-jvm-2.1.0.jar" "$@"

Section ④ is completely optional and defines some tests to check the installation worked properly.

Step 5: validate and test the formula

Now, let's install the formula locally (note that it will download the world if you don't have openjdk already installed):

HOMEBREW_NO_INSTALL_FROM_API=1 brew install bitdowntoc

If this works, the next steps are to run the tests and audit the formula:

brew test bitdowntoc
brew audit --strict bitdowntoc

The last step is to commit and push everything to GitHub. Once done, anyone can install your software using either:

brew tap derlin/bitdowntoc
brew install bitdowntoc

Or in one line (will fetch the tap and run the install in one command):

brew install derlin/bitdowntoc/bitdowntoc

Keep the homebrew tap in sync

The homebrew tap lives in a different repository and must be updated every time a new version is available. In the age of the "*** as Code", this manual step is frowned upon. This section shows how I automated it for bitdowntoc.

⚠ Note that this assumes you use GitHub Releases and have a GitHub Actions release workflow.

The idea is simple:

create a workflow on the homebrew tap repository able to check for new versions of the app and update the formula accordingly.
trigger this workflow from the release workflow of the app.

(Note that we could also schedule 1 to run e.g. every week, but this adds latency - imagine you release on Monday but the workflow runs on Sunday! GitHub may also disable scheduled workflows e.g. due to a lack of activity in the repo. So meh.)

Step 1: the tap workflow

First, let's create a workflow on the homebrew tap repository. Its role is to:

check the latest version available upstream (the main repository),
if a new version is available, update the formula and commit the changes.

This is what it looks like for bitdowntoc (homebrew-bitdowntoc/.github/workflows/update.yml):

name: update version from upstream

on:
  # Allow manual triggers
  workflow_dispatch:
permissions:
  # Necessary to commit the changes back
  contents: write

jobs:
  update_version:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
        with:
          ref: ${{ inputs.ref }}

      # ↓↓ ①
      - name: get upstream version
        id: upstream
        run: |
          res=$(curl -s https://api.github.com/repos/derlin/bitdowntoc/releases/latest)
          url=$(jq -r '.assets[] | select( .content_type == "application/java-archive" ) | .browser_download_url' <<< "$res")
          version=$(echo $url | sed -E 's/.*-([0-9]\.[0-9]\.[0-9]).*/\1/')

          curl -L -o /tmp/bitdowntoc.jar $url
          sha=$(sha256sum /tmp/bitdowntoc.jar | cut -d' ' -f1)

          echo "✅ Found version $version: $url" | tee -a $GITHUB_STEP_SUMMARY
          echo "url=$url" >> $GITHUB_OUTPUT
          echo "version=$version" >> $GITHUB_OUTPUT
          echo "sha=$sha" >> $GITHUB_OUTPUT

      ## ↓↓ ②
      - name: update bitdowntoc formula
        id: update
        run: |
          formula="Formula/bitdowntoc.rb"
          sed -i'' -E 's#^  url ".*"$#  url "${{ steps.upstream.outputs.url }}"#' $formula
          sed -i'' -E 's#^  sha256 ".*"$#  sha256 "${{ steps.upstream.outputs.sha }}"#' $formula

          if [[ -z "$(git status -s $formula)" ]]; then
            echo "✅ Already up-to-date" | tee -a $GITHUB_STEP_SUMMARY
            echo "changed=" >> $GITHUB_OUTPUT
          else
            echo "🟠 Updated $formula" | tee -a $GITHUB_STEP_SUMMARY
            git add $formula
            cat $formula
            echo "changed=true" >> $GITHUB_OUTPUT
          fi

      ## ↓↓ ③
      - name: commit changes
        if: steps.update.outputs.changed == 'true'
        run: |
          git config --global user.name 'Github Workflow'
          git config --global user.email 'workflow@bot.github.com'
          git commit -am "chore: update bitdowntoc to ${{ steps.upstream.outputs.version }}"
          git push

          echo "✅ Changes committed" | tee -a $GITHUB_STEP_SUMMARY

It may seem daunting, but this boils down to some basic bash. Let's break it down together.

In ①, I use the GitHub API to fetch the list of artifacts from the latest GitHub release of bitdowntoc, and find the URL of the one with a application/java-archive type. Next, I extract the version from the URL (looking for a X.X.X pattern), download the jar, and compute its SHA. The URL, SHA, and version are stored as a GitHub output of this step.

In ②, I update the Formula/bitdowntoc.rb file with the URL and SHA of step ① using a regex (sed). I can now ask git if there is any change, and store the result as an output of this step.

Step ③ only runs if a change was detected in step ②. I configure the git user and commit the changes back to the repo.

This workflow is idempotent: I can run it as often as I want, it will only update the tap if a new release is present upstream.

Step 2: the app workflow

The last step is to trigger the tap workflow on every release of bitdowntoc. This means adding a new job to my release workflow that uses the GitHub API to trigger the tap workflow:

  update_homebrew:
    runs-on: ubuntu-latest
    needs: [release-please, upload-jar]
    steps:
      - name: trigger homebrew update
        # The PAT should have actions:read-write
        run: |
          curl -L \
            -X POST \
            --fail-with-body \
            -H "Accept: application/vnd.github+json" \
            -H "Authorization: Bearer ${{ secrets.HOMEBREW_PAT }}" \
            -H "X-GitHub-Api-Version: 2022-11-28" \
            https://api.github.com/repos/derlin/homebrew-bitdowntoc/actions/workflows/${{ secrets.HOMEBREW_WORKFLOW_ID }}/dispatches \
            -d '{"ref":"main","inputs":{}}'

For the trigger to work, I had to create a PAT (Personal Access Token) with write permissions on the tap repository and add it as a secret (HOMEBREW_PAT) on the bitdowntoc repository (Settings > Secrets and variables > Actions: Repository secrets).

The needs here is specific to my workflow, and is necessary to ensure the GitHub release exists (with the jar!) before the job runs.

And that's it!

Conclusion

I always loved homebrew (as a user), but never tried to publish anything. I was pleasantly surprised by the overall architecture and the great documentation. Setting up a personal tap was a breeze. My only reproach is the whole brew-related terminology... For a non-native speaker, cask, tap, keg, cellar, etc are not straightforward without a dictionary close by.

The syncing of the tap was more "experimental", but I am quite happy with the result. If you are a tap maintainer, let me know how you solved this challenge!

I hope this will be useful to some, and remember, bitdowntoc can now be installed with homebrew, so give it a try 😉.

Happy coding!

How to mock an API in 2 minutes

Lucy Linder — Thu, 09 May 2024 12:43:00 +0000

I don't need to enumerate all the situations in which you would need to mock an API or an HTTP server. There are many options available, such as faker. This article delves into MockServer and shows how to set up a mock server using only Docker and a JSON file. Let's get started!

🤯 🔥 MockServer can also be used as a proxy (to record and potentially modify requests/responses), or a combination of mock and proxy!

The docker-compose
Defining the expectations (spec.json)
- The most basic expectation
- Using templates
- Importing the request spec from OpenAPI
- Going further
Conclusion

The docker-compose

First, let's create a docker-compose for running the latest version of MockServer:

services:
  mockserver:
      image: mockserver/mockserver:latest
      ports:
        - "1080:1080"
      environment:
        MOCKSERVER_PROPERTY_FILE: /config/mockserver.properties
        MOCKSERVER_INITIALIZATION_JSON_PATH: /config/spec.json
        # ↓ hot reload when the spec.json changes 😍
        MOCKSERVER_WATCH_INITIALIZATION_JSON: "true"
      volumes:
        # bind the config for easy editing
        - ./config:/config

Nothing special here. We are using a volume for the config and telling MockServer to load the endpoints to mock from the ./config/spec.json file.

Before starting it, let's create the necessary files (we will see how to use them later):

mkdir config
touch config/mockserver.properties
echo '[]' > config/spec.json

Start the mock server by executing:

docker compose up -d

MockServer is now running on http://localhost:1080! However, since the spec.json - defining the expectations, or endpoints to mock - is empty, it only returns 404's. Let's change that!

💡 the mockserver.properties will stay empty for this article, but I recommend you check out the configuration documentation for all the possibilities it offers! Dynamic SSL certificates, logging, metrics, etc.

Defining the expectations (spec.json)

🔥 More examples available at https://github.com/mock-server/mockserver/blob/master/mockserver-examples/json_examples.md 🔥

The most basic expectation

The spec.json file defines expectations, aka the endpoints to mock and what to return. It contains an object or an array of objects defining the HTTP request to match, and the response to return.

Unclear? Here is a basic example:

{
    "httpRequest": {
        "path": "/hello"
    },
    "httpResponse": {
        "statusCode": 200,
        "headers": {"Content-Type": "application/json"},
        "body": "{\"hello\": \"world\"}"
    }
}

Save the spec.json, wait a second and try hitting http://localhost:1080/hello. You should see the body {"hello": "world"}.

This is just a simple example, but we can go way deeper. The request may define headers, method, path and query parameters, authentication, and more, all using either string or regex matching. MockServer even supports stuff such as adding delay in the response or behaving differently depending on how often the endpoint is called!

Using templates

The response often depends on the request, and enumerating all the possible input/output pairs is tedious. MockServer supports loops, conditionals, and 3 response template engines: Javascript, mustache, and velocity.

Let's see a velocity example (at the time of writing, Javascript is broken in Docker, see the related issue):

{
    "httpRequest": {
        "path": "/cart/item/{id}/",
        "pathParameters": { "id": ["\\d+"] }
    },
    "httpResponseTemplate": {
        "templateType": "VELOCITY",
        "template": "#set($id = $!request.pathParameters['id'][0]) #if ($id == 1) {\"statusCode\": 500} #else {\"statusCode\": 200, \"body\": \"{\\\"name\\\": \\\"foo\\\"}\"} #end",
    }
}

What is happening? Instead of specifying the httpResponse directly, we ask MockServer to "compute" it by interpreting a template. In the example above, the server returns a 500 for the item 1 and a JSON body otherwise.

Things to note:

#set, #if/#end, #foreach/#end, etc. are MockServer constructs independent of the template type, allowing for complex logic inside the response.
$id is a MockServer local variable, but $!request (notice the !) is a velocity template directive.
there is full support for regular expressions in all request attributes.
Request paths and query parameters can be named and accessed in the response. The index [0] is necessary because MockServer uses Java multivalued maps under the hood. Here, using the request path /cart/item/\\d+/ and the condition ($request.path == '/cart/item/1/') would have been simpler.
Since the template itself is a JSON string, multiple escapes of " are necessary and no wrap is possible, making it difficult to read... The joys of JSON 😆.

Let's check the result of the above template:

➜ curl --fail-with-body http://localhost:1080/cart/item/13/
{"name": "foo"}
➜ curl --fail-with-body http://localhost:1080/cart/item/1/
curl: (22) The requested URL returned error: 500

Request attributes (headers, body, path, remote address, etc.) aside, other magic variables are at our disposal in templates. For example:

$!now_epoch → get the current timestamp seconds since January 1, 1970 (UNIX epoch)
$!uuid → get a random UUID
$!rand_int_100 → get a random number between 0 and 99
...

Check the docs for more!

Importing the request spec from OpenAPI

Often, you want to simulate existing endpoints from a known API. If the latter has an OpenAPI schema, MockServer can parse it to construct the request matchers.

Let's use the dev.to API as an example, and simulate the GET /api/articles endpoint: https://developers.forem.com/api/v1#tag/articles/operation/getArticles.

{
    "httpRequest": {
        "specUrlOrPayload": "https://developers.forem.com/redocusaurus/plugin-redoc-1.yaml",
        "operationId": "getArticles"
    },
    "httpResponse": {
        "statusCode": 200,
        "body": "[{\"name\": \"super article\"}]"
    }
}

The two lines of the httpRequest above transform into a 93-line long request matcher. Here is an abridged version:

{
  "method": "GET",
  "path": "/api/articles",
  "headers": { "api-key": [".+"] },
  "queryStringParameters":
    {
      "keyMatchStyle": "MATCHING_KEY",
      "?username": {"parameterStyle": "FORM_EXPLODED", "values": [{"schema": {"type": "string"}}]},
      "?top": { "parameterStyle": "FORM_EXPLODED", "values": [{"schema": {"format": "int32", "minimum": 1, "type": "integer"}}]},
      // ...
    }
}

When importing from OpenAPI, the request matchers are way more precise. For instance, the dev.to schema requires an authentication header. Without it in a request, there is no match, and the MockServer returns a 404. It doesn't complain about invalid query parameters though, which is expected.

# Missing the authorization header
➜ curl --fail-with-body http://localhost:1080/api/articles
curl: (22) The requested URL returned error: 404
# ok
➜ curl --fail-with-body -H 'api-key: x' http://localhost:1080/api/articles
[{"name": "super article"}]

Going further

To see more, check out their documentation and the examples. Even better, try it out yourself!

Conclusion

MockServer is a lightweight, convenient solution for mocking APIs. The Docker image is only 276MB! A basic setup only requires Docker and a JSON file, which is powerful enough for more use cases. The hot reloading feature makes it easy to set up and test.

However, while a JSON file has its advantages, especially when paired with Docker, it can be challenging in terms of readability and comprehension. For complex APIs, opting to define specifications in Java or Javascript and then exporting them to JSON might be a better alternative. To learn more, check out Creating expectations.

This article only covered the basics, but I hope it got you interested in trying out MockServer the next time you need to fake an HTTP(s) endpoint.

Thanks for reading, and happy coding!

Exploring The Magic of Python Through The Awesome Slumber Library

Lucy Linder — Mon, 08 Apr 2024 12:10:00 +0000

Slumber is one of those libraries you don't need, but can't live without once you learn about it (much like attrs!). It covers a generic use case - interacting with RESTful services - and is a prime example of what only the Python language lets you do. Intrigued? Let me explain!

🤲 (NOTE) I recently published a new library - mantelo - that leverages the magic of slumber to provide a fully-fledged Keycloak Admin Client. Read more in my other article → https://blog.derlin.ch/introducing-mantelo.

Introduction to slumber
- In a few words
- Slumber in action: an example with the dev.to API
- A more formal explanation of the "translation"
Delving into the magic
- Theory first: dunder methods
- A simple implementation (< 20 lines!)
Conclusion

Introduction to slumber

In a few words

From the documentation:

Slumber is a python library that provides a convenient yet powerful object oriented interface to ReSTful APIs. It acts as a wrapper around the excellent requests library and abstracts away the handling of urls, serialization, and processing requests.

In short, slumber wraps a requests.Session and allows you to use Pythonic constructs to call RESTful APIs. Put even more simply, it translates Python to HTTP calls.

Still a bit fuzzy, isn't it? Let's look at a concrete example!

Slumber in action: an example with the dev.to API

To better understand, let's assume we need to interact with dev.to's Forem API v1. First, let's spin up a REPL, import slumber (after a pip install) and create an API object:

>> import slumber
>> api = slumber.API("https://dev.to/api")

Remember slumber doesn't know about the dev.to API. Yet, we can use it to query any of its endpoints in the following manner:

# Get my profile image
# ⮕ GET https://dev.to/api/profile_images/{username}
>>> api.profile_images("derlin").get()
{'type_of': 'profile_image', ...}

# Get one of my articles on dev.to
# ⮕ GET https://dev.to/api/articles?username=derlin&per_page=1
>>> api.articles.get(username="derlin", per_page="1")
[{'type_of': 'article',
  'id': 1750725,
   ...
}]

# Try to create a user (oops, I am not allowed!)
# ⮕ POST https://dev.to/api/admin/users <body>
>>> api.admin.users.post({"email": "a@a.com", "name": "a"})
HttpClientError: Client Error 401: https://dev.to/api/admin/users/

The last call raised an exception, 401: unauthorized. Normal, I am not authenticated. To change this, let's recreate the api object, this time passing some auth.

Since the Forem API uses custom headers for authentication instead of a known authentication mechanism, we can't rely on the built-ins offered by requests (and thus slumber). So let's create our own authentication class:

from requests.auth import AuthBase

# You can create an API key in dev.to's Settings > Extensions
DEVTO_API_KEY = "<xxx>"

# A custom Auth class that adds the right header
# to every request
class DevToAuth(AuthBase):
   def __call__(self, request):
      request.headers['api-key'] = DEVTO_API_KEY
      return request

# Tell slumber to use the custom auth
api = slumber.API("https://dev.to/api", auth=DevToAuth())

To test it, let's query my articles again:

>>> api.articles.me.get()
[{'type_of': 'article',
  'id': 1750725,
  # ...
}]

Magical, right?

Under the hood, slumber uses a requests.Session , which can be tuned in case we need to add headers or other things. Another (simpler) way of authenticating would thus be:

import requests

session = requests.Session()
session.headers['api-key'] = DEVTO_API_KEY

api = slumber.API("https://dev.to/api", session=session)

This example shows all you need to know about slumber.

A more formal explanation of the "translation"

How does this translation from Python to an HTTP call actually works?

As you may have guessed from the dev.to example, the slumber api object starts with the base URL. Every property or method is then used to add to this base. When it reaches a method that looks like an HTTP method (.get(), .post(), .delete(), ...), slumber puts together the final URL, makes the call, parses the response, and returns either the response body (as a dictionary) or an exception.

And since an image is worth a thousand words:

Delving into the magic

The "translation" explained above seems rather complex. Yet, the whole slumber library is less than 1000 lines of code! Let's see how the "Python magic" makes it possible by re-implementing the translation ourselves.

Theory first: dunder methods

In Python, dunder methods, short for "double underscore" methods, are special methods (also called magic methods) that define behavior for built-in Python operations. For example, __init__ initializes newly created objects, __repr__ returns a string representation of an object, and __add__ defines the behavior of the + operator. The ability to define/override such methods at the class level is one of the distinguishing traits of Python.

For our purpose, we need to familiarize ourselves with two dunders:

__call__(self) : this method enables instances to be called as if they were functions / lets you define what happens when using parentheses on class instances (my_instance()).
__getattr__(self, item): this method is invoked when an undefined attribute is accessed on an object. If not defined, the normal behavior is to raise an AttributeError.

A simple implementation (< 20 lines!)

First, let's create a class that has a base URL and implements two HTTP methods (body left as an exercise 😉):

class Resource:
    def __init__(self, url):
        self.url = url

    def get(self, **query_params): ...  # GET <url>
    def post(self, body=None, **query_params): ...  # POST <url>

With this base, we now have to implement the URL construction. How? Let's start with the addition of a path to the URL. In slumber, we do this by using an attribute unknown to the instance. Does it ring a bell?

class Resource:
    # ... rest of the implementation ...
    def __getattr__(self, item):
        return Resource(f"{self.url}/{item}")

Now, whenever we call an attribute on a Resource, it returns a new Resource with the path segment added to the URL. What about path parameters? Well, same principle, just another dunder:

class Resource:
    # ... rest of the implementation ...
    def __call__(self, path_param):
        return Resource(f"{self.url}/{path_param}")

The final touch is to bootstrap the whole thing by making the API object also return a Resource when an unknown attribute is accessed. A full implementation would thus look like:

class Resource:
    def __init__(self, url):
        self.url = url

    def get(self, **query_params):
        qs = "&".join([f"{k}={v}" for k, v in query_params.items()])
        print(f"GET {self.url}?{qs}")

    def post(self, body=None, **query_params):
        qs = "&".join([f"{k}={v}" for k, v in query_params.items()])
        print(f"POST {self.url}?{qs}")

    def __getattr__(self, item):
        return Resource(f"{self.url}/{item}")

    def __call__(self, path_param):
        return Resource(f"{self.url}/{path_param}")


class API:
    def __init__(self, base_url):
        self.base_url = base_url

    def __getattr__(self, item):
        return Resource(self.base_url)

Let's try this out!

>>> api = API("https://example.com/api/v1")
>>> api.users("lala").foo_bar.get(x=1, y="buzz")
GET https://example.com/api/v1/lala/foo_bar?x=1&y=buzz

With these 20 lines of code, we demystified all of slumber's magic. For this kind of thing, Python is quite awesome 😎.

Conclusion

Even though I am more and more drawn to typed languages, Python has some nice tricks in its sleeves. I love how slumber leveraged it to provide a simple yet useful library. It is a prime example of a good use of dunder methods.

I hope you'll remember slumber the next time you need to interact with an API!

If you liked this article, leave a comment or a thumbs up, or share it around ... This would help keep my motivation up!

Introducing Mantelo - The Best Keycloak Admin Client for Python

Lucy Linder — Tue, 02 Apr 2024 12:10:00 +0000

I'm thrilled to present my newest open-source project! Mantelo is a super small yet super powerful Python library for interacting with the Keycloak Admin API.

Mantelo [manˈtelo], from German "Mantel", from Late Latin "mantum" means "cloak" in Esperanto.

Born from the frustration of encountering incomplete support in existing libraries, this is the only Python client that guarantees full coverage of all the routes in the Keycloak Admin RESTful API. How? By wrapping instead of implementing. Let me explain 😊.

The motivation behind mantelo
Getting started with mantelo
- 🏁 Installing
- 🔐 Authenticating
- 📞 Making calls
The best library out there, really?
Mantelo needs YOU

TOC generated with bitdowntoc.

The motivation behind mantelo

When I started working with Keycloak in Python, I first opted for the renowned python-keycloak, considered the most complete library.

It worked well until I had to query a resource that was not supported. I diligently submitted a pull request to add a get_idp method, which took three months to be merged, and a bit more to be released. This doesn't mean the maintainers are incompetent (they are great!), just that PRs in significant open-source projects take time.

Everything proceeded smoothly until I encountered another wall: python-keycloak implements none of the endpoints related to TOTP. I had three choices:

Submit PR(s) and postpone everything for months (slow),
Use raw HTTP requests instead of python-keycloak in part of the code (ugly),
Create a library that supports ALL the endpoints, so I will never be in this position again (fun!).

It took a weekend for the code, and a few more for the publishing, documentation, and all the other ornaments of an open-source project. But there it is!

Getting started with mantelo

Mantelo is hosted on GitHub, available on PyPi, and hosts its documentation at readthedocs: https://mantelo.readthedocs.io/en/latest/.

🏁 Installing

To no one's surprise:

pip install mantelo

🔐 Authenticating

The first step is to create a KeycloakAdmin and specify a way of authenticating. Mantelo supports natively username/password and client credentials (OpenID), taking care of tokens and refreshes behind the scenes.

More details are available in the docs, but for the demo let's use the default admin user with the default admin-cli client and connect to the master realm running on a local Keycloak instance:

from mantelo import KeycloakAdmin

adm = KeycloakAdmin.from_username_password(
    # base Keycloak URL
    server_url="http://localhost:8080",
    # realm to interact with (and authenticate to if not specified otherwise)
    realm_name="master",
    # Client to use for authentication
    client_id="admin-cli",
    # User to use for authentication
    username="admin",
    # Password to use for authentication
    password="CHANGE-ME",
)

ℹ This code doesn't interact with Keycloak: Mantelo is lazy and only fetches/refreshes the token when needed (e.g. on the first request).

📞 Making calls

You are now ready to interact with the Admin API. One advantage of mantelo is that it doesn't provide any specific method, it only offers an object-oriented interface to the RESTful API. It ensues any endpoint present in the Admin API is callable using the adm object.

First, let's look at an example. Let's say you need to manage users. Here is a sample of what you can do:

# Get the list of users
# ⮕ GET /admin/realms/{realm}/users
adm.users.get()
# Search users for "foo bar"
# ⮕ GET /admin/realms/{realm}/users?search=foo+bar
adm.users.get(search="foo bar")
# Get user count
# ⮕ GET /admin/realms/{realm}/users/count
adm.users.count.get()
# Get a specific user by ID
# ⮕ GET /admin/realms/{realm}/users/725209cd-9076-417b-a404-149a3fb8e35b
adm.users("725209cd-9076-417b-a404-149a3fb8e35b").get()
# Create a new user, with credentials
# ⮕ POST /admin/realms/{realm}/users
adm.users.post({
    "username": "example",
    "firstName": "Winston",
    "lastName": "Smith",
    "email": "orwell@1984.uk",
    "emailVerified": True,
    "enabled": True,
    "credentials": [
        {
            "type": "password",
            "value": "juliaMyLove",
        }
    ],
})

Did you get it? Instead of implementing methods to reach endpoints, mantelo uses simple rules to translate Python code to HTTP calls. More formally:

Each property is a path appended to the base URL. If the server_url is https://localhost:8080 and the realm is master, the base URL is https://localhost:8080/admin/realms/master. Underscores are automatically translated to dashes.
The same goes for methods, useful for specifying path parameters: the first argument is appended to the path.
The actual HTTP call is triggered by ending a "chain" with a call to one of:
- .get(**kwargs)
- .options(**kwargs)
- .head(**kwargs)
- .post(data=None, files=None, **kwargs)
- .patch(data=None, files=None, **kwargs)
- .put(data=None, files=None, **kwargs)
- .delete(**kwargs)
All the above methods have kwargs which are appended as query parameters.
POST/PUT/PATCH support data (a dictionary converted to JSON) or files for specifying the body of the request.

Every request returns either the content of the response's body (parsed from JSON) or an instance of HttpException if the response's status code is not in the 2XX range.

And that's it! This magic is possible thanks to slumber, a library I will feature in a separate article (stay tuned 😉).

The best library out there, really?

I stated earlier I believe mantelo is the best Keycloak Admin library. Let me summarize the premises of such a bold claim:

mantelo's footprint is ridiculous: it is currently less than 1000 lines of code and has only 3 direct dependencies.
mantelo is the only library that can rightfully claim it supports the whole Keycloak Admin interface and is always up-to-date.
mantelo abstracts away authentication (and refresh tokens), which is always tricky to get right.
mantelo gives you access to the exact URL that was called (or the requests.Response in case of an error) so debugging is easy.
mantelo is flexible: you can tweak it easily if you need to.

I am aware wrapping instead of implementing has a drawback: if Keycloak changes the Admin API, your code has to be updated (no encapsulation). But I believe it is a small price to pay, especially since Keycloak proved to be careful about retro-compatibility. What do you think?

Mantelo needs YOU

I've put a lot of work into this library, not just on the code, but everything around it—like documentation, making it easy to test, setting up the GitHub repo, and more.

But mantelo isn't complete without users and a community. So please, give it a shot, star it, criticize it, open an issue... I'm excited to see where we can take it together!

Ever wondered how cloud providers (PaaS) integrate with GitHub? I did.

Lucy Linder — Tue, 06 Feb 2024 13:15:00 +0000

With GitHub standing out as the leading platform for hosting public code, most cloud providers offer dedicated features for deploying code hosted on GitHub. I have always used them without much thought. That is, until recently.

Dissecting their inner workings, I found various approaches, some better than others. This gave me a deeper understanding of the GitHub tools available, how they work, and, well... I found this fascinating and wanted to share! So here it is.

The contenders
Google Cloud Run: all the power of GitHub Apps
- Overview
- What is a GitHub App?
- The process
- Wait... What?
- Analysis
Azure Static Web Apps: reversing the responsibility
- Overview
- What is a GitHub OAuth App?
- What is a GitHub Actions workflow?
- The process
- More about the Azure GitHub Action
- Analysis
Heroku: keep it simple
- Overview
- The process
- Analysis
Summary

The contenders

Today, we look at Google Cloud Run, Azure Static Web Apps, and Heroku. They are not the sole cloud providers out there (not even the bests, check out divio 😉), but all three are relatively famous and have some sort of GitHub integration.

By GitHub integration, I mean the cloud provider has a dedicated process to link with a repository hosted on GitHub and provides additional services such as automatic deployments on push or preview apps on pull requests.

Google Cloud Run: all the power of GitHub Apps

From their landing page, Google Cloud Run allows you to:

Run frontend and backend services, batch jobs, deploy websites and applications, and queue processing workloads without the need to manage infrastructure.

I use Cloud Run to deploy my little rickroller app.

Overview

When setting up a new service in Cloud Run, you can choose to "continuously deploy revisions from a source repository". This supports BitBucket, Cloud Repositories, and of course, GitHub. When the latter is selected, here is what the wizard looks like:

There are two steps here, let's break them down.

What is a GitHub App?

Cloud Run uses a GitHub App (called Google Cloud Build), which is a compelling concept. In short, GitHub Apps can act on your behalf, act on themselves (for example open issues), or act outside GitHub by reacting to events (via webhooks). Each App must declare the permissions it requires (e.g. only read repository contents), which determines the subset of API endpoints they can access.

Something fundamental is that GitHub Apps are installed at the organization level. An App just needs to exist to impersonate you, but it needs to be installed to act on its own. When installing the app in an organization, it is up to you to define the list of repositories it can act on (all, or a selected few).

The process

Back to Cloud Run. In the first step, Google tells you to authenticate. Here, it is using OAuth2 to retrieve an access token to act on your behalf. This token will have the permissions both you AND the app have (impersonation). It is important to note that currently, GitHub App user tokens are valid for 8 hours (with a refresh token valid for 6 months)! So don't forget to revoke access after setup.

Once authenticated, Google uses the token to ask the GitHub API about the installations of the GitHub App you have access to, and for each installation which repositories the installation can manage. This list is used to populate the second dropdown field.

The "manage connected repositories" redirects you to GitHub, where you can see your organizations and potentially install (or update the installation of) the App. After installing an app on an organization (and giving it access to some repos), Google updates the dropdown list displayed in the wizard.

Finally, once you select a repository in the list, Google saves somewhere the repository information and the corresponding installation ID of the App. With this, Google can authenticate as a GitHub App installation and start doing things. In this case, "doing things" is reacting to push events (via a webhook) to pull and redeploy your code on Cloud Run (and many other cool things that I won't list there). Its range of actions is limited by the set of permissions of the App.

Wait... What?

Why all this complexity you ask? Let's break it down one more time:

Why the need for a user access token in the first step? A GitHub App can list all its installations, but cannot determine which of those you have access to.
Why the need to install the app in the second step (and not just use the user access token)? The user access token generated by a GitHub App has an expiration date. Google needs to perform actions without you being around to give your blessing. Only installations can request access tokens to act on themselves.

Analysis

The way Google Cloud Run works may seem overly complicated, but there is currently no easier way to associate a GitHub App with a specific repository on GitHub (except by asking the user to enter manually the name of the repo and the installation ID, which is not user-friendly). Once Google associates a GitHub App Installation ID to the repository, it has all the power a GitHub App entails: it can do anything, being only limited by its permissions (which can be updated). This is a very stable and future-proof way of integrating with GitHub.

Another advantage of GitHub Apps is that they act on their own. If they publish a comment on a pull request or do a push, you will see the name of the App as the author. This allows for easy monitoring.

A user is free to opt out at any point by deleting the GitHub App installation.

Azure Static Web Apps: reversing the responsibility

Azure Static Web Apps offers "static content hosting and dynamic scale for integrated serverless APIs".

Overview

Contrary to Google, which implements a "pull" mechanism, Azure went for the "push" approach, taking advantage of GitHub OAuth Apps and GitHub Actions.

What is a GitHub OAuth App?

Unlike GitHub Apps, you do not install OAuth Apps because they can only act on behalf of a user. They cannot act on themselves as GitHub Apps can. It is not possible to limit the set of repositories an OAuth app can access, only their permissions (e.g. read vs read/write), and their access tokens never expire. There is no reason to use OAuth Apps nowadays, but they were the first historically, thus many integrations still use them.

To know more, read Differences between GitHub Apps and OAuth apps (the GitHub documentation is amazing, btw).

What is a GitHub Actions workflow?

From the documentation,

A workflow is a configurable automated process that will run one or more [GitHub Actions] jobs. Workflows are defined by a YAML file checked in to your repository and will run when triggered by an event in your repository, or they can be triggered manually, or at a defined schedule.

To paraphrase, workflows allow running automated tasks on GitHub (using runners) to implement CI (Continuous Integration, such as running tests) and CD (Continuous Deployment, such as building and publishing a Docker image or deploying a package to NPM).

Gitlab and others also have CI/CD. What is specific to GitHub is the notion of GitHub Actions. A workflow is composed of jobs that have a list of steps. A step can be a shell script or a call to an action. Think of an action as a way of packing complex logic into a simple component that can be reused, thus extending GitHub capabilities. Public Actions are stored on public GitHub repositories and may be published to the MarketPlace.

The process

So, back to the wizard. First, Azure asks you to authenticate to GitHub (using the OAuth App), so it can act on your behalf. Next, it retrieves the list of your repositories and branches from the GitHub API to populate the dropdowns. Once you choose the target repository, it asks you to select a "build preset" (it currently supports frameworks such as React or Angular, and static website generators such as Gatsby or Hugo). Those presets are used to generate a GitHub Workflow file, which itself uses a GitHub Action maintained by Azure called static-web-apps-deploy.

When hitting deploy, Azure:

commits the workflow file in the .github/workflows directory of your branch (with you as the author), and
updates your repository settings to add environment secrets (under Settings > Secrets and Variables > Actions), such as the secret token used by the workflow to authenticate to Azure.

Note that this is a one-time action. Afterward, everything is done from GitHub through the magic of the Azure GitHub Action. In other words, it is the workflow running in a GitHub runner that calls an Azure API endpoint to deploy a new version. The responsibility is reversed.

Note that you don't need to use the wizard, you can also manually create the workflow file in your repo (and add the necessary secrets) and the result would be the same. I would even encourage you to go the manual way, to avoid granting Azure any permissions.

More about the Azure GitHub Action

The default workflow committed by Azure can be seen here (with slight modifications depending on the preset): https://learn.microsoft.com/en-us/azure/static-web-apps/build-configuration?tabs=github-actions#build-configuration. Unfortunately, the GitHub Action, Azure/static-web-apps-deploy, hides its source code inside an executable from a Docker image. A shame, because I would love to analyze it!

With the default workflow, your Azure static web app is redeployed with the latest changes on every push to the main branch. Here is an output of the GitHub Action run:

Zipping App Artifacts
Done Zipping App Artifacts
Uploading build artifacts.
Finished Upload. Polling on deployment.
Status: InProgress. Time: 0.0628832(s)
Status: Succeeded. Time: 15.1286488(s)
Deployment Complete :)
Visit your site at: https://blue-sea-084206610-1.centralus.4.azurestaticapps.net
Thanks for using Azure Static Web Apps!

Whenever a pull request targeting the main branch is opened, the action deploys instead a preview environment and adds a comment to the pull request with the preview URL. Every change on the pull request will update the preview environment. The preview environment is torn down when the pull request is closed (merged or discarded).

It is possible to configure preview environments for other branches than main with a few changes to the GitHub Action's parameters.

And all of this works flawlessly! The only thing I had to change was the default permissions of the workflows (in Settings > actions > workflow permissions) from read to read/write to allow the Action to post comments on pull requests (see this issue for details).

Analysis

I find the Azure approach brilliant.

Thanks to the GitHub Action, Azure itself doesn't need to know anything about git or GitHub (the content to deploy is sent as a zipped archive). There are no permissions involved since the workflow runs inside your repository. Reacting to pull requests is also easier since it is a built-in feature of workflows. To do the same with a GitHub App, you would require additional permissions, plus a server somewhere reacting to webhooks.

As a bonus, users can freely choose the version of the action they use, and parametrize it to their liking in a Configuration as Code (CAC) manner. There is no magic outside the repo itself. A GitHub Action is thus more transparent, and its power is only limited to the permissions of the GITHUB_TOKEN configured in the workflow.

The only drawback is the use of an OAuth App for automatic setup. OAuth App tokens remain active until they're revoked by the user, meaning Azure may still act on your behalf on all your private repositories after the setup unless you manually do something. I believe Azure discards the token after creating the workflow, but still. My two cents: revoke access to the Azure OAuth App after setup!

In short, the GitHub Action approach is less expensive, more sustainable, and user-friendly. Good job!

Heroku: keep it simple

Heroku is a container-based cloud Platform as a Service (PaaS). Acquired in 2010 by Salesforce, it hosts your applications in AWS (without you having to know this fact).

Overview

Heroku takes a simpler approach, combining a GitHub OAuth App and a repository webhook, while still offering advanced features such as preview apps (deployed on pull requests), automatic deployments, and more.

The process

To enable GitHub integration, Heroku asks you to allow the Heroku Dashboard OAuth application and give it full control over all your public and private repositories across all the organizations you are admin of. Because it uses an OAuth app, there are no fine-grained permissions: it has access to all or nothing.

Once authorized, you can search (no dropdown 😳) the name of a repository and "enable" it. Upon pressing enable, Heroku adds a webhook to your repository. From now on, every action on your repository will notify https://kolkrabbi.heroku.com/hooks/github. The webhook is visible under your repository's Settings > Webhooks.

With the webhook in place and the ability to act as you on GitHub (remember that OAuth user tokens do not expire), Heroku is now ready. In the Heroku settings, you can now enable automatic deployments, Review Apps, and more.

In other words, when something happens on the repository, Heroku gets notified via the webhook and takes action depending on the Heroku settings. If it needs to interact with GitHub, it impersonates you. This is how you can see messages on pull requests such as "<your username> deployed <your app> X seconds ago".

Analysis

This approach is efficient because, with only one action from you (the authorization of the App), Heroku can do everything. Adding new features is completely transparent because it listens to all events and asks for all the permissions you have on GitHub.

The downside? You are asked to trust Heroku completely and have no way of ensuring it doesn't abuse its power. A security breach such as the one of April 2022 (attackers accessed OAuth2 tokens from Heroku and used them to steal secrets stored in private GitHub repositories) can yield big consequences with unrestrained OAuth tokens. Remember: "OAuth tokens remain active until they're revoked by the customer". It is up to Heroku to ensure it secures and rotates the tokens properly. Hopefully, they have competent engineers :).

To be fair, Heroku supported GitHub integrations before GitHub Apps appeared, and is aware of the issue:

GitHub Apps allow for finer grained control on repositories, but that has not been integrated into the GitHub integration yet. We cannot commit to a timeline but it’s definitely on our radar. Keep an eye on the Heroku Changelog.

Given their current implementation works perfectly, I doubt the migration to GitHub Apps will be prioritized soon (unless another security breach forces their hands).

Summary

This article looked at three different ways for cloud platforms to interact with GitHub:

Provider	Approach	Notes
Google Cloud Run	Pull → GitHub App (+app webhook)	Limited permissions. Acts as you during setup, acts as itself otherwise.
Azure Static Web Apps	Push → GitHub Action	No permissions. Optional GitHub OAuth for setup only (theoretically one shot).
Heroku	Pull → GitHub OAuth App (+ repository webhook)	Full permissions, always acts as you.

In my opinion, Azure is the clear winner with an original push approach. You are responsible for adding a workflow in your repository and you control what happens end-to-end. There is no hidden magic, and you are not required to give Azure any access to GitHub whatsoever, although you can choose to do so to benefit from a guided setup.

Google comes second, with the most correct way of implementing a pull approach as of 2023. GitHub Apps have fine-grained permissions and act as themselves, making their actions more transparent on GitHub. GitHub Apps are a bit more complex to manage though, with this whole authorization versus installation concept, and requires a very good wizard to make it simple for users. Google's wizard is not the best in class in this regard (I am not surprised).

Heroku's implementation based solely on a GitHub OAuth App is outdated and suffers from all the downsides pushing GitHub to ship GitHub Apps in the first place: lack of fine-grained permissions, Apps acting on behalf of (admin) users, no expiration of tokens, and more. It is however the easiest to put in place and maintain from a vendor perspective, and also the easiest for the user.

The main takeaways from this (already too long) article are:

implementing a GitHub integration is not straightforward and must be planned carefully.
GitHub is an amazing product with many points of extensions: GitHub Apps, GitHub OAuth Apps, and GitHub Actions are today the most prevalent.
As a GitHub user, you may often be asked to authorize an extension. Learn how to read the prompts properly and always think of the security implications before saying yes.
Don't forget to review permissions regularly, and revoke access to Apps you do not use anymore. On GitHub, click on your avatar on the top-right, Settings > Applications.

I hope you enjoyed this article and learned something, let me know in the comments what you think and if you have a better integration approach undocumented here.

Diving Deeper into Python Exceptions

Lucy Linder — Tue, 09 Jan 2024 14:00:00 +0000

I have been coding in Python for a long time, yet I am puzzled by how little I knew about Exceptions. This post is about some of my recent findings on this topic.

Content

A little story
Exception chaining (and the magic of __context__ )
Bare except vs except Exception
Raising shorthands
Annotating exceptions (3.11+)
What about UserWarning?
Bonus

A little story

I had an interesting use case at work lately: some external library code was "swallowing" another exception, and I needed to get back the message of the initial one somehow, without touching the library itself.

The library code looked something like this:

class TokenBackend:
   def decode(self, token: str) -> dict:
      # Decode a JWT token,
      # It may fail for many reasons, detailed in the
      # TokenError message
      if not token:
         raise TokenError("Empty token")
      if len(token.split(".")):
         raise TokenError("A JWT must have 3 parts")
      if ...:
         raise TokenError("Invalid signature")
      # ...

class Token:

   def __init__(self, raw_token):
      # ...
      try:
         self.backend.decode(token) # <- calls TokenBackend.decode
      except TokenError:
         # Here, the library swallows the exception,
         # we LOSE the detailed error message!
         raise DecodeError("Token could not be decoded")

I had no clue how to do this except to override the Token class in my codebase. When I asked my boss about this, he looked at me and said: "just use __context__". Huh? Never heard of it. I started digging, and long story short: he was right. This was the perfect solution.

Those small discoveries happened a lot lately, and I wanted to share them. If this intrigues you, keep reading!

Exception chaining (and the magic of `context` )

So, what is this __context__??

Formalised in PEP 3134 (I love PEPs), exceptions in Python 3 have three dunder attributes that provide information about the context in which they were raised: __cause__, __context__ and __suppress_context__. To understand, let's first make sense of implicit and explicit exception chaining.

An exception chain starts when a new exception is raised during the handling of another, for example from an except clause:

try:
    open("foo.bar")
except OSError:
    raise RuntimeError("oops")

Traceback (most recent call last):
  File "<stdin>", line 2, in <module>
FileNotFoundError: [Errno 2] No such file or directory: 'foo.bar'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 4, in <module>
RuntimeError: oops

This is called an implicit chain (hence the "During handling ..."). To make it explicit and clearly state an exception is the cause of another, one can use the special raise ... from :

try:
    open("foo.bar")
except OSError as e:
    raise RuntimeError("oops") from e

Traceback (most recent call last):
  File "<stdin>", line 2, in <module>
FileNotFoundError: [Errno 2] No such file or directory: 'foo.bar'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "<stdin>", line 4, in <module>
RuntimeError: oops

As you can see, we now have another log message: "was the direct cause of". Of course, chains can be longer than two.

To go back to the context attributes of an exception:

__suppress_context__ is false by default.
When raising a new exception while another exception is already being handled (in except, finally or with), the new exception’s __context__ attribute is automatically set to the handled exception.
When using raise ... from , the supplied exception will additionally be saved in the __cause__ attribute of the raised exception, and __suppress_context__ will be set to true.

The default traceback uses those attributes to display stacktraces in the following way:

if __cause__ is present, always show it
if __cause__ is None, show the __context__ only if __suppress_context__ is false.

To ensure you followed, what does this valid Python code prints?

try:
   1/0
except ZeroDivisionError:
   raise RuntimeError("zero!") from None

It only shows the RuntimeError, because the from will set the __cause__ (to None) and the __suppress_context__ (to true).

Now, this doesn't completely swallow the original exception. Even when using a from, the initial ZeroDivisionError is still in the __context__, just ignored when printing the stacktrace.

Back to the problem in the introduction, I simply catch the exception e raised by the library (in Token.__init__), and then use e.__context__.args[0] to get the initial exception message.

Bare except vs except Exception

I learned this one from the ruff rule bare-except (E722). When you don't care about which exception is raised, you may be tempted to use a bare except (but should NOT):

try:
   do_something()
except: # <- no exception class is called bare except
   print("oops")

A bare except catches BaseException

BaseException is the common base class of all exceptions. One of its subclasses, Exception, is the base class of all the non-fatal exceptions. Exceptions which are not subclasses of Exception are not typically handled, because they are used to indicate that the program should terminate.

This except thus catches Exception, but also KeyboardInterrupt, SystemExit, and other fatal errors, making it hard to interrupt the program (e.g., with Ctrl-C) and potentially disguising other problems or leaving the program in an unexpected state.

So instead, always specify an exception type, or simply Exception if you are in doubt:

try:
   do_something()
except Exception: # now we are good
   print("oops")

Raising shorthands

When re-raising inside an except clause, you don't need to pass an argument to raise, as it re-raises the caught exception by default.

try:
   x / y
except ZeroDivisionError: # no need to add "as e"
   log.error("we got a zero here")
   raise

Similarly, when raising an exception with no argument, no need for parentheses. If an exception class is passed to raise, it will be implicitly instantiated by calling its constructor with no arguments.

Hence the following is perfectly valid and more concise (see ruff rule unnecessary-paren-on-raise-exception (RSE102)):

raise ValueError

Annotating exceptions (3.11+)

Since Python 3.11, it is possible to attach notes to exceptions, effectively enriching their context. This is a very interesting feature, that could replace re-raising an exception with a different message.

try:
  try:
    raise ValueError
  except Exception as e:
    e.add_note("This was raised as an example")
    raise
except Exception as e:
  e.add_note("Great article btw!")
  raise

Traceback (most recent call last):
  File "<stdin>", line 3, in <module>
ValueError
This was raised as an example
Great article btw!

Those notes are saved in the __notes__ attribute (list of strings).

What about `UserWarning`?

If you look at the Python documentation, you will see some strange built-in exceptions such as UserWarning, DeprecationWarning, etc. They all inherit from Warning (which itself inherits from Exception) but they are NOT meant to be raised. Instead, they are used as warning categories.

In short, Warning exceptions are to be used with the warnings module. There is much more to it, but let's look at a simple example:

import warnings

def foo():
    # Explicit category
    warnings.warn("Don't use me anymore!", DeprecationWarning)
    # Implicit category
    warnings.warn("bar") # <- default to UserWarning

What is nice about warning is that users have complete control over what is reported, thanks to the warning filter:

foo()
# <stdin>:3: DeprecationWarning: Don't use me anymore!
# <stdin>:5: UserWarning: bar
foo() # second time, no more warnings printed
---
warnings.simplefilter("ignore")
foo() # -> nothing printed
---
warnings.simplefilter("error")
foo() # -> raises!
# Traceback (most recent call last):
#   File "<stdin>", line 1, in <module>
#   File "<stdin>", line 3, in foo
# DeprecationWarning: Don't use me anymore!

For all available filters, see The Warnings Filter.

So, why use exceptions for that? You guessed it, it simplifies turning warnings into exceptions (error filter): one just has to raise it.

Bonus

This article is already way too long, so here is a bullet list of other interesting subjects and picks:

Python 3.11 introduced ExceptionGroup, a nice way to pack multiple exceptions into one. The new syntax except* allows filtering groups efficiently. Find out more in the documentation.
Python supports try-except-else-finally, although I never found a good use case for the else (also supported in for loops). The else block is executed after the try block, but before the finally block (if the except block doesn't run). Exceptions raised inside the else block are not caught by the except. See Handling exceptions for more info.
The NotImplementedError (not to confuse with the constant NotImplemented) signals a missing implementation that should come one day. If the feature will never be implemented, raise a TypeError instead.
Exceptions store their __init__ arguments in the args attribute.
I am always tempted to name my custom exception classes with the Exception suffix (a remnant of Java perhaps?). However, PEP 8 clearly states we should use the suffix Error for exception class names.
It is possible to catch multiple exceptions using parentheses: except (FooException, BarException)
Never return in a finally: this will override whatever return you may have inside the try or the except.

I haven't written in a while, so I hope you enjoyed this article.

- With ❤️, @derlin

Python, type hints, and future annotations

Lucy Linder — Wed, 13 Sep 2023 12:10:00 +0000

Not long ago, I came across a bug in one of my projects that highlighted very interesting changes in how Python handles type hints at runtime.

This made me dig into the future of type annotations, and what it means for libraries manipulating type hints at runtime. Spoiler alert: It has to do with PEP 563.

The bug
A brief history of type hints
- Basics of type hints
Introducing PEP 563, or what lies behind future annotations
The explanation
How to get the return type of a function after PEP 563?
- In Python 3.10 and newer
- In Python 3.9 and older
A word of caution
Conclusion

TOC generated with bitdowntoc.

The bug

I have a Django API (django-rest-framework - drf) which uses drf-yasg to generate an OpenAPI schema at runtime. You don't need to know the intricacies of those libraries, just that API endpoints return serializers, and that drf-yasg inspects them at runtime to generate a schema with the proper fields and types.

Now, the problem arose with a snippet similar to this:

from rest_framework import serializers

class MySerializer(serializers.Serializer):
   # This allows to return something that is not a direct
   # attribute of a model, but a "computed" value (function)
   is_foo = serializers.SerializerMethodField()

   # The type hint (-> bool) is used by drf-yasg to
   # guess the proper type is "is_foo"
   def get_is_foo(self, obj) -> bool:
      return obj.is_foo

Here, drf-yasg determines the type of is_foo based on the type hints of get_is_foo. The annotation says bool, so this converts correctly to the OpenAPI schema {"is_foo": "BOOLEAN"}.

However, upon refactoring, I added the following at the top of the file:

from __future__ import annotations

Suddenly, the schema became {"is_foo": "STRING"}. What?? (Note that it took me a while to pinpoint that the problem came from this import, but let's keep the story short).

If you understand why, you can stop here. If, like me, you find this utterly confusing and want to understand what is going on, keep reading!

A brief history of type hints

(I digress briefly, so feel free to skip this section).

In 2006, Python 3 introduced a syntax for adding arbitrary metadata annotations to Python functions (PEP 3107: Function Annotations). (Ab)used by many 3rd parties in different ways, Python 3.5 finally formalized the semantics in PEP 484: Type Hints (Python 3.5, October 2014 - see also PEP 483 – The Theory of Type Hints) and added the typing module to the standard library.

Over the years, the type hinting system evolved, offering new syntaxes and meta-classes. This is not only drastically changing the way we write Python code, but also the way we ship it to the masses.

Since Python 3.7, packages can incorporate type information and stubs files, making the life of IDEs and type checkers such as MyPy, pytype, etc. easier (It does, however, make the life of library maintainer a bit more complicated). If you want to know more, read PEP 561: Distributing and Packaging Type Information.

This is to say, type hints are now really taking off. I would not be surprised to find type hints ubiquitous in a few years. Javascript is taking the same route, deviating from the purely scripted language to something more structured.

Basics of type hints

I wasn't aware of most of the intricacies of type hints until I tried migrating a large project to mypy (a good read if you are in a similar process), but I assume most of you are now familiar with the basics of type hints:

from typing import List, Dict, Optional

def foo(a: List[int], b: Dict[str, object], c: int) -> Optional[None]:
  ...

def bar() -> bool:
  ...

Note that foo can now be rewritten without imports, thanks to PEP 585 – Type Hinting Generics In Standard Collections (Python 3.9, March 2019) and PEP 604: Allow writing union types as X | Y (Python 3.10, May 2020):

def foo(a: list[int], b: dict[str, object], c: int) -> str | None:
  ...

Introducing PEP 563, or what lies behind future annotations

If you ever tried using new type hinting features in old Python versions or struggled with forward references, you may have stumbled upon (and used) the famous future import:

from __future__ import annotations

def foo(a: list[int], b: dict[str, object], c: int) -> str | None:
  ...

Adding this line at the top of the file makes the snippet "work" (that is, not throw exceptions) from Python 3.7. Ever wondered why?

Well, the only thing this import does is turn on PEP 563: Postponed Evaluation of Type Annotations:

This PEP proposes changing function annotations and variable annotations so that they are no longer evaluated at function definition time. Instead, they are preserved in __annotations__ in string form.

Treating all annotations as string at runtime have clear advantages, including:

It removes issues with forward references. Annotations can now refer to a name that has not yet been defined.
It gives a simple way to deal with cyclic references. We can import the names when running type checks (if typing.TYPE_CHECKING: <imports>) and leave them out at runtime.
It improves runtime performances. Type hints are no longer executed at module import time, which is computationally expensive.

This little trick, turning type hints into strings, also explains why we can use new type hint features in old Python versions. Even if str | None is invalid in Python 3.9 (TypeError: unsupported operand type(s) for |: 'str' and 'NoneType'), the string 'str | None' is perfectly fine!

All those advantages explain why PEP 563 is planned to become the default behaviour soon:

from __future__ import annotations was previously scheduled to become mandatory in Python 3.10, but the Python Steering Council twice decided to delay the change (announcement for Python 3.10; announcement for Python 3.11). No final decision has been made yet.

The explanation

So, back to our bug.

For a long time, the inspect module was the best way to determine a method's return type using annotations. It is what the drf-yasg library uses in https://github.com/axnsan12/drf-yasg/blob/1.21.7/src/drf_yasg/inspectors/field.py#L616:

import inspect

def foo() -> bool:
  return True

sig = inspect.signature(foo)
assert sig.return_annotation == bool

This worked perfectly until PEP 563: Postponed Evaluation of Type Annotations came along. When enabled, all annotations are suddenly str, so that was previously the type bool becomes the string 'bool'!

How to get the return type of a function after PEP 563?

For now, library maintainers need to assume annotations may or may not be stringized, and support both. This has been made easier for Python 3.10, but the transition will take time. Let's review our options.

In Python 3.10 and newer

From the documentation:

Python 3.10 adds a new function to the standard library: inspect.get_annotations(). In Python versions 3.10 and newer, calling this function is the best practice for accessing the annotations dict of any object that supports annotations. This function can also “un-stringize” stringized annotations for you.

To turn on the "un-stringize" feature, simply pass the parameter eval_str=True :

from __future__ import annotations

def foo() -> bool:
  return True

import inspect
# eval_str does the magic
ann = inspect.get_annotations(foo, eval_str=True)
assert ann["return"] == bool

Under the hood, this new function looks up the foo.__annotations__ dictionary and calls eval on values of type str.

If you never heard of __annotations__, it is a mutable dictionary "mapping parameter names to the evaluated annotation expression". It was introduced in Python 3.0 in 2006 (see PEP 3107 – Function Annotations). As its name suggests, return is a special entry key reserved for return annotations.

Note that inspect.signature is also updated to support eval_str, but it just defers the work to get_annotations(), so better to call the latter directly.

In Python 3.9 and older

In older versions, getting the actual type is tricky, even before PEP 563.

First, as explained in Accessing The Annotations Dict Of An Object In Python 3.9 And Older, just getting the annotation has some flaws. Without going into the details, the safest is to use this code snippet:

def get_return_annotation(func):
  if isinstance(o, type):
    ann = o.__dict__.get('__annotations__', None)
  else:
    ann = getattr(o, '__annotations__', None)
  return (ann or {})["return"]

This can return a type, a str, or None.

In the case of a str, we may need to "un-stringize" it. Again, the doc gives some pointers in Manually Un-Stringizing Stringized Annotations, but the gist is to use eval and see if this works. But there are many catches... So what do we do?

From my own experience (and deviating from the docs), I suggest one of two approaches:

Either give a shot at get-annotations. This is a very small library that backports inspect.get_annotations() (with eval_str support 🤗) to older Python versions. It doesn't have many stars on GitHub but seems to deserve more.

Or try typing.get_type_hints. Python 3.7's doc says "In addition, forward references encoded as string literals are handled by evaluating them in globals and locals namespaces", which roughly translates to "if str, I will try my best to still return a type".

A word of caution

future.annotations lets you use anything as annotations, including features that are not yet available (or even "garbage"). This is very dangerous if you plan on reading types at runtime.

For example, if you are using list[str] or str | None in a version that does not yet incorporate them, any attempt to unstringize the annotation will raise an exception (e.g. unsupported operands type(s) for |).

Thus, the safest (especially for libraries) is to stick with the features present in the lowest version of Python you support. In the example above, this means using typing.List and typing.Union instead before Python 3.9.

Conclusion

The advent of PEP 563: Postponed Evaluation of Type Annotations will greatly change the way Python code can access types as runtime, as it turns all annotations from type to str representing types.

Before this PEP becomes the default (Python 3.12?), it can already be activated by importing __future__.annotations. Any annotation in the same file automatically becomes stringized before being stored in the __annotations__ dict.

If a piece of code needs to get the return type of a function (or any other annotation) at runtime, it needs to take this new reality into account.

In summary, here are the different ways of properly getting types at runtime:

inspect.get_annotations(f) → only available for Python 3.10+, it is the new best practice. It deals properly with stringized annotations when eval_str=True.
inspect.signature(f) → properly get the types, but only supports eval_str since Python 3.10.
f.__annotations__ → dangerous before Python 3.10, due to some quirks. Never access it directly unless you know what you are doing!
get_annotations.get_annotations(f) → very small library that offers eval_str starting from Python 3.6.
typing.get_type_hints(f) → states in the docs that it will do its best to deal with stringized annotations, but test thoroughly before use.

Keep in mind that stringized annotations let you write anything, but as soon as you try to unstringize them, exceptions will be thrown. So no str | None before python 3.10!