DEV Community: Alexis Tacnet

The Modern Way To Call APIs In Python

Alexis Tacnet — Wed, 13 May 2020 13:32:42 +0000

Even if I was quite skeptical at first, I now truly believe that asynchronous in Python will shape the future of the language. The ecosystem is more and more developed and a lot of amazing people are getting on board the async train to help Python developers manage this new way of coding. For sure, it's more complicated to write ansynchronous than synchronous code, but a lot has changed in the past few years, making the entry barrier lower that it has ever been.

At Strio, we use mainly FastAPI for our front API. It is an asynchronous web framework that allows us to achieve good performance with little effort. During a request from our frontend, our other clients, or directly from our customers, we usually have to call a bunch of other parties: external APIs, AMQP brokers, SQL or NoSQL databases, ... Our API plays the role of glue between all kind of services, and since all those calls are quite long, we can leverage asynchronous code to maintain impressive performance for our clients.

Among the services that we call, we have many HTTP requests to make. In the modern era of microservices, this is typically what you expect from a front API: when something quite complex has to be done, you just call the service in charge of it and let it do it. So, let's try to build the modern way of communicating asynchronously with those HTTP APIs.

The HTTP library

The first block that we need is the HTTP library. Why use a HTTP library? HTTP is a standard and you can't really avoid it, however this protocol can be hard sometimes and the reference is rather full of tricks. Fortunately, some people made some great HTTP library for python that we can use.

The one I really like is httpx, created by the encode team, in which you will find some of the most famous Pythonistas out there. httpx is basically requests, but with async support, typing and even more. Its simple syntax inspired by requests makes it really easy to use and understand, especially for someone that starts with asynchronous code and needs to read it.

Let's just take a glance at how we could make a simple GET requests, but asynchronously:

>>> import httpx
>>> async with httpx.AsyncClient() as client:
...     r = await client.get('https://ifconfig.co/json')
...
>>> r
<Response [200 OK]>

This is the Asyncio interpreter, launched with python -m asyncio

The goal of this brick that I introduced is to make the HTTP call and return the response in a typed and verified model. 90% of the time we will then need to access the JSON body of this response, that we need to validate.

Response data validation

The next brick is the data validation of the response's body given by HTTPX. Why do we need validation? This helps us as developers to build typing for the response, which is a good thing for auto-completion and developer productivity, but also validate the data before going further. If the API returns something unexpected, we want to return a friendly exception now, instead of an AttributeError later on in the request.

For this job, I selected pydantic, a really good library that makes checking and validating data simple. Let's see how we can integrate it in our request pipeline:

>>> from pydantic import BaseModel
>>> from ipaddress import IPv4Address
>>> class IfConfig(BaseModel):
...     ip: IPv4Address
...     ip_decimal: int
... 
>>> ifconfig = IfConfig.parse_obj(r.json())
>>> ifconfig
IfConfig(ip=IPv4Address('78.153.21.75'), ip_decimal=1807729179)

Now that we have static typing for this response data, we know that ifconfig.ip exists and is an IPv4Address. However the data is also validated, so if the field ip was missing from the response, or if we tried to parse an ipv6, for example, we would get an ValidationError exception, and we could take action for this unusual event.

Creation of an API client

In order to glue those two bricks together, we can integrate them into a class that will be called an API client.

The role of this class is to abstract the API reference logic for different pieces of our codebase, so developers can ignore the underlying layer and focus only on what resources they want to obtain or modify. This class also allows us to separate the external API's definition in a single place in order to test it and refactor it easily.

Let's create our IfConfigClient so that everything in our code, from our own API to our background jobs can query this service easily:

from ipaddress import IPv4Address

from pydantic import BaseModel
from httpx import AsyncClient

IFCONFIG_URL = "https://ifconfig.co/"

class IfConfig(BaseModel):
    ip: IPv4Address
    ip_decimal: int

class IfConfigClient(AsyncClient):
    def __init__(self):
        super().__init__(base_url=IFCONFIG_URL)

    async def get_ifconfig(self):
        request = await self.get('json')

        try:
            ifconfig = IfConfig.parse_obj(request.json())
        except ValidationError:
            print("Something went wrong!")

        return ifconfig

The trick here is to make our client inherit from the AsyncClient class from httpx. This is something that makes everything really simple to develop but also to use. Let's keep this code in a ifconfig.py file, that we can import in our python -m asyncio interpreter:

>>> from ifconfig import IfConfigClient
>>> async with IfConfigClient() as client:
...     ifconfig = await client.get_ifconfig()
... 
>>> ifconfig
IfConfig(ip=IPv4Address('78.153.21.75'), ip_decimal=1807729179)

The good way to call APIs

We built an API client that is quite modern: it is asynchronous, supports typing and validates data. But furthermore, this is the way I would like people to write API wrappers in the future, for several reasons.

Separations of concerns

With this configuration, the API is completely opaque to the user and can be tested properly on its own. If something changes in the API itself, like a new parameter on some endpoint, it can be done globally without breaking all the code currently using this endpoint. This is something that we have been doing in the Python ecosystem for quite some time already, but it is now even simpler to create a client.

Mocking for tests

I really like unit tests for complex functions, but intercepting everything that is going out from my code like HTTP calls is usually quite a mess. Thanks to our proxy, we have a standard way to patch our tests and even create fixtures with typing:

from unittest.mock import patch

import ifconfig

mock_ip = IPv4Address('78.153.21.75')
mock_ip_decimal = 1807729179
mock_ifconfig = ifconfig.IfConfig(ip=mock_ip, ip_decimal=mock_ip_decimal)

@patch.object(ifconfig.IfConfigClient, "get_ifconfig", return_value=mock_ifconfig)
def test_ifconfig_processing(get_ifconfig):
    ...
    assert get_ifconfig.assert_awaited_once()
    ...

Typing everywhere

The client now has integrated typing: no more JSON and dictionnaries that you need to rummage through to get your data. You will have auto-completion with all the fields that you can access from the request response, with their types in your favorite IDE. Coupled with mypy or pyright, it also allows you to perform static type checking ahead of tests and commits.

Data validation

The API client is now in charge of taking an action if something uncommon happens in the API response, like a field disappearing, instead of having an issue when trying to access this field in the code made by someone else. This is essential for testing, but also for maintainability of the code when the API changes.

I am truly happy to see new tools like httpx and pydantic emerge in the Python landspace, I think they make building clean code easier and enforce good standards for complex codebases. This new way of coding external requests will spread. Even the Elasticsearch python maintainer is agreeing when discussing about the future of their python clients.

Demystifying authentication with FastAPI and a frontend

Alexis Tacnet — Sun, 10 May 2020 18:18:06 +0000

Authentication is definitely a hard and complicated problem. Protocols like OAuth2 try to make it simpler, but in fact they make it harder to understand for beginners, as the reference is quite complex and there aren't a lot of good practices available online.

The new framework FastAPI is now our go-to web library for all our projects, as it is very efficient to develop with and it supports amazing typing out of the box. The only issue we have is dealing with authentication when using a JS Frontend in front of it. Let's close this debate once and for all by describing the authentication scheme that I think everyone needs for a simple web application with FastAPI, using an external provider.

The authentication plan

First, we need to have a plan for what we want to do. I have always wondered why authentication tutorials cover hundreds of different cases. In fact, I am convinced that a modern web application shall just need one.

What I mean by modern web application here is a FastAPI Python backend (but you can go with whatever you like as long as it outputs JSON) with a JS frontend (with React or Vue for example). What we need to do is therefore authenticate someone navigating on the frontend, and establish a secure connection with the backend to know who is calling your API.

Simple right? And because we are in the 21st century, let's not use a lame email/password form on the frontend : they are not really secure, and SSO (Single Single-On) is clearly a marketing advantage nowadays. We will therefore use a OAuth authentication scheme to get information and authenticate the user on our website.

The goal of OAuth

You may have heard of it, because every major auth provider supports: Google, Twitter, Github, Facebook, ... but you may wonder why it exists in the first place!

The goal of OAuth in general is to streamline and standardize the use of authorization around the web, especially when it comes to multiple parties like our website and Google for example. How do you access the Google information from our website with a Google login? This is what OAuth is made for.

In this tech story, we have as always both sides : the server and the client. In the case of a external provider like Github, Github is the OAuth server, and our application is the client. The goal of OAuth is to enable the client to query the API server attached to the OAuth server in a secure way.

What we want to do is ask Google who the user on the website is to achieve authentication. We will store this user information in a database, and then secure the connection for this user between the frontend and the backend to identify and authorize API calls.

But keep in mind: OAuth2 is made to make it easy to authorize our server to interact with other parties like Google or Github on behalf of an user. You can then ask them who the user on the website is to make authentication, for authenticating the link between the frontend and our API.

Securing the connection between the frontend and our API

What we need in the end is to authenticate our connection between our frontend (usually a SPA or SSR javascript application) and our own API. For that, we will use the famous JWT token that has several advantages and that I really like.

The protocol is the following:

Google redirects to our application with a code
We ask Google for more information, like the profile picture
We then issue a JWT token with this user data that we will share between the frontend and the backend

This token will be used for every API call to our own backend and will therefore identify the user making the calls.

We don't use OAuth directly to secure the connection between the frontend and our API, because we don't need it, and that is not what it's made for. We only issue a small token, and we verify it for each request in the backend. Secure. Simple :)

Authentication pattern

If you already looked at OAuth2, you may have noticed that several schemes are available and are needed for different types of applications. The two that we are interested in are the Implicit one and the Authorization code scheme.

The Implicit scheme

With the Implicit scheme, our user is redirected to the SSO login (1) and then redirected after authentication to our frontend (2) with the access_token needed to access the OAuth resources (profile picture remember). This is typically useful when we need to access this type of resource directly in the frontend, without bothering making the backend in charge of that.

To secure the connection between the frontend and our own API, you need to give this token to the API (3) that will check with the OAuth server that it is valid (4). If the token is valid, then this user is clearly on the website, so let's give him another token (5) (to check it super quickly without asking the OAuth server every time and control its expiration for example).

As mentioned, our frontend is also capable of querying directly the external API (6), with the access_token, to process on its side the resources the user gave access to.

The Authorization code scheme

With the Authorization code scheme, our user is also redirected to the SSO login (1) and then redirected to our frontend (2), but with a mysterious code that we need to give to the backend (3).

The backend will then give this code (along with the OAuth secret client data, like client_id and client_secret) to the OAuth server that will then return an access_token if the code is correct (4).

If the code is correct, we can use the token given by the OAuth server to access user resources directly from the backend (5), to store a profile picture or a Google doc file that belongs to the user for example. Then, our user is authenticated, so let's finally give him his token (6).

The advantage of this scheme is we don't put the frontend in charge of the access_token of the OAuth server, which has a few security advantages:

The frontend doesn't have to give this token to the backend, which is better for security as a Man-in-the-middle attack could steal your identity. This is not possible with a code as it needs a client_secret which only the backend is aware of.
The backend is the one with the access_token, preventing any malicious extensions from stealing it in your front-end domain.

In our experience, we prefer this scheme for production applications. This is the only one supported by Github for example.

FastAPI implementation

We really like FastAPI at Strio : this framework is simple, efficient, and typing friendly. Creating a authentication scheme on top of it was not that hard, and is really clean. We just have to keep in mind the few tips I described earlier:

OAuth is only for external API access
Simple JWT encoding and decoding between our frontend and backend

In this example, we will use the Authorization Code scheme, with this code forwarded between our frontend and our backend, and the Github external provider.

URLs

The first thing to configure are the different URLs we will use for this scheme:

The LOGIN_URL is the base URL needed to create the URL our frontend will be redirected to
The TOKEN_URL is the one our backend will query with the Authorization code to get an access_token.
The USER_URL is an api endpoint used to get data about the user, it is a typical example of the external API described earlier.
The REDIRECT_URL is the URL of our frontend that Github will need to redirect to, it is useful because it will be passed with the LOGIN_URL.

from app.settings import settings

LOGIN_URL = "https://github.com/login/oauth/authorize"
TOKEN_URL = "https://github.com/login/oauth/access_token"
USER_URL = "https://api.github.com/user"

REDIRECT_URL = f"{settings.app_url}/auth/github"

Login URL creation

The frontend needs to redirect the user's browser to a URL generated from the LOGIN_URL but also with some information specific to our application:

The client_id, given by Github for our application
The redirect_uri, that we want Github to forward the user to, with the Authorization code
The state, a random generated string that Github will check again when we want to create the access_token, for security reasons

The code is quite straightforward, we created a route called /login so that we can tell the frontend which URL to use. We use an APIRouter here because we want to integrate this piece of code directly into an existing FastAPI application.

from urllib.parse import urlencode

from fastapi import APIRouter

from app.settings import settings
from .schemas import Url
from .helpers import generate_token

LOGIN_URL = "https://github.com/login/oauth/authorize"
REDIRECT_URL = f"{settings.app_url}/auth/github"

router = APIRouter()

@router.get("/login")
def get_login_url() -> Url:
    params = {
        "client_id": settings.github_client_id,
        "redirect_uri": REDIRECT_URL,
        "state": generate_token(),
    }
    return Url(url=f"{LOGIN_URL}?{urlencode(params)}")

The schema Url is defined like this:

from pydantic import BaseModel

class Url(BaseModel):
    url: str

Authorization code verification and token creation

Once the frontend receives the Authorization code, it can forward it to a specific API endpoint, here /authorize for the backend to check it and get all it needs from the User API. The first thing we want to define are the schemas that will be used:

from pydantic import BaseModel

class AuthorizationResponse(BaseModel):
    state: str
    code: str

class GithubUser(BaseModel):
    login: str
    name: str
    company: str
    location: str
    email: str
    avatar_url: str

class User(BaseModel):
    id: int
    login: str
    name: str
    company: str
    location: str
    email: str
    picture: str

    class Config:
        orm_mode = True

class Token(BaseModel):
    access_token: str
    token_type: str
    user: User

The AuthorizationResponse is the body of the request made by the frontend with the state and authorization code, while the GithubUser and User represent users from different sources. The Token schema defines what we will send to the frontend to authenticate our requests between our API and the interface.

On the route side, we use httpx to make requests to Github and check the authorization code, as well as retrieving the user's information. We then create a database entry if the user is not yet present in the database, using the sqlachemy ORM. This is a plain implementation of the Github documentation on their auth API.

from urllib.parse import parse_qsl
from typing import Dict

import httpx
from fastapi import APIRouter, Depends
from sqlalchemy.orm import Session

from app.database import get_db
from .helpers import create_access_token
from .schemas import AuthorizationResponse, GithubUser, User, Token
from .crud import get_user_by_login, create_user

TOKEN_URL = "https://github.com/login/oauth/access_token"
USER_URL = "https://api.github.com/user"

router = APIRouter()

@router.post("/authorize")
async def verify_authorization(
 body: AuthorizationResponse, db: Session = Depends(get_db)
) -> Token:
    params = {
        "client_id": settings.github_client_id,
        "client_secret": settings.github_client_secret,
        "code": body.code,
        "state": body.state,
    }

    async with httpx.AsyncClient() as client:
        token_request = await client.post(TOKEN_URL, params=params)
        response: Dict[bytes, bytes] = dict(parse_qsl(token_request.content))
        github_token = response[b"access_token"].decode("utf-8")
        github_header = {"Authorization": f"token {github_token}"}
        user_request = await client.get(USER_URL, headers=github_header)
        github_user = GithubUser(**user_request.json())

    db_user = get_user_by_login(db, github_user.login)
    if db_user is None:
        db_user = create_user(db, github_user)

    verified_user = User.from_orm(db_user)
    access_token = create_access_token(data=verified_user)

    return Token(access_token=access_token, token_type="bearer", user=db_user)

The function create_access_token creates a simple JWT token, and is therefore in the helpers file:

from datetime import datetime, timedelta

import jwt

from app.settings import settings
from .schemas import User

def create_access_token(*, data: User, exp: int = None) -> bytes:
    to_encode = data.dict()
    if exp is not None:
        to_encode.update({"exp": exp})
    else:
        expire = datetime.utcnow() + timedelta(minutes=60)
        to_encode.update({"exp": expire})
    encoded_jwt = jwt.encode(
        to_encode, settings.jwt_secret_key, algorithm=settings.jwt_algorithm
    )
    return encoded_jwt

Get user from the JWT token

Now that our frontend has a JWT token, we just need to secure our private routes with a FastAPI Dependency that will decode the token and raise an Exception if needed. The dependency is made like this, in its own file:

import jwt
from fastapi import Header, HTTPException, status
from fastapi.security.utils import get_authorization_scheme_param
from pydantic import ValidationError

from app.settings import settings
from .schemas import User

def get_user_from_header(*, authorization: str = Header(None)) -> User:
    credentials_exception = HTTPException(
        status_code=status.HTTP_401_UNAUTHORIZED,
        detail="Could not validate credentials",
        headers={"WWW-Authenticate": "Bearer"},
    )

    scheme, token = get_authorization_scheme_param(authorization)
    if scheme.lower() != "bearer":
        raise credentials_exception

    try:
        payload = jwt.decode(
            token, settings.jwt_secret_key,
            algorithms=[settings.jwt_algorithm]
        )
        try:
            token_data = User(**payload)
            return token_data
        except ValidationError:
            raise credentials_exception
    except jwt.PyJWTError:
        raise credentials_exception

We can use this dependency in some private endpoints, like the common /me that will give information of the user making the request (through its JWT token):

from fastapi import APIRouter, Depends, HTTPException
from sqlalchemy.orm import Session

from app.database import get_db

from .crud import get_user
from .schemas import User
from .models import User as DbUser
from .dependency import get_user_from_header

router = APIRouter()

@router.get("/me", response_model=User)
def read_profile(
    user: User = Depends(get_user_from_header),
    db: Session = Depends(get_db),
) -> DbUser:
    db_user = get_user(db, user.id)
    if db_user is None:
        raise HTTPException(status_code=404, detail="User not found")
    return db_user

Conclusion

Authentification is a very difficult beast to tame, and improvements can be made on our own approach. However, this FastAPI example should give you a good start to implement your own scheme, using whatever external provider you like (Google, Facebook, Twitter, ...).

What is GitOps and why it could be great for your team

Alexis Tacnet — Wed, 15 Apr 2020 18:11:56 +0000

Have you heard of GitOps? This newly coined term describes an innovative pattern in configuration management, used to scale deployment on Kubernetes, especially in a microservices environment.

GitOps refers to something that you might already have implemented in your company. "Everyone who successfully did infrastructure-as-code… is the true creator of the concept of GitOps" says Priyanka Sharma, evangelist at Gitlab. GitOps allows developers to perform more tasks related to infrastructure, by relying on familiar git-based workflows and pull requests.

Currently used by companies such as Weaveworks or the Financial Times, the GitOps pattern can bring some great benefits to your organization. Let's dig into this!

What is GitOps?

GitOps is "pushing code, not containers", says Alexis Richardson, CEO of WeaveWorks. The underlying idea behind GitOps is that changes to a cloud-native system can be implemented through Git. It implies that a version control system - such as Git - hosts all configuration and code to deploy on Kubernetes. It is used for automatically creating, updating and deleting system resources such as containers, infrastructure, proxies, ...

With GitOps, developers can manage their operational workflow and deploy on Kubernetes in the same way they perform merge requests or pull requests. While changes to Kubernetes were done previously through manual actions, the GitOps pattern brings automation to the deployment process.

How GitOps works and what it implies for your organization

GitOps ensures that the state of an application or service is reproducible based on the state of a Git repository. Once a pull request is approved and merged, it modifies the state of the repository that will be the trigger to automatically reconfigure and synchronize the live application and its infrastructure.

Let's imagine you want to create a deployment on Kubernetes. You can for example implement a GitOps pattern with Helm charts: you put your charts in a subfolder of your service Git repository, and every time it changes, you can ask for a deployment of the new Helm chart. Basically, you trigger deployments with pull requests and merges.

In order to listen to the changes and trigger a deployment, you need what it is called a controller. This controller will hook to your Git repository and synchronize the deployments every time it is needed. For instance, a controller such as FluxCD is the operator responsible for listening to changes and deploying them to your Kubernetes clusters.

GitOps is not limited to Helm charts only. It can also be a useful pattern for plain Kubernetes manifests, Ksonnet applications, Jsonnet files or even Terraform files and more globally, Infrastructure-as-Code. Having an agent (your controller) fetching your changes where they are defined allows you to set up Continuous Deployment processes easily and quickly.

Benefits of GitOps

GitOps provides significant benefits to your team.

Improved productivity and speed

GitOps allows developers to work with tools they already use and methods they already rely on. Because developers are already familiar with Git, they are able to participate more efficiently to the DevOps process. This common language increases their productivity and avoids useless back and forth with the Ops team.

Better visibility and auditability

GitOps allows for more visibility, transparency and clarity by relying on a central repository. GitOps provides a single source of truth through the version control system, allowing to track changes and issues easily. Teams can easily review the history of changes and identify where an unexpected behavior occurred, which can be of great help in the case of an audit for instance.

More security and reliability

GitOps brings more security to your infrastructure. Git has strong correctness and security guarantees and it allows an easy identification of authorship and origin. This creates a secure and correct definition of what the desired state of the cluster is.

In case a security breach occurs, GitOps enables the recreation of a new system that is independent from the compromised one.

Reduced time to recovery

Since you have a single source of truth in Git, it is much easier to identify and troubleshoot cluster failures. Furthermore, GitOps allows automatic rollbacks, which means you can simply revert any change that caused a failure.

Cost effective

Less downtime and a much better productivity means that IT spendings are used more wisely and with more efficiency. Your team is able to ship features faster and more safely, while dowtime is significantly reduced.

As you can see, GitOps is a very powerful way of creating and managing your modern cloud infrastructure. GitOps can generate many benefits to your team including improved productivity, visibility and system reliability.

Strio, the current tool I'm working on, is based on this pattern, making it even more efficient. Strio allows you to set up more easily your GitOps process: our tool generates, manages and monitors all your pipeline configuration files in your Git repositories, making it a breeze to manage for SRE teams and easy to configure for everyone in the engineering team.