Open sourcing apiron: A Python package for declarative RESTful API interaction

This post originally appeared on Build Smarter.

---

At ITHAKA our web teams write applications that each interact with a large handful of services—sometimes as many as ten. Each of those services provide multiple endpoints, each with their own set of path variables and query parameters.

Gathering data from multiple services has become a ubiquitous task for web application developers. The complexity can grow quickly: calling an API endpoint with multiple parameter sets, calling multiple API endpoints, calling multiple endpoints in multiple APIs. While the business logic can get hairy, the code to interact with those APIs doesn’t have to.

We created a module some time ago for low-level HTTP interactions, and use it throughout our code base. For a good while, though, the actual details of each service call—the service name, endpoint path, query parameters—were scattered throughout the code. This inevitably led to duplication as well as a bug or two when we made an update in one place and forgot about the other.

To reduce the pains from this, we eventually took stock of these scattered configurations and centralized them in one registry module. This module essentially contains a giant dictionary of all the services we interact with:

service_endpoints = {
    'CONTENT_SERVICE': {
        'SERVICE': 'content-service',
        'METADATA_ENDPOINT': '/content/{id}',
        'CITATION_ENDPOINT': '/citation/{citation_type}/{id}',
    },
    'SEARCH_SERVICE': {
        'SERVICE': 'search',
        'SEARCH_ENDPOINT': '/search',
        'EXCERPTS_ENDPOINT': '/excerpt?contentId={content_id}',
    },
    ...
}

Each service has a 'SERVICE' key containing the name of the service used to discover hosts, and some number of '*_ENDPOINT' keys that describe an endpoint and its parameters. Calling these services looks like this:

from http import make_get_request_with_timeout
from services.registry import service_endpoints

CONTENT_SERVICE = service_endpoints.get('CONTENT_SERVICE', {})
METADATA_ENDPOINT = CONTENT_SERVICE.get('METADATA_ENDPOINT', '')

# determine content_id...

metadata = make_get_request_with_timeout(
    service_name=CONTENT_SERVICE.get('SERVICE'),
    endpoint=METADATA_ENDPOINT.format(id=content_id),
    headers={'Accept': 'application/json'},
    request_timeout=5,
)

As you can see, there are a variety of shapes to these endpoints. This solved the issue of duplication across the codebase, but we still faced a couple of problems with this approach:

1. Strings as endpoint descriptors don’t result in structured data. This is pretty difficult to introspect or validate.
2. Even with fully-formattable strings, sometimes a call needed to exclude a parameter all together, or add a new one. This had to be done ad-hoc after the fact.
3. Our HTTP module still had a laundry list of methods, each with slightly different behavior and unclear names like make_get_request_fast (how fast?). Many of these methods called the same underlying methods with different default parameters, and the stack got pretty deep sometimes. Choosing the right method for a call was hard.

In order to address the high variability of behaviors and lack of structured data of this problem, we built a new paradigm for HTTP interactions that provided a declarative interface for configuring services. We wanted a few things out of it:

1. Code describes how a service interaction looks, not the details of how to make the underlying HTTP call happen.
2. The endpoint descriptors are structured and support introspection.
3. Default behaviors can be declared in the service configuration, but can also be easily overridden dynamically at call time.

With these desires in mind, we came up with apiron. With apiron the same definition from above looks more like this:

from services import IthakaDiscoverableService
from apiron.endpoint import Endpoint

class ContentService(IthakaDiscoverableService):
    service_name = 'content-service'

    metadata = Endpoint(path='/content/{id}')
    citation = Endpoint(path='/citation/{citation_type}/{id}')

And the code to call the service looks more like this:

from apiron.client import ServiceCaller, Timeout
from services import ContentService

CONTENT_SERVICE = ContentService()

# determine content_id...

metadata = ServiceCaller.call(
    service=CONTENT_SERVICE,
    endpoint=CONTENT_SERVICE.metadata,
    path_kwargs={'content_id': content_id},
    headers={'Accept': 'application/json'},
    timeout_spec=Timeout(read_timeout=5),
)

We can now define what ContentService looks like and easily refer back to that class whenever we need to understand its shape. Service discovery is now a plugin system. Endpoints can be introspected and have their parameters validated and enforced.

With apiron we’ve been able to replace many of our existing service calls quickly and with little pain. The code has become clearer and with the cognitive load out of the way we can begin focusing on other gains like streaming responses and data compression. It’s been nice for us, and we’d like to make it nice for you too.

You can install apiron from PyPI with pip (or your favorite package manager):

$ pip install apiron

There are a few other helpful tools in the package, so give it a try today!

ithaka / apiron

🍳 apiron is a Python package that helps you cook a tasty client for RESTful APIs. Just don't wash it with SOAP.

apiron

apiron helps you cook a tasty client for RESTful APIs. Just don't wash it with SOAP.

Gathering data from multiple services has become a ubiquitous task for web application developers The complexity can grow quickly calling an API endpoint with multiple parameter sets calling multiple API endpoints, calling multiple endpoints in multiple APIs. While the business logic can get hairy, the code to interact with those APIs doesn't have to.

apiron provides declarative, structured configuration of services and endpoints with a unified interface for interacting with them.

Defining a service

A service definition requires a domain and one or more endpoints with which to interact:

from apiron import JsonEndpoint, Service

class GitHub(Service):
    domain = 'https://api.github.com'
    user = JsonEndpoint(path='/users/{username}')
    repo = JsonEndpoint(path='/repos/{org}/{repo}')

Interacting with a service

Once your service…

View on GitHub