DEV Community

Shingo Kawamura
Shingo Kawamura

Posted on

I built a transport-independent foundation for JSON API clients in Perl

#perl #api #opensource #webdev

Writing a simple API client is easy.

my $response = $http->get($url);
Enter fullscreen mode Exit fullscreen mode

Then the client starts being used in production.

Suddenly, it needs to handle:

  • query parameters
  • JSON encoding and decoding
  • structured errors
  • retries and Retry-After
  • rate limits
  • pagination
  • authentication
  • request IDs
  • logging and metrics
  • idempotency

At that point, what started as a small API wrapper often becomes a collection of infrastructure code.

I found myself looking at this problem and asking:

How much of this is actually specific to the API being wrapped?

Not very much.

That led me to build HTTP::API::Core, a small, dependency-light foundation for building JSON HTTP API clients in Perl.

The key idea is simple: keep HTTP transport separate from API-client policy.

Use HTTP::Tiny, LWP, Mojo::UserAgent, Furl, or your preferred transport for HTTP. Let HTTP::API::Core handle the repetitive policy around it.

The project is available on GitHub.

It is deliberately not another HTTP client

One of the main design decisions was not to replace the existing HTTP ecosystem.

Perl already has good HTTP implementations:

  • HTTP::Tiny
  • LWP
  • Mojo::UserAgent
  • Furl

HTTP::API::Core sits above the transport layer.

The HTTP implementation is responsible for making the request.

HTTP::API::Core is responsible for the application-level policies surrounding that request.

Conceptually:

Your API client
      |
HTTP::API::Core
      |
HTTP::Tiny / LWP / Mojo / Furl / custom transport
      |
     HTTP
Enter fullscreen mode Exit fullscreen mode

This separation means an API wrapper does not have to couple its retry, pagination, error handling, or authentication logic to a particular HTTP library.

A small example

A client can start with:

use HTTP::API::Core;

my $api = HTTP::API::Core->new(
    base_url => 'https://api.example.com',
    headers  => {
        Authorization => "Bearer $ENV{API_TOKEN}",
    },
    timeout => 10,
    retry => {
        attempts   => 3,
        base_delay => 0.25,
        max_delay  => 5,
        jitter     => 1,
    },
);

my $response = $api->get('/users');
my $data = $response->json;
Enter fullscreen mode Exit fullscreen mode

A service-specific client can then remain small:

sub get_user {
    my ($self, $id) = @_;

    return $self->{api}
        ->get("/users/$id")
        ->json;
}
Enter fullscreen mode Exit fullscreen mode

The resource method describes the API.

The core handles the repetitive infrastructure around it.

Retries are more complicated than they look

Retries were one of the areas where I wanted conservative defaults.

HTTP::API::Core retries safe/idempotent methods by default:

GET
HEAD
PUT
DELETE
OPTIONS
Enter fullscreen mode Exit fullscreen mode

It does not automatically retry POST or PATCH.

Retryable failures include transport failures and HTTP responses such as:

408
425
429
5xx
Enter fullscreen mode Exit fullscreen mode

It also understands Retry-After and rate-limit reset metadata.

Delays use exponential backoff with optional jitter.

If an API explicitly supports idempotent POST requests, that behavior can be enabled rather than silently assumed.

For example:

$api->post('/jobs',
    json => { task => 'sync' },
    retry => {
        attempts => 2,
        methods  => ['POST'],
    },
);
Enter fullscreen mode Exit fullscreen mode

The goal is to make the safe behavior the boring default.

Rate limits should be data, not special cases

Different APIs expose rate limits differently.

Instead of making application code repeatedly inspect raw headers, responses expose normalized rate-limit information:

my $rate = $response->rate_limit;

say $rate->limit
    if defined $rate->limit;

say $rate->remaining
    if defined $rate->remaining;

say $rate->wait_seconds
    if $rate->exhausted;
Enter fullscreen mode Exit fullscreen mode

HTTP::API::Core understands the commonly used X-RateLimit-* family as well as standard-style rate-limit fields and Retry-After.

The retry system can use the same normalized information.

This is one of the ideas behind the project: parse infrastructure information once and expose it consistently.

One pagination interface, different API styles

Pagination is another thing that looks trivial until you support several APIs.

Some APIs return a next URL.

Others use page numbers.

Others use opaque cursors.

HTTP::API::Core provides one iterator interface for these styles.

For cursor pagination:

my $pager = $api->paginate(
    '/users',
    mode  => 'cursor',
    items => 'data.users',
    next  => 'meta.next_cursor',
    query => { limit => 100 },
);

while (my $user = $pager->next) {
    ...
}
Enter fullscreen mode Exit fullscreen mode

For page-number pagination:

my $pager = $api->paginate(
    '/users',
    mode      => 'page',
    items     => 'users',
    page_size => 100,
);

my @users = $pager->all;
Enter fullscreen mode Exit fullscreen mode

Repeated cursors or next URLs are detected so a broken API response does not accidentally create an infinite pagination loop.

Authentication is implemented through hooks

I did not want authentication to become a growing collection of special cases inside the HTTP client.

Instead, HTTP::API::Core provides lifecycle hooks.

For example:

my $api = HTTP::API::Core->new(
    base_url => 'https://api.example.com',

    hooks => {
        before_request => sub {
            my ($ctx) = @_;

            $ctx->{headers}{Authorization}
                = "Bearer $token";
        },

        after_response => sub {
            my ($response, $ctx) = @_;

            log_status($response->status);
        },

        on_error => sub {
            my ($error, $ctx) = @_;

            record_failure($error->category);
        },
    },
);
Enter fullscreen mode Exit fullscreen mode

The same mechanism can be used for:

  • authentication
  • logging
  • metrics
  • tracing
  • request customization

Small helpers for Bearer, Basic, and API-key authentication are included.

OAuth token acquisition and refresh deliberately remain outside the core.

Observability without choosing an observability framework

The library records useful request metadata without requiring a specific logging or tracing stack.

For example:

say $response->elapsed;

say $response->request_id
    if defined $response->request_id;
Enter fullscreen mode Exit fullscreen mode

Common request ID headers are normalized, including:

X-Request-Id
Request-Id
X-Correlation-Id
Enter fullscreen mode Exit fullscreen mode

Hooks can then send this information wherever an application wants.

The library provides the data without deciding how the application should observe it.

Structured errors

Another design goal was to avoid application code parsing human-readable error strings.

Errors have machine-readable categories such as:

encode
decode
transport
http
hook
Enter fullscreen mode Exit fullscreen mode

Application code can inspect fields including:

$error->category;
$error->status;
$error->retryable;
$error->request_id;
Enter fullscreen mode Exit fullscreen mode

HTTP errors retain their response information, so callers can still inspect headers, text, JSON, or rate-limit state.

Transport independence

The transport boundary is intentionally small.

A transport receives:

($method, $url, \%options)
Enter fullscreen mode Exit fullscreen mode

and returns a response-like hash containing at least:

status => 200
Enter fullscreen mode Exit fullscreen mode

That means the same API client policy can sit on top of different HTTP implementations.

It also makes testing API-specific code possible without requiring real network requests.

Testing against real API conventions

The repository includes example clients demonstrating different API styles:

  • GitHub — page-number pagination and rate-limit metadata
  • Slack — cursor pagination
  • Cloudflare — page-number pagination using response metadata

These are not intended to replace their official SDKs.

They are examples showing that the abstraction can map onto APIs with significantly different conventions.

What I intentionally left out

A reusable core can easily become a framework that tries to solve everything.

I am trying to avoid that.

HTTP::API::Core intentionally does not handle:

  • service-specific SDK behavior
  • complete OAuth flows
  • OpenAPI generation
  • GraphQL-specific behavior
  • WebSockets
  • HTTP server functionality
  • async runtime concerns

The project is intended to remain small, predictable, dependency-light, and transport-independent.

The broader idea

Although this implementation is for Perl, the architectural question is language-independent.

Where should the boundary be between:

  1. the HTTP transport,
  2. reusable API-client policy, and
  3. service-specific domain methods?

Putting everything into the HTTP layer makes the transport too opinionated.

Putting everything into every service SDK means repeatedly implementing retries, pagination, errors, authentication, and rate-limit handling.

HTTP::API::Core is my attempt to put that boundary in the middle.

Try it

HTTP::API::Core is available on GitHub:

👉 github.com/kawamurashingo/HTTP-API-Core

I'd be especially interested in feedback from people who maintain API clients or SDKs:

What functionality do you consider generic API-client infrastructure, and what would you deliberately keep out of a core like this?

Top comments (0)