Certificate-Pinned HTTPS for Video API Clients Without Downtime

The morning a transparent proxy almost ate our API keys

A fetch worker on one of our Asia-Pacific edge boxes started returning HTTP 200 responses with subtly wrong JSON: video IDs that did not exist, thumbnail URLs pointing at a domain we had never seen. The TLS handshake succeeded. curl was happy. PHP's stream wrapper was happy. Every certificate in the chain validated against the system trust store.

The problem was that the datacenter's upstream transit provider ran a transparent TLS-terminating proxy, and that proxy shipped with a root CA that was — of course — already in the OS trust bundle. From the worker's point of view, nothing was wrong. From our point of view, a third party was decrypting every outbound request, including the Authorization: Bearer headers carrying our upstream API keys.

Standard TLS answers the question is this certificate signed by a CA I trust? For a server pulling data from a fixed set of known upstreams, that is the wrong question. The right question is is this the specific endpoint I expect? Certificate pinning answers that one. This post walks through how we implemented public-key pinning for the outbound API clients that feed TopVideoHub, in PHP 8.4, Python, and Go, and — more importantly — how we did it without paging ourselves at 3am every time an upstream rotated a leaf certificate.

Why the default trust model fails server-to-server

The web PKI is built for browsers talking to arbitrary strangers. A browser cannot know in advance which of a million sites it will visit, so it delegates trust to ~150 root CAs and their intermediates. Any one of those CAs can mint a valid certificate for any domain. That is a feature for the open web and a liability for a backend that only ever talks to four or five known hosts.

Our outbound surface is tiny and static:

• A handful of trending-video metadata APIs
• Two translation endpoints we use for CJK title normalization
• One image CDN we prefetch thumbnails from

Every one of those hosts is known at deploy time. There is no reason to accept a certificate signed by a random Turkish, Israeli, or US government CA when we know exactly whose key should be on the other end. Pinning collapses the trust anchor set from all CAs down to this specific key.

Three things you can pin

1. The leaf certificate. Simple, but it breaks every time the cert is renewed — often every 90 days or less for cloud-fronted APIs. Do not do this in production.
2. The Subject Public Key Info (SPKI). You pin the SHA-256 of the DER-encoded public key. As long as the operator reuses the same key across renewals — or you pin the intermediate — the pin survives rotation. This is the sweet spot.
3. The intermediate or root CA's SPKI. The most stable choice for APIs behind Google Trust Services or Cloudflare, where leaf keys rotate aggressively but the issuing intermediate is long-lived.

We pin SPKI, and for the two upstreams fronted by Cloudflare we pin the intermediate's SPKI plus a backup. The rule of thumb: pin the highest, most stable point in the chain that still uniquely identifies the operator.

Computing an SPKI pin

The pin is base64(sha256(DER-encoded SubjectPublicKeyInfo)). You can compute it from a live host in one pipeline with OpenSSL. This is the exact command we bake into our provisioning scripts:

# Compute the SPKI pin for the leaf certificate of a host
openssl s_client -connect api.example.com:443 -servername api.example.com </dev/null 2>/dev/null \
  | openssl x509 -pubkey -noout \
  | openssl pkey -pubin -outform der \
  | openssl dgst -sha256 -binary \
  | openssl base64

To pin an intermediate instead, dump the full chain with openssl s_client -showcerts, save the second certificate in the chain to a file, and run the last three stages of the pipeline against it. The output looks like k3XnEYQCK79AtL9GYnT/nyhsabas03V+bhRQYHQdS6g=. That base64 string, prefixed with sha256//, is what libcurl and most HTTP clients expect.

Store the pins somewhere your workers can read them without a redeploy. We keep them in the same SQLite database that backs the rest of the site, in a tiny table:

CREATE TABLE tls_pins (
  host        TEXT NOT NULL,
  spki_sha256 TEXT NOT NULL,
  role        TEXT NOT NULL DEFAULT 'primary',  -- primary | backup
  added_at    INTEGER NOT NULL,
  PRIMARY KEY (host, spki_sha256)
);

Keeping pins in a table rather than a constant means a rotation is a one-row insert plus a cache flush, not a code deploy across every edge box.

PHP 8.4: pinning with libcurl

libcurl has native SPKI pinning through CURLOPT_PINNEDPUBLICKEY. It accepts one or more sha256// pins joined with semicolons, and — critically — it validates in addition to normal chain verification, not instead of it. Keep CURLOPT_SSL_VERIFYPEER on. Pinning is defense in depth, not a replacement for verification.

Here is the client we actually use in the fetch worker. It pulls the allowed pins for a host out of SQLite and refuses to make the request if none are configured — fail closed, never fail open:

<?php
declare(strict_types=1);

final class PinnedHttpClient
{
    public function __construct(private readonly PDO $db) {}

    /** @return list<string> sha256// pin strings for the host */
    private function pinsFor(string $host): array
    {
        $stmt = $this->db->prepare(
            'SELECT spki_sha256 FROM tls_pins WHERE host = :host'
        );
        $stmt->execute(['host' => $host]);
        $rows = $stmt->fetchAll(PDO::FETCH_COLUMN);

        return array_map(static fn(string $p): string => 'sha256//' . $p, $rows);
    }

    public function get(string $url, array $headers = []): string
    {
        $host = parse_url($url, PHP_URL_HOST);
        if (!is_string($host)) {
            throw new InvalidArgumentException('Unparseable URL: ' . $url);
        }

        $pins = $this->pinsFor($host);
        if ($pins === []) {
            // Fail closed: an unpinned host is a configuration error, not a fallback.
            throw new RuntimeException('No TLS pin configured for host ' . $host);
        }

        $ch = curl_init($url);
        curl_setopt_array($ch, [
            CURLOPT_RETURNTRANSFER  => true,
            CURLOPT_FOLLOWLOCATION  => false,          // redirects can dodge the pin
            CURLOPT_SSL_VERIFYPEER  => true,
            CURLOPT_SSL_VERIFYHOST  => 2,
            CURLOPT_PINNEDPUBLICKEY => implode(';', $pins),
            CURLOPT_HTTPHEADER      => $headers,
            CURLOPT_TIMEOUT         => 15,
            CURLOPT_CONNECTTIMEOUT  => 5,
        ]);

        $body = curl_exec($ch);
        $errno = curl_errno($ch);
        $error = curl_error($ch);
        curl_close($ch);

        if ($errno === CURLE_SSL_PINNEDPUBKEYNOTMATCH) {
            // Distinct from a generic TLS error so alerting can treat it as SECURITY, not NETWORK.
            throw new RuntimeException('PIN MISMATCH for ' . $host . ': possible MITM');
        }
        if ($body === false) {
            throw new RuntimeException('cURL error for ' . $host . ': ' . $error);
        }

        return (string) $body;
    }
}

The important design decisions here are not the curl options — they are the two throws. An unpinned host throws before any bytes leave the box. A pin mismatch throws a distinct exception type so our log pipeline can route CURLE_SSL_PINNEDPUBKEYNOTMATCH straight to a security channel instead of burying it in the noise of ordinary timeouts. On a video aggregator that makes tens of thousands of outbound calls per cron cycle, a pin failure buried among network flakes is a pin failure you will never see.

Disabling CURLOPT_FOLLOWLOCATION matters more than it looks. If an upstream 302-redirects you to a different host, libcurl applies the same pin to the new host and fails — which sounds fine until you realize the failure looks identical to an attack. We handle redirects explicitly in application code so a legitimate CDN hop resolves its own pins.

Python: pinning at the socket layer

Python's ssl module does not expose SPKI pinning directly, and urllib3's assert_fingerprint pins the whole certificate, which rots on every renewal. To pin the public key we grab the peer certificate in DER form after the handshake, extract its SPKI, and compare. The cryptography library does the DER surgery cleanly:

import socket
import ssl
import hashlib
import base64

from cryptography import x509
from cryptography.hazmat.primitives.serialization import Encoding, PublicFormat


def spki_pin_from_der(cert_der: bytes) -> str:
    cert = x509.load_der_x509_certificate(cert_der)
    spki = cert.public_key().public_bytes(
        Encoding.DER, PublicFormat.SubjectPublicKeyInfo
    )
    return base64.b64encode(hashlib.sha256(spki).digest()).decode()


def assert_pinned(host: str, port: int, allowed_pins: set[str]) -> None:
    """Handshake, verify the chain normally, then enforce the SPKI pin."""
    ctx = ssl.create_default_context()          # keeps hostname + chain checks ON
    with socket.create_connection((host, port), timeout=5) as raw:
        with ctx.wrap_socket(raw, server_hostname=host) as tls:
            der = tls.getpeercert(binary_form=True)
            if der is None:
                raise RuntimeError(f"No peer certificate from {host}")
            pin = spki_pin_from_der(der)
            if pin not in allowed_pins:
                raise RuntimeError(
                    f"PIN MISMATCH for {host}: got {pin}, "
                    f"expected one of {sorted(allowed_pins)}"
                )


if __name__ == "__main__":
    pins = {"k3XnEYQCK79AtL9GYnT/nyhsabas03V+bhRQYHQdS6g="}
    assert_pinned("api.example.com", 443, pins)
    print("pin OK")

Note getpeercert(binary_form=True) returns only the leaf, so this snippet pins the leaf. To pin an intermediate in Python you need the full chain, which means either the newer SSLSocket.get_verified_chain() (available on recent CPython builds linked against OpenSSL 1.1.1+) or dropping down to a library like pyOpenSSL that exposes get_peer_cert_chain(). For a batch ingester we run this check once against each upstream at worker startup, cache the result, and only re-verify on a fresh connection — pinning every request in a hot loop is wasted CPU when the TLS session is being reused anyway.

Go: pinning through VerifyConnection

Go gives you the cleanest hook of the three. tls.Config.VerifyConnection runs after the normal chain and hostname verification (as long as InsecureSkipVerify stays false), and it hands you every peer certificate. That is exactly where SPKI pinning belongs — you get MITM protection layered on top of, not instead of, the standard checks.

package main

import (
    "crypto/sha256"
    "crypto/tls"
    "crypto/x509"
    "encoding/base64"
    "errors"
    "fmt"
    "net/http"
    "time"
)

func spkiPin(cert *x509.Certificate) string {
    sum := sha256.Sum256(cert.RawSubjectPublicKeyInfo)
    return base64.StdEncoding.EncodeToString(sum[:])
}

func pinnedClient(allowed map[string]bool) *http.Client {
    tr := &http.Transport{
        TLSClientConfig: &tls.Config{
            MinVersion: tls.VersionTLS12,
            VerifyConnection: func(cs tls.ConnectionState) error {
                // Chain + hostname already verified here; we only add pinning.
                for _, cert := range cs.PeerCertificates {
                    if allowed[spkiPin(cert)] {
                        return nil
                    }
                }
                return errors.New("tls: no pinned SPKI found in presented chain")
            },
        },
    }
    return &http.Client{Transport: tr, Timeout: 15 * time.Second}
}

func main() {
    allowed := map[string]bool{
        "k3XnEYQCK79AtL9GYnT/nyhsabas03V+bhRQYHQdS6g=": true, // primary
        "backupPinGoesHereBase64Sha256OfSpkiXXXXXXXXX=": true, // backup
    }
    client := pinnedClient(allowed)

    resp, err := client.Get("https://api.example.com/v3/trending")
    if err != nil {
        fmt.Println("request failed:", err)
        return
    }
    defer resp.Body.Close()
    fmt.Println("status:", resp.Status)
}

Because VerifyConnection iterates PeerCertificates — the whole presented chain — the same code pins a leaf, an intermediate, or a root depending purely on which pins you load into the allowed map. That is the flexibility you want when different upstreams sit behind different fronting infrastructure.

The part everyone gets wrong: rotation

The reason most teams try pinning once, get burned by an unannounced certificate rotation, and rip it out, is that they pin exactly one key with no backup. When the upstream rotates, every request fails simultaneously and there is no way to recover without a deploy. Mobile apps famously bricked themselves this way.

The fix is boring and non-negotiable: always pin a set, and always keep at least one backup pin for a key that is not yet in production. Our runbook:

• Pin two keys per host at all times — the current key and a pre-generated next key the upstream has committed to rotating into. For third-party APIs where you cannot get the next key, pin the current leaf and its issuing intermediate. When the leaf rotates under the same intermediate, the intermediate pin carries you.
• Rotate as insert-then-remove, never remove-then-insert. Add the new pin, deploy/flush, confirm traffic validates against it, and only then retire the old pin. Because pins live in SQLite, both steps are one-row writes and a cache clear — no code change.
• Alert on approaching expiry. A nightly job connects to each pinned host, reads the leaf's notAfter, and warns if it is under 21 days out. A pin failure should never be your first warning that a certificate rotated.
• Monitor mismatch rate as a security signal. A single mismatch from one worker is probably a stale pin. A coordinated spike of mismatches from workers in one region is what an actual interception attack looks like — exactly the transparent-proxy scenario that started this whole exercise.

Pinning through Cloudflare and other fronts

Most of our upstreams — and TopVideoHub itself — sit behind Cloudflare. Two things follow. First, when you front your API with Cloudflare, your clients cannot pin your origin key; they can only pin the edge, and Cloudflare rotates edge keys freely. For those hosts, pin the Cloudflare or Google Trust Services intermediate, which is stable across leaf rotations, and accept that you are trusting the front's operator. Second, this pinning applies strictly to outbound calls your backend makes. It has nothing to do with LiteSpeed serving your own pages or with Cloudflare terminating inbound TLS in Flexible mode — those are a separate trust boundary. Do not confuse the two directions; pinning your inbound TLS from a PHP worker is meaningless.

What pinning bought us

After rolling this out across every outbound client, the transparent-proxy datacenter started throwing CURLE_SSL_PINNEDPUBKEYNOTMATCH on exactly the requests being intercepted — loudly, immediately, and routed to the right alert channel instead of validating silently. We moved that worker to a clean network within the hour. Before pinning, the interception was invisible; after, it was a hard failure with a security label.

The cost is real but bounded: two extra pins per host in a SQLite table, a nightly expiry check, and the discipline to rotate insert-first. In exchange, a compromised CA, a rogue corporate proxy, or a nosy transit provider can no longer silently read your API keys or feed your ingester poisoned data. For a backend whose entire outbound surface is a known, static handful of hosts, that trade is lopsided in your favor. Pin the public key, keep a backup, fail closed, and treat every mismatch as the security event it is.