ahmet gedik

Posted on Aug 13

Certificate Pinning for Video API Clients Without Breaking Production

#security #php #go #python

Eleven months of clean cron runs, then this row showed up in our upstream TLS audit table:

host=youtube.googleapis.com  leaf_cn=*.googleapis.com  issuer_cn=Web Filter CA  verified=1

verified=1. curl was satisfied. CURLOPT_SSL_VERIFYPEER was on, the hostname matched, and the chain built cleanly to a root in the system bundle — a root belonging to a filtering appliance that someone had added to the trust store on one of our shared LiteSpeed hosts. Every video metadata request from that machine, API key sitting in the query string, had been decrypted, inspected, and re-encrypted by hardware we do not own and cannot audit.

Nothing about that is exotic or even a bug. It is exactly what X.509 is designed to permit. The web PKI is a logical OR across roughly 150 root certificates: if any one of them signs a certificate for youtube.googleapis.com, your client accepts it. Certificate pinning is how you turn that OR into an AND for the specific hosts you care about.

This is what we run in production for our outbound video API clients — the PHP fetchers, the Python enrichment workers, and one Go sidecar — and, more importantly, how we rotate pins without waking anybody up.

Why chain validation is not the control you think it is

Standard TLS verification answers one question: did some trusted CA vouch for this name? It does not answer did the CA I expect vouch for this name? The gap has been exploited repeatedly and mundanely:

Locally installed CAs. Corporate proxies, hosting-provider "optimizers", antivirus TLS interception, and developer tooling like mitmproxy all work by adding a root. Nothing in your code can tell the difference.
Misissuance. DigiNotar, the Symantec distrust, and a steady trickle of CT-detected misissuance incidents mean the trusted set is only as strong as its weakest member.
Compelled issuance. A CA in a jurisdiction you have never thought about can sign for your API host, and your client will accept it.

Our threat model is narrow and concrete: outbound requests that carry credentials (YouTube Data API keys, IndexNow keys, webhook secrets) leaving machines on shared hosting where we do not control the OS trust store. Pinning closes that specific hole, and it costs almost nothing at runtime. The entire cost is operational: you now own a new outage class, and the rest of this post is mostly about not triggering it.

Pin the public key, not the certificate

The first mistake people make is hashing the certificate DER. Certificates rotate — Google's leaf certs for googleapis.com have short lifetimes and change constantly. What you want is the SubjectPublicKeyInfo (SPKI): the DER structure containing the algorithm identifier and the public key itself. A renewed certificate that reuses the same key has the same SPKI hash. The convention, borrowed from HPKP and still used by curl, is sha256//<base64 of sha256(SPKI DER)>.

The second decision is which certificate in the chain to pin. This matters more than the hashing details:

Leaf pin. Strongest guarantee, worst operational profile. For a third-party API you do not control, a leaf pin is a scheduled outage. Do not do this to googleapis.com.
Intermediate pin. The sweet spot for third-party APIs. Google's issuing CAs (the GTS/WE-series intermediates) change on a slow, announced cadence, and pinning them still rejects every unrelated root — including the web filter that started this story.
Root pin. Weakest, because the root signs many intermediates, but effectively never rotates. Reasonable as a backup pin.

The rule we enforce in code review: every pinned host has at least two active pins, and they must not both be able to disappear in the same event. In practice that means one intermediate pin plus the root pin above it, or two sibling intermediates from the same CA.

Extracting pins from a live host

This is the script we run before adding any host to the pin store. It prints one pin per certificate in the served chain, labelled with the subject so you can pick the level you want.

#!/usr/bin/env bash
# spki-pins.sh — print sha256 SPKI pins for every cert a host serves
set -euo pipefail

host="${1:-youtube.googleapis.com}"
tmp=$(mktemp -d)
trap 'rm -rf "$tmp"' EXIT

openssl s_client -servername "$host" -connect "$host:443" -showcerts </dev/null 2>/dev/null > "$tmp/raw"
awk '/BEGIN CERT/,/END CERT/' "$tmp/raw" > "$tmp/chain.pem"
csplit -sz -f "$tmp/cert-" -b '%02d.pem' "$tmp/chain.pem" '/BEGIN CERT/' '{*}'

for f in "$tmp"/cert-*.pem; do
  subj=$(openssl x509 -in "$f" -noout -subject | sed 's/^subject=//')
  exp=$(openssl x509 -in "$f" -noout -enddate | sed 's/^notAfter=//')
  pin=$(openssl x509 -in "$f" -pubkey -noout | openssl pkey -pubin -outform der | openssl dgst -sha256 -binary | base64)
  echo "sha256//$pin"
  echo "    subject: $subj"
  echo "    expires: $exp"
done

Two details worth stealing: use openssl pkey -pubin rather than openssl rsa -pubin, because Google and Cloudflare both serve ECDSA leaves and the RSA subcommand will simply fail on them. And note that the served chain is not necessarily the verified chain — the server may omit a cross-signed root your client actually uses. Always confirm your pin against what your client library reports, not just against s_client.

Enforcing pins in PHP

curl has native support via CURLOPT_PINNEDPUBLICKEY, which accepts a semicolon-separated list of pins evaluated as OR. The critical caveat that is not obvious from the docs: curl pins only the server's leaf certificate. If you want to pin an intermediate — which you should, for third-party APIs — you have to verify the chain yourself using CURLINFO_CERTINFO.

That creates an ordering problem. CURLINFO_CERTINFO is only readable after the transfer completes, by which point your API key has already been transmitted. Our solution is a credential-free preflight: probe the host with a HEAD request carrying no secrets, verify the whole chain against the intermediate pins, cache the resulting leaf pin for a few minutes, and then send real requests with CURLOPT_PINNEDPUBLICKEY set to that verified leaf.

<?php
declare(strict_types=1);

final class PinFailure extends RuntimeException {}

final class PinnedHttpClient
{
    private const int PROBE_TTL = 300;
    private const int TIMEOUT   = 15;

    /** @var array<string, array{pin: string, until: int}> */
    private array $verified = [];

    public function __construct(
        private readonly PinStore $store,
        private readonly bool $enforce = true,
    ) {}

    public function get(string $url, array $headers = []): string
    {
        $host = parse_url($url, PHP_URL_HOST) ?: throw new PinFailure('bad url');
        $leafPin = $this->preflight($host);

        $ch = curl_init($url);
        curl_setopt_array($ch, [
            CURLOPT_RETURNTRANSFER => true,
            CURLOPT_SSL_VERIFYPEER => true,
            CURLOPT_SSL_VERIFYHOST => 2,
            CURLOPT_SSLVERSION     => CURL_SSLVERSION_TLSv1_2,
            CURLOPT_HTTPHEADER     => $headers,
            CURLOPT_TIMEOUT        => self::TIMEOUT,
        ]);

        if ($this->enforce && $leafPin !== null) {
            curl_setopt($ch, CURLOPT_PINNEDPUBLICKEY, 'sha256//' . $leafPin);
        }

        $body = curl_exec($ch);
        $errno = curl_errno($ch);
        curl_close($ch);

        if ($errno === CURLE_SSL_PINNEDPUBKEYNOTMATCH) {
            unset($this->verified[$host]);
            throw new PinFailure("leaf pin mismatch for {$host}");
        }
        if ($body === false) {
            throw new PinFailure("transfer failed for {$host}: " . $errno);
        }

        return $body;
    }

    /** Credential-free handshake; validates the chain, returns the leaf pin. */
    private function preflight(string $host): ?string
    {
        $now = time();
        if (isset($this->verified[$host]) && $this->verified[$host]['until'] > $now) {
            return $this->verified[$host]['pin'];
        }

        $ch = curl_init("https://{$host}/");
        curl_setopt_array($ch, [
            CURLOPT_NOBODY         => true,
            CURLOPT_RETURNTRANSFER => true,
            CURLOPT_CERTINFO       => true,
            CURLOPT_SSL_VERIFYPEER => true,
            CURLOPT_SSL_VERIFYHOST => 2,
            CURLOPT_TIMEOUT        => 10,
        ]);
        curl_exec($ch);
        $info = curl_getinfo($ch, CURLINFO_CERTINFO) ?: [];
        curl_close($ch);

        if ($info === []) {
            throw new PinFailure("no chain returned for {$host}");
        }

        $chainPins = [];
        foreach ($info as $cert) {
            if (isset($cert['Cert'])) {
                $chainPins[] = $this->spkiPin($cert['Cert']);
            }
        }

        $this->store->observe($host, $chainPins);
        $accepted = $this->store->activePins($host);

        if (array_intersect($chainPins, $accepted) === []) {
            $this->store->alert($host, $chainPins);
            if ($this->enforce) {
                throw new PinFailure("no pinned key in chain for {$host}");
            }
            return null;
        }

        $leaf = $chainPins[0];
        $this->verified[$host] = ['pin' => $leaf, 'until' => $now + self::PROBE_TTL];

        return $leaf;
    }

    private function spkiPin(string $certPem): string
    {
        $key = openssl_pkey_get_public($certPem);
        if ($key === false) {
            throw new PinFailure('unparseable certificate in chain');
        }
        $pem = openssl_pkey_get_details($key)['key'] ?? '';
        $b64 = str_replace(
            ['-----BEGIN PUBLIC KEY-----', '-----END PUBLIC KEY-----', "\r", "\n", ' '],
            '',
            $pem,
        );
        $der = base64_decode($b64, true);
        if ($der === false) {
            throw new PinFailure('bad SPKI encoding');
        }

        return base64_encode(hash('sha256', $der, true));
    }
}

openssl_pkey_get_details()['key'] hands you a PEM-wrapped SubjectPublicKeyInfo, so base64-decoding it gives you exactly the DER bytes you need to hash — no ASN.1 parsing required. The PROBE_TTL of five minutes bounds how long a stale verdict can survive; with our fetch cadence the preflight cost disappears into the noise.

The pin store lives in SQLite

We already ship a SQLite file per site — it holds the video catalog and the FTS5 search index — so the pin store is three more tables in the same database. No extra service, no config file to keep in sync across hosts, and the admin panel can flip enforcement without a deploy.

<?php
declare(strict_types=1);

final class PinStore
{
    public function __construct(private readonly PDO $db) {}

    public static function migrate(PDO $db): void
    {
        $db->exec(<<<SQL
            CREATE TABLE IF NOT EXISTS tls_pin (
                host       TEXT NOT NULL,
                pin        TEXT NOT NULL,
                level      TEXT NOT NULL CHECK (level IN ('leaf','intermediate','root')),
                status     TEXT NOT NULL CHECK (status IN ('active','candidate','retired')),
                label      TEXT NOT NULL DEFAULT '',
                not_after  INTEGER,
                added_at   INTEGER NOT NULL,
                PRIMARY KEY (host, pin)
            );
            CREATE TABLE IF NOT EXISTS tls_observation (
                host      TEXT NOT NULL,
                pin       TEXT NOT NULL,
                first_at  INTEGER NOT NULL,
                last_at   INTEGER NOT NULL,
                hits      INTEGER NOT NULL DEFAULT 1,
                PRIMARY KEY (host, pin)
            );
        SQL);
    }

    /** @return list<string> */
    public function activePins(string $host): array
    {
        $st = $this->db->prepare(
            "SELECT pin FROM tls_pin WHERE host = ? AND status = 'active'"
        );
        $st->execute([$host]);

        return $st->fetchAll(PDO::FETCH_COLUMN) ?: [];
    }

    /** @param list<string> $pins */
    public function observe(string $host, array $pins): void
    {
        $now = time();
        $st = $this->db->prepare(<<<SQL
            INSERT INTO tls_observation (host, pin, first_at, last_at)
            VALUES (:h, :p, :t, :t)
            ON CONFLICT(host, pin) DO UPDATE SET last_at = :t, hits = hits + 1
        SQL);
        foreach ($pins as $pin) {
            $st->execute([':h' => $host, ':p' => $pin, ':t' => $now]);
        }
    }

    /** Record an unknown chain as a candidate so a human can promote it. */
    public function alert(string $host, array $pins): void
    {
        $st = $this->db->prepare(<<<SQL
            INSERT OR IGNORE INTO tls_pin (host, pin, level, status, label, added_at)
            VALUES (?, ?, 'intermediate', 'candidate', 'auto-discovered', ?)
        SQL);
        foreach ($pins as $pin) {
            $st->execute([$host, $pin, time()]);
        }
        error_log("TLS pin alert host={$host} pins=" . implode(',', $pins));
    }
}

The tls_observation table is the part that actually earns its keep. It is a running census of every public key we have ever seen from every upstream host. That table is what would have flagged the web filter in week one instead of month eleven — a new pin appearing on exactly one of six hosts is not subtle once you are looking at the data.

Rotating pins without a 3 a.m. incident

Pinning fails closed. Everything below exists so that failing closed never happens for a boring reason.

Report mode first. TLS_PIN_MODE=report runs the whole preflight, writes observations, logs mismatches, and then proceeds. We run every new host in report mode for at least a full rotation window before flipping to enforce. The constructor flag $enforce in the client above is wired to that env var.
A daily canary. One cron job per host connects, computes chain pins, and inserts anything unrecognised as status='candidate'. Candidates page nobody; they show up in the admin panel. When a CA announces a new issuing intermediate, the candidate row usually appears days before that intermediate starts serving real traffic — you promote it to active and the rotation is a non-event.
Two active pins, always. A constraint check in the canary refuses to let any host drop to one active pin. A single pin plus a surprise rotation equals a hard outage on every host at once.
Expiry monitoring. not_after is populated from the certificate; anything inside 45 days gets flagged next to the candidate list.
A kill switch that is not a deploy. Ours is a row in the settings table the admin panel writes to, checked on every preflight. FTP-deploying a config change to several hosts during an incident is not a recovery plan.

One more rule specific to our stack: never pin your own origin's edge certificate when it sits behind Cloudflare. The edge cert rotates on Cloudflare's schedule with no notice to you. If you want pinning on the Cloudflare-to-origin hop instead, pin the Cloudflare Origin CA certificate — it is valid for fifteen years and you control the rotation. And pinning in browsers is not an option at all any more: HPKP was removed from Chrome and Firefox precisely because sites bricked themselves with it. Certificate Transparency monitoring plus HSTS preload is the browser-side answer.

The Python worker

Our enrichment workers use requests. The naive approach — check the peer certificate in Session.send() — is detection after the fact, because the request body has already gone out. The pin check has to run at connect time, before any application bytes are written, which means hooking urllib3's connection class.

"""pinned.py — SPKI-pinned requests session (urllib3 2.x, Python 3.11+)."""
import base64
import hashlib
import ssl

import requests
from cryptography import x509
from cryptography.hazmat.primitives.serialization import Encoding, PublicFormat
from requests.adapters import HTTPAdapter
from urllib3.connection import HTTPSConnection
from urllib3.connectionpool import HTTPSConnectionPool


def spki_pin(der_cert: bytes) -> str:
    cert = x509.load_der_x509_certificate(der_cert)
    spki = cert.public_key().public_bytes(Encoding.DER, PublicFormat.SubjectPublicKeyInfo)
    return base64.b64encode(hashlib.sha256(spki).digest()).decode("ascii")


class PinMismatch(ssl.SSLCertVerificationError):
    pass


class PinnedHTTPSConnection(HTTPSConnection):
    pins: frozenset[str] = frozenset()

    def connect(self) -> None:
        super().connect()
        chain = []
        # Python 3.13+ exposes the full verified chain; older versions see the leaf only.
        getter = getattr(self.sock, "get_verified_chain", None)
        if getter is not None:
            chain = [c.public_bytes(_format=1) if hasattr(c, "public_bytes") else c for c in getter()]
        if not chain:
            chain = [self.sock.getpeercert(binary_form=True)]

        seen = {spki_pin(der) for der in chain if der}
        if not (seen & self.pins):
            self.close()
            raise PinMismatch(f"pin mismatch for {self.host}: saw {sorted(seen)}")


class PinnedAdapter(HTTPAdapter):
    def __init__(self, pins, **kwargs):
        self._pins = frozenset(pins)
        super().__init__(**kwargs)

    def init_poolmanager(self, connections, maxsize, block=False, **kwargs):
        super().init_poolmanager(connections, maxsize, block=block, **kwargs)
        conn_cls = type("PinnedConn", (PinnedHTTPSConnection,), {"pins": self._pins})
        pool_cls = type("PinnedPool", (HTTPSConnectionPool,), {"ConnectionCls": conn_cls})
        self.poolmanager.pool_classes_by_scheme = {
            **self.poolmanager.pool_classes_by_scheme,
            "https": pool_cls,
        }


def pinned_session(host: str, pins) -> requests.Session:
    s = requests.Session()
    s.mount(f"https://{host}", PinnedAdapter(pins))
    return s


if __name__ == "__main__":
    sess = pinned_session("youtube.googleapis.com", {"REPLACE_WITH_YOUR_INTERMEDIATE_PIN"})
    print(sess.get("https://youtube.googleapis.com/generate_204", timeout=10).status_code)

Mounting the adapter on a host-specific prefix matters: pinning applies only to the hosts you have pins for, and everything else keeps using the default adapter with normal verification. A global mount is how you accidentally break your metrics endpoint.

The Go sidecar

Go makes this the cleanest of the three, because VerifyPeerCertificate runs during the handshake — before a single application byte moves — and with InsecureSkipVerify left false you get the verified chains for free, so you are pinning in addition to normal validation rather than instead of it.

package pinned

import (
    "crypto/sha256"
    "crypto/tls"
    "crypto/x509"
    "encoding/base64"
    "errors"
    "net"
    "net/http"
    "time"
)

var ErrNoPinnedKey = errors.New("tls: no pinned key present in verified chain")

// Client returns an http.Client that requires at least one certificate in the
// verified chain to match a pin. Pins are base64(sha256(SPKI DER)).
func Client(pins map[string]struct{}) *http.Client {
    verify := func(_ [][]byte, chains [][]*x509.Certificate) error {
        for _, chain := range chains {
            for _, cert := range chain {
                sum := sha256.Sum256(cert.RawSubjectPublicKeyInfo)
                if _, ok := pins[base64.StdEncoding.EncodeToString(sum[:])]; ok {
                    return nil
                }
            }
        }
        return ErrNoPinnedKey
    }

    tr := &http.Transport{
        ForceAttemptHTTP2:   true,
        MaxIdleConnsPerHost: 4,
        IdleConnTimeout:     60 * time.Second,
        TLSHandshakeTimeout: 5 * time.Second,
        DialContext:         (&net.Dialer{Timeout: 5 * time.Second}).DialContext,
        TLSClientConfig: &tls.Config{
            MinVersion:            tls.VersionTLS12,
            VerifyPeerCertificate: verify,
        },
    }

    return &http.Client{Transport: tr, Timeout: 15 * time.Second}
}

Because RawSubjectPublicKeyInfo is already the exact DER bytes we hash everywhere else, the same pin string works unchanged across all three implementations — which is the whole point of keeping the pin store in one SQLite table rather than three config formats.

What pinning does not buy you

Be honest about the boundary, or you will over-trust it:

It does not protect a key that leaks through logs, a repo, or an error page. Pinning secures the channel, not the secret.
It does not help browser traffic. HPKP is gone; use CT monitoring for your own domains.
It does not authenticate behavior. A correctly pinned endpoint can still return garbage, and your parser still needs to treat upstream data as hostile.
It converts a silent confidentiality failure into a loud availability failure. That is the trade you are making deliberately.

For us, the whole thing came to roughly 200 lines of PHP, one canary cron, and two pins per host. The first real test came when our upstream CA introduced a new issuing intermediate: the canary filed it as a candidate, someone promoted it during business hours, and the actual rotation produced no alert at all.

Conclusion

Start with the audit table, not the enforcement. Log the issuer and SPKI pin of every outbound TLS connection for a couple of weeks — that alone will tell you whether anything is sitting between your servers and your APIs, and it costs you nothing but a few rows of SQLite. Then pin the intermediate, keep two active pins, ship in report mode, and give yourself a kill switch that is not a deploy. Everything I have described here runs today behind DailyWatch on PHP 8.4 and shared LiteSpeed hosting, which is to say: you do not need dedicated infrastructure or a service mesh to close this hole. You need a hash, a table, and the discipline to rotate before you are forced to.

DEV Community