The 2 a.m. page that started this
Last spring one of our multi-region cron workers — the PHP process that pulls trending titles from an upstream streaming-metadata provider — started returning empty result sets. No errors, no timeouts, just clean 200 OK responses with JSON we did not recognize. The worker dutifully wrote garbage into SQLite, our FTS5 index rebuilt itself around it, and for about forty minutes one of our eight regions served recommendations that pointed at nothing.
The cause was mundane and terrifying at once: a misconfigured transparent proxy at the hosting edge had inserted itself into the TLS path, presented a certificate that chained to a CA our system trusted, and happily re-serialized the upstream response through a caching layer that was hours stale. Standard TLS validation passed. The hostname matched. The chain was valid. And we still got served data from a machine that was not the API we thought we were talking to.
That incident is why every outbound HTTPS call from our backend now pins the certificate or public key of the endpoint it expects. This article is the playbook we wrote afterwards, with the actual code we run across our PHP 8.4 ingestion layer, our Python tooling, and the Go sidecar that fronts a couple of high-volume regions. If you operate something like TrendVidStream — many regions, many cron workers, a thin FTP-based deploy pipeline and no room for a full service mesh — you want pinning that is boring, auditable, and deployable as a flat file.
What pinning actually buys you (and what it does not)
Let me be precise, because pinning is routinely oversold.
Default TLS answers one question: is the peer presenting a certificate that chains to a CA my system trusts, and does the name match? That is necessary but not sufficient. Any of the ~150 root CAs in a typical trust store — plus any intermediate they have signed, plus any corporate MITM root your hosting provider quietly installed — can mint a valid certificate for your upstream's hostname.
Pinning narrows the question to: is the peer presenting the specific key (or cert) I expect? It defends against:
- A rogue or compromised CA issuing a certificate for your API's domain.
- Transparent proxies and TLS-terminating middleboxes at the edge or hosting layer.
- An attacker who has tricked a CA into mis-issuance via a domain-validation bypass.
It does not defend against:
- A compromise of the upstream server itself (you would be pinning the attacker's legitimate key).
- Your own DNS being hijacked to a host that somehow holds the real private key (vanishingly unlikely, but pinning is not the control for it).
- Operator error during key rotation — which is, ironically, the single largest risk pinning introduces.
That last point dominates everything. The famous failure mode of pinning is not getting attacked; it is pinning yourself out of your own dependency when the upstream rotates its certificate and you forgot to ship the new pin. Every design decision below is shaped by that reality.
Pin the public key, not the certificate
There are two things you can pin:
- The full leaf certificate (or its SHA-256 fingerprint).
- The Subject Public Key Info (SPKI) — the public key — extracted from the certificate.
Pin the SPKI. Here is why it matters for an operation that deploys over FTP and cannot afford emergency redeploys: a well-run upstream renews its certificate every 60–90 days but reuses the same key pair across several renewals, or at least lets you fetch the next key ahead of time from their CSR. If you pin the certificate fingerprint, every renewal breaks you. If you pin the SPKI, renewals that reuse the key sail through untouched, and you only act on genuine key rotation.
You compute the SPKI pin like this — this is the canonical incantation, and it produces the same base64 value every major library expects:
# Extract the SPKI SHA-256 pin from a live host
openssl s_client -connect api.example.com:443 -servername api.example.com < /dev/null 2>/dev/null \
| openssl x509 -pubkey -noout \
| openssl pkey -pubin -outform der \
| openssl dgst -sha256 -binary \
| openssl enc -base64
That emits something like K7tZ...=, a 44-character base64 string. Run it against the leaf today, then run it against the upstream's published backup key if they offer one. Always pin at least two keys: the current one and a backup. This is the same discipline HPKP mandated years ago, and it is the only thing standing between a key rotation and a region-wide outage.
Store your pins in a flat, version-controlled config file. In our case it ships alongside the code in the same FTP deploy, which means a pin update is a one-line diff and a deploy-all, not a code change:
; pins.conf — one upstream per line, comma-separated SPKI pins (current,backup)
api.metadata-provider.example = sha256//K7tZqf+0n3xV...=,sha256//9c2Bmtr1kQp...=
images.cdn-provider.example = sha256//Lm4Up0aWqZ...=,sha256//Hh8Yt3sLbN...=
PHP 8.4: pinning with cURL
Our ingestion workers are PHP, and the cleanest path is libcurl's built-in CURLOPT_PINNEDPUBLICKEY. It accepts exactly the sha256//... format above and validates the SPKI of the leaf certificate during the handshake. No userland crypto, no manual chain walking — libcurl rejects the connection before a single byte of response body is read.
Here is the client we actually use, trimmed of logging and retry glue:
<?php
declare(strict_types=1);
final class PinnedHttpClient
{
/** @var array<string,string> host => "sha256//A=,sha256//B=" */
private array $pins;
public function __construct(string $pinFile)
{
$parsed = parse_ini_file($pinFile, false, INI_SCANNER_RAW);
if ($parsed === false) {
throw new RuntimeException("Cannot read pin file: {$pinFile}");
}
$this->pins = $parsed;
}
public function getJson(string $url, int $timeout = 10): array
{
$host = parse_url($url, PHP_URL_HOST);
if ($host === null || !isset($this->pins[$host])) {
throw new RuntimeException("No pin configured for host: {$host}");
}
$ch = curl_init($url);
curl_setopt_array($ch, [
CURLOPT_RETURNTRANSFER => true,
CURLOPT_FOLLOWLOCATION => false, // never follow off-host
CURLOPT_SSL_VERIFYPEER => true, // keep normal chain checks
CURLOPT_SSL_VERIFYHOST => 2, // and hostname checks
CURLOPT_PINNEDPUBLICKEY => $this->pins[$host],
CURLOPT_CONNECTTIMEOUT => 5,
CURLOPT_TIMEOUT => $timeout,
CURLOPT_HTTPHEADER => ['Accept: application/json'],
]);
$body = curl_exec($ch);
$errno = curl_errno($ch);
$error = curl_error($ch);
$code = curl_getinfo($ch, CURLINFO_RESPONSE_CODE);
curl_close($ch);
if ($errno === CURLE_SSL_PINNEDPUBKEYNOTMATCH) {
// Pin mismatch is a security event, not a transient failure.
throw new PinMismatchException("SPKI pin mismatch for {$host}");
}
if ($errno !== 0) {
throw new RuntimeException("Transport error ({$errno}): {$error}");
}
if ($code !== 200) {
throw new RuntimeException("Unexpected status {$code} from {$host}");
}
$data = json_decode((string) $body, true, 512, JSON_THROW_ON_ERROR);
return $data;
}
}
final class PinMismatchException extends RuntimeException {}
Two details earn their keep here. First, CURLOPT_FOLLOWLOCATION => false: a pinned client that follows redirects can be bounced to a host you have no pin for, and depending on your config that either fails confusingly or — worse — silently drops the pin. We resolve redirects explicitly and re-enter getJson() so each hop is checked. Second, the dedicated PinMismatchException. A pin mismatch must never be retried or treated like a 503. In our cron workers it halts that upstream for the run, writes nothing to SQLite, and pages a human. The whole point is to fail closed: a mismatch means something is sitting between us and the API, and serving stale-but-real data beats serving fresh-but-forged data every time.
Python: pinning under requests via a custom adapter
Our operational tooling — the scripts that backfill regions, reconcile the FTS5 index, and audit pin freshness — is Python. requests does not expose pinning directly, but urllib3 lets you assert a fingerprint on the connection pool, and you can bolt that onto a requests session with a small adapter. urllib3's assert_fingerprint pins the certificate (DER) fingerprint, not SPKI, so we keep a separate cert-fingerprint column in our pin audit for the Python side. For pure SPKI parity I prefer to drop to a verify callback, shown second.
The pragmatic adapter:
import hashlib
import ssl
from requests.adapters import HTTPAdapter
from urllib3.poolmanager import PoolManager
class FingerprintAdapter(HTTPAdapter):
"""Pin the leaf certificate's SHA-256 fingerprint (DER)."""
def __init__(self, fingerprint: str, **kwargs):
# fingerprint: lowercase hex, no colons
self._fingerprint = fingerprint.lower().replace(":", "")
super().__init__(**kwargs)
def init_poolmanager(self, connections, maxsize, block=False, **kwargs):
ctx = ssl.create_default_context()
ctx.check_hostname = True
ctx.verify_mode = ssl.CERT_REQUIRED
self.poolmanager = PoolManager(
num_pools=connections,
maxsize=maxsize,
block=block,
ssl_context=ctx,
assert_fingerprint=self._fingerprint,
**kwargs,
)
def spki_pin_from_der(cert_der: bytes) -> str:
"""Compute the SPKI sha256// pin so Python and PHP agree."""
from cryptography import x509
from cryptography.hazmat.primitives import serialization
import base64
cert = x509.load_der_x509_certificate(cert_der)
spki = cert.public_key().public_bytes(
encoding=serialization.Encoding.DER,
format=serialization.PublicFormat.SubjectPublicKeyInfo,
)
digest = hashlib.sha256(spki).digest()
return "sha256//" + base64.b64encode(digest).decode()
Using it is a two-line mount, and the spki_pin_from_der helper means my audit scripts can verify that the Python-observed SPKI matches the sha256// string PHP has in pins.conf — they must agree to the byte, which makes cross-language drift impossible to ignore:
import requests
session = requests.Session()
session.mount(
"https://api.metadata-provider.example",
FingerprintAdapter("3a:7b:...:9f"), # leaf DER sha256
)
resp = session.get(
"https://api.metadata-provider.example/v1/trending?region=DE",
timeout=10,
)
resp.raise_for_status()
If the fingerprint does not match, urllib3 raises SSLError during the handshake and the body is never delivered. In my backfill jobs I catch that specifically and treat it the same way the PHP worker does: abort the upstream, write nothing, alert.
For genuine SPKI pinning across rotation in Python, prefer a context-level verify hook over the DER fingerprint, because it survives certificate renewal that reuses the key:
import ssl, socket, hashlib, base64
from cryptography import x509
from cryptography.hazmat.primitives import serialization
EXPECTED = {"sha256//K7tZqf+0n3xV...=", "sha256//9c2Bmtr1kQp...="}
def verify_spki(host: str, port: int = 443) -> None:
ctx = ssl.create_default_context()
with socket.create_connection((host, port), timeout=5) as raw:
with ctx.wrap_socket(raw, server_hostname=host) as tls:
der = tls.getpeercert(binary_form=True)
cert = x509.load_der_x509_certificate(der)
spki = cert.public_key().public_bytes(
serialization.Encoding.DER,
serialization.PublicFormat.SubjectPublicKeyInfo,
)
pin = "sha256//" + base64.b64encode(hashlib.sha256(spki).digest()).decode()
if pin not in EXPECTED:
raise SystemExit(f"PIN MISMATCH for {host}: got {pin}")
I run exactly this as a standalone preflight in cron, before any region fetch fires, against every upstream in pins.conf. If a pin is about to expire or has silently rotated, I find out from the preflight at a quiet hour rather than from a worker exploding mid-fetch across eight regions.
Go: pinning with VerifyPeerCertificate
The Go sidecar that fronts our two busiest regions does its own pinning, and Go gives you the most surgical hook of the three: tls.Config.VerifyPeerCertificate. It runs after the normal chain build, receives the verified chains, and lets you assert whatever you want. We walk the leaf's SPKI and compare against the allowed set — current plus backup, same as everywhere else.
package pinned
import (
"crypto/sha256"
"crypto/tls"
"crypto/x509"
"encoding/base64"
"errors"
"net/http"
"time"
)
var errPinMismatch = errors.New("spki pin mismatch")
// NewClient returns an http.Client that only completes a handshake when the
// leaf certificate's SPKI matches one of the allowed base64 SHA-256 pins.
func NewClient(allowedPins map[string]bool) *http.Client {
tlsConf := &tls.Config{
MinVersion: tls.VersionTLS12,
VerifyPeerCertificate: func(rawCerts [][]byte, _ [][]*x509.Certificate) error {
if len(rawCerts) == 0 {
return errors.New("no peer certificate")
}
leaf, err := x509.ParseCertificate(rawCerts[0])
if err != nil {
return err
}
spki, err := x509.MarshalPKIXPublicKey(leaf.PublicKey)
if err != nil {
return err
}
sum := sha256.Sum256(spki)
pin := base64.StdEncoding.EncodeToString(sum[:])
if !allowedPins[pin] {
return errPinMismatch
}
return nil
},
}
return &http.Client{
Timeout: 10 * time.Second,
Transport: &http.Transport{
TLSClientConfig: tlsConf,
ForceAttemptHTTP2: true,
MaxIdleConns: 32,
IdleConnTimeout: 90 * time.Second,
TLSHandshakeTimeout: 5 * time.Second,
},
}
}
Note what we did not do: we left InsecureSkipVerify at its default false. A distressing amount of Go pinning code on the internet sets InsecureSkipVerify: true and then "replaces" validation inside VerifyPeerCertificate. Do not do that — it disables hostname verification and SAN checks, so you would be pinning the key but accepting it for any hostname. Keep normal verification on and add the pin check on top. Defense in layers, never instead.
Rotation: the part everyone skips and then regrets
Pinning without a rotation plan is a loaded gun pointed at your own uptime. Here is the discipline we settled on, and it is genuinely the most important section of this article:
- Always carry a backup pin. Two pins minimum in every config, every language. The backup corresponds to a key the upstream has committed to rotating into. When they rotate, you are already trusting the new key; you remove the old one on your own schedule.
-
Preflight before every batch. The Python
verify_spkicheck runs at the top of every multi-region cron run. A pin that no longer matches stops the run before it writes to SQLite, instead of poisoning the FTS5 index region by region. - Alert on mismatch, never auto-update. A pin mismatch is a security signal. Auto-fetching and trusting the new key on mismatch turns pinning into theatre — it is exactly what an active attacker wants you to do. Humans approve pin changes.
-
Make pin updates cheap to ship. Because our pins live in a flat
pins.confdeployed by the same FTP automation as everything else, updating a pin is a one-line diff reviewed in git and pushed withdeploy-all. No build, no container, no restart. The cheaper the legitimate update path, the less tempted anyone is to build a dangerous automatic one. -
Track expiry out of band. Keep a small table — we keep it in the same SQLite the workers use — of each upstream's current cert
notAfterand the date you last verified the backup pin. A weekly cron emails you anything within 21 days of expiry. Pinning failures should be a calendar event, not a surprise.
One more operational note for multi-region setups: pin the same upstream identically across all regions. If your provider serves region-specific certificates from a geo-distributed CDN, each edge may present a different leaf — but they almost always share an SPKI or a small known set. Enumerate them once, from each region's vantage point if you can, and union the pins. The preflight from a single host will miss a region-specific key you never fetched, so make the audit region-aware if your provider's TLS is.
Putting it together
The shape that has held up for us across eight regions and three languages is small and unglamorous:
- One flat
pins.conf, version-controlled, SPKI pins, current plus backup per upstream. - A pinned HTTP client in each language that fails closed on mismatch with a dedicated, non-retryable error type.
- A preflight verifier that runs before any batch and refuses to let workers touch the database when a pin is wrong.
- A rotation process that is human-approved, cheap to deploy, and driven by expiry tracking rather than incident response.
None of this requires a service mesh, a secrets manager, or a sidecar per pod. It requires a fingerprint, a config file, and the discipline to fail closed. After that one bad afternoon where a middlebox fed us forty minutes of forged recommendations, the cost of getting this right has paid for itself many times over — mostly in afternoons that stayed boring.
If you take one thing away: pin the public key, carry a backup, and treat a mismatch as a page, not a retry. The rest is plumbing.
Top comments (0)