crt.sh has no pagination, and its wildcard is a percent sign you must spell %25

Quick answer

crt.sh is the best free subdomain-enumeration source on the internet, and it has no pagination. None. You ask for a domain, you get every certificate ever issued for it as one JSON array in one response body.

For example.com that is fine. For a domain behind a CDN that rotates certs weekly, that is tens of thousands of records and hundreds of megabytes, arriving whether you wanted it or not.

Three things that cost us time:

1. The wildcard is a literal %, which you must send URL-encoded as %25. Send % raw and you get a different query than you think.
2. name_value is newline-delimited, holding every SAN on the certificate — so one record is N subdomains, not one.
3. There is nowhere to put a limit. The cap has to live in your client, and where you apply it decides whether it saves you anything.

The URL is the first trap 🔎

Here is the entire API:

https://crt.sh/?q=%25.example.com&output=json

That %25 is not a typo and not an encoding artifact of this blog post. crt.sh is a Postgres front-end, the query goes into a SQL LIKE, and the SQL wildcard is %. So the query you want is %.example.com — "anything, then a dot, then example.com" — and because % is also the URL percent-escape character, it has to be written %25 in the URL.

This is the kind of detail that produces a bug report rather than an error. Most HTTP clients will happily send a bare % followed by .e and let the server decide what that means. You do not get a 400. You get a 200, with results, that are not the results you asked for.

CRTSH_URL_TEMPLATE = "https://crt.sh/?q=%25.{domain}&output=json"

Written as a module constant precisely so nobody "tidies" the %25 into a % while reading it as a typo.

One record is not one subdomain

The obvious mental model is one row per certificate. crt.sh returns something more interesting:

{
  "id": 12345678901,
  "issuer_name": "C=US, O=Let's Encrypt, CN=R11",
  "common_name": "example.com",
  "name_value": "example.com\nwww.example.com\napi.example.com\n*.staging.example.com",
  "not_before": "2026-06-01T00:00:00",
  "not_after":  "2026-08-30T23:59:59",
  "serial_number": "03a1..."
}

name_value is the Subject Alternative Name list, newline-separated inside a single JSON string. A modern certificate routinely covers a dozen hostnames. If you treat each record as one subdomain you throw away most of the attack surface you came for — and if you split it without deduplicating you count example.com once per certificate that mentions it, which for a Let's Encrypt domain is once every 60 days forever.

def _san_entries(record, common_name):
    """Newline-separated SAN entries from name_value, deduped, falling back to common_name."""

The fallback matters too: some older records carry an empty name_value and only a common_name. Drop those and you silently lose the oldest history for the domain — which is exactly the part an attack-surface audit cares about, because that is where the forgotten hosts are.

Where you put the cap is the whole design 🚧

Since crt.sh will not limit the response, the limit is yours. The naive place to put it is at the end:

rows = [build_row(c) for c in candidates]   # build everything
return rows[:max_certs]                     # then throw most of it away

That caps your dataset and saves you nothing. You already paid for the transfer, you already paid the CPU to construct every row, and on a big domain you already risked the memory.

The order that actually helps is filter → dedupe → cap → compute → construct:

"""Filter -> dedupe/split -> CAP -> compute -> construct, in that order (REQ-3..REQ-6)."""

Cap before the expensive per-row work, and after dedupe so the cap counts distinct things rather than counting the same hostname forty times and calling it a day. maxCertsPerDomain then means what a user expects it to mean: "give me up to N real results", not "give me whatever survives the first N raw records".

There is one thing this cannot fix. The HTTP response itself is un-capped, because the server offers no way to ask for less. That transfer happens no matter how disciplined your parser is — worth knowing before you point a wide sweep at a CDN domain.

The timestamps have drifted

crt.sh datetimes arrive naive — no offset, no Z:

"not_before": "2026-06-01T00:00:00"

The documented convention is UTC, and the format has changed more than once over the years. So parsing is deliberately tolerant, and naive values are explicitly stamped UTC rather than being handed to the host's local timezone:

def _parse_crtsh_datetime(raw):
    """Tolerant datetime parsing — crt.sh's format has drifted historically.

    Naive values are assumed UTC (crt.sh's documented convention); a
    trailing 'Z' is normalized to an explicit offset for
    `datetime.fromisoformat` compatibility.
    """

If you skip that, days_until_expiry is wrong by your server's UTC offset — which for a cert-expiry alert is the difference between a warning and an outage. And a record with an unparseable date is skipped with a warning rather than crashing the domain, because one bad row out of 40,000 is not a reason to lose the other 39,999.

The part that generalises 🧭

crt.sh is an unusually honest API: it makes no promises about volume and gives you no controls, so the discipline has to be yours. That is rarer than it sounds. Most APIs let you pass ?limit= and quietly cap you anyway, which trains you to believe the server is protecting you.

When a service has no pagination, the response size is an input you do not control — so treat it as a hostile one. Cap early, dedupe before you cap, and know which part of the cost you genuinely cannot avoid. Ours is the transfer; everything downstream of it we bounded.

What the Actor gives you

One row per certificate, or one per unique subdomain — your choice:

• every SAN expanded out of name_value, deduplicated, with the newest certificate kept per hostname
• days_until_expiry, is_expired and is_wildcard already computed, not left as raw timestamps
• issuer name, serial, certificate id and the CT log entry timestamp
• maxCertsPerDomain applied after dedupe, so the cap counts real results
• ISO-8601 timestamps with explicit UTC offsets throughout
• retries with exponential backoff on 408 / 429 / 503 — crt.sh runs on a shared Postgres box and does have slow moments

A domain that fails is a per-domain skip with a reason, never a dead run.

The honest limitations 🚧

• CT logs record certificates, not live hosts. A subdomain here may have been decommissioned years ago — this is discovery, not a port scan.
• Hosts that never had a public certificate do not appear. Internal-only names are invisible to CT by design.
• The un-capped HTTP response is a property of crt.sh. We bound everything after it; we cannot bound that.
• crt.sh is a free community service run by Sectigo. Be a good citizen with your sweep sizes.

Pricing

$0.20 per run plus $0.006 per certificate — about $6.20 per 1,000 results. A domain that returns nothing costs the start fee and nothing else.

→ Certificate Transparency Subdomain Scraper on Apify

---

Built by Devil Scrapes. We handle the SQL wildcards, the newline-packed SAN lists, the missing pagination and the drifting timestamps, so you get a flat table instead of a weekend.