5 ways subdomain enumeration breaks (and how to handle each)

Osiris Technical Institute — Wed, 13 May 2026 13:28:16 +0000

Subdomain enumeration looks easy. There's a wordlist. There are CT logs. There's a DNS resolver. Plug them together, return a list. Maybe sort it.

Then you run it on 50 different domains for the first time and notice that the results are wildly inconsistent. Sometimes you get 3 subdomains. Sometimes you get 30,000. Sometimes you get an empty array on a domain that should have an obvious hit. The tool isn't broken — it's quietly failing in five different ways, depending on the input.

Here's what each one looks like in production, and how to build a tool that actually returns useful results across arbitrary inputs.

1. CT log sources go down silently

The most common single source is crt.sh. It has fantastic coverage, it's free, and its uptime is... not consistent. A naive implementation hits crt.sh, gets a non-200 or an empty array, and treats the result as "no subdomains found." Which is technically what crt.sh returned. But it's wrong.

The fix is parallelism plus honesty:

async def enumerate_subdomains(domain):
    sources = [crtsh, certspotter, hackertarget]
    results = await asyncio.gather(
        *[run_with_timeout(s, domain, timeout=10) for s in sources],
        return_exceptions=True,
    )
    ok, failed = [], []
    for source, r in zip(sources, results):
        if isinstance(r, Exception) or not r:
            failed.append(source.__name__)
        else:
            ok.extend(r)
    return {"subdomains": sorted(set(ok)), "warnings": failed}

Three independent sources query in parallel. Return the union of whatever succeeded, plus a warnings[] array naming which sources failed so the caller can reason about completeness. If only one source succeeded, the caller knows. If all failed, the caller knows. Silent partial failures are the actual bug — not the upstream outages.

2. Pure CT misses dormant subdomains

Certificate Transparency logs catch every subdomain that's ever had a TLS certificate issued. They don't catch:

Internal-only subdomains that never got a public cert
Subdomains using self-signed certs
Subdomains provisioned this week that haven't issued a cert yet
Subdomains behind cloud providers that don't propagate certs to the CT log ecosystem

The realistic gap is somewhere between 10% and 30% of an organization's actual subdomain inventory, depending on how much internal infrastructure they expose.

The fix is a DNS bruteforce fallback. Critically, this fallback only runs when the CT sources return suspiciously few results — not on every query, because bruteforce is slow and noisy:

async def fallback_bruteforce(domain, wordlist):
    # Resolve a random subdomain first - if it returns an IP,
    # the domain has wildcard DNS and bruteforcing is meaningless
    nonce = "doesnotexist-" + secrets.token_hex(8)
    if await resolve(f"{nonce}.{domain}"):
        return None  # wildcard DNS - skip
    return [w for w in wordlist if await resolve(f"{w}.{domain}")]

Bigger wordlist = better coverage, slower runtime. The standard wordlists from ProjectDiscovery work well as a starting point.

3. Wildcard DNS poisons results

*.example.com is legitimate DNS configuration — it lets a domain resolve for any subdomain to a single IP. CDNs and SaaS platforms use it routinely. If you're running a DNS bruteforce against a domain with a wildcard, every word in your wordlist resolves successfully, and you return a list of 10,000 fake subdomains.

The fix is the wildcard-detection probe shown above: query a random nonsense subdomain before running the wordlist. If the nonsense resolves, the domain has a wildcard, and you skip bruteforcing entirely (or aggressively filter to entries whose IP differs from the wildcard's IP — which catches some real subdomains but is fragile).

This is the failure mode that does the most damage to a tool's credibility. A user who sees 10,000 results, half of which are obviously made up, doesn't trust the output again.

4. Rate limiting and IP bans

crt.sh especially is aggressive about per-IP rate limiting. If your tool runs queries serially with no backoff, you'll hit the limit somewhere around the 5th or 10th query of a session and start getting 429s or 503s — which look identical to "no results" if you're not reading status codes carefully.

Three pieces of the fix:

Per-source timeouts. Cap each upstream call so a slow source doesn't block the whole pipeline.
Exponential backoff with jitter on retries. Standard pattern, applied per-source.
Bounded parallelism. Three sources in parallel is fine. 50 queries against crt.sh in parallel is suicide. Use a semaphore or a small worker pool.

Also, narrow your queries when possible. crt.sh supports specific TLD filters (?q=%.example.com) that return faster and use less server resources than broader searches.

5. IDN / punycode domains return nothing

International domain names (IDNs) — Cyrillic, Chinese, Arabic, etc. — get encoded as ASCII via punycode (the xn-- prefix) before going into DNS. A user looking up мой-сайт.com needs the tool to convert that to its xn-- form before any DNS query.

Most modern libraries handle this — dnspython and httpx both do. But if your tool is doing raw string concatenation anywhere in the pipeline (f"{prefix}.{domain}" without normalization), you'll silently fail on IDNs and the user won't know why. The fix is one line:

domain = idna.encode(user_input).decode("ascii")

Run that as the first step of every domain-handling function. Always, even if you "don't expect IDNs." You'll get one eventually.

The pattern

Every one of these failures is silent. The tool returns something — usually an empty list or a contaminated one — and the user assumes the tool worked. The single biggest improvement you can make to a subdomain enumeration tool isn't a new source or a faster algorithm. It's surfacing failure honestly. Return what succeeded. Name what didn't. Let the caller decide whether the result is complete enough to act on.

This is exactly the pattern we built into Domain Intelligence — bundles subdomain enum (CT logs + DNS bruteforce fallback, wildcard-aware) with DNS, WHOIS/RDAP, SSL, and email security in one call, and surfaces partial failures in a warnings field rather than swallowing them. Free tier on RapidAPI, MIT-licensed source on GitHub.

But the principles above hold whether you use a service or roll your own. Honest failures beat confident wrong answers every time.