DEV Community

Cover image for Working out what a site is running — and where every check lies to you
Hammad Shams Uddin
Hammad Shams Uddin

Posted on

Working out what a site is running — and where every check lies to you

#webdev #devops #tooling #tutorial

You want to know what some site is built on. Maybe a competitor, maybe a client's old site nobody has the credentials for, maybe you just saw something fast and want to know why.

There are four places to look, and each one will confidently tell you something false under conditions you should know about. The checks are easy. Knowing when to disbelieve them is the actual skill.

1. DNS, and the CDN that hides everything behind it

The first instinct is to resolve the domain and see where it lands.

dig +short example.com
Enter fullscreen mode Exit fullscreen mode

This is genuinely useful and it is also the check that misleads people most often, because an IP in front of a CDN is the CDN, not the host.

I ran this on my own site recently. The IP that came back belongs to Cloudflare. My actual server is on a shared host in a different country entirely, and nothing in DNS will tell you that — which is most of the point of putting Cloudflare there.

So the rule is: if the IP resolves to Cloudflare, Fastly, Akamai or a big cloud edge, you have learned who fronts the site and nothing about who hosts it. That is still worth knowing. Just do not write "hosted on Cloudflare" in your notes.

The records that survive the CDN are the ones it does not proxy. MX almost always points at the real mail provider. TXT is a gift — SPF records list every service allowed to send mail as that domain, which in practice is a list of the SaaS products the company pays for.

dig +short example.com MX
dig +short example.com TXT
Enter fullscreen mode Exit fullscreen mode

My own domain demonstrates the whole point in two lines. The A record says Cloudflare (AS13335). The MX record says mx1.hostinger.com. The first tells you nothing about where the site lives; the second quietly names the host, because mail does not route through the CDN.

If you are not at a terminal, a browser DNS lookup will pull the same records, and there is a domain to IP tool for the plain A-record case.

2. Response headers, which are honest by accident

Headers are the highest-signal-per-second check there is, because they are mostly emitted by machines that have no interest in your investigation.

curl -sI https://example.com
Enter fullscreen mode Exit fullscreen mode

What to read:

  • server — sometimes the real thing (nginx, Apache), often replaced by the CDN.
  • x-powered-by — when present, usually accurate and usually an oversight.
  • cf-cache-status — Cloudflare, and more interesting than it looks. HIT means the edge served it. DYNAMIC means Cloudflare was told not to cache this. BYPASS means something upstream explicitly said no.
  • via, x-vercel-id, x-amz-cf-id — the infrastructure signing its own work.

That cf-cache-status line is worth dwelling on, because it varies by path on the same site and the variation is the configuration, visible from outside. Two requests against my own domain:

/public/css/style.min.css   HIT       cache-control: public, max-age=31536000, immutable
/tools/dns-lookup           DYNAMIC   cache-control: private, max-age=0, must-revalidate
Enter fullscreen mode Exit fullscreen mode

Assets cached at the edge for a year, HTML never cached at all. That is a deliberate setup and you can read it off in about four seconds without any access to the site.

The lie to watch for: headers describe the machine that answered, which on a CDN is an edge node in your own city, not the origin. A missing x-powered-by means the CDN stripped it, not that the site is not running PHP.

If you want status codes and the full redirect chain rather than raw headers, this status code checker follows the chain and shows each hop, which is the part that matters when you are chasing a redirect loop.

3. The HTML, which is where the CMS gives itself away

Frameworks leave litter. Fetch the page and look for it:

curl -s https://example.com | grep -oE '<meta name="generator"[^>]*>'
curl -s https://example.com | grep -oE '/wp-(content|includes)/[^"]*' | head
Enter fullscreen mode Exit fullscreen mode

/wp-content/themes/something/ in an asset URL names the WordPress theme outright. <meta name="generator" content="WordPress 6.x"> does the same more politely. Shopify, Ghost, Wix and Squarespace all announce themselves similarly if nobody has bothered to turn it off.

There is a WordPress theme detector that does this pattern-matching for you and also reports the plugins it can see from asset paths, which is often the more interesting half.

The lie here is the biggest one on the list, and it is not the HTML's fault — it is yours. Plenty of pages build their content in JavaScript, so what curl returns is a shell and what a browser shows is the page. If you grep the raw HTML and find nothing, you have not learned that there is nothing there.

I lost an evening to exactly this last week. I was checking whether a publication's guest-author page linked to an application form, grepped the HTML for mailto: and forms, found neither, and concluded the route was elsewhere. The page renders its body client-side. Everything I wanted was there, in the browser, the whole time.

When the raw HTML looks empty, that is the signal to open devtools and read the rendered DOM instead — not to conclude anything.

4. robots.txt, which tells you what exists

This one is under-used and it costs one request.

curl -s https://example.com/robots.txt
Enter fullscreen mode Exit fullscreen mode

Disallow lines are a list of things the site owner knows about and would rather you did not crawl. That is a site map of sorts, drawn by someone with full knowledge.

A concrete example. I was evaluating a service to see whether their listings link back to the products they list. Their robots.txt disallowed /provider/ and /developer/ — which told me those sections existed, that they were considered low-value, and that whatever lives there is not meant to be indexed. Combined with the listing page carrying no outbound link at all, the picture was complete in about four minutes.

The lie: robots.txt is a request, not a wall. Paths listed there are frequently still reachable and sometimes still indexed. It tells you about intent, not access.

Putting it in order

For a site you know nothing about, this is about five minutes:

  1. curl -sI — status, server, CDN, redirect chain
  2. curl -s .../robots.txt — what exists, what they hide
  3. dig MX and dig TXT — real mail host, and every SaaS they pay for
  4. Page source for generator and /wp-content/ — the CMS
  5. If the source is thin, open devtools and read the rendered DOM

And the corresponding list of what each one cannot tell you, which is the part worth memorising:

Check Confidently wrong about
A record the origin, whenever a CDN is in front
Response headers the origin, again — it is the edge answering
Raw HTML anything rendered client-side
robots.txt what is actually reachable

None of these are exotic failure modes. They are the normal case on any site built in the last five years, which is why the checks are worth running together rather than one at a time. Each one covers the specific blind spot of another.

The version of this I keep having to relearn: a check that returns cleanly has not necessarily answered your question. It has answered its question, and it is on you to know whether that was the one you asked.

I build Utilorax, a set of free browser-based tools — the ones linked above among them. They exist because I got tired of not having a terminal handy on someone else's laptop.

Top comments (0)