You have a column of domains (sign-ups, a trade-show list, a CRM export) and you want a name, logo, one-line description, industry and the company's LinkedIn page next to each. Enrichment APIs sell this as a subscription. But for most companies the data is published on their own home page, in machine-readable form.
Where the data actually lives
-
schema.org JSON-LD. Many sites ship an
Organizationblock for Google:name,logo,sameAs(their official social profiles),foundingDate,address, sometimesnumberOfEmployees. -
OpenGraph and meta tags.
og:site_name,og:imageandmeta descriptionare on almost every site, because they make link previews look right. -
The header and footer. Social icons,
mailto:andtel:links, and links to "About" and "Contact". -
The about and contact pages. "Founded in 2014", the postal address and the
info@mailbox.
A minimal extractor in Python
import json, re
from curl_cffi import requests # pip install curl_cffi lxml
from lxml import html
def profile(domain):
r = requests.get(f"https://{domain}", impersonate="chrome", timeout=20)
doc = html.fromstring(r.content)
meta = {m.get("property") or m.get("name"): m.get("content") for m in doc.xpath("//meta") if m.get("content")}
org = {}
for s in doc.xpath('//script[@type="application/ld+json"]'):
try:
data = json.loads(s.text_content())
except ValueError:
continue
for node in (data if isinstance(data, list) else data.get("@graph", [data])):
if isinstance(node, dict) and node.get("@type") in ("Organization", "Corporation", "LocalBusiness"):
org = node
links = doc.xpath("//a/@href")
return {
"name": org.get("name") or meta.get("og:site_name"),
"description": meta.get("description") or meta.get("og:description"),
"logo": org.get("logo") if isinstance(org.get("logo"), str) else meta.get("og:image"),
"linkedin": next((l for l in links if "linkedin.com/company/" in l), None),
"emails": sorted({l[7:].split("?")[0] for l in links if l.startswith("mailto:")}),
}
print(profile("shipbob.com"))
That gets you surprisingly far. The hard part is the edge cases.
The edge cases that eat your week
-
Names. The
<title>is often a slogan ("Love your bank | Spend, save and invest"). Score each candidate (schema name,og:site_name, title segments) by how well it matches the domain, and fall back to a word in the page text that equals the domain label (N26,3M). -
Social links. Footers link to articles and share buttons too:
twitter.com/intent/tweet,facebook.com/sharer, a GitHub repo mentioned in a blog post. Keeplinkedin.com/company/…and droplinkedin.com/in/…(that's a person). TrustsameAsfirst. -
Personal data. A contact page can list a named person's email. If you only want company data, keep role mailboxes on the company's own domain (
info@,sales@,press@) and drop the rest. - "Founded in 1983". On an about page, that can be from a person's bio ("she co-founded … in 1983"). Ignore years near personal context.
- Bot walls, parked domains and maintenance pages. These return HTTP 200 with junk. Detect them and don't count them as results.
-
Encodings. Not every site is UTF-8. Read the header and
<meta charset>before decoding.
What fill rates to expect
From test runs on October 27, 2026 (home + about + contact pages, no browser):
| 60 well-known companies | 500 random domains from a top-sites list | |
|---|---|---|
| Profile found | 90% | 61% (the rest were dead, parked, blocked or empty) |
| Name / description | 100% / 100% | 100% / 90% |
| Logo | 78% | 84% |
| LinkedIn company page | 67% | 32% |
| 85% | 51% | |
| Generic email | 35% | 24% |
| Founded year | 41% | 20% |
Without a browser, JavaScript-only sites give you nothing. With browser-like TLS (curl_cffi), most "403 for bots" pages go away, but sites with real bot protection still block cloud IPs.
The hosted version
Disclosure: we (SteadyScrape) built the tool below and get paid when it's used.
Company Data by Domain (steady-scrape/company-profile-by-domain) is an Apify Actor that does the above for lists of up to 20,000 domains per run. It adds an industry guess with a confidence label (high, medium or low), the postal address, tech hints and a per-row status, so you can see why a domain didn't return a profile. It reads only the company's own public site (home, about and contact pages), doesn't scrape LinkedIn and doesn't return personal data.
import os
from apify_client import ApifyClient # pip install apify-client (v3)
client = ApifyClient(os.environ["APIFY_TOKEN"])
run = client.actor("steady-scrape/company-profile-by-domain").call(
run_input={"domains": ["stripe.com", "shipbob.com", "levainbakery.com"]})
for row in client.dataset(run.default_dataset_id).iterate_items():
print(row["domain"], row["status"], row.get("companyName"),
row.get("industryGuess"), row.get("industryConfidence"), row.get("linkedinUrl"))
Output from a run today (October 29, 2026), about 12 seconds end to end:
shipbob.com ok ShipBob Logistics / Transportation high https://www.linkedin.com/company/shipbob
stripe.com ok Stripe Fintech / Payments medium https://www.linkedin.com/company/stripe
levainbakery.com ok Levain Bakery Food & beverage high https://www.linkedin.com/company/levain-bakery
Rows with status other than ok (unreachable, blocked, parked, empty, invalid or duplicate domains) come back with the reason, so you can retry or drop them.
What it costs
It's pay per result: $0.004 per company profile found ($4 per 1,000), plus Apify's standard $0.00005 start event per run. Dead, blocked, parked and empty domains are free, and Apify paid plans get volume discounts on the same event.
Example: you upload 1,000 domains, 700 return a profile and 300 are dead, parked or blocked. You pay 700 × $0.004 = $2.80. You can set a maximum cost per run, and the Actor stops before it goes over.
If you need revenue or headcount estimates, this isn't the tool: those come from data vendors, not from the company's website. If you want to rank the same list by size, Bulk Domain Rank & Traffic Checker adds traffic estimates, backlinks and spam score per domain.
Top comments (0)