If you scrape or analyze Facebook data, the same page will arrive under half a dozen different URLs — and if you count anything before resolving that, your numbers are quietly wrong. This is the reference I wish had existed: the formats, the traps, and a canonicalization procedure that never guesses.
Why your counts are wrong
Pull posts from any Facebook scraper and look at the page URLs. The same fish market shows up as:
https://www.facebook.com/HelensFreshMarket
https://m.facebook.com/helensfreshmarket/
https://www.facebook.com/profile.php?id=100077000000001
https://www.facebook.com/p/Helens-Fresh-Market-100077000000001/
Group by URL and you have four "pages," each with a quarter of the real engagement. Every downstream number — rankings, share-of-voice, averages — inherits the error. Post URLs have the same disease: the same post arrives with and without tracking parameters, from mobile and desktop hosts, and through share redirects.
The zoo, part 1: hosts
All of these serve the same content and must be folded together:
| Host | What it is |
|---|---|
www.facebook.com |
canonical desktop |
| facebook.com (bare) | redirects to www |
| m.facebook.com | mobile web (very common in scraped data) |
| mbasic.facebook.com | legacy low-bandwidth mobile |
| web.facebook.com | regional variant (common in Africa/Asia exports) |
| touch.facebook.com | legacy touch interface |
Rule 1: fold every host to www.facebook.com, force https, drop the trailing slash. Trivial, and it already collapses a third of the zoo.
The zoo, part 2: page-identity URLs
These are the formats that name a page (strongest evidence first):
| Format | Example | Identity signal |
|---|---|---|
| profile.php?id={N} | /profile.php?id=100077… | numeric ID — authoritative |
| /p/{Name}-{N}/ | /p/Helens-Fresh-Market-100077…/ | numeric ID — authoritative (newer format; the trailing number is the page ID) |
| /people/{Name}/{N}/ | /people/Helen-M/100077…/ | numeric ID — authoritative |
| /pages/{Name}/{N} | legacy | numeric ID — authoritative |
| /{vanity} | /HelensFreshMarket | slug — reliable but case-varies; case-fold before comparing |
| /pg/{vanity}/… | legacy page-tab wrapper | slug after stripping /pg/ |
Rule 2: extract identity by hierarchy — a numeric ID always outranks a slug. When both are observed for the same entity (a slug URL and a profile.php?id= URL that share posts or metadata), the numeric ID is the canonical key and the slug becomes an alias.
Rule 3: slugs are case-insensitive — case-fold before comparing. HelensFreshMarket and helensfreshmarket are the same page; a case-sensitive groupby splits them.
The zoo, part 3: post URLs
| Format | Example | Note |
|---|---|---|
| /{vanity}/posts/{id} | /helensfreshmarket/posts/101 | id may be numeric or an opaque pfbid… token |
| permalink.php?story_fbid={S}&id={P} | desktop permalink | the params are the identity — keep them |
| story.php?story_fbid={S}&id={P} | mobile permalink | same as above |
| /groups/{g}/posts/{id} | group post | group ≠ page; keep the distinction |
| photo.php?fbid={N} / /photo/?fbid={N} | photo post | fbid is the identity param |
| /watch/?v={N} | video watch page | carries no page identity (see traps) |
| /reel/{id} | reel | post-level only |
| /share/p/{token}/, /share/v/{token}/ | share redirect | opaque (see traps) |
Rule 4: for deduplication, strip tracking parameters but keep identity parameters. Strip: ref, refid, fbclid, mibextid, rdid, utm_*, tn, comment_id, notif_id, notif_t, locale. Keep: story_fbid, id, fbid, v. The same post with and without ?ref=share&utm=x is one post; the same path with a different story_fbid is not.
The traps (where guessing corrupts datasets)
Trap 1 — display names are not identity. Two pages named "Caladan Bay Tours" can be genuinely different entities (rebrand, fan page, impersonator, franchise). Merging on display name is how datasets rot. The correct behavior is to report the collision as a possible-same-entity hint with the evidence, and let a human decide.
Trap 2 — /watch?v= links carry no page identity. The URL names a video, not a page. Attribute it to a page only if the row carries separate page metadata; otherwise it is honestly unresolvable.
Trap 3 — share links are opaque. /share/p/{token}/ is a server-side redirect; the token doesn't encode the destination. Offline, you cannot resolve it — flag it, don't guess it.
Trap 4 — renames. A page that changed its vanity slug between scrapes, and never exposed its numeric ID, looks like two pages. Without an ID observation there is no safe automatic merge; surface it, don't force it.
Trap 5 — pfbid tokens. Modern post IDs are opaque pseudonymized tokens. Treat them as exact-match strings only; never parse meaning out of them.
The procedure, end to end
for each row:
1. fold host → www.facebook.com, https, strip trailing slash
2. classify URL: page-identity form vs post form
3. page identity:
numeric ID present → key = fb:{numeric}
else vanity slug → key = fb:{casefold(slug)}
else → UNRESOLVED (keep the URL as evidence)
4. post dedup key: canonical path + kept identity params
(tracking params stripped)
5. merge pages only when keys match
same display name, different keys → report hint, do NOT merge
The principle underneath: an unresolved row with its evidence attached is useful; a silently guessed merge is poison. Every downstream consumer can act on "here are 3 rows I couldn't resolve, and why" — nobody can undo a wrong merge they never knew happened.
Disclosure: I maintain a small free tool that implements exactly this procedure (including the refusals) for any posts-scraper export — Facebook Page Identity Resolver & URL Deduplicator on Apify, with a 3-second no-signup demo. But the procedure above works fine without it — that's rather the point.
Top comments (3)
I particularly appreciated the breakdown of the various URL formats in the "zoo" sections, which highlights the complexity of Facebook's URL structure. The distinction between numeric IDs and slugs as identity signals is crucial, and Rule 2 provides a clear hierarchy for extracting identity. I've encountered similar issues when working with social media data, where inconsistent URL formats can lead to incorrect counts and skewed analytics. Have you found any specific challenges when applying these canonicalization rules to larger datasets, especially when dealing with legacy or regional variants like
mbasic.facebook.comorweb.facebook.com?Thanks Luis — good question. Honest answer: the canonicalization itself scales trivially — it's deterministic string work, and in our stress runs 50,000 rows go end-to-end in ~16 seconds, with the parsing a rounding error inside that. The genuine large-dataset challenges turned out to live elsewhere:
Host-folding is necessary but not sufficient for posts. m./mbasic often expose the same post under a different path shape (story.php?story_fbid=…&id=… on mobile vs /{page}/posts/{id} on desktop), so the dedup key has to come from the identity params, not the folded URL string.
Ambiguity grows with scale. Same-display-name collisions are rare in a 30-page dataset and routine in a 3,000-page one — which is exactly why the procedure reports hints instead of auto-merging. A silent merge error compounds invisibly at scale; a reported hint stays a human decision.
And a mundane one: output I/O beats parsing as the bottleneck. Batching result writes gave us a ~28× speedup; the canonicalization was never the slow part.
web.facebook.com specifically we've only ever needed to fold — very common in African/Asian exports, structurally identical otherwise.
"Count anything before resolving that and your numbers are quietly wrong" applies way beyond FB honestly. Trailing slashes and utm junk will do the same to you on any site.
Canonical key first, count second. Learned that one the expensive way.