I would validate Page identity and post identity separately. A collector can resolve the correct Page and still return an incomplete or incorrectly segmented post history.
The workflow here concerns public organization Pages. Personal profiles, groups, messages, and commenter identities are outside the default contract.
Page snapshots and post observations get separate tables
A Page snapshot would contain Page identity, submitted URL, final URL, observed name, website, observation time, access method, and validation status.
A post observation would contain native post ID where available, Page identity, permalink, publication time, text, media references, observation time, and extraction status.
I keep engagement snapshots separate from the stable post content. If a count changes, I want another observation rather than an overwrite that loses measurement history.
Where a native ID is unavailable, I document the fallback identity rule. I don't invent a native ID from a URL hash and label it as platform-provided.
My acceptance checks are source-specific
| Check | Failure I want to catch |
|---|---|
| Intended Page resolved | Profile, group, or similarly named organization |
| Post boundary preserved | Two posts merged into one text block |
| Permalink mapped correctly | Link belongs to a recommended post |
| Time interpreted explicitly | Relative display text treated as exact UTC |
| Pagination bounded and reported | Partial collection labeled complete |
| Error page classified | Sign-in prompt stored as Page content |
For relative times, I preserve the source text and parsing assumptions. A display such as “yesterday” depends on observation time and locale; it is not enough for an exact publication timestamp without further evidence.
The test fixtures would include text posts, media posts, pinned content, quiet Pages, redirects, and unavailable post URLs. These are planned cases, not reported collection results.
Access conditions are part of the adapter
For a supported official API route, I record API version, requested fields, and the relevant application permission context. I check Meta's current documentation and the application's actual approved access before implementation.
For a third-party extractor, I ask what surface it observes and how it represents errors. If it delivers a stored dataset, I would carry its collection time into my records rather than assigning the import time as observation time.
For a custom browser check, I use an authorized scope and record the viewing context needed to interpret the result. An unavailable page remains unavailable; it does not justify an access-control workaround.
I compare collections by post identity
Two runs can return different row counts because of pagination, pinned posts, removal, visibility, or observation time. I compare the sets of native IDs or reviewed permalinks before interpreting the difference.
When a post disappears, I retain a missingness state that matches the evidence. “Not returned by this run” is more defensible than “deleted” when the collector cannot establish the latter.
I keep extraction versions too. A provider parser change can alter measured counts without any source behavior changing.
My deployment gate checks the dataset
Before scheduling collection, I inspect a small reviewed sample and report identity accuracy, required-field completeness, post fidelity, and unresolved failures. A successful task status is insufficient.
I set request budgets and a retention policy, and exclude commenter identities unless a separate justified task requires them. The default exported schema should reflect that exclusion, not merely hide those fields in the dashboard.
My first monitoring run would stay small enough to compare with the reviewed fixtures. I would expand only after the collector can explain both its accepted posts and its missing ones.
Top comments (1)
The separate engagement table avoids overwriting history. I'd put parser version and pagination limit on each run too, then compare post-ID sets only between runs with comparable coverage. For BrowserAct Cloud exports, that metadata can travel alongside the rows into your database rather than disappearing during import.