Quick answer: we asked Facebook's own internal pagination for 40 posts per Page. Across three unrelated Pages — NASA, Nike, Microsoft — we got back exactly 3, 3, and 3. Not roughly three. Exactly three, every time, on every Page. The cause was one isinstance() check that was always False: it expected the pagination cursor's path field to be a string, and on the real endpoint it's a JSON array. Fix the type check, and the same three Pages returned 40, 40, and 40 — 120 rows total, 120 distinct post IDs.
How does a type check silently cap every scrape at exactly 3 posts?
extract_next_cursor read line["path"] and tested isinstance(path, str) before trying to pull the next-page cursor out of it. On a real refetch response, path is never a string — it's always a JSON array, something like ["node", "timeline_list_feed_units"]. That check was False on every single response, which meant the function always returned (None, False): no cursor, no more pages. The scraper walked exactly one refetch page per Page and then, correctly by its own logic, stopped — regardless of what max_posts_per_page asked for.
The identifying signal was actually one level up from where the code was looking: the page-info chunk in the response is tagged by its Relay label (a string ending ..._timeline_list_feed_units$page_info), and the cursor itself sits one layer deeper than the old code read — data["page_info"]["end_cursor"], not data["end_cursor"]. Two wrong assumptions stacked on top of each other, and both had to be found before pagination worked at all.
Why did this look like a healthy Actor for so long?
Because our own QA fixture was built at a depth the bug couldn't be seen at. The prior test asked for 2 Pages at 10 posts each, hit the exact same 3-per-page ceiling on both, and got 6 rows back — which passes a lenient "SUCCEEDED with some rows" check even though it delivered 15% of what was requested. A ceiling that caps every run at the same small number doesn't look like a bug from inside a small fixture; it looks like a thin result. It took a deliberate deep run — 3 Pages, 40 posts requested each — to turn "a bit thin" into "exactly 3, 3, and 3," which is the shape that can't be an accident.
We also found the fixtures themselves were part of the problem. The NDJSON test files that exercised this code were hand-synthesized to the shape the buggy parser expected — path as a dotted string, cursor fields sitting flat under data — which is exactly why the unit suite stayed green through the whole thing. We replaced them with two real, live-captured pages from facebook.com/nasa, trimmed of irrelevant bookkeeping but never reshaped to match what the code wanted to see.
What else broke in the same commit?
Four fields that read from node keys that don't exist on the real payload: text, reaction_count, share_count, and comment_count. The code looked for them at the top level of a post node (message, reaction_count, share_count, a comments_count_summary_renderer key); the real values live several comet_sections layers down — the engagement counts specifically at comet_sections.feedback.story.story_ufi_container.story.feedback_context.feedback_target_with_context.comet_ufi_summary_and_actions_renderer.feedback.*. Same bug class as the pagination cursor: code written against a guessed shape instead of a captured one.
One field we checked and did not fix, on purpose: is_cross_post was reading False for every post in the sample, and it looked like a fifth instance of the same bug. It wasn't. The actor-id-vs-page-id comparison the field runs is correct as written — none of NASA's or Nike's sampled posts happened to be cross-posts. Grouping a correctly-negative field in with four genuinely broken ones would have been the wrong fix for a field that was never broken.
What you get per row
| Field | Notes |
|---|---|
post_id / permalink / creation_time
|
Stable post identity and timing — 100% fill rate, measured |
text |
Post body. ~92.5% fill — the gap is real photo/video-only posts with no caption, not a parsing miss |
author_name / author_id / author_url
|
Post author — 100% fill |
reaction_count / share_count / comment_count
|
Engagement counts — 100% / 98.3% / 100% measured across a 120-row deep run |
is_cross_post |
true when the post's author differs from the Page you requested |
attachments |
Photo/video attachments, structured as {type, attachment_id, url}
|
What does it actually cost?
Pay-Per-Event: $0.20 per run start + $0.003 per post landed — $3.20 per 1,000 posts. Measured, not estimated: the same 120-row deep run that proved the pagination fix settled at roughly $0.0356 total, about $0.30 per 1,000 rows of underlying platform cost — comfortably inside the price band. A Page with no new posts still costs only the flat start fee; there's nothing to bill per post for posts that weren't there.
FAQ
Does this need a Facebook account or access token?
No. It reads the same public timeline anyone can see logged out.
Why did an earlier version of this only return a few posts per Page?
A pagination cursor check was comparing the wrong type against the wrong field, so every Page's walk stopped after one internal page regardless of how many posts you asked for. Fixed — verified at 40/40/40 across three unrelated Pages.
Can it scrape groups or personal profiles?
No — public Page timelines only. Groups, personal profiles, Reels, Marketplace, and Events are out of scope.
What happens if a Page is private, deleted, or login-walled?
That identifier is reported as failed on its own — it never stops the rest of the Pages in the same run.
😈 Facebook Posts Scraper walks any public Facebook Page's timeline and exports one clean row per post — text, permalink, timestamp, author, and engagement counts — through the same internal endpoint the Page itself loads. We rotate browser fingerprints, self-heal Meta's rotating persisted-query id, and isolate a bad Page from the rest of your batch. $3.20 per 1,000 posts.
Top comments (0)