DEV Community

Cover image for Python Facebook Posts Scraper with Requests and BeautifulSoup4

Python Facebook Posts Scraper with Requests and BeautifulSoup4

Hugo Sandoval on May 13, 2020

A few weeks ago I wrote about the Importance of using User Agents when we scrap data, and my examples shows the response from Twitter when we used ...
Collapse
 
sis4insa profile image
sis4insa • • Edited

Hello Every one I need for research purpose 100 users already posted text.Per user i need from 120 to 200 posted text.But this python code download only 5 days post.Can I use this code for this purpose pls help me I kill 2 month for this purpose.I need to download a specific users profile post text all

Collapse
 
sis4insa profile image
sis4insa •

please any one help me to this problem

Collapse
 
menbiamel profile image
menbiamel • • Edited

hello Hugo thanks for sharing this .am working on scraping facebook data but the facebook blocked my account several times . please make a video to help as to see how it works.
how many posts does it extract ?

Collapse
 
hhsm95 profile image
Hugo Sandoval •

The key to avoid being blocked is scrap slowly (about 3-5 between profiles) and use an User Agent like above (an old phone) that don't requires javascript so the detection is harder for Facebook.

Collapse
 
jazzman16 profile image
jazzman16 •

Hugo, this is great. When I run this, as is, I'm only getting about 6 posts. Is this normal? Also, how would you recommend I edit to search for group posts with a specific tag_id? I tried to rewrite a few lines to specify my url with "groups/{group_id}/post_tags/?post_tag_id={tag_ID}" but I was only ever brought to the group's standard timeline.

Collapse
 
nassera2014 profile image
nassera2014 •

Hi, i had the same problem, when i run, i'm only getting 6 posts ! you resolved this ?
Thank you.

Collapse
 
Sloan, the sloth mascot
Comment deleted
Collapse
 
hhsm95 profile image
Hugo Sandoval •

Hi, I'm not sure if I can post an email here, but ask me any issues in the Github project.
github.com/adeoy/FacebookPostsScraper

Collapse
 
ds123 profile image
data science •

getting error TypeError: 'module' object is not callable while running the code. What may be the problem ? Thanks in advance

Collapse
 
rakshit888 profile image
rakshit888 •

same error

Collapse
 
nassera2014 profile image
nassera2014 •

Hi, Thank you for sharing this, that's works well, but when i run the program, i get just 6 posts from a facebook page? how can i get more posts ?
Thank you.

Collapse
 
chalst1988 profile image
Surya Teja Challapalli • • Edited

Hi @nassera2014

In order to get more number of posts I suggest to use Facebook_Scrapper Library where one can configure get_posts() with number of pages and posts per page to be scrapped. And by the way you could get all granular level info of reactions, like, comments, post_id, post date, etc.,

Please find the below link for your information,
pythonzero.com/2021/12/how-to-extr...

Collapse
 
benjaminbenteke profile image
Benjamin Benteke Longau •

Hello Sir thank you for you tuto. How can we also collect for each post all comments, likens and number of share?

Collapse
 
moukouba profile image
Moukouba •

Hi!

Could you please advise on how to build a scraper that extracts comments on a specific subject on Facebook?
Thanks for assisting.

Collapse
 
hhsm95 profile image
Hugo Sandoval •

Hi, the key to scraping on Facebook is the use of a User Agent from an old phone, as in this case use a Nokia C3. To extract the comments from the posts, it is necessary to enter the post and locate the CSS selectors that identify the comments, it is possible that I update the post adding this functionality.

Collapse
 
mezaouryounes profile image
Younes Mezaour •

Hello :) thank you for this share, can you please tell me the equivalent of "post_url_text" for the French version?

thanks :)

Collapse
 
omejiasq profile image
omejiasq •

hello guys. Can i to use today this steps and code to download comments and post of facebook?