DEV Community: Braydon Buckner

Scrape Instagram Followers | Extract Follower Data (2021)

Braydon Buckner — Fri, 05 Mar 2021 07:31:12 +0000

Important note: Please be advised that automatically scraping follower data from Instagram is against their terms of service.

This is just an educational resource and for those of you who don’t have the time, resources, and want to be 100% on the legal side of things, get in touch with Influencers Club and just buy email addresses from Instagram followers.

A lot of you want to know how to scrape data from your Instagram followers or anyone else's followers (cough-competitors-cough).

The reason being, it's an awesome way to get contact information like emails and phone numbers from a targeted audience.

In this guide I'll share how you can scrape follower data yourself or completely outsource the scraping.

What Kind of Data You Can Scrape From Instagram?

First things first, other than email addresses and phone numbers what other data you can extract from one Instagram account?

• Instagram Bio
• Number of Followers
• Location (10% of all users have country/city/street)
• Instagram Username
• Instagram Category (1 out of 1500)
• Number of Posts
• Whether it's a business or a regular Instagram account

All of these data points will help you further analyze your database. For example if you want to find influencers among your followers (or competitors) you can simply sort your dataset by number of followers. That way you'll instantly see which Instagram users are popular and can be used to spread the word about your brand.

How to find emails and phone numbers from Instagram followers

Few people know this but 30-40% of all Instagram users have their contact information right on their profile.

Some of them add it willingly and some not...let me explain.

There are two ways to find an email address or phone number on Instagram:

Instagram Biography
10% of all users intentionally add their contact details in their bio. Businesses, influencers and even regular users hope someone will reach out to them when they leave their email and contact number in bio.
Instagram Business Accounts
Another 30% of all Instagram users have a so-called Instagram business account. With this type of profile they access additional statistics about their followers such as engagement %.

However, what they (probably) don't know is that when creating a business account they also agree to make their personal contact information publicly available.

How to Instantly Get Data From Instagram Followers?

There are a lot of data providers that collect Instagram user data and can scrape anyone's Instagram followers instantly.

Before I continue explaining how you can do this yourself I want to explain why is sometimes better to simply outsource the scraping.

I use this Instagram email scraper because all of their data is validated upon scraping and they offer advanced targeting options like gender, age, location, interest.

Here's why that's important:

Why you need additional targeting?

Although scraping Instagram followers is pretty targeted you still end up with a lot of people that are not relevant to your business.

• 10% or 90 million Instagram profiles are fake accounts
• A lot of people will simply don't fall in your ideal niche (no gender, location, age or other types of targeting)

And if you plan on scraping emails for cold email marketing then this will negatively affect your campaigns i.e. very high bounce rates.

Why you must clean the scraped data?

Web-scraping means a lot of trash in your net. Some examples would be:

• Spam-traps
• Catch all emails
• Fake emails

Here's how this affects your plans:

Scraped email lists from Instagram can be used in two ways - either for cold emailing or using them as custom audiences in Facebook Ads.

When you create your cold email campaign and reach out to so many invalid emails, there's a good chance your company domain will end up in SPAM. This means that even regular emails will be considered spam and not reach the desired destination.

Or, if you import the list as Custom Audiences, Facebook will notice that a lot of your contacts are invalid which may result in permanent ban.

That's why the Instagram scraper I use also validates the emails and offers additional targeting like gender, age, location or emails based on keywords in user bio.

How to Scrape Instagram Follower Data For Free?

If you have less than 1000 followers you can extract data manually. Note that to see more information on a profile you need to visit all users with the Instagram application i.e. from a mobile device.

Collect the data by copy-pasting it in a spreadsheet.

PS. I really don't recommend visiting more than 1k profiles manually (at least not with your personal profile) because you will get your account suspended.

Building an Instagram Follower Scraper with Python and GitHub

Python is the best way to scrape anything really.

Extracting data from Instagram followers with Python comes down to two simple things.

Scraping the usernames (or user IDs) that follow a given profile (yours, your competitor’s followers etc), hashtag or location (yes you can do that too)
Scraping additional details (email, category, etc) for all usernames/ids collected

Extract Instagram Followers (User IDs) List

If you only want to scrape follower usernames, here's a code sample by @tonoli (Github).

from datetime import datetime
from modules import compare
from modules import file_io
from modules import stats

from modules.scraper import Scraper
from modules.utils import ask_input, ask_multiple_option


groups = ['followers', 'following']

# Ask for input
target = ask_input('Enter the target username: ')
group = ask_multiple_option(options = groups + ['both']);
print('\nEnter your Instagram credentials')
username = ask_input('Username: ')
password = ask_input(is_password = True)

def scrape(group):
    differs = False
    scraper = Scraper(target)
    startTime = datetime.now()

    scraper.authenticate(username, password)
    users = scraper.get_users(group, verbose=True)
    scraper.close()

    last_users = file_io.read_last(target, group)
    if last_users:
        differs = bool(compare.get_diffs(users, last_users))

    if (differs or not last_users):
        file_io.store(target, group, users)
    # Stats
    stats.numbers(len(users), scraper.expected_number)
    if (differs): stats.diff(users, last_users)
    print('Took ' + str(datetime.now() - startTime))

if (group == 'both'):
    for group in groups:
        scrape(group)
else:
    scrape(group)

This is enough to extract the followers of specific Instagram account.

One thing worth mentioning is that profiles with huge follower base will be harder to scrape with this method (Instagram is pretty strict with call-limits). But for a small account, it should do the job.

Here's all you'll need to execute this:
• Python 3
• PIP
• Chrome installed (Selenium uses the chromedriver contained in the drivers folder)
• and, Instagram credentials to log in

Scraping additional follower details (email, category, etc)

If you only needed to scrape list of Instagram usernames, that follow a specific account, then you’ll be done with the first part. But if you want to market to those people, then you’ll need to scrape additional details, like email addresses, phone numbers, category of the profile etc.

This is a bit harder to do as you’ll need Instagram accounts and log them through the mobile API. Here is a sample on how to do that with the browser, but again, you are limited and PLEASE don’t do any scraping with your main Instagram account:

from requests import get
from bs4 import BeautifulSoup
import sys
import re
import json
from datetime import datetime

URL = 'https://www.instagram.com'

def get_data(username):
    url = '%s/%s/' % (URL, username)
    page = get(url, timeout=5)
    soup = BeautifulSoup(page.content, 'html.parser')
    data = soup.find_all('meta', attrs={'property': 'og:description'})
    photo = soup.find_all('meta', attrs={'property': 'og:image'})
    text = data[0].get('content').split()
    retext = re.findall(
        '<script type="text\/javascript">([^{]+?({.*profile_pic_url.*})[^}]+?)<\/script>', str(soup))[0][1]
    jsontext = json.loads(retext)['entry_data']['ProfilePage'][0]
    print(jsontext)
    media_likes = 0
    media_comments = 0
    media_views = 0
    media_count = 0
    media_videos = 0
    media_list = []
    for media in jsontext['graphql']['user']['edge_owner_to_timeline_media']['edges']:
        media_count += 1
        media_likes += media['node']['edge_liked_by']['count']
        media_comments += media['node']['edge_media_to_comment']['count']
        if media['node']['is_video']:
            media_videos += 1
            media_views += media['node']['video_view_count']
            media_list.append({
                'id': '2042452950535483363',
                'img_url': media['node']['thumbnail_src'],
                'date_posted': datetime.fromtimestamp(media['node']['taken_at_timestamp']),
                'likes': media['node']['edge_liked_by']['count']  
            })
        else:
            media_list.append({
                'id': '2042452950535483363',
                'img_url': media['node']['thumbnail_src'],
                'date_posted': datetime.fromtimestamp(media['node']['taken_at_timestamp']),
                'likes': media['node']['edge_liked_by']['count'],
            })
    media_video_like_engagement = 0
    media_video_comment_engagement = 0
    if media_videos != 0:
        media_video_like_engagement = (
            ig_tv_likes / ig_tv_videos) / (ig_tv_views / ig_tv_videos)

    return ....

Full code available here.

If you don’t want to bother with that, you can just upload the list of usernames you already got to this API and enrich it with the email addresses. It’s free to sign up and you’ll also get 1500 credits so use it to speed up the process.

Get Instagram follower data (no-code)

If you have no previous experience with Python but want to export a list of your followers (without contact data) you can use IGExport. If you are targeting less than 500 followers you can export the list for free. The max amount of followers you can download is 50k.

And for those of you who also want to get valid & targeted emails and phone numbers from anyone's followers, visit Influencers Club.

Instagram Scraping | The Top 2021 Guide [Updated]

Braydon Buckner — Wed, 03 Mar 2021 11:43:34 +0000

In this guide, you will find the best ways to scrape Instagram data. You can extract Instagram emails, phone numbers, images/photos, user bio, hashtags, etc.

To get the data you can scrape based on the followers of a specific Instagram account (your own or your competitors) or relevant hashtags.

Sit tight.

Instagram Scraper Tool - How to Build One Yourself

As you’ll see in this article, it’s money and time consuming to develop an Instagram scraper yourself so if you want to skip the whole process and get targeted and validated data use this Instagram email scraper.

Scraping Instagram data with the unofficial API

To access the unofficial Instagram API you need to use the mobile app since we’re after the mobile endpoints. If we intercept the traffic coming from Instagram’s servers and to we’ll be able to collect Instagram data.

Read more about how to access the API and scrape data from Instagram visit this Instagram scraper tool.

Instagram scraping with Python and GitHub

The above-mentioned mobile endpoints can be accessed with Python, PHP, or any other language that can log into Instagram profiles and extract data. While it’s only a couple of Python lines, there is so much more to it but here it goes.

To log into a profile and use the unofficial Instagram API use the code sample from this GitHub:

def login(self, force=False):
        """
            Authenticate this API instance.
            If already logged in (and not later logged out) does nothing (unless forced).
        :param force: if true, will attempt to log in even if already logged in.
        :return: dictionary of responses.
        """
        if not self._isloggedin or force:
            self._session = requests.Session()
            # if you need proxy make somethin://proxyip:proxyport"}
            full_response = self._sendrequest(
                'si/fetch_headers/?challenge_type=signup&amp;guid=' + self.generate_uuid(False), login=True)
            data = {
                'phone_id': self.generate_uuid(True),
                '_csrftoken': full_response.cookies['csrftoken'],
                'username': self._username,
                'guid': self._uuid,
                'device_id': self._deviceid,
                'password': self._password,
                'login_attempt_count': '0'}
            try:
                full_response = self._sendrequest(
                    'accounts/login/',
                    post=self._generatesignature(json.dumps(data)),
                    login=True)
            except InstagramAPIBase._2FA_Required as exception:
                # In order to login, need to provide the second factor (i.e. SMS code or backup code).
                # Use call-back to get this string.
                if not self._two_factor_callback:
                    raise AuthenticationError("This account requires support for Two-Factor Authentication")
                two_factor_info = exception.two_factor_info = exception.two_factor_info
                verification_string = self._two_factor_callback(two_factor_info)
                data = {
                    'verification_code': verification_string,
                    'two_factor_identifier': g like this:
            # self.s.proxies = {"https": "httptwo_factor_info['two_factor_identifier'],
                    '_csrftoken': full_response.cookies['csrftoken'],
                    'username': self._username,
                    'device_id': self._deviceid,
                    'password': self._password,
                }
                full_response = self._sendrequest(
                    'accounts/two_factor_login/',
                    post=self._generatesignature(json.dumps(data)),
                    login=True)
            self._isloggedin = True
            decoded_text = json.loads(full_response.text)
            self._loggedinuserid = decoded_text["logged_in_user"]["pk"]
            self._ranktoken = "%s_%s" % (self._loggedinuserid, self._uuid)
            self._csrftoken = full_response.cookies["csrftoken"]
            return decoded_text

From my experience, logging in is pretty straightforward. Use a proxy (if possible from the location you’re at) and fill out any captchas necessary. After the accounts rest for a few days you can start your scraping.

P.s. be extremely cautious with the login request because it can trigger Instagram’s algorithm to keep an eye on you.

Instagram scraping profiles for getting the data

While I started manually collecting emails and phone numbers from Instagram I now own 1500 Instagram profiles that simultaneously crawl the platform. And it’s not too much! You need a large number of accounts because the API call limit is very...limited. Meaning you can only gather so much data from one account.

So, if you’re serious about building an Instagram tool online with Python you have to remember two things:

Never use your personal Instagram account
And the Instagram profiles you purchase need to be aged and validated with a phone number (if you don’t want to get suspended)

Another thing to remember is that Instagram has gotten pretty smart in detecting fake accounts that are purchased at scale. Mainly they study the gray market and track the source.

My suggestion is to diversify and buy accounts from multiple sources and see which one is safe.

To keep the Instagram scraping safe use proxies

When you use proxies, Instagram no longer tracks your IP address but the address of the proxy used. While you can scrape from one server, you still need to make sure to keep the IP simulations low because logging in from more than 5 different IG accounts can be a huge problem.

Just as I previously mentioned the problem of the gray market and Instagram, the same holds true for proxies as well.

Instagram has the ability to detect millions of proxy providers, and you need to find the perfect one for long-term scraping.

Benefits and drawbacks of having your own Instagram scraper

It’s pretty hard creating your own tool that can extract Instagram user data, so let’s see whether building your own tool is viable or not.

Starting with the pros

Having full control over the whole process
You can use and reuse the email addresses and phone numbers
And sell and resell the database

And the drawbacks of an in-house scraper are:

No segmentation or additional targeting
A lot of bot accounts scraped
With it comes tons of invalid emails, spam traps, catch-alls

Skip the whole process and get an Instagram email list instantly?

There are data scraping companies that have been crawling Instagram for years. They’ve become so advanced that now they offer additional targeting options that you otherwise wouldn’t get.

Some of them are

Gender
Interest
Country/city
Age

These are advanced targeting options that you can’t get from scraping your competitors’ followers or a hashtag relevant to your brand. But they are crucial in a marketing campaign.

One provider that’s separating from the bunch is Influencers Club that offers all of these options + the data you get is validated i.e. safe to use. You can check out their pricing here.

Codes for Instagram email scraping (Python)

All code samples are from this GitHub repository.

After you’re logged in with an Instagram scraping account from a safe proxy you can start scraping emails.

The only mobile endpoints (API) you need are the ones below and the one for email addresses is

/api/v1/users/{{user_id}}/info/


User.public_email
Email address

user.username
The Username

user.is_private
If this is a private account

user.full_name
User’s full name

user.profile_pic_url
User’s profile photo URL

user.biography
User’s bio

user.external_url
User’s website

user.follower_count
Follower count

user.following_count
Following count

user.media_count
Number of posts

The table shows the data points you can get with this code.

Create Instagram Phone Number Scraper Tool

Scraping phone numbers from Instagram uses the same API call as the one for email addresses. Of course, you can only find an Instagram phone number if the user has agreed to share it publicly.

/api/v1/users/{{user_id}}/info/

According to my stats, only 10-15% of all Instagram users have their personal number publicly available which is close to 200 million users.

Final Thoughts on Instagram Scraping

In very few cases it’s actually ROI positive to build an Instagram scraping tool. It takes a lot of time and money to gather data, just to find out that the data is not really actionable.

I’m saying that because having Instagram data like email addresses will definitely ruin your cold email campaign because of high bounce rate.

Or if you plan on using the list in Facebook ads as custom audiences you can trigger their algorithm as you import a lot of invalid emails.

So maybe most of you should just visit one of these Instagram scraping companies that will make all of this safe and instant.

I recommend you check out Influencers Club.

How To Build Your Own Instagram Email Scraper

Braydon Buckner — Tue, 02 Feb 2021 07:09:23 +0000

Important note: Please be advised that automatically scraping emails from Instagram is against their terms of service.

Here's how you can build your own Instagram email scraper and get a lot of other data points like phone numbers, bio text, images, and more.

1. Access the unofficial Instagram API

To access the unofficial Instagram API you need to use mobile endpoints with Python or PHP.

You can take the code sample from this Github public repos.

def login(self, force=False): """ Authenticate this API instance. If already logged in (and not later logged out) does nothing (unless forced). :param force: if true, will attempt to log in even if already logged in. :return: dictionary of responses. """ if not self._isloggedin or force: self._session = requests.Session() # if you need proxy make somethin://proxyip:proxyport"} full_response = self._sendrequest( 'si/fetch_headers/?challenge_type=signup&guid=' + self.generate_uuid(False), login=True) data = { 'phone_id': self.generate_uuid(True), '_csrftoken': full_response.cookies['csrftoken'], 'username': self._username, 'guid': self._uuid, 'device_id': self._deviceid, 'password': self._password, 'login_attempt_count': '0'} try: full_response = self._sendrequest( 'accounts/login/', post=self._generatesignature(json.dumps(data)), login=True) except InstagramAPIBase._2FA_Required as exception: # In order to login, need to provide the second factor (i.e. SMS code or backup code). # Use call-back to get this string. if not self._two_factor_callback: raise AuthenticationError("This account requires support for Two-Factor Authentication") two_factor_info = exception.two_factor_info = exception.two_factor_info verification_string = self._two_factor_callback(two_factor_info) data = { 'verification_code': verification_string, 'two_factor_identifier': g like this: # self.s.proxies = {"https": "httptwo_factor_info['two_factor_identifier'], '_csrftoken': full_response.cookies['csrftoken'], 'username': self._username, 'device_id': self._deviceid, 'password': self._password, } full_response = self._sendrequest( 'accounts/two_factor_login/', post=self._generatesignature(json.dumps(data)), login=True) self._isloggedin = True decoded_text = json.loads(full_response.text) self._loggedinuserid = decoded_text["logged_in_user"]["pk"] self._ranktoken = "%s_%s" % (self._loggedinuserid, self._uuid) self._csrftoken = full_response.cookies["csrftoken"] return decoded_text

Use a proxy from the location you're already at and make sure to complete any captchas if necessary. Then let the profile rest for couple of days before the scraping because Zucker is now watching you.

2. Get a lot of Instagram profiles

Two main things to remember here - is to always use aged Instagram profiles that are phone-validated otherwise they'll get banned immediately.

The second thing is to never use your personal profiles.

You can buy Instagram accounts from Facebook groups, DM's on Instagram or online stores.

3. Proxies

Don't stimulate too many IPs because logging in more than 5 accounts on the same IP a huge no-no.

Same as with Instagram accounts, the issue occurs with proxies. Zucker can detect proxy providers and before you find a good one you're in deep trouble.

To check whether the provider is not on the radar use this website and simply paste your proxies IP.

Building an Instagram email scraper

Here's a Github Repo for all the code samples you need for creating your own Instagram scraper.

Once you're logged with an IG profile from a proxy, getting the data is "easy enough".

You only need the API endpoints.

And the one for getting emails is:
/api/v1/users/{{user_id}}/info/

User.public_email Email address user.username The Username user.is_private If this is a private account user.full_name User’s full name user.profile_pic_url User’s profile photo URL user.biography User’s bio user.external_url User’s website user.follower_count Follower count user.following_count Following count user.media_count Number of posts

For more information check Influencers Club.