Using Twitter's API to Gather Tweet Stats (and Follower Data) in Python

In this little tutorial, we will learn how to use Twitter's API to download statistics about your tweets, like number of impressions, profile clicks etc. Also, we download a list of your followers, which would also allow you to track your unfollowers.

Getting Started

Before we begin, we need an API key for the Twitter API. You can get one by signing up here.

Once you have your Twitter Developer Account, head over to the Dashboard to create a new App.

Click on your newly created app and head to the "Keys and tokens" tab.

Ensure that you generate Authentication tokens with "Read, Write and Direct Messages Permissions" for your Twitter user.

In general, we need two pairs of key and secret to make use of our app:

1. API Key and Secret: This tells Twitter that it's your App which makes a request.
2. Access Token and Secret: Since apps can be used by multiple users, the access token and secret authenticates your user to Twitter.

Please store these secrets somewhere as you have no chance to display them again on the Twitter Dashboard. If you lose them, you have to regenerate them.

Installing the TwitterAPI PyPI package

Since the Twitter API is a well-documented REST-API, we could build all our queries to it ourselves. However, there are Python packages that allow us to access the Twitter API in a much more comfortable way. The most prominent packages for that purpose are tweepy and TwitterAPI. For this guide, we use the latter.

We install it as always by using pip:

pip install TwitterAPI

Accessing the API

Getting Tweet Stats

Now, we have everything to get started. Let's create a file stats.py and first import the TwitterAPI classes. We also create a client object by providing our credentials.

from TwitterAPI import TwitterAPI, TwitterPager

consumer_key        = "<YOUR API KEY>"
consumer_secret     = "<YOUR API KEY SECRET>"
access_token        = "<YOUR ACCESS TOKEN>"
access_token_secret = "<YOUR ACCESS TOKEN SECRET>"

client = TwitterAPI(consumer_key, consumer_secret, access_token, access_token_secret, api_version="2")

The first request to the API is for retrieving our own user_id, which is a numeric value rather than your self-chosen username. We can get that by calling the /users/me endpoint like so:

my_user = client.request(f'users/:me')
USER_ID = my_user.json()['data']['id']

Now, we can get stats for our tweets:

params = {
        "max_results": 100,
        "tweet.fields": "created_at,public_metrics,non_public_metrics,in_reply_to_user_id",
    }
r = client.request(f"users/:{USER_ID}/tweets", params)

There are two things to note here:

1. the params dictionary contains a value for max_results set to 100. This is the maximum Twitter allows for one API request.
2. To access the statistic fields, we have to instruct Twitter to attach these values to the API response. We do that by requesting a comma-separated list of fields for the tweet (tweet.fields). If you have paid ads running on Twitter's platform, you can also include the values organic_metrics,promoted_metrics to the tweet.fields params. However, these fields are available on promoted tweets only so that non-promoted tweets won't appear in your result when you request organic_metrics,promoted_metrics. So, you should make two API requests, one for promoted and one for non-promoted tweets.

Getting Follower Data

Just like other social networks, Twitter doesn't really seem to want you to get information about your followers other than their sheer number. In particular, Twitter is reluctant to give out information about who has unfollowed you. The following API accesses are therefore more strictly limited, and you should take care not to make too many requests.

That being said, let's get our list of followers:

params = {
        "max_results": 1000,
        "user.fields": "id,name,username,created_at,description,profile_image_url,public_metrics,url,verified",
    }
followers = client.request(f"users/:{USER_ID}/followers", params)

Even though Twitter is a bit reluctant to access the follower endpoint, we can still increase the number of users returned to 1000. That is good news.

Paging results

However, what do we do if you're lucky enough to have more than 1000 followers on Twitter? The TwitterAPI package gets us covered and provides a TwitterPager object for that purpose. If a request produces more results than fit into one response, the further results are hidden in further responses that can be accessed with a next_token. This means that the response is "accompanied" by a link to be followed for the next results. Basically, it works like pages in your favourite online shop: "Page 1 of 453", you get the idea. That's why the tool we need is called a Pager. Here is how to use it:

params = {
        "max_results": 200,
        "user.fields": "id,name,username,created_at,description,profile_image_url,public_metrics,url,verified",
    }
pager = TwitterPager(client, f"users/:{USER_ID}/followers", params)
followers = list(pager.get_iterator())

Getting Unfollowers

How to get the users who unfollowed you on Twitter now?

There is no particular API endpoint for getting unfollowers. To circumvent this limitation, we store the result of our follower query on our disk. When we make a subsequent request, we can compare the old and new results. User IDs not present in the old result but present in the new result are new followers, and User IDs present in the old result but not present in the new result anymore are therefore unfollowers.

If you query the followers, let's say, every two hours, you will know who unfollowed you in the meantime.

The logic looks like this:

follower_ids = [follower['id'] for follower in followers]

old_followers = json.load(open('last_run.json'))

old_follower_ids = [follower['id'] for follower in old_followers]

new_followers = [follower_id for follower_id in follower_ids if follower_id not in old_follower_ids]
unfollowers = [follower_id for follower_id in old_follower_ids if follower_id not in follower_ids]

The Whole Package

The complete source code for our stats.py file is on GitHub.

The TwitterStats PyPI package

For added convenience, I created a package on PyPI, which combines all the ideas of this article on one class for fetching all the stats. You can install it as always with pip:

pip install TwitterStats

and use it in this simple way:

from twitterstats import Fetcher

consumer_key        = "<YOUR API KEY>"
consumer_secret     = "<YOUR API KEY SECRET>"
access_token        = "<YOUR ACCESS TOKEN>"
access_token_secret = "<YOUR ACCESS TOKEN SECRET>"

fetcher = Fetcher(consumer_key, consumer_secret, access_token, access_token_secret)

promoted_tweets = fetcher.get_tweets(promoted=True)
unpromoted_tweets = fetcher.get_tweets(promoted=False)
followers = fetcher.get_followers()

tl;dr

The TwitterAPI package makes it easy to download your tweet and follower/unfollower stats from Twitter. The TwitterStats package makes it even more convenient to access these data.

If you enjoyed this article, consider following me on Twitter where I regularly share tips on Python, SQL and Cloud Computing.

twttr.conversion.trackPid('q5o0o', { tw_sale_amount: 0, tw_order_quantity: 0 });

Comments

[Andy Piper]

Related to the "reluctance" to provide access to who has unfollowed - this is really a reason to use the API. I think that providing that information from within the system would require a ton more historical state management with every individual user. "diff'ing" the lists at points in time is an OK (not ideal) way to figure this out; there's an Account Activity API (webhooks, v1.1) which does provide unfollow events when they happen, but holding specific point-in-time snapshots of followings all the time could get really complicated, I think. Note that this is me thinking it through, rather than having direct knowledge of how it works.

Thanks for sharing the TwitterStats package, I love to see code built on the API!

[Bas Steins]

Thank you, Andy for these insights. Indeed, I was not aware of the Account Activity API, but I will have a look. Thanks for the hint!

As for managing the "diff'ing" of follower snapshots: In the next article on this series I will show how this can be achieved with Azure Functions quite easily.

[Andy Piper]

I cannot remember (without checking the docs in detail) whether it lets an account see when it unfollows, or is unfollowed by, another account, to be honest - it is webhook-based so quite different to what you are currently doing (and you will still want a list of followers, I guess). Will keep reading your updates!

[Bas Steins]

According to the docs (developer.twitter.com/en/docs/twit...) it is possible to get notified (among others) of:

• Follows (by user or of user)
• Unfollows (by user)

If I read that correctly, you wouldn't get notified of unfollows.

Since other social media platforms handle this in the same way, the reason might be more in the space of community management rather than it being a technical decision. With not exposing unfollow events, I guess platforms disincentivise strategic follows and a tit for tat like behaviour.