DEV Community: Suhem Parack

How to search for Tweets about various 'Topics' using the Twitter API v2

Suhem Parack — Tue, 18 Oct 2022 17:09:35 +0000

The Twitter API v2 allows developers and researchers to search for Tweets programmatically using the recent search endpoint (available for all developers) and full-archive search endpoint (available in the academic researcher product track). In order to search for Tweets using these endpoints, you have to specify your request in terms of a search query. In your search query you can use specify that keywords that you want the Tweets returned for and also use various operators such as from:, has:, is: etc. The Twitter API v2 supports a new search operator: context:. This operators let you search for Tweets on certain topics and entities that are available in the Twitter API as part of Tweet Annotations.

So for example, if you want to search for all Tweets that are about the LA Lakers, instead of building a search query like 'LA Lakers OR #LakeShow' etc., your can be 'context:12.706083845846597632' and it will give you all the Tweets about the Los Angeles Lakers. The question is, how do you know what the value is for the Los Angeles Lakers entity, in order to use the context: operator? In this guide, I will show you how to get the ID value for a topic to use with the context: operator to search for Tweets about a topic.

Step 1: Find a Tweet that represents a topic that you want to search for Tweets on

To do so simply, search for that topic using a keyword of your choice on Twitter. In this example, I want all Tweets about the Los Angeles Lakers so I simply searched for a Tweet with the hashtag #LakeShow on Twitter. I got a Tweet with the following URL: https://twitter.com/Lakers/status/1582026396917866497.

Step 2: Paste this link in the Tweet entity extractor tool

I put together a sample app that gives you the key & value combination to use with the context: operator. To use this app, open your browser and go to: https://tweet-entity-extractor.glitch.me/. In this app, pase the Tweet URL from step 1, and click on Get context ID! This will give you a response like the one shown below:

[
  {
    "context": "3.10000607734",
    "entity_name": "NBA Basketball"
  },
  {
    "context": "11.706083902411055104",
    "entity_name": "Basketball"
  },
  {
    "context": "12.706083845846597632",
    "entity_name": "Los Angeles Lakers"
  },
  {
    "context": "26.706083889454813185",
    "entity_name": "NBA"
  },
  {
    "context": "46.1557697289971322880",
    "entity_name": "Sports & Fitness Business"
  }
]

From this response, you can see that the ID to use for the Los Angeles Lakers topic is 12.706083845846597632.

Note: Some Topics may have multiple key.value pairs so you can chose one of those.

Step 3: Use the ID value from the previous step in your search query & get Tweets for a Topic

Now that you have the ID for the topic of your choice, simply use it in your search query. Below is a code snippet that shows how you can use context: operator in Python to get Tweets on a topic (Los Angeles Lakers in this case) using the key value pair from the previous step. (The code below uses the Tweepy package in Python, so make sure you have it installed, using the steps shown here)

import tweepy

client = tweepy.Client(bearer_token='REPLACE_ME')

response = client.search_recent_tweets(query='context:12.706083845846597632')

for tweet in response.data:
    print(tweet.id)

Note: Not all Tweets are annotated. You can find more information and FAQs on Tweet annotations here.

I hope you are now able to get Tweets based on topics of your interest. The Tweet Annotations support various topics, so feel free to explore the available topics using the steps show above. If you have any questions, feel free to DM me on Twitter.

Whats new with the Twitter API v2 in 2022

Suhem Parack — Tue, 11 Oct 2022 16:56:32 +0000

When the Twitter API v2 had just gone public in August 2020, I had made a video (available here) with the technical overview of the new features and functionality available in v2. Since then, there have been quite a few good updates available for developers and researchers in the Twitter API v2. In this post, I will share some of my favorites features and functionality that are available in the Twitter API v2, and why I think developers and researchers should give it a try!

Full-archive search

My favorite feature in the Twitter API v2 is the support for full-archive search. This feature (available only in the academic research product track) lets researchers search for Tweets from the full-archive of public Tweets for free. In v1.1, developers and researchers could only search for Tweets from the last 7 days for free. To use the full-archive search, they had to use the paid premium API. So the full-archive search in v2 enables researchers to study topics from the past for their research studies for free. Learn more about full-archive search here.

Tweet Counts

When you use the search endpoints, you get the Tweets (ID, text etc.) that match a search query. However, in some cases (e.g. for building visualizations) you may just want to know the volume of Tweets on Twitter for a certain topic for a given time period. The Twitter API v2 supports the Tweet counts endpoint that gives you the volume of Tweets for a search query so that you can understand trends and patterns for a topic and build graphs and visualizations for it. The response from these endpoints for a search query looks something like this:

[
      {
        "end": "2021-07-02T00:00:00.000Z",
        "start": "2021-07-01T22:57:15.000Z",
        "tweet_count": 1
      },
      {
        "end": "2021-07-03T00:00:00.000Z",
        "start": "2021-07-02T00:00:00.000Z",
        "tweet_count": 36
      },
      {
        "end": "2021-07-04T00:00:00.000Z",
        "start": "2021-07-03T00:00:00.000Z",
        "tweet_count": 9
      },
      {
        "end": "2021-07-05T00:00:00.000Z",
        "start": "2021-07-04T00:00:00.000Z",
        "tweet_count": 4
      }
    ]

As you can see, instead of the Tweets, this endpoint returns the Tweet volume for a time period. This guide showcases how you can use this Tweet counts endpoint to build visualizations in Python.

Ability to get the full conversation with Quote Tweets and Conversation Threads

The Twitter API v2 supports the Quote Tweets endpoint that gives you all the Quote Tweets for a Tweet. Additionaly, the search endpoints support the conversation_id operator that lets you easily get all the replies to a Tweet. Thus, using these two endpoints, you can get and understand the entire conversation around a Tweet. The Twitter API v1.1 did not support a Quote Tweets endpoint or a conversation_id search operator, thus getting the entire conversation required some custom coding. This guide explains how you can lookup and understand the entire conversation around a Tweet in Python.

Support for Tweet Edits metadata

Since October 2022, the Twitter API v2 supports metadata for edited Tweets. Using the Twitter API v2, you can get a list of revisions to a Tweet as well as information such as how many edits remaining for a Tweet, how much more time left for a Tweet to be edited etc. as shown below:

{
  "edits_remaining": "5",
  "is_edit_eligible": true,
  "editable_until": "2022-09-30T18:30:01.000Z"
}

This guide showcases how to analyze edited Tweets with the Twitter API v2 in Python.

Batch compliance solution

The Twitter Developer API policy requires developers and researchers to keep their Tweet & User datasets in compliance. The Twitter API v2 supports a batch compliance solution that lets developers and researchers upload millions of Tweet or User IDs in bulk and get the current status of those Tweet and User IDs i.e. which of these Tweets are now deleted, suspended etc. or which of these user accounts went from public to protected, so that you can take action on your side. The response from this endpoint looks somethign like this:

{
  "id": "1265324480517361664",
  "action": "delete",
  "created_at": "2019-10-29T17:02:47.000Z",
  "redacted_at": "2020-07-29T17:02:47.000Z",
  "reason": "deleted"
}

This tells us that the Tweet with ID 1265324480517361664 was deleted. This guide showcases how to work with the batch compliance solution in Python.

Context Annotations

The Twitter API v2 supports a cool feature called context annotations. Context annotations provide contextual information about a Tweet and named entity recognition for entities present in a Tweet. There are two components to context annotations:

Domain: that is the topic about which a Tweet is
Entity: that is any named entity present in a Tweet

So for example, a Tweet about the Los Angeles Lakers may contain context annotations like:

[
  {
    "context": "12.706083845846597632",
    "entity_name": "Los Angeles Lakers"
  },
  {
    "context": "26.706083889454813185",
    "entity_name": "NBA"
  },
  {
    "context": "131.706083902411055104",
    "entity_name": "Basketball"
  }
]

This information tells us that this Tweet is contextually about Basketball, NBA, Lakers etc. Learn more about context annotations here.

Bookmarks

Finally, the Twitter API v2 supports the ability for developers to work with Bookmarks for an authenticated user. You can get, create or delete Bookmarks for an authenticated user. This functionality does not exist in the Twitter API v1.1. Check out this app that showcases this functionality and how a user can export their Twitter bookmarks to Notion.

I hope that some of these features sound exciting to you and that you will give the Twitter API v2 a try! If you have any questions or feedback, feel free to reach out to me on Twitter

How to analyze edited Tweets with the Twitter API v2 using Python

Suhem Parack — Tue, 04 Oct 2022 17:32:28 +0000

As announced in this blog post, the Twitter API v2 supports the ability to get metadata about edited Tweets. In this short guide, I will showcase how developers and researchers can get information about edited Tweets from the Twitter API v2 in Python using Tweepy.

Prerequisite

In order to use the Twitter API v2, you need to apply for a Twitter developer account. Once you have an approved Twitter developer account, follow the instruction here to obtain your BEARER TOKEN that you will use to connect to the Twitter API v2 in you code in Python. We will be using the Tweepy package in Python to get Tweets from the Twitter API v2, so you will need to have Python as well as the Tweepy package installed on your machine. Instructions on installing the Tweepy package can be found in this tutorial.

Some fundamental background on Edit Tweets

Currently, some Tweets on Twitter Blue are editable. These Tweets are editable for 30 minutes and up to 5 times. Each time a Tweet is edited, a new Tweet ID is created for the edited Tweet.

There are three components to a Tweet’s edit history:

By default, the Tweet response from the Twitter API v2 will contain an array of Tweet IDs that are part of a Tweet’s edit history. This information is specified by the edit_history_tweet_ids, which is a default field in the Tweet response. This array will contain at least one ID, the ID of the original, unedited Tweet. When there is only one ID that means the Tweet has no edit history.
You can get information such as whether a Tweet was editable at the time it was created, how much time (if any) is remaining for a Tweet to be edited, and how many edits remain. This information is specified by requesting the edit_controls field in the tweet.fields.
```
{
"edits_remaining": "5",
"is_edit_eligible": true,
"editable_until": "2022-09-30T18:30:01.000Z"
}
```
Finally, you can get the Tweet objects for each Tweet in a Tweet’s edit history, by specifying the edit_history_tweet_ids using the expansion parameter

Also, while most Tweets are eligible for edit, Tweets such as those with polls, nullcast, promoted etc. are not eligible for edit. The is_edit_eligible field in edit_controls will indicate this. Full list of editable Tweet categories can be found here.

Getting Edit Tweets metadata with Tweepy in Python

In the example below, the Tweet with ID 1575590109000323073 is a Tweet that has been edited. By default, the edit_history_tweet_ids will be returned containing a list of Tweet IDs associated with the revised version(s) of this Tweet. To get information such as how many more edits remaining, how much more time left to make edits to this Tweet, we include the edit_controls field. We are also including the created_at field to get the timestamp for when the Tweet was created. We simply print the Tweet ID, text, the creation time, edit controls related information and the list of Tweet IDs associated with this Tweet (in terms of revisions) to the console.

import tweepy

# Replace bearer token below
client = tweepy.Client(bearer_token='REPLACE_ME')

response = client.get_tweet(id='1575590109000323073', tweet_fields=['edit_controls,created_at'])

tweet = response.data

print(tweet.id)
print(tweet.text)
print(tweet.created_at)
print(tweet.edit_controls)
print(tweet.edit_history_tweet_ids)

The response will look something like this:

1575590109000323073
hello

this is a test to make sure the edit button works https://t.co/NBkYlCIPrA
2022-09-29 20:55:29+00:00
{'edits_remaining': '4', 'is_edit_eligible': True, 'editable_until': '2022-09-29T21:25:29.000Z'}
['1575590109000323073', '1575590534529556480']

Because this Tweet is not a poll, nullcast, promoted Tweet etc., is_edit_eligible is marked as true.

If you want the Tweet objects for each Tweet in a Tweet’s edit history, you will have to use the edit_history_tweet_ids expansion as shown below:

import tweepy

# Replace bearer token below
client = tweepy.Client(bearer_token='REPLACE_ME')

response = client.get_tweet(id='1575590109000323073', tweet_fields=['edit_controls,created_at'],
                            expansions=['edit_history_tweet_ids'])

for tweet in response.includes['tweets']:
    print(tweet.id)
    print(tweet.text)
    print(tweet.created_at)

The response will look something like this:

hello

this is a test to make sure the edit button works https://t.co/NBkYlCIPrA
2022-09-29 20:55:29+00:00
1575590534529556480
hello

this is a test to make sure the edit button works, we’ll let you know how it goes https://t.co/XbZmX572Xc
2022-09-29 20:57:11+00:00

Checking if a Tweet can be edited any longer

In order to check if the Tweet is editable any longer, we can check if:

edits_remaining is greater than 0 AND
editable_until is in the future from the current time (this will also give us how many more minutes left until a Tweet can no longer be edited)

The example below shows how you can do this with Tweepy:

import tweepy
from datetime import datetime

# Replace bearer token below
client = tweepy.Client(bearer_token='REPLACE_ME')

response = client.get_tweet(id='1575590109000323073', tweet_fields=['edit_controls,created_at'],
                            expansions=['edit_history_tweet_ids'])

tweet = response.data

edits_remaining = tweet.edit_controls['edits_remaining']

editable_until = tweet.edit_controls['editable_until']

time_now = datetime.now()

formatted_time_now = time_now.strftime("%Y-%m-%dT%H:%M:%S.%f")[:-3]

if edits_remaining == 0:
    print("This Tweet can no longer be edited")
else:
    if editable_until[:-1] < formatted_time_now:
        print("This Tweet can no longer be edited")
    else:
        print("This Tweet can be edited until {}".format(editable_until))

How to check if a Tweet with a given ID is the most recent version of a Tweet

If a Tweet has edits, the edit_history_tweet_ids will have more than one Tweet ID. So, we will simply check if the length of this array is equal to 1. If it is, that means there are no edits for this Tweet. If the length of this array is greater than 1, then the last Tweet ID in this array is the latest 'edited' Tweet.

import tweepy

# Replace bearer token below
client = tweepy.Client(bearer_token='REPLACE_ME')

response = client.get_tweet(id='1575590109000323073')

tweet = response.data

edit_history_tweet_ids = tweet.edit_history_tweet_ids

if len(edit_history_tweet_ids) == 1:
    print("There are no edits to this Tweet")
else:
    print("The latest version of this Tweet is {}".format(edit_history_tweet_ids[-1]))

I hope this tutorial is helpful in studying and understanding edited Tweets. Feel free to reach out to me with any questions and feedback!

Getting COVID-19 Tweets using the Twitter API v2

Suhem Parack — Wed, 31 Aug 2022 16:33:17 +0000

In April 2020, Twitter launched the COVID-19 stream endpoint that enabled qualified developers & researchers to study the public conversation around COVID in real-time. This data can be obtained using the filtered stream endpoint in the Twitter API v2. In this tutorial, I will show you how to get COVID-19 related Tweets using the filtered stream endpoint in Python.

Note: In order to use the Twitter API v2, you need to apply for a developer account. To do so, simply visit t.co/signup and sign up for access. Once you have access to the Twitter API v2, you can connect an app to your project and get your bearer token that you will need to connect to the Twitter API v2 in order to get Tweets.

The filtered stream endpoint lets you filter for Tweets in real time based on the filters you set. In our case, we want all Tweets about the COVID-19 topic. For this, we will use the COVID-19 Tweet Annotation. Tweet Annotations provide contextual information about Tweets. So, if we want to filter for all Tweets about COVID-19 instead of using a filter like COVID OR COVID-19 OR COVID19 etc. we can use the context operator in our filter and use the value of the COVID-19 Tweet Annotation, which is 123.1220701888179359745. Thus, our search query or filter will be context:123.1220701888179359745. This will gives us all Tweets about COVID-19. An example of search terms that power this annotation can be found here.

In order work with the Twitter API v2 in Python, we will use the Tweepy package. To install it, you can running the following in your terminal:

pip3 install tweepy

Once this package is installed, you can run the following script. Replace the BEARER_TOKEN with your own token.

import tweepy

class IDPrinter(tweepy.StreamingClient):

    # This will print the Tweet ID and Tweet text for each Tweet
    def on_tweet(self, tweet):
        print(tweet.id)
        print(tweet.text)

# Replace with your own bearer token below
printer = IDPrinter('BEARER_TOKEN')
# This is where we set our filter rule
printer.add_rules(tweepy.StreamRule("context:123.1220701888179359745"))
printer.filter()

To learn about the common questions about Tweet Annotations, check out our FAQ section.

If you have any additional questions, feel free to DM me on Twitter

Analyzing images using the Twitter API v2 and AWS

Suhem Parack — Tue, 12 Jul 2022 15:17:25 +0000

The Twitter API v2 lets you get Tweets with images programmatically. Many times, researchers want to study these images to better understand the conversation on a topic. An example use-case of researchers analyzing images is user's studying memes on Twitter. For this, they may want to identify objects in the image or extract text from these images (memes). In this tutorial (meant for academic researchers), I will showcase how you can study images in Twitter, using the Twitter API v2 and Amazon Rekognition.

Note: Twitter does not own or maintain Amazon Rekognition. Amazon Rekognition is developed and owned by Amazon Web Services and I have used it for education purposes.

Prerequisites

In order to use the Twitter API v2, you need to apply for a Twitter developer account. Once you have an approved Twitter developer account, follow the instruction here to obtain your BEARER TOKEN that you will use to connect to the Twitter API v2 in you code in Python. We will be using the Tweepy package in Python to get images from the Twitter API v2, so you will need to have Python as well as the Tweepy package installed on your machine. Instructions on installing the Tweepy package can be found in this tutorial.

You will also need an Amazon Web Services (AWS) account in order to use the Amazon Rekognition. Instructions on setting up your AWS credentials locally can be found here. Please set it up in order to get the sample code working.

Getting a Tweet's image url with the Twitter API v2

In order to use the Twitter API v2 in Python, you need to first import Tweepy and initialize the Tweepy client, which makes the API calls for you. You can initialize it as:

client = tweepy.Client('BEARER_TOKEN')

Next, you need to specify the Tweets that you are looking for. If you want to get Tweets from a certain account, which contain images, you can specify this condition in the search query. has:images tells the Twitter API that you want Tweets that contain images in them.

# Replace with your own search query
query = 'from:twitterdev -is:retweet has:images'

Next, we will call the search_recent_tweets method, which searches for Tweets from the last 7 days. Because we want the image information, we do two additional things:

we set the expansions to attachments.media_keys
we set the media_fields to url

This will give us the URL for images in the Tweets. In this example, I am only getting 10 Tweets from my search result. You can get upto 100 Tweets with the search_recent_tweets method by setting the max_results to 100. If you want more than 100 Tweets, you will need to use pagination. Check out this example that shows how you can use pagination.

# This method gives us Tweets from the last 7 days with images
tweets = client.search_recent_tweets(query=query,
                                    media_fields=['url'],
                                    expansions='attachments.media_keys',
                                    max_results=10)

If your search query returns Tweets with images, the image information will be available in the tweets.includes under media. So, we check if this information is present. If it is, we create a dictionary with media key and the media information. In the snippet below, we print the image URL to the console.

# Check to see if this search query gives any Tweets with images
if 'media' in tweets.includes:

    # Get list of media from the includes object
    media = {m["media_key"]: m for m in tweets.includes['media']}

    for tweet in tweets.data:
        attachments = tweet.data['attachments']
        media_keys = attachments['media_keys']
        if media[media_keys[0]].url:
            image_url = media[media_keys[0]].url
            print(image_url)
else:
    print('No images found for this search query')

Analyzing images with Amazon Rekognition

In order to analyze images obtained from the Twitter API v2, we need to import boto3 library and initialize the rekognition client. This will allow us to call the different methods available such as detect_text, detect_moderation_labels etc.

client = boto3.client('rekognition')

Next, we need to pass the rekognition client the image data. So we make a GET call to the Tweet image URL and obtain the response, and pass it to our methods as shown below:

def get_image(url):
    response = requests.get(url)
    return response.content

Text extraction

To extract text from images, we will call the detect_text method. If the response contains TextDetections, we will print those to the console.

def detect_text(photo):
    client = boto3.client('rekognition')
    response = client.detect_text(Image={'Bytes': photo})
    text_detections = response['TextDetections']
    for text in text_detections:
        print(text['DetectedText'])

In this example, if the image was:

The response from this method will look like:

Using the Twitter API
tpRespor
(0)
decY0c
v2 with GitHub Copilot
pEntd
Knull
ye Lom
Join us on twitch.tv/twitterdev
14th July 2022 @ 2PM ET / 11AM PT

Detecting objects and scenes

To detect labels from images, we will call the detect_labels method. If the response contains Labels, we will print those to the console.

def detect_labels(photo):
    client = boto3.client('rekognition')
    response = client.detect_labels(Image={'Bytes': photo})
    for label in response['Labels']:
        print("Label: " + label['Name'])

For each Tweet image, the output of this method will look something like this:

Label: Person
Label: Human
Label: Car
Label: Automobile
Label: Vehicle
Label: Transportation

Image moderation

You can do image moderation with Rekognition. To detect moderation lables for images, we will call the detect_moderation_labels method. If the response contains ModerationLabels, we will print those to the console.

def get_moderation_labels(photo):
    client = boto3.client('rekognition')
    response = client.detect_moderation_labels(Image={'Bytes': photo})
    for label in response['ModerationLabels']:
        print(label['Name'])

For each Tweet image, the output of this method will look something like this:

Suggestive
Revealing Clothes

Celebrity rekognition

You can also detect celebrities in images using Rekognition. To detect celebrities in images, we will call the recognize_celebrities method. If the response contains CelebrityFaces, we will print the names and IDs to the console.

def detect_celebrity(photo):
    client = boto3.client('rekognition')
    response = client.recognize_celebrities(Image={'Bytes': photo})
    for celebrity in response['CelebrityFaces']:
        print('Name: ' + celebrity['Name'])
        print('Id: ' + celebrity['Id'])

For each Tweet image, the output of this method will look something like this:

Name: Joe Biden
Id: 3Zw7fr

Putting it all together

Below is the complete code snippet that you can run in Python. The example below uses the detect_labels method but you can replace it with the other methods that we discussed above.

import boto3
import requests
import tweepy


def get_image(url):
    response = requests.get(url)
    return response.content

# Feel free to replace this method with any of the other methods
# shared above 
def detect_labels(photo):
    client = boto3.client('rekognition')
    response = client.detect_labels(Image={'Bytes': photo})
    for label in response['Labels']:
        print("Label: " + label['Name'])


def main():
    client = tweepy.Client('BEARER_TOKEN')

    # Replace with your own search query
    query = 'from:twitterdev -is:retweet has:images'

    tweets = client.search_recent_tweets(query=query,
                                         media_fields=['url'],
                                         expansions='attachments.media_keys',
                                         max_results=10)

    if 'media' in tweets.includes:
        # Get list of media from the includes object
        media = {m["media_key"]: m for m in tweets.includes['media']}

        for tweet in tweets.data:
            attachments = tweet.data['attachments']
            media_keys = attachments['media_keys']
            if media[media_keys[0]].url:
                image_url = media[media_keys[0]].url
                print(image_url)
                photo = get_image(image_url)
                print(detect_labels(photo))
    else:
        print('No images found for this search query')


if __name__ == "__main__":
    main()

That is it for the tutorial. If you have any questions or feedback about this guide, please feel free to reach out to me on Twitter!

Detecting toxicity in Tweets using the Twitter API v2 and the Perspective API

Suhem Parack — Fri, 01 Jul 2022 06:18:24 +0000

The Perspective API allows developers and researchers to study and analyze text to detect toxicity in it. In this tutorial, I will show you how you can detect toxicity in a Twitter user's mentions, using the Perspective API and the Twitter API v2 in Python.

Note: This is just sample code. Twitter does not own the Perspective API. It is a product of a collaborative research effort by Jigsaw and Google’s Counter Abuse Technology team. According to their website, they define 'toxicity' as a rude, disrespectful, or unreasonable comment that is likely to make someone leave a discussion.

Prerequisite

Because this tutorial uses code in Python, make sure you have Python installed on your machine. In order to get a Twitter user's mentions with the Twitter API v2, you need a Twitter developer account and your bearer token to connect to the Twitter API. Similarly, to use the Perspective API to analyze the a Twitter user's mentions, you need to apply for access to Perspective API and obtain your API key.

Applying for a Twitter developer account and getting your bearer token for the Twitter API v2

To sign up for access to the Twitter API v2, visit this page and follow instructions to sign up for essential access. You will be asked to verify your email as part of this process. Additionally, you need to have phone number registered with your account in order to use the API.

After you have created your account, you will be asked some basic questions:

Review and agree to the developer agreement and policy

Next, you will be asked to verify your email

You should received an email that looks like this:

Once you click confirm, you will be directed to the developer portal. You can create an app by giving it a name

Once you click Get Keys, you will be able to see your keys and bearer token. Please keep those in a safe place, as you will be using those to connect to the Twitter API v2.

Applying for access to the Perspective API and getting your API key

For instructions on signing up for a Perspective API developer account and obtaining your API key, check out the Perspective API developer documentation

Installing the Tweepy & googleapiclient packages in Python

To work with the Twitter API v2 in Python, we will use the Tweepy package. To install it, run the following:

pip3 install tweepy

To work with the Perspective API in Python, we will use the Google API Client. To install it, run the following:

pip3 install googleapiclient

Building the function for using the Perspective API

We have a method that will take the Tweet text as input and call the Perspective API with that Tweet text. Replace the API_KEY value in this function with your own API key from Perspective.

def analyze(text):

    # Replace with your own Perspective API key
    API_KEY = 'PERSPECTIVE_API_KEY'

    client = discovery.build(
        "commentanalyzer",
        "v1alpha1",
        developerKey=API_KEY,
        discoveryServiceUrl="https://commentanalyzer.googleapis.com/$discovery/rest?version=v1alpha1",
        static_discovery=False,
    )

    analyze_request = {
        'comment': {'text': text},
        'requestedAttributes': {
            'TOXICITY': {}
        }
    }

    return client.comments().analyze(body=analyze_request).execute()

Getting a user's mentions using the Twitter API v2

In order to get the user's Twitter mentions, you will have to use the get_users_mentions method and pass it the user_id. By default, you will get 10 mentions per API call, and you can get up to 100 mentions per API call. If you want additional mentions, you can specify the number in the limit parameter below.

    # Replace with the number of Tweets you'd like
    limit = 25
    # Replace your bearer token below
    client = tweepy.Client('TWITTER_BEARER_TOKEN')
    # Replace with the username of your choice
    user = client.get_user(username='twitterdev')
    user_id = user.data['id']
    for tweet in tweepy.Paginator(client.get_users_mentions, id=user_id, max_results=25).flatten(limit=limit):
        tweet_id = tweet['id']
        tweet_text = tweet['text']

Passing the Tweet text to the Perspective API for analysis

We pass the Tweet text to the analyze method. The response of the analyze method is a JSON that looks like this:

{
  "attributeScores": {
    "TOXICITY": {
      "spanScores": [
        {
          "begin": 0,
          "end": 99,
          "score": {
            "value": 0.010204159,
            "type": "PROBABILITY"
          }
        }
      ],
      "summaryScore": {
        "value": 0.010204159,
        "type": "PROBABILITY"
      }
    }
  },
  "languages": [
    "en"
  ],
  "detectedLanguages": [
    "en"
  ]
}

We call the analyze method in a try and except block and are interested in the summaryScore from this response. In the except block, we print the reason for the error. In most cases in this sample, the reason will be if a Tweet is in a language that the Perspective API does not support for detecting toxicity. For a list of supported languages check out the Perspective API documentation

try:
    response = analyze(tweet_text)
    if 'attributeScores' in response:
        if 'TOXICITY' in response['attributeScores']:
            toxicity = response['attributeScores']['TOXICITY']
            summary_score = toxicity['summaryScore']
            toxicity_str = "Toxicity score: {}".format(summary_score['value'])
            print(tweet_id)
            print(tweet_text)
            print(toxicity_str)
except HttpError as error:
    print(error.reason)

Putting it all together

Below is the complete working script for this code sample:

import tweepy
from googleapiclient.errors import HttpError
from googleapiclient import discovery


def analyze(text):
    # Replace with your own Perspective API key
    API_KEY = 'PERSPECTIVE_API_KEY'
    client = discovery.build(
        "commentanalyzer",
        "v1alpha1",
        developerKey=API_KEY,
        discoveryServiceUrl="https://commentanalyzer.googleapis.com/$discovery/rest?version=v1alpha1",
        static_discovery=False,
    )

    analyze_request = {
        'comment': {'text': text},
        'requestedAttributes': {
            'TOXICITY': {}
        }
    }

    return client.comments().analyze(body=analyze_request).execute()


def main():
    # Replace with the number of Tweets you'd like
    limit = 25
    # Replace your bearer token below
    client = tweepy.Client('TWITTER_BEARER_TOKEN')
    # Replace with the username of your choice
    user = client.get_user(username='twitterdev')
    user_id = user.data['id']
    for tweet in tweepy.Paginator(client.get_users_mentions, id=user_id, max_results=25).flatten(limit=limit):
        tweet_id = tweet['id']
        tweet_text = tweet['text']
        try:
            response = analyze(tweet_text)
            if 'attributeScores' in response:
                if 'TOXICITY' in response['attributeScores']:
                    toxicity = response['attributeScores']['TOXICITY']
                    summary_score = toxicity['summaryScore']
                    toxicity_str = "Toxicity score: {}".format(summary_score['value'])
                    print(tweet_id)
                    print(tweet_text)
                    print(toxicity_str)
        except HttpError as error:
            print(error.reason)


if __name__ == "__main__":
    main()

That is it for the tutorial. If you have any questions or feedback about this guide, please feel free to reach out to me on Twitter!

A guide to working with the Twitter API v2 in Java using twitter-api-java-sdk

Suhem Parack — Fri, 17 Jun 2022 17:54:00 +0000

twitter-api-java-sdk is Twitter's official SDK for Java that supports the Twitter API v2 and the academic research product track. In this guide, we will learn how to use the various functionality available in the Twitter API v2, using this twitter-api-java-sdk.

Prerequisite

In order to work with the Twitter API, you need to have a developer account and your API keys and tokens to connect to the API. Learn more on how to obtain these keys and tokens here.

In order to work with twitter-api-java-sdk, make sure you have Java installed on your machine. If you use Maven, add the following dependency:

<dependency>
  <groupId>com.twitter</groupId>
  <artifactId>twitter-api-java-sdk</artifactId>
  <version>2.0.0</version>
</dependency>

For all the examples in this section, you will first need to create an instance of the TwitterApi class and pass it your bearer token. Once you have the instance set up, you will be ready to start using the various functions in this library.

TwitterApi apiInstance = new TwitterApi();
TwitterCredentialsBearer credentials = new TwitterCredentialsBearer("BEARER_TOKEN");
apiInstance.setTwitterCredentials(credentials);

1. Searching for Tweets from the last 7 days

In order to search Tweets from the last 7 days, you can use the tweetsRecentSearch method. You will have to pass it a search query to specify the data that you are looking for. In the example below, we are searching for Tweets from the last 7 days that contain the term covid and we are excluding retweets using -is:retweet.

By default, in your response, you will only get the Tweet ID and Tweet text for each Tweet. If you need additional Tweet fields such as context_annotations, created_at (the time the tweet was created) etc., you can specifiy those fields using the tweetFields parameter, as shown in the example 2. Learn more about available fields here. By default, a request returns 10 Tweets. If you want more than 10 Tweets per request, you can specify that using the maxResults parameter. The maximum Tweets per request for this recent search endpoint is 100.

import java.util.*;

import com.twitter.clientlib.TwitterCredentialsBearer;
import com.twitter.clientlib.ApiException;
import com.twitter.clientlib.api.TwitterApi;
import com.twitter.clientlib.model.*;

public class TwitterApiExample {

    public static void main(String[] args) throws ApiException {

        TwitterApi apiInstance = new TwitterApi();
        TwitterCredentialsBearer credentials = new TwitterCredentialsBearer("BEARER_TOKEN");
        apiInstance.setTwitterCredentials(credentials);

        String query = "covid -is:retweet";

        int maxResults = 100;

        TweetSearchResponse result = apiInstance.tweets().tweetsRecentSearch(query, null, null, null, null, maxResults,
                null, null, null, null, null, null, null, null, null);

        if (result.getData() != null) {
            for (Tweet tweet : result.getData()) {
                System.out.println(tweet.getId());
                System.out.println(tweet.getText());
            }

        }

    }

}

2. Searching for Tweets from the full-archive of public Tweets

If you have access to the academic research product track, you can get Tweets older than 7 days i.e. from the full archive of publicly available Tweets. In order to get these Tweets, make sure to use the bearer token from an app connected to your academic project in the Twitter developer portal. You can use the tweetsFullarchiveSearch method and pass it the search query. As shown in the example below, you can also specify additional Tweet fields such as created_at, lang etc. that you'd like returned for your request to the Twitter API.

Note: Not all Tweets have context_annotations associated with them, which is why, in the sample below, we only print the context_annotations if those are available for a Tweet.

import java.util.*;

import com.twitter.clientlib.TwitterCredentialsBearer;
import com.twitter.clientlib.ApiException;
import com.twitter.clientlib.api.TwitterApi;
import com.twitter.clientlib.model.*;

public class TwitterApiExample {

    public static void main(String[] args) throws ApiException {

        TwitterApi apiInstance = new TwitterApi();
        TwitterCredentialsBearer credentials = new TwitterCredentialsBearer("BEARER_TOKEN");
        apiInstance.setTwitterCredentials(credentials);

        Set<String> tweetFields = new HashSet<>(Arrays.asList("created_at", "lang", "context_annotations"));

        String query = "covid -is:retweet";

        int maxResults = 500;

        TweetSearchResponse result = apiInstance.tweets().tweetsFullarchiveSearch(query, null, null, null, null,
                maxResults, null, null, null, null, tweetFields, null, null, null, null);

        if (result.getData() != null) {
            for (Tweet tweet : result.getData()) {
                System.out.println(tweet.getId());
                if (tweet.getContextAnnotations() != null) {
                    for (ContextAnnotation contextAnnotation : tweet.getContextAnnotations()) {
                        System.out.println(contextAnnotation.getDomain().getName());
                        System.out.println(contextAnnotation.getEntity().getName());
                    }
                }
            }

        }

    }

}

3. Getting Tweets from the full-archive of public Tweets for a specific time-frame

If you want to get Tweets from the full-archive for a specific time-period, you can specify the time-period using the startTime and endTime parameters, as shown in the example below:

import java.time.OffsetDateTime;

import com.twitter.clientlib.TwitterCredentialsBearer;
import com.twitter.clientlib.ApiException;
import com.twitter.clientlib.api.TwitterApi;
import com.twitter.clientlib.model.*;

public class TwitterApiExample {

    public static void main(String[] args) throws ApiException {

        TwitterApi apiInstance = new TwitterApi();
        TwitterCredentialsBearer credentials = new TwitterCredentialsBearer("BEARER_TOKEN");
        apiInstance.setTwitterCredentials(credentials);

        String query = "covid -is:retweet";

        int maxResults = 100;

        OffsetDateTime startTime = OffsetDateTime.parse("2020-01-01T00:00:00Z");

        OffsetDateTime endTime = OffsetDateTime.parse("2021-01-01T11:59:59Z");

        TweetSearchResponse result = apiInstance.tweets().tweetsFullarchiveSearch(query, startTime, endTime, null, null,
                maxResults, null, null, null, null, null, null, null, null, null);

        if (result.getData() != null) {
            for (Tweet tweet : result.getData()) {
                System.out.println(tweet.getId());
                System.out.println(tweet.getText());
            }

        }

    }

}

4. Writing Tweets to a text file

This example shows how you can write the Tweet IDs for each Tweet obtained for a search result, to a text file. Make sure to replace the fileName with the a name of your chosing. If you wish to write other fields to the text file, make sure to adjust the script below accordingly.

import java.io.FileNotFoundException;
import java.io.PrintWriter;
import java.io.UnsupportedEncodingException;
import java.time.OffsetDateTime;
import java.util.*;

import com.twitter.clientlib.TwitterCredentialsBearer;
import com.twitter.clientlib.ApiException;
import com.twitter.clientlib.api.TwitterApi;
import com.twitter.clientlib.model.*;

public class TwitterApiExample {

    public static void main(String[] args) throws ApiException, FileNotFoundException, UnsupportedEncodingException {

        TwitterApi apiInstance = new TwitterApi();
        TwitterCredentialsBearer credentials = new TwitterCredentialsBearer("BEARER_TOKEN");
        apiInstance.setTwitterCredentials(credentials);

        String query = "covid -is:retweet";

        int maxResults = 100;

        String fileName = "tweet_ids.txt";

        TweetSearchResponse result = apiInstance.tweets().tweetsFullarchiveSearch(query, null, null, null, null,
                maxResults, null, null, null, null, null, null, null, null, null);

        if (result.getData() != null) {
            PrintWriter writer = new PrintWriter(fileName, "UTF-8");
            for (Tweet tweet : result.getData()) {
                writer.println(tweet.getId());
            }
            writer.close();
        }

    }

}

5. Getting Tweets with user information, for each Tweet

In the examples so far, we have seen how to get Tweet information for the Tweets that we have searched for. However, if you need additional information associated with the Tweet, such as the information of the user Tweeting it, you can use expansions and pass it to the tweetsRecentSearch method.

In this example, we want user information, which is why the value for expansions that we pass is author_id. Additionally, we will specify the userFields that we want returned such as profile_image_url, description (aka user's bio) etc. Learn more about available fields here.

From the response, we will get the users list from the includes object. Then, we will look up the user information for each Tweet, using the author_id of the Tweet.

import java.util.*;

import com.twitter.clientlib.TwitterCredentialsBearer;
import com.twitter.clientlib.ApiException;
import com.twitter.clientlib.api.TwitterApi;
import com.twitter.clientlib.model.*;

public class TwitterApiExample {

    public static void main(String[] args) throws ApiException {

        TwitterApi apiInstance = new TwitterApi();
        // Replace with your own bearer token below
        TwitterCredentialsBearer credentials = new TwitterCredentialsBearer("BEARER_TOKEN");
        apiInstance.setTwitterCredentials(credentials);

        Set<String> tweetFields = new HashSet<>(Arrays.asList("created_at", "lang", "conversation_id"));

        Set<String> userFields = new HashSet<>(Arrays.asList("description", "profile_image_url"));

        Set<String> expansions = new HashSet<>(Arrays.asList("author_id"));

        // Replace with your own query below
        String query = "covid -is:retweet";

        TweetSearchResponse result = apiInstance.tweets().tweetsRecentSearch(query, null, null, null, null, 100, null,
                null, null, expansions, tweetFields, userFields, null, null, null);

        // Create a map with User ID and User Object
        Map<String, User> userMap = new HashMap<>();
        if (result.getIncludes() != null && result.getIncludes().getUsers() != null) {
            List<User> users = result.getIncludes().getUsers();
            for (User user : users) {
                userMap.put(user.getId(), user);
            }
        }

        if (result.getData() != null) {
            for (Tweet tweet : result.getData()) {
                User user = userMap.get(tweet.getAuthorId());
                System.out.println(tweet.getId());
                System.out.println(user.getUsername());
                System.out.println(user.getDescription());
            }

        }

    }

}

6. Searching for geo-tagged Tweets

Some Tweets have geographic information associated with them. The search endpoints in the Twitter API v2 support operators such as place, place_country, bounding_box, point_radius etc. (these operators are currently only available in the academic research product track) that allow you to get these geo-tagged Tweets.

In the example below, we want all Tweets that are from the country US, which is why we specify it in our search query using the place_country:US operator. Now because by default, only the Tweet ID and Tweet text are returned, we have to specify the fact that we want the geo information in our response as well. We do this by setting the expansions to geo.place_id in the tweetsRecentSearch method and we also specify the geo fields that we are looking for.

In our response, we get the list of places from the includes object, and we match on the place_id to get the relevant geo information associated with the Tweet (the place name in the example below)

import java.util.*;

import com.twitter.clientlib.TwitterCredentialsBearer;
import com.twitter.clientlib.ApiException;
import com.twitter.clientlib.api.TwitterApi;
import com.twitter.clientlib.model.*;

public class TwitterApiExample {

    public static void main(String[] args) throws ApiException {

        TwitterApi apiInstance = new TwitterApi();
        TwitterCredentialsBearer credentials = new TwitterCredentialsBearer("BEARER_TOKEN");
        apiInstance.setTwitterCredentials(credentials);

        Set<String> tweetFields = new HashSet<>(Arrays.asList("created_at", "lang", "geo"));

        Set<String> placeFields = new HashSet<>(Arrays.asList("name", "place_type"));

        Set<String> expansions = new HashSet<>(Arrays.asList("geo.place_id"));

        String query = "covid -is:retweet place_country:US";

        TweetSearchResponse result = apiInstance.tweets().tweetsRecentSearch(query, null, null, null, null, 100, null,
                null, null, expansions, tweetFields, null, null, placeFields, null);

        // Create a map with Place ID and Place Object
        Map<String, Place> placeMap = new HashMap<>();
        if (result.getIncludes() != null && result.getIncludes().getPlaces() != null) {
            List<Place> places = result.getIncludes().getPlaces();
            for (Place place : places) {
                placeMap.put(place.getId(), place);
            }
        }

        if (result.getData() != null) {
            for (Tweet tweet : result.getData()) {
                Place place = placeMap.get(tweet.getGeo().getPlaceId());
                System.out.println(tweet.getId());
                System.out.println(place.getName());
            }

        }

    }

}

7. Getting Tweet counts (volume) for a search query

So far, we have seen examples of how to get Tweets for a search query. However, if you want to get the count of Tweets (volume) for a search term, you can use the tweetCountsRecentSearch function. You can pass it the search query and specify the granularity for the data aggregation e.g. if you want the daily count, hourly count or 15-minute count for a search term.

import com.twitter.clientlib.TwitterCredentialsBearer;
import com.twitter.clientlib.ApiException;
import com.twitter.clientlib.api.TwitterApi;
import com.twitter.clientlib.model.*;

public class TwitterApiExample {

    public static void main(String[] args) throws ApiException {

        TwitterApi apiInstance = new TwitterApi();
        TwitterCredentialsBearer credentials = new TwitterCredentialsBearer("BEARER_TOKEN");
        apiInstance.setTwitterCredentials(credentials);

        String query = "covid -is:retweet";

        TweetCountsResponse result = apiInstance.tweets().tweetCountsRecentSearch(query, null, null, null, null, null,
                null, Granularity.DAY);

        if (result.getData() != null) {
            for (SearchCount searchCount : result.getData()) {
                System.out.println(searchCount.getTweetCount());
            }

        }

    }

}

8. Getting a user's timeline

In order to get the most recent 3200 Tweets from a user's timeline, we can use the usersIdTweets method and pass it the user id. We can also specify additional fields that we want using the tweetFields and expansions (as shown in the examples above).

import com.twitter.clientlib.TwitterCredentialsBearer;
import com.twitter.clientlib.ApiException;
import com.twitter.clientlib.api.TwitterApi;
import com.twitter.clientlib.model.*;

public class TwitterApiExample {

    public static void main(String[] args) throws ApiException {

        TwitterApi apiInstance = new TwitterApi();
        TwitterCredentialsBearer credentials = new TwitterCredentialsBearer("BEARER_TOKEN");
        apiInstance.setTwitterCredentials(credentials);

        String userID = "2244994945";

        int maxResults = 100;

        GenericTweetsTimelineResponse result = apiInstance.tweets().usersIdTweets(userID, null, null, maxResults, null,
                null, null, null, null, null, null, null, null, null);

        if (result.getData() != null) {
            for (Tweet tweet : result.getData()) {
                System.out.println(tweet.getId());
            }
        }

    }

}

9. Getting a user's mentions

In order to get the most recent 3200 mentions for a user, we can use the get_users_musersIdMentionsentions method and pass it the user id. We can also specify additional fields that we want using the tweetFields and expansions (as shown in the examples above).

import com.twitter.clientlib.TwitterCredentialsBearer;
import com.twitter.clientlib.ApiException;
import com.twitter.clientlib.api.TwitterApi;
import com.twitter.clientlib.model.*;

public class TwitterApiExample {

    public static void main(String[] args) throws ApiException {

        TwitterApi apiInstance = new TwitterApi();
        TwitterCredentialsBearer credentials = new TwitterCredentialsBearer("BEARER_TOKEN");
        apiInstance.setTwitterCredentials(credentials);

        String userID = "2244994945";

        int maxResults = 100;

        GenericTweetsTimelineResponse result = apiInstance.tweets().usersIdMentions(userID, null, null, maxResults,
                null, null, null, null, null, null, null, null, null);

        if (result.getData() != null) {
            for (Tweet tweet : result.getData()) {
                System.out.println(tweet.getId());
            }
        }

    }

}

10. Getting a user's followers

In order to get the followers of a user, we can use the usersIdFollowers function and pass it the user ID. If we want additional fields for the User object such as profile_image_url, we can specify those using the userFields parameter.

import com.twitter.clientlib.TwitterCredentialsBearer;
import com.twitter.clientlib.ApiException;
import com.twitter.clientlib.api.TwitterApi;
import com.twitter.clientlib.model.*;

public class TwitterApiExample {

    public static void main(String[] args) throws ApiException {

        TwitterApi apiInstance = new TwitterApi();
        TwitterCredentialsBearer credentials = new TwitterCredentialsBearer("BEARER_TOKEN");
        apiInstance.setTwitterCredentials(credentials);

        String userID = "2244994945";

        GenericMultipleUsersLookupResponse result = apiInstance.users().usersIdFollowers(userID, null, null);

        if (result.getData() != null) {
            for (User user : result.getData()) {
                System.out.println(user.getId());
                System.out.println(user.getUsername());
            }
        }

    }

}

11. Getting users that a user follows

In order to get the list of users that a user follows, we can use the usersIdFollowing function and pass it the user ID. If we want additional fields for the User object such as profile_image_url, we can specify those using the userFields parameter.

import com.twitter.clientlib.TwitterCredentialsBearer;
import com.twitter.clientlib.ApiException;
import com.twitter.clientlib.api.TwitterApi;
import com.twitter.clientlib.model.*;

public class TwitterApiExample {

    public static void main(String[] args) throws ApiException {

        TwitterApi apiInstance = new TwitterApi();
        TwitterCredentialsBearer credentials = new TwitterCredentialsBearer("BEARER_TOKEN");
        apiInstance.setTwitterCredentials(credentials);

        String userID = "2244994945";

        UsersFollowingLookupResponse result = apiInstance.users().usersIdFollowing(userID, null, null);

        if (result.getData() != null) {
            for (User user : result.getData()) {
                System.out.println(user.getId());
                System.out.println(user.getUsername());
            }
        }

    }

}

12. Getting users that like a Tweet

In order to get the list of users that like a Tweet, we can use the tweetsIdLikingUsers function and pass it the Tweet ID. If we want additional fields for the User object such as profile_image_url, we can specify those using the userFields parameter.

import com.twitter.clientlib.TwitterCredentialsBearer;
import com.twitter.clientlib.ApiException;
import com.twitter.clientlib.api.TwitterApi;
import com.twitter.clientlib.model.*;

public class TwitterApiExample {

    public static void main(String[] args) throws ApiException {

        TwitterApi apiInstance = new TwitterApi();
        TwitterCredentialsBearer credentials = new TwitterCredentialsBearer("BEARER_TOKEN");
        apiInstance.setTwitterCredentials(credentials);

        String tweetId = "20";

        GenericMultipleUsersLookupResponse result = apiInstance.users().tweetsIdLikingUsers(tweetId, null, null);

        if (result.getData() != null) {
            for (User user : result.getData()) {
                System.out.println(user.getId());
                System.out.println(user.getUsername());
            }
        }

    }

}

13. Getting users that retweeted a Tweet

In order to get the list of users that retweeted a Tweet, we can use the tweetsIdRetweetingUsers function and pass it the Tweet ID. If we want additional fields for the User object such as profile_image_url, we can specify those using the userFields parameter.

import com.twitter.clientlib.TwitterCredentialsBearer;
import com.twitter.clientlib.ApiException;
import com.twitter.clientlib.api.TwitterApi;
import com.twitter.clientlib.model.*;

public class TwitterApiExample {

    public static void main(String[] args) throws ApiException {

        TwitterApi apiInstance = new TwitterApi();
        TwitterCredentialsBearer credentials = new TwitterCredentialsBearer("BEARER_TOKEN");
        apiInstance.setTwitterCredentials(credentials);

        String tweetId = "20";

        GenericMultipleUsersLookupResponse result = apiInstance.users().tweetsIdRetweetingUsers(tweetId, null, null);

        if (result.getData() != null) {
            for (User user : result.getData()) {
                System.out.println(user.getId());
                System.out.println(user.getUsername());
            }
        }

    }

}

14. Getting Tweets that a user liked

In order to get the list of Tweets that a user liked, we can use the usersIdLikedTweets function and pass it the User ID. If we want additional fields for the Tweet object such as context_annotations, created_at etc. we can specify those using the tweetFields parameter.

import com.twitter.clientlib.TwitterCredentialsBearer;
import com.twitter.clientlib.ApiException;
import com.twitter.clientlib.api.TwitterApi;
import com.twitter.clientlib.model.*;

public class TwitterApiExample {

    public static void main(String[] args) throws ApiException {

        TwitterApi apiInstance = new TwitterApi();
        TwitterCredentialsBearer credentials = new TwitterCredentialsBearer("BEARER_TOKEN");
        apiInstance.setTwitterCredentials(credentials);

        String userId = "2244994945";

        UsersIdLikedTweetsResponse result = apiInstance.tweets().usersIdLikedTweets(userId, null, null, null, null,
                null, null, null, null);

        if (result.getData() != null) {
            for (Tweet tweet : result.getData()) {
                System.out.println(tweet.getId());
            }
        }

    }

}

15. Lookup Tweets using Tweet IDs

In some cases, if we have a list of Tweet IDs and want to build the Tweet dataset for these Tweet IDs, we can use the findTweetsById function and pass it the list of Tweet IDs. By default, only the Tweet ID and Tweet text are returned. If we want additional Tweet fields, we can specify those using tweetFields parameter.

import com.twitter.clientlib.TwitterCredentialsBearer;
import com.twitter.clientlib.ApiException;
import com.twitter.clientlib.api.TwitterApi;
import com.twitter.clientlib.model.*;

import java.util.Arrays;
import java.util.HashSet;
import java.util.List;
import java.util.Set;

public class TwitterApiExample {

    public static void main(String[] args) throws ApiException {

        TwitterApi apiInstance = new TwitterApi();
        TwitterCredentialsBearer credentials = new TwitterCredentialsBearer("BEARER_TOKEN");
        apiInstance.setTwitterCredentials(credentials);

        List<String> tweetIds = Arrays.asList("1460323737035677698", "1519781379172495360", "1519781381693353984");

        Set<String> tweetFields = new HashSet<>(Arrays.asList("created_at", "lang", "context_annotations"));

        MultiTweetLookupResponse result = apiInstance.tweets().findTweetsById(tweetIds, null, tweetFields, null, null,
                null, null);

        if (result.getData() != null) {
            for (Tweet tweet : result.getData()) {
                System.out.println(tweet.getId());
                System.out.println(tweet.getLang());
                System.out.println(tweet.getCreatedAt().toString());
            }
        }

    }

}

These are some common examples of working with the Twitter API v2 using twitter-api-java-sdk.

If you have any questions or feedback about this guide, please feel free to reach out to me on Twitter!

Exploring Tweets from a user's reverse chronological timeline

Suhem Parack — Thu, 02 Jun 2022 17:54:38 +0000

The reverse chronological timeline endpoint in the Twitter API v2 returns Tweets that appear in a user's home timeline, with the most recent Tweets first. In this short tutorial, I will show how you can get these Tweets using the Tweepy package in Python and perform some basic exploratory analysis on them. Specifically, we will learn how to get:

Recent Tweets that appear in a user's home timeline
Timestamp for the first and last Tweet in the timeline
Most liked Tweet in the timeline
Different languages that appear in the timeline
Common topics that appear in the timeline
Accounts with the most Tweets that appear in the timeline
Types of Tweets that appear in the timeline

In order to use the Twitter API v2, you need to sign up for a Twitter developer account. Once you have signed up, you will need to obtain your keys and tokens to connect to the Twitter API in Python, using Tweepy. Note: the reverse chronological timeline endpoint only works with user access token (and not with app-only auth). Finally, make sure you have Python installed on your machine and that you have the most recent version of Tweepy installed by running:

pip3 install tweepy --upgrade

Getting recent 800 Tweets from the reverse chronological timeline

In order to get Tweets from the reverse chronological timeline with Tweepy, you will first have a to initializa the client (which makes the API calls for you) with your consumer_key, consumer_secret, access_token and access_token_secret. Next, you can use the get_home_timeline function. You can get maximum 100 Tweets per call, so if you want more Tweets, you will have to use the Paginator functionality in Tweepy, and specify how many Tweets you want returned. So, for example, if you want 800 Tweets (as shown in the code below), you can specify limit=8 for the Paginator.

Also, by default the Twitter API v2 returns only the Tweet ID and text for a Tweet. If you want additional data such as the time the Tweet was created, language of the Tweet, metrics (such as like_count), you will have to request those individually using fields and expansions. In this example, because we want the username of the person Tweeting, we will have to set expansions=['author_id']. Then, we will create a users dictionary with the user ID as the key and the user information such as name, username etc as the value, so that we can easily lookup user information for each Tweet.

In the example below, we are creating a dictionary called tweets_dict which contains the Tweet ID as the key and the value is an object with the Tweet text, the time when the Tweet was created, the number of likes for the Tweet, context annotations and user name of the person Tweeting it.

import tweepy

client = tweepy.Client(consumer_key='REPLACE_ME',
                       consumer_secret='REPLACE_ME',
                       access_token='REPLACE_ME',
                       access_token_secret='REPLACE_ME')

tweets_dict = dict()

# Limit = 8 below will result in recent 800 Tweets being returned because for each request we are requesting 100 Tweets
for response in tweepy.Paginator(client.get_home_timeline,
                                 max_results=100,
                                 tweet_fields=['created_at', 'lang', 'context_annotations', 'public_metrics', 'referenced_tweets'],
                                 expansions=['author_id'],
                                 limit=8):
    tweets = response.data
    users = {u["id"]: u for u in response.includes['users']}

    for tweet in tweets:
        print(tweet.id)
        user = users[tweet.author_id]
        tweets_dict[tweet.id] = {
            "id": tweet.id,
            "text": tweet.text,
            "created_at": tweet.created_at,
            "lang": tweet.lang,
            "like_count": tweet.public_metrics['like_count'],
            "context_annotations": tweet.context_annotations,
            "username": user.username
        }

print(len(tweets_dict))

First and last Tweet creation timestamp from the timeline

Now that we have the recent Tweets from the user's home timeline, we can simply get the first and last Tweet from the tweets_dict to determine the first and last Tweet in the users home timeline. In the example below, we also print the difference in time between these 2 Tweets.

first_tweet = tweets_dict[list(tweets_dict)[0]]
last_tweet = tweets_dict[list(tweets_dict)[-1]]

print("First Tweet in timeline is {} created at {}".format(first_tweet['id'], first_tweet['created_at']))
print("Last Tweet in timeline is {} created at {}".format(last_tweet['id'], last_tweet['created_at']))

print("Number of days between first and last Tweet: {}".format(first_tweet['created_at'] - last_tweet['created_at']))

In my case, I got the following response:

First Tweet in timeline is 1532378597620473856 created at 2022-06-02 15:08:22+00:00
Last Tweet in timeline is 1444321306330107905 created at 2021-10-02 15:20:08+00:00
Number of days between first and last Tweet: 242 days, 23:48:14

Most liked Tweet from the timeline

To get the most liked Tweet from the timeline, we can reverse sort tweets_dict on the like_count and that will give us Tweets based on the like_count (most to least).

for k,v in sorted(tweets_dict.items(), key=lambda x: x[1]['like_count'], reverse=True):
    print(k,v['like_count'])

Different languages present in the timeline

To see the common languages present in the timeline, we will create a languages dictionary and then count how many times a language appears and then reverse sort the dictionary based on the count and print it.

languages = dict()

for key, value in tweets_dict.items():
    if value['lang'] not in languages:
        languages[value['lang']] = 1
    else:
        languages[value['lang']] = languages[value['lang']] + 1

for k, v in sorted(languages.items(), key=lambda item: item[1], reverse=True):
    print(k, v)

In my case, it gave me the following response:

en 727
und 9
fr 5
in 3
es 2
tr 2
hi 1
pl 1
ar 1
ja 1
ro 1
it 1
tl 1
ca 1

Most common topics that appear in the timeline

The Twitter API v2 supports Tweet annotations that provide contextual information about a Tweet and return named entities present in a Tweet. Each context_annotation contains a domain and entity. Check out the complete list of supported domains here. We create a topics dictionary and count the entity name and add to it.

topics = dict()

for key, value in tweets_dict.items():
    if "context_annotations" in value:
        annotations = tweet['context_annotations']
        for annotation in annotations:
            if 'entity' in annotation:
                entity = annotation['entity']
                if 'name' in entity:
                    name = entity['name']
                    if name in topics:
                        topics[name] = topics.get(name) + 1
                    else:
                        topics[name] = 1

for k, v in sorted(topics.items(), key=lambda item: item[1], reverse=True):
    print(k, v)

In my case, the response I got is:

Services 756
Twitter 756

Most common accounts that appear in the timeline

In order to get the most common accounts that appear in the timeline, we can create a usernames dictionary and count the number of times a username appears in the timeline, and then we can reverse sort and print it.

usernames = dict()

for key, value in tweets_dict.items():
    if value['username'] not in usernames:
        usernames[value['username']] = 1
    else:
        usernames[value['username']] = usernames[value['username']] + 1

for k, v in sorted(usernames.items(), key=lambda item: item[1], reverse=True):
    print(k, v)

In my case, I got the following response:

suhemparack 374
icahdq 228
TwitterDev 114
hackingcommsci 18
TwitterAPI 16
SentimentsDev 6

Types of Tweets present in the timeline

Sometimes, you may want to understand how many of the Tweets that appear in the timeline are Original Tweets, Replies, Retweets or Quote Tweets. In order to do so, use the referenced_tweets field and then in the determine_tweet_type function, we check whether it is replied_to, quoted or retweeted. If it is neither, then we know that it is an original Tweet.

import tweepy


def determine_tweet_type(tweet):
    if 'referenced_tweets' in tweet:
        # Check for reply indicator
        if tweet['referenced_tweets'][0]['type'] == "replied_to":
            return "Reply Tweet"
        # Check for quote tweet indicator
        elif tweet['referenced_tweets'][0]['type'] == "quoted":
            return "Quote Tweet"
        # Check for retweet indicator
        elif tweet['referenced_tweets'][0]['type'] == "retweeted":
            return "Retweet"
        else:
            return "Original Tweet"
    else:
        return "Original Tweet"


client = tweepy.Client(consumer_key='REPLACE_ME',
                       consumer_secret='REPLACE_ME',
                       access_token='REPLACE_ME',
                       access_token_secret='REPLACE_ME')

tweets_dict = dict()

# Limit = 8 below will result in recent 800 Tweets being returned because for each request we are requesting 100 Tweets
for response in tweepy.Paginator(client.get_home_timeline,
                                 max_results=100,
                                 tweet_fields=['created_at', 'lang', 'context_annotations', 'public_metrics',
                                               'referenced_tweets'],
                                 expansions=['author_id'],
                                 limit=8):
    tweets = response.data
    users = {u["id"]: u for u in response.includes['users']}

    for tweet in tweets:
        user = users[tweet.author_id]
        tweets_dict[tweet.id] = {
            "id": tweet.id,
            "type": determine_tweet_type(tweet),
            "text": tweet.text,
            "created_at": tweet.created_at,
            "lang": tweet.lang,
            "like_count": tweet.public_metrics['like_count'],
            "context_annotations": tweet.context_annotations,
            "username": user.username
        }

types = dict()

for key, value in tweets_dict.items():
    if value['type'] not in types:
        types[value['type']] = 1
    else:
        types[value['type']] = types[value['type']] + 1

for k, v in sorted(types.items(), key=lambda item: item[1], reverse=True):
    print(k, v)

In my case, I got the following response:

Original Tweet 348
Retweet 195
Quote Tweet 112
Reply Tweet 101

I hope this tutorial is helpful to you in learning how to do exploratory analysis on the reverse chronological timeline. If you have any questions or feedback, feel free to reach out to me on Twitter.

Understanding the entire conversation around a Tweet with the Twitter API v2

Suhem Parack — Fri, 06 May 2022 16:31:44 +0000

The Twitter API allows developers and researchers to get Twitter data programmatically. Researchers interested in studying the conversation around a topic often use the search endpoints to get Tweets on a topic. Each of these Tweets returned from the search endpoints may have important follow-up conversations happening in replies to them, and in the Tweets quoting them, which may provide additional information to better understand the entire conversation. In this tutorial, you will learn how to get the entire conversation for a Tweet.

Step one: Search for Tweets about a topic

In the first step, you can search for Tweets on a topic, containing certain keywords and meeting certain criteria using the search endpoints. Currently, there are 2 search endpoints available in the Twitter API v2:

Recent search, available to all developers and gives Tweets from the last 7 days
Full-archive search, currently only available to academic researchers and gives Tweets from the entire archive of public Tweets

The search endpoints require developers to specify a search query to return Tweets meeting a certain criteria. Optionally, you can specify the time period from which you want the Tweets for.

By default, you will get 10 Tweets (Tweet ID and Tweet text) per request. Per request, using the max_results parameter, you can get upto:

100 Tweets for the recent search and quote Tweets lookup endpoints
500 Tweets for the full-archive search endpoint

If you want additional Tweets, you can use pagination with the next_token.

Below is an example of searching for Tweets from the TwitterDev account, using the recent search endpoint in Python using the Tweepy package.

Note: You will first need to sign up for a developer account and have your bearer token to use in the code samples below. Follow the steps in this tutorial to get your bearer token. Also, you will need to install the Tweepy package in Python to run the code sample below. Follow the instructions in this guide to install Tweepy.

import tweepy

client = tweepy.Client(bearer_token='REPLACE_ME')

# Replace with your own search query
query = 'from:TwitterDev -is:retweet'

# Replace the limit=1000 with the maximum number of Tweets you want
for tweet in tweepy.Paginator(client.search_recent_tweets, query=query,
                              tweet_fields=['context_annotations', 'created_at'], max_results=100).flatten(limit=1000):
    print(tweet.id)

Once you have the Tweets on the topic of your choice, for each Tweet you can get all its replies.

Step two: For a Tweet, get all replies to it

You can get all replies to a Tweet using the conversation_id parameter available in the search endpoints. The value for this conversation_id is the Tweet ID of the original Tweet. The example below shows how to get all replies for the Tweet with ID 1503863303709286407

import tweepy

client = tweepy.Client(bearer_token='REPLACE_ME')

# Replace with your own search query
query = 'conversation_id:1503863303709286407 -is:retweet'

# Replace the limit=1000 with the maximum number of Tweets you want
for tweet in tweepy.Paginator(client.search_recent_tweets, query=query,
                              tweet_fields=['context_annotations', 'created_at'], max_results=100).flatten(limit=1000):
    print(tweet.id)

Step three: For a Tweet, get all its Quote Tweets

To get Quote Tweets for a Tweet, you can use the Quote Tweets lookup endpoint. You can pass it the Tweet ID and set the exclude parameter to retweet. This will give you all original Quote Tweets of a Tweet. The example below shows how to get all Quote Tweets for the Tweet with ID 1503863303709286407.

import tweepy

client = tweepy.Client(bearer_token='REPLACE_ME')

# Replace the limit=1000 with the maximum number of Tweets you want
for tweet in tweepy.Paginator(client.get_quote_tweets, id=1503863303709286407, max_results=100,
                              exclude='retweets').flatten(limit=1000):
    print(tweet.id)

That’s it. You now have the entire conversation for a Tweet including all replies and quote Tweets. The conversation_id and the quote Tweets lookup endpoints make it really easy for developers and researchers to understand the entire conversation around a Tweet. If you have any questions or feedback about this tutorial, please reach out to me on Twitter.

Resources

A guide to working with the Batch Compliance endpoints in the Twitter API v2

Suhem Parack — Mon, 25 Apr 2022 18:56:25 +0000

The batch compliance endpoints allow Twitter developers to to upload large datasets of Tweet or User IDs to get their compliance status in order to determine what data requires action in order to keep their datasets in compliance with Twitter's developer policy. In this guide, I will show you how to use these batch compliance endpoints in Python using the Tweepy library.

Typically, there are 4 steps involved in working with this endpoint:

Step 1: Creating a compliance job

First, you need to create a compliance job.

If you will be uploading a list of Tweet IDs, you can set the job_name as tweets.
If you will be uploading a list of USER IDs, you can set the job_name as users.

Note: You can have one concurrent job, per job_type at any time.

import tweepy

client = tweepy.Client('BEARER_TOKEN')

# Replace the job_name with appropriate name for your compliance job
response = client.create_compliance_job(name='job_name', type='tweets')

upload_url = response.data['upload_url']

download_url = response.data['download_url']

job_id = response.data['id']

print(upload_url)

Once your job is succesfully created, please note the following fields in the response for the created job:

upload_url which is the link where you will be uploading your file with Tweet or User IDs to
download_url which is the link where you will be downloading your compliance results from
id which is the unique ID for your compliance job, that you will use to check the status of the job

Step 2: Upload your dataset

Next, you will upload your dataset as a plain text file to the provided upload_url(from the previous step), with each line of the file containing a single Tweet ID or user ID. Replace file_path in the code below, with the appropriate path to your file that contains the Tweet or User IDs.

Note: The upload_url expires after 15 minutes.

import requests

# Replace with your job upload_url
upload_url = ''

# Replace with your file path that contains the list of Tweet IDs or User IDs, one ID per line
file_path = ''

headers = {'Content-Type': "text/plain"}


def connect_to_endpoint(url):
    response = requests.put(url, data=open(file_path, 'rb'), headers=headers)
    print(response.status_code)
    if response.status_code != 200:
        raise Exception(response.status_code, response.text)
    return response.text


def main():
    response = connect_to_endpoint(upload_url)
    print(response)


if __name__ == "__main__":
    main()

Step 3: Check the status of your compliance job

Once you have uploaded your dataset, you can check the status of your job, which can be:

created
in_progress
failed
complete

Get all compliance jobs for a given type

To get all jobs for a specific type, run the following code:

import tweepy

client = tweepy.Client('BEARER_TOKEN')

# If your dataset has User IDs instead, replace the type with users instead of tweets
jobs = client.get_compliance_jobs(type='tweets')

for job in jobs.data:
    print(job)
    print(job['status'])

Get a specific compliance job by job ID

If you want to check the status of a specific job by job ID, run the following code. Replace job_id with your job_id from step 1.

import tweepy

client = tweepy.Client('BEARER_TOKEN')

# Replace the job_id below with your own job ID
job = client.get_compliance_job(id=job_id)

print(job.data['status'])

Step 4: Download the results

Once your job status is set to complete, you can download the results, by running the following code. Replace download_url with the download_url you got in step 1.

import requests

# Replace with your job download_url
download_url = ''


def connect_to_endpoint(url):
    response = requests.request("GET", url)
    print(response.status_code)
    if response.status_code != 200:
        raise Exception(response.status_code, response.text)
    return response.text


def main():
    response = connect_to_endpoint(download_url)
    entries = response.splitlines()
    for entry in entries:
        print(entry)


if __name__ == "__main__":
    main()

Note: The download_url expires after one week (from when the job was created).

This result will contain a set of JSON objects (one object per line). Each object will contain a Tweet ID, the Tweet’s creation date (useful to locate Tweets organized by date), required action, the reason for the compliance action, and its date.

I hope this guide is helpful in understanding how you can work with the batch compliance endpoints in the Twitter API v2. If you have any questions, feel free to reach to me on Twitter @suhemparack

A guide to teaching with the Twitter API v2

Suhem Parack — Mon, 04 Apr 2022 17:54:24 +0000

Are you an educator looking to use Twitter API in your course? In this guide, you will learn:

What is the Twitter API
Why do educators use Twitter API in their course
What type of data is available in the Twitter API
What type of classes are being taught using the Twitter API
Available access levels and which ones should your students choose
How can students get keys and tokens to connect to the Twitter API in their code
How to download Twitter data about a topic in CSV or JSON format without writing code
What tools and resources are available for students to learn about the Twitter API

Note: In this guide, wherever I use the term Twitter API, I am referring to the Twitter API v2 only which is the primary API. We will not be using the old Twitter API v1.1.

Introduction

Twitter is a platform that is used by people across the world to exchange thoughts, ideas and information with one another, using Tweets. Each Tweet consists of up to 280 characters and may include media such as links, images and videos. The Twitter API provides a programmatic way to retrieve Twitter data.

Professors at various schools around the world use Twitter API in their class to teach students:

data science, text mining, machine learning etc.
how to write code
how to work with APIs
how to work with real-world social media data

The data from the Twitter API is available in JSON format. An example of how a Tweet on Twitter is represented in JSON format is shown below:

Examples of courses that use the Twitter API

Applying for access to the Twitter API

In order to use the Twitter API, students need to get their keys and tokens. In order to do so, they need to first apply for the appropriate Twitter API access. Currently, the Twitter API v2 has 3 access levels:

Essential access

This is the fastest and recommended way for students in a classroom to get started with the Twitter API. Students can simply sign up for a developer account here and they will get instant access to the Twitter API v2 without waiting for a use-case review. This access level lets them retrieve up to 500,000 Tweets per month from the Twitter API v2. When using search functionality in the Twitter API, they can get Tweets from the last 7 days in this access level.

Elevated access

If a student in your class needs more than 500,000 Tweets per month, then they can apply for elevated access here. This does require a use-case review, which is why it takes longer to get this access. This access level lets them retrieve up to 2 million Tweets per month from the Twitter API v2. When using search functionality in the Twitter API, they can get Tweets from the last 7 days in this access level.

Academic access

If your students are part of a Masters or PhD program and have a valid research use-case, they can apply for academic access here. This does require a use-case review, which is why it takes longer to get this access and students have to provide additional documentation. This access level lets them retrieve up to 10 million Tweets per month from the Twitter API v2. When using search functionality in the Twitter API v2, they can get any public Tweet from the full archive of public Tweets dating back to 2006. They can also get advanced search operators (such as geo) that let them filter the data they want returned, more precisely.

How to get your API Keys and tokens

The steps for getting the API keys and tokens vary based on the access levels. Below are the instruction for each access-level:

Getting keys and tokens for essential access

First, you need to apply for essential access. You will be asked to verify your email as part of this process. Additionally, you need to have phone number registered with your account in order to use the API.

After you have created your account, you will be asked some basic questions:

Review and agree to the developer agreement and policy

Next, you will be asked to verify your email

You should received an email that looks like this:

Once you click confirm, you will be directed to the developer portal. You can create an app by giving it a name

Once you click Get Keys, you will be able to see your keys and bearer token. Please keep those in a safe place, as you will be using those to connect to the Twitter API v2.

Getting keys and tokens for elevated access

Complete steps for getting keys and tokens for academic access can be found here

Getting keys and tokens for academic access

Complete steps for getting keys and tokens for academic access can be found here

What if my students just want Twitter data without writing any code?

Sometimes professors don't want their students spending too much time writing code to get Twitter data using the API. Twitter recently launched the Downloader tool that lets researchers specify the search query for which they want the data, the time period and the format in which they would like the data i.e. JSON or CSV. Once specified, they can download the data and use for their assignments and projects. Learn more about how to write search queries here

Step one: Enter your academic bearer token

Step two: Create a new download

Step three: Specify your query, time range and download format

Note: The downloader tool currently works for academic access only.

Tools and resources for students to get started with the Twitter API v2

There are a lot of resources available to students looking to get started with the Twitter API v2. Some of those are listed below:

Libraries and packages

If you are looking to use the Twitter API v2 in Python, two popular libraries that you can use:

Tweepy
Twarc

If you are looking to use the Twitter API v2 in R, you can use the academictwitteR package.

Twitter also has its own first-party libraries in Java and Typescript

Getting started with academic research with the Twitter API v2 (course)

Twitter has a dedicated course on getting started with academic research using the Twitter API v2. This course includes plug-n-play code samples and cheatsheets on writing search queries etc.

Videos

Blogs and articles

Twitter Community Forums

If students have any questions about the Twitter API, they can ask on the Twitter Community Forums.

Community datasets

Many researchers share their research datasets (Tweet IDs) on Github etc. An example of such a dataset is the COVID-19-Dataset from Emily Chen. Some professors have their students hydrate such datasets using a tool like DocNow's Hydrator and use those in their classroom.

Final thoughts

I hope this guide is helpful in understanding how you can teach with the Twitter API in your classroom. If you teach a course that uses the Twitter API and have any questions, feel free to reach to me on Twitter @suhemparack and please share what course you are teaching!

Year in review: Academic Research with the Twitter API v2

Suhem Parack — Fri, 03 Dec 2021 18:58:39 +0000

In 2021, we launched product improvements and various community initiatives to help the academic research community succeed in their research studies with Twitter data. In this post, I present a recap of all the API launches and programs - from Twitter as well as the academic research community - that has made it easier for researchers to do research with Twitter data.

Improvements for academics & students with the Twitter API v2

Academic Research Product Track

Launched in January 2021, the academic research product track is one of the biggest updates to the Twitter API v2 for the academic research community. It provides qualified academic researchers free access to the full-archive of public Tweets (previously, academics had to use the paid premium API to get Tweets older than 7 days). In addition to access to the full-archive search functionality, academics get 10 million Tweets per month. This is higher than the other access levels - the essential access provides 500K Tweets per month and the Elevated access provides 2 Million Tweets per month. The academic research track also lets academics further refine their search queries by supporting search operators for geo-filtering.

Tweet counts endpoints

In June 2021, we launched the Tweet counts endpoints that give the volume of Tweets over a period of time for any search query. The Full-archive Tweet counts endpoint is currently available only for academics, so you can get the Tweet counts from the entire archive of public Tweets. This is particularly helpful for visualization use-cases and to identify trends and pattterns for topics over a period of time.

Batch compliance endpoints

In August 2021, we launched the batch compliance endpoints that let developers and researchers upload large number of Tweet or User IDs and get the current status of those Tweets or Users (whether they are deleted, suspended etc). This lets developers and researchers keep their datasets compliant with Twitter's developer policy. In the past, developers had to use the Tweet lookup endpoints to do this, which was quite slow and supported maximum 100 IDs per request, so this new solution is very helpful for researchers.

Essential access

In November 2021, the essential access was added to the Twitter API v2. With essential access, anyone can get access to basic functionality in the Twitter API just by signing up - without applying for a developer use-case review. This gives you instant access to the Twitter API v2 and is thus a good option for teaching use-cases. Professors can just ask their students to sign up for essential access ahead of class, if they want to use Twitter data in their classes.

Livestreams and virtual events for academic research with the Twitter API

We hosted livestreams and virtual events with various guests from the academic community to showcase how researchers can use the Twitter API v2 to get data for their research studies. Below is a list of livestreams and virtual events we did in 2021:

February 25th 2021 - Introduction to the academic research product track
March 18th 2021 - Deep-dive into the full-archive search with the Twitter API v2
April 22nd 2021 - Getting started with the Twitter API v2 in Python using Twarc2
May 20th 2021 - Getting started with the Twitter API v2 in R using academictwitteR
June 23rd 2021 - Exploratory data analysis in R with the Twitter API v2 with Dr. Maria Rodriguez
July 9th 2021 - Building visualizations with the Tweet counts endpoints in the Twitter API v2
August 7th 2021 - Postman Student Summit

August 26th 2021 - Back to school event - how PhD & Postdocs use the Twitter API in their research studies
September 3rd 2021 - Getting started with the Twitter API v2 in Python using OSoMeTweet package
September 30th 2021 - Academic Research with the Twitter API v2 with Dr. Ernesto Calvo
October 5th 2021 - Academic Research with the Twitter API v2 with CSMaP NYU
Octover 29th 2021 - Academic Research with the Twitter API v2 with Dr. Deen Freelon
November 12th 2021 - Twitter API in the classroom

Office hours

In March 2021, we announced monthly office hours for academics that need help with their usage of the Twitter API v2 and the academic research product track. During these office hours, researchers can get 1:1 technical help and get their questions about the Twitter API v2 answered.

Getting started with the Twitter API v2 course

In June, we launched a comprehensive course on getting started with the Twitter API v2 for academic research. This course is available for free and contains content, cheatsheets and code samples in Python and R to help academics learn how to get the Twitter data for research studies using the Twitter API. Since its launch, this course has been adopted as a prerequisite model by various professors who teach courses on computational social sciences, text mining etc.

Academic advisory board

In August 2021, we announced our inaugural Academic Research Advisory Board aimed at creating regular and frequent dialogue between members of the academic community and our Developer Platform.

Sharing your research with Twitter

In August 2021, we also provided researchers with a way to share their research (done with the Twitter API) with us so that we can better learn how researchers use the Twitter API and how we can better serve their needs.

Sample apps, videos, blogs and code samples

In 2021, we shared various code samples, videos, blogs and sample apps for researchers to learn how to work with the Twitter API v2. Below is a list of these.

Tutorials and guides

Videos

Sample apps

Below are some some sample apps, for educational purposes that we built in 2021 to showcase different use-cases pertaining to research with the Twitter API v2.

Twitter Timeline Explorer - a sample app that shows the common topics that an account Tweets about
What's in this image - a demo app to show how you can use AWS Rekognition to extract text from images
Tweet tone analyzer - a sample app that uses IBM Tone Analyzer and Perpective APIs toxicity measurement functionality
Tweet entity extractor - a sample app that shows how to use the Twitter API v2's Tweet annotations

From the community

In 2021, we saw some amazing updates and resources from the academic community that make it easy for researchers to work with the Twitter API for their research studies. Below is a list of some of these:

The Document The Now project launched the twarc2 library in Python (with command line option) that supports the Twitter API v2 and the academic research product track
academictwitteR - a new package in R from Chris Barrie and Justin Ho was launched that supports the Twitter API v2 and the academic product track in R.
focalevents - open source codebase from Ryan Gallagher with tools to work with Twitter data for research
OSoMeTweets - a package in Python to work with the Twitter API
A course on cultural analytics in Python from Melanie Walsh that uses the Twitter API v2

This was a great year for academics that work with the Twitter API for research. If you want to share with the community how you work with the Twitter API for academic research use-cases or have any feedback on our community programs, please feel free to reach out to me @suhemparack