DEV Community: Aurelia Specker

Retrieve a list of user mentions from a thread of Tweet replies

Aurelia Specker — Tue, 25 Jan 2022 17:10:07 +0000

The code for this tutorial is available on GitHub.

Have you ever wanted a quick way to retrieve all usernames mentioned in a thread of replies on Twitter? This Python script does just that using the Twitter API.

Set up

Clone the repository here.
Ensure you have a Twitter developer account with access to GET /2/tweets/search/all. If you don't have access to this endpoint, follow the instructions below to use the recent search endpoint instead. More information on getting access to the API is available here.
In the root directory, rename .env_example to .env and add the credentials for your developer App in between the "". Remember to add .env to your .gitignore file to keep your credentials secure.

Running the code

Use the following command to run the code:

$ python3 replies.py -t <TWEET-ID>

The following flags are available to use:

Flags
`-t`	Tweet ID	Required	ID of the Tweet for which you want to pull data.
`-h`	Help	Optional	View a list of available flags and a description for each of these.
`-s`	Start time	Optional	The oldest UTC timestamp from which the replies will be provided. Format: YYYY-MM-DDTHH:mm:ssZ; for example: 2021-12-04T01:30:00Z. If unspecified, will default to returning replies from up to 30 days ago.
`-e`	End time	Optional	The newest, most recent UTC timestamp to which the replies will be provided. Format: YYYY-MM-DDTHH:mm:ssZ; for example: 2021-12-04T01:30:00Z. If unspecified, will default to [now - 30 seconds].

Output

Returned in the command life interface

Ordered dictionary containing a list of all usernames present in the replies and the number of times each username was mentioned. Note: the user who authored the original Tweet is not included.
Number of replies to the Tweet.
Number of usernames mentioned in these replies.
Number of requests made to the Search Tweets endpoint to pull all replies.

Returned in a new file

A new file, replies.txt, will be created. This file will contain a list of Tweet IDs for the replies.

What if I don't have access to Twitter's full-archive search endpoint?

If you only have access to the recent search endpoint, you can still use this script. Simply make the following two modifications to replies.py:

On line 63, change the endpoint to:

search_url = "https://api.twitter.com/2/tweets/search/recent"

On line 49, make sure the max_results parameter is set to:

"max_results": "500"

When you run the code, if you specify a start time and/or end time using the -s and -e flags, these must be within the last seven days, otherwise you will get back a 400 Bad Request error.

You can read more about Search Tweets and the difference between the recent search and full-archive search endpoints here: https://developer.twitter.com/en/docs/twitter-api/tweets/search/introduction

Use case examples

The above can be helpful in the following scenarios:

You've asked a question on Twitter and want to quickly retrieve a list of all accounts mentioned in the replies.
Identify a list of influencers, as mentioned in a conversation thread.

Migrate to Twitter’s newly released Labs recent search endpoint

Aurelia Specker — Tue, 04 Feb 2020 14:51:24 +0000

In January, Twitter announced the launch of a new search endpoint. This endpoint is available in Twitter Developer Labs and is meant to preview what the Tweet search endpoints will look like in the future.

The framework of the Labs recent search endpoint is very similar to how Twitter will eventually structure other historical search endpoints available in the next generation of the Twitter API. This search endpoint returns the same, new, and modernized version of Tweet and user JSON objects as other endpoints available in Twitter Developer Labs. It also includes expansions and selectable response formats.

In an attempt to test this new endpoint, I refactored my London commute app and replaced the Premium 30-Day Search API with the new Labs search endpoint. This article highlights the migration changes required to do this. Please refer to the original blog post that walks you through the steps required to build the initial London commute app when reading this migration article.

You can find the refactored code that uses the new endpoint on GitHub. Make sure to select the labs-1-tweets-search branch.
You can find the original code using the Premium 30-Day Search API on GitHub on the master branch.
A full comparison of the changes required to migrate from using the Premium 30-Day Search API to using the new Labs search endpoint can be found on GitHub.
If you already have code written for the standard, premium, or enterprise versions of the Search API, you can also check out the "Compare" section of the Twitter developer docs.

Set up

Follow these steps to gain access to the new Labs search endpoint:

Head over to the Labs section of the Twitter developer portal and click "Join Labs"
Select "Activate" next to Recent search, then select a Twitter developer app (your app should be linked to the Twitter account of the notification sender, in this example the Twitter account @maddie_testing).

In the previous version of this app, we use the Python wrapper searchtweets to access the premium search API. We no longer require this. Remove the following lines from alerts.py:

from searchtweets import ResultStream, gen_rule_payload, load_credentials, collect_results
import pandas as pd

Instead, insert the following lines at the top of alerts.py:

import urllib.parse
import requests
from requests.auth import AuthBase

The new version looks like this:

from requests_oauthlib import OAuth1Session
import urllib.parse
import requests
from requests.auth import AuthBase
import datetime as dt
import yaml
import json

Handling credentials

Open your credentials.yaml file and replace the previous version of your credentials with the new one:

Previous version

search_tweets_api:
    account_type: premium
    endpoint: https://api.twitter.com/1.1/tweets/search/30day/{ENV}.json
    consumer_key: XXXXXXXXXXX
    consumer_secret: XXXXXXXXXXX
    access_token: XXXXXXXXXXX
    access_token_secret: XXXXXXXXXXX

New version

labs_search_tweets_api:
    consumer_key: XXXXXXXXXXX
    consumer_secret: XXXXXXXXXXX
    access_token: XXXXXXXXXXX
    access_token_secret: XXXXXXXXXXX

The consumer_key and consumer_secret will be used to generate a bearer_token to access the Labs search endpoint.

The consumer_key, consumer_secret, access_token, and access_token_secret are required to access the POST statuses/update endpoint.

Remember to always add the credentials.yaml file to your .gitignore file, to avoid accidentally pushing your private credentials to GitHub.

Handling authentication

In alerts.py, remove the following lines of code that refer to the Python wrapper searchtweets:

creds = load_credentials(filename="./credentials.yaml",
                       yaml_key="search_tweets_api",
                       env_overwrite=False)

Make sure to replace the four references to "search_tweets_api" with "labs_search_tweets_api" to reflect the changes made in your credentials.yaml file.

Keep the section that uses OAuth1Session to manage credentials for the POST statuses/update endpoint (as described in the original blog post).

In order to authenticate with the Labs search endpoint, we will use OAuth 2.0 Bearer Token authentication.

This is what the authentication section of your code should look like in this newer version:

# Authentication
with open("./credentials.yaml") as file:
    data = yaml.safe_load(file)

consumer_key = data["labs_search_tweets_api"]["consumer_key"]
consumer_secret = data["labs_search_tweets_api"]["consumer_secret"]
access_token = data["labs_search_tweets_api"]["access_token"]
access_token_secret = data["labs_search_tweets_api"]["access_token_secret"]

oauth = OAuth1Session(
    consumer_key,
    client_secret=consumer_secret,
    resource_owner_key=access_token,
    resource_owner_secret=access_token_secret,
)

# Generate bearer token with consumer key and consumer secret via https://api.twitter.com/oauth2/token
class BearerTokenAuth(AuthBase):
    def __init__(self, consumer_key, consumer_secret):
        self.bearer_token_url = "https://api.twitter.com/oauth2/token"
        self.consumer_key = consumer_key
        self.consumer_secret = consumer_secret
        self.bearer_token = self.get_bearer_token()

    def get_bearer_token(self):
        response = requests.post(
            self.bearer_token_url,
            auth=(self.consumer_key, self.consumer_secret),
            data={"grant_type": "client_credentials"},
            headers={"User-Agent": "LabsMetTutorialPython"})

        if response.status_code is not 200:
            raise Exception(f"Cannot get a Bearer token (HTTP %d): %s" % (response.status_code, response.text))

        body = response.json()
        return body["access_token"]

    def __call__(self, r):
        r.headers["Authorization"] = f"Bearer %s" % self.bearer_token
        r.headers["User-Agent"] = "LabsMetTutorialPython"
        return r

# Create Bearer Token for authenticating
bearer_token = BearerTokenAuth(consumer_key, consumer_secret)

Request parameters

Timestamps

The new Labs search endpoint offers two optional timestamp parameters (start_time and end_time) in the ISO 8601/RFC 3339 date format: YYYY-MM-DDTHH:mm:ssZ

As this format is different to the format required with the Premium Search API, you will have to edit the output format of datetime like so:

# Generate start_time and end_time parameters
utc = dt.datetime.utcnow() + dt.timedelta(minutes = -1)
utc_time = utc.strftime("%Y-%m-%dT%H:%M:%SZ")
print("end_time:", utc_time)

two_hours = dt.datetime.utcnow() + dt.timedelta(hours = -2, minutes = -1)
two_hours_prior = two_hours.strftime("%Y-%m-%dT%H:%M:%SZ")
print("start_time", two_hours_prior)

Generating a query

As we’re no longer using the searchtweets Python wrapper, we can delete the following lines of code:

rule = gen_rule_payload("from:metline -has:mentions",from_date=str(two_hours_prior), to_date=str(utc_time), results_per_call=100)
print("rule:", rule)

tweets = collect_results(rule, 
                        max_results=100,
                        result_stream_args=creds)

[print(tweet.created_at_datetime, tweet.all_text, end='\n\n') for tweet in tweets[0:10]];

Instead, we’re going to use the Python library urllibe to encode non-ASCII text and make the parameters safe for use as a URL component:

# Generate query and other parameters
query = urllib.parse.quote(f"from:metline -has:mentions")
print(query)
start_time = urllib.parse.quote(f"{two_hours_prior}")
print(start_time)
end_time = urllib.parse.quote(f"{utc_time}")
print(end_time)
tweet_format = urllib.parse.quote(f"compact")
print(tweet_format)

Note the use of Python 3’s f-Strings formatting syntax to pass in the timestamp variables. We are going to use this syntax for all instances of string formatting in this version of the tutorial.

The request URL is generated by appending the different encoded parameters to the endpoint.

# Request URL
endpoint = "https://api.twitter.com/labs/1/tweets/search" 
url = f"{endpoint}?query={query}&start_time={start_time}&end_time={end_time}&format={tweet_format}"
print(url)

Define headers. This tells your client that it is able to parse gzip requests and decompress them:

# Request headers
headers = {
    "Accept-Encoding": "gzip"
}

Getting a response

Store the response in a variable called response:

response = requests.get(url, auth = bearer_token, headers = headers)

if response.status_code is not 200:
   raise Exception(f"Request returned an error:{response.status_code}, {response.text}")

And then convert the response to JSON. This will enable you to pull out the Tweet text and store it in a new variable called combined_tweet_text:

# Convert response to JSON & pull out Tweet text
parsed_response = json.loads(response.text)
print(parsed_response)
try:
    tweet_text = [tweet["text"] for tweet in parsed_response["data"]]
    combined_tweet_text = " ".join(tweet_text)
    print(combined_tweet_text)
except:
    combined_tweet_text = " "

Delete the previous version of this:

tweet_text = []
tweet_date = []
combined_tweet_text = ''

for tweet in tweets: 
   tweet_text.append(tweet.all_text)
   tweet_date.append(tweet.created_at_datetime)
   combined_tweet_text += tweet.all_text

Analyse Tweets

The Tweet analysis section that determines if the commuter has to be notified of a delay remains similar to the first version of this application. This is where you can input details specific to your use case.

In order to provide greater flexibility, we can pull out the Twitter @handles of the commuters into two variables (commuter_1 and commuter_2). However, this change is not mandatory.

If a notification is required, we use the POST statuses/update endpoint to send a Tweet to the commuter.

# Analyse Tweets & notify commuter (details specific to use case)
all_trigger = {"closure", "wembley", "delays", "disruption", "cancelled", "sorry", "stadium"}

david_trigger = {"hillingdon", "harrow"}

aurelia_trigger = {"baker"}

tweet_words = set(combined_tweet_text.lower().split())

commuter_1 = "@AureliaSpecker"
commuter_2 = "@dormrod"

if len(tweet_words.intersection(all_trigger)) != 0: 
    message = f"{commuter_1} and {commuter_2} 👋 check https://twitter.com/metline for possible delays, [{utc_time}]"
elif len(tweet_words.intersection(david_trigger)) != 0:
    message = f"{commuter_2} 👋 check https://twitter.com/metline for possible delays, [{utc_time}]" 
elif len(tweet_words.intersection(aurelia_trigger)) != 0:
    message = f"{commuter_1} 👋 check https://twitter.com/metline for possible delays, [{utc_time}]"
else:
    message = "There are no delays"
    pass

print("Message:", message)

params = {"status": message}

oauth.post(
    "https://api.twitter.com/1.1/statuses/update.json", params=params
)

Conclusion

I used several libraries and services beyond the Twitter API to make this tutorial, but you may have different needs and requirements and should evaluate whether those tools are right for you.

Let us know if this inspires you to build anything on the Twitter community forums or by Tweeting us at @TwitterDev. You can also give us feedback on our feedback platform.

Using the Twitter API to make your commute easier

Aurelia Specker — Thu, 14 Nov 2019 14:41:40 +0000

The code for this tutorial is available on GitHub.

I use London’s TfL Metropolitan line on a daily basis to commute to and from work. Being aware of disruptions as they occur helps me save time on a daily basis. Luckily, the official Twitter account of the Metropolitan line (@metline) sends accurate information about the status of the line.

I had originally enabled notifications for the @metline account, so that each time @metline sent a new Tweet, I would receive a push notification on my phone. But I ended up receiving too many notifications and many weren’t relevant to me.

I needed a way to filter Tweets of interest and only receive notifications for information that is relevant to my personal commuting pattern.

Enter... 🥁🥁🥁 the Twitter API!

We’ll be using two different endpoints:

The Premium 30-Day Search API (you can use the Sandbox version for free) to find the Tweets that we’re interested in.
The POST statuses/update endpoint to create a new Tweet that will notify you of potential disruptions. This endpoint is available for free.

Set up

In order to reproduce this application, you will need the following:

Choose a Twitter account that Tweets information of interest.
In this example, we will use @metline for Tube line information.
Twitter account(s) of the notification recipient(s), in other words the commuter(s). For this, I used my primary Twitter account (@AureliaSpecker).
Twitter account of the notification sender. For this, I used a secondary Twitter account (@maddie_testing).
Twitter Developer account: if you don’t have one already, you can apply for one. The developer account should be linked to the Twitter account of the notification sender (in this example: @maddie_testing).
Create a Twitter Developer app.
Access keys and tokens for the app you created in the step above.
Set up your dev environment for "Search Tweets: 30-Days Sandbox" to something like dev or prod.
Have pip installed on your laptop.
Python version 3.6 or above (in your command line, you can check what version of Python is installed by running $ python --version).

From your command line, create a new directory for your project.

$ mkdir commute-alerts

Navigate into it and create the following two files:

$ cd commute-alerts
$ touch alerts.py
$ touch credentials.yaml

At this point, your project tree should look something like this (you can install tree with $ sudo install tree and then use $ tree to get the following output):

└── alerts.py
└── credentials.yaml

If you plan on pushing your project to GitHub, you should also add a .gitignore file in your project directory:

$ touch .gitignore

It is very important that you carefully guard your access keys and tokens, as they represent your unique access to the Twitter API. Keeping these in a separate file (credentials.yaml) will ensure that you exclude them from source and that no one steals them from you. Make sure to add the following line to your .gitignore file:

credentials.yaml

Check that you have the necessary dependencies installed, by running the following in your command line:

$ pip install searchtweets
$ pip install requests_oauthlib
$ pip install pandas

Open alerts.py in your preferred code editor (I use Visual Studio Code) and add the following lines of code to import the required modules:

from searchtweets import ResultStream, gen_rule_payload, load_credentials, collect_results
from requests_oauthlib import OAuth1Session
import yaml
import json
import datetime as dt
import pandas as pd

Handling credentials

Now open credentials.yaml and add your access keys and tokens in the format below. Replace {ENV} with the dev environment name you previously created and insert your access keys and tokens for each field accordingly.

search_tweets_api:
  account_type: premium
  endpoint: https://api.twitter.com/1.1/tweets/search/30day/{ENV}.json
  consumer_key: XXXXXXXXXX
  consumer_secret: XXXXXXXXXX
  access_token: XXXXXXXXXX
  access_token_secret: XXXXXXXXXX

Back in alerts.py, use load_credentials from the Search Tweet Python wrapper. This will automatically handle credentials for you:

creds = load_credentials(filename="./credentials.yaml",
                        yaml_key="search_tweets_api",
                        env_overwrite=False)

For the POST statuses/update endpoint, we will manually load the credentials and store them into a new set of variables.

with open('./credentials.yaml') as file:
    data = yaml.safe_load(file)

consumer_key = data["search_tweets_api"]["consumer_key"]
consumer_secret = data["search_tweets_api"]["consumer_secret"]
access_token = data["search_tweets_api"]["access_token"]
access_token_secret = data["search_tweets_api"]["access_token_secret"]

We will then use OAuth1Session to manage credentials:

oauth = OAuth1Session(
    consumer_key,
    client_secret=consumer_secret,
    resource_owner_key=access_token,
    resource_owner_secret=access_token_secret,
)

Using the Premium 30-Day Search API

This endpoint lets you access the last 30 days of Twitter data. You can find out more about this endpoint in the Twitter Developer documentation.

This endpoint takes four different parameters:

query
fromDate
toDate
maxResults (in other words, the number of results returned per call).

We will use Python’s datetime module to generate a fromDate and a toDate and establish a timeframe during which we want to be searching for Tweets from @metline. In our case, we want to be looking for all Tweets that were created in the past two hours. Note that, with the Twitter API, times are in UTC. Add the following lines of code to alerts.py:

utc = dt.datetime.utcnow() + dt.timedelta(minutes=-1)
utc_time = utc.strftime("%Y%m%d%H%M")
print("toDate:", utc_time)

two_hours = dt.datetime.utcnow() + dt.timedelta(hours=-2, minutes=-1)
two_hours_prior = two_hours.strftime("%Y%m%d%H%M")
print("fromDate:", two_hours_prior)

This will return something like this when you run $ python alerts.py in the command line (which is the format required for the toDate and fromDate parameters):

toDate: 201910071658
fromDate: 201910071458

The Search Tweets Python Wrapper enables us to use the 30Day Search endpoint without too much setup on our side. In this case, we are looking to get all Tweets from @metline that do not contain any mention of other Twitter accounts. Our query is therefore "from:metline -has:mentions". You can read more about building queries in the Twitter Dev documentation. We also want to return no more than 100 Tweets per call (this is the maximum we can have with the free Sandbox access).

rule = gen_rule_payload("from:metline -has:mentions",from_date=str(two_hours_prior), to_date=str(utc_time), results_per_call=100) 
print("rule:", rule)

Here, the print function will return something like this:

rule: {"query": "from:metline -has:mentions", "maxResults": 100, "toDate": "201910071658", "fromDate": "201910071458"}

We store the Tweets returned by this query in a new variable "tweets" and then we use list comprehensions to print out the first 10 lines of Tweets returned by our query:

tweets = collect_results(rule, 
                         max_results=100,
                         result_stream_args=creds)

[print(tweet.created_at_datetime, tweet.all_text, end='\n\n') for tweet in tweets[0:10]];

This will return something like the below. Note that if no Tweets were created in the previous two hours, nothing will be returned.

2019-10-07 05:06:17 Minor delays between Uxbridge and Harrow-on-the-Hill, southbound only, due to a temporary shortage of train operators. Good service on the rest of the line.

2019-10-07 04:36:30 Minor delays between Harrow-on-the-Hill and Watford / Rayners Lane northbound only due to the temporary unavailability of train operators. Good service on the rest of the line.

2019-10-06 22:27:02 A good service is now operating to all destinations.

2019-10-06 21:49:52 Minor delays between Chesham/Amersham and Rickmansworth, southbound only, due to a signal failure at Chalfont &amp; Latimer.

Setting up trigger words and notifications

We need to establish what words will trigger a notification for the commuter(s). We will use Python sets to do this.

In this example, there are two commuters: David and Aurelia. Certain words will trigger a notification for both commuters, whereas other words will only trigger a notification for one of the commuters.

all_trigger = {'closure', 'wembley', 'delays', 'disruption', 'cancelled', 'sorry', 'stadium'}

david_trigger = {'hillingdon', 'harrow'}

aurelia_trigger = {'baker'}

Note that if you want to add strings made up of two words in these sets you will have to add an extra level of data processing with REs (this is not included in this tutorial).

Next, we create two new lists in which to store the Tweet dates and Tweet text returned by the call above:

tweet_text = []
tweet_date = []

We will also want all of the words included in the Tweets from the previous two hours to be concatenated in one string:

combined_tweet_text = ' '

We then use a For Loop to populate the lists we created with the Tweet text and Tweet dates, and to concatenate the text from the different Tweets into one string.

for tweet in tweets: 
    tweet_text.append(tweet.all_text)
    tweet_date.append(tweet.created_at_datetime)
    combined_tweet_text += tweet.all_text

The following line creates a new set containing all of the words in lowercase from Tweets created in the past two hours:

tweet_words = set(combined_tweet_text.lower().split())

We then use an If statement, as well as the Python len() function and set intersection() method to define whether a notification has to be sent or not:

if len(tweet_words.intersection(all_trigger)) != 0: 
    message = "@AureliaSpecker & @_dormrod 👋 check https://twitter.com/metline for possible delays, [{}]".format(utc_time)
elif len(tweet_words.intersection(david_trigger)) != 0: 
    message = "@_dormrod 👋 Check https://twitter.com/metline for possible delays, [{}]".format(utc_time)
elif len(tweet_words.intersection(aurelia_trigger)) != 0:
    message = "@AureliaSpecker 👋 Check https://twitter.com/metline for possible delays, [{}]".format(utc_time)
else:
    message = "There are no delays"
    pass

You can print the message output, to check that the correct information is returned when you run $ python alerts.py:

print("Message:", message)

Using the POST statuses/update endpoint to post on Twitter

The final step involves sending the notification, in the form of a new Tweet, when one of the trigger words appears in the Tweet text.

We use Twitter’s POST statuses/update endpoint and set the parameter "status" (i.e. the Tweet body) to our variable "message" defined above:

params = {"status": message}

oauth.post(
    "https://api.twitter.com/1.1/statuses/update.json", params=params
)

As long as the receiving Twitter account(s) have notifications enabled, the commuter(s) will receive a notification every time their @username is included in the Tweet body.

Deploying your app with DigitalOcean

I chose to use DigitalOcean to deploy this app to a remote server.

Login to your DigitalOcean account (or sign up for one)
Create a new project and, within that project, create a new droplet by selecting "Create" on the top right hand side corner. Here is a good tutorial on how to create new droplets.

A "droplet" is a Linux-based virtual machine. In other words, each new droplet you create acts as a new server you can use to deploy projects.

Once you’ve created a droplet, you will have to connect to your droplet. This tutorial explains how you can do this.

Once you have connected to the droplet (i.e. you are connected to the server) you will have to upload your project to that server. You can think of your droplet (or server) as another local machine on which you can add your project by cloning the repository from GitHub.

If you haven’t already, add your project to GitHub by following this tutorial. Make sure to add your credentials.yaml file to your .gitignore file, as described above. Remember not to accidentally push any credentials to GitHub. Check out this guide on how to secure your keys and access tokens.
This article explains how to clone GitHub repositories.
Once you’ve added your project, you will have to create a new credentials.yaml file (as the one you previously did) and save it in the project folder. You can use vim or nano as a text editor to add your credentials to the file:

# touch credentials.yaml
# vim credentials.yaml

Type ESC :wq to save your changes and quit VIM mode.

Also make sure to add a .gitignore file and add credentials.yaml to that file, to make sure you don’t accidentally commit any access keys and tokens:

# touch .gitignore
# vim .gitignore

Once again, you can use ESC :wq to save your changes and quit VIM mode.

Going forward, if you want to pull any changes from GitHub, you can run the following in your project directory:

# git pull

Then, install all required dependencies (as described previously) just as you would if you were on another local machine.
You’re now ready to run your app:

# python3 alerts.py

Congratulations! 🥳 Your app is now installed on a remote server.

Scheduling your app to automatically run with Cron

Cron allows you to schedule your app to run automatically at certain times and on certain days. For example, I only want alerts.py to run on weekdays, at 6.30am and at 4.30pm.

I used the following tutorials to learn how to set up a Cron job:

Here are the steps I followed:

On your remote server, open alerts.py with vim and change the two mentions of ./credentials.yaml to root/met-line-alerts/credentials.yaml
Install crontab-python: $ pip install python-crontab
Create a new file cron.txt and open it with vim:

# touch cron.txt
# vim cron.txt

In that file, add the following line (type the letter "i" to insert text in vim):

30 6,16 * * 1-5 python3 /root/met-line-alerts/alerts.py

Save and quit vim mode with ESC :wq

This line of code basically tells Cron to execute the command python3 /root/met-line-alerts/alerts.py at minute 30 past hour 6 and 16 on every day-of-week from Monday through Friday.

You can check your Cron schedule expressions on this website: https://crontab.guru/

You should also make sure to check the time on your Linux remote server, to ensure that you’re setting the time in the correct time zone. You can do this by typing the date command: # date

Run the following command to initiate the Cron job:

# crontab cron.txt

You can list your active Cron jobs with:

# crontab -l

You can remove existing Cron jobs with:

# crontab -r

Conclusion

Hopefully this tutorial inspires you to build with the Twitter API to solve problems in your daily life. I used several libraries and services beyond the Twitter API to make this tutorial, but you may have different needs and requirements and should evaluate whether those tools are right for you.

Let us know if this inspires you to build anything on the Twitter community forums or by Tweeting us at @TwitterDev. You can also give us feedback on our Feedback platform.

#100DaysOfCode

Aurelia Specker — Fri, 13 Sep 2019 13:53:30 +0000

I don't have a technical background.

I used to work in Market Research, and before that I was doing a degree in Modern Languages. In 2017, I took a coding course with Code First: Girls. This is how I got into tech and what led me to start a new job as a Partner Engineer at Twitter.

A few months into my new job, I felt like I wasn't improving my coding skills fast enough. I needed to find the motivation to code outside of work and force myself outside of my comfort zone to learn new things. This is what led me to take on this challenge.

This post is based on a talk I recently gave about my experience of the #100DaysOfCode challenge.

What

The #100DaysOfCode challenge was created by Alexander Kallaway when he became frustrated at the "slow speed of [his] progress in learning to code" (you can read more here). Alexander wrote an article about this and, as others read about the challenge, many decided to join in. Since 2017, thousands of people have taken up the challenge.

There are two rules:

Commit to coding for minimum 1 hour every day for the next 100 days.
Publicly commit to the challenge and Tweet your progress every day with the #100DaysOfCode hashtag.

You can add any other rule that will help you reach your goals and fit your schedule. For example, I added the following rule: "In order for the hour to count, I have to be working on something new." This meant that the regular day-to-day coding I did as part of my job didn't count.

Why

Join a community of supportive, friendly, motivated and like-minded individuals

The community is very active on Twitter and Slack, for example, and the support I got from others during the challenge was very motivating.

Make new friends

I was surprised at how many people (both at work and outside of work) who follow me on Twitter approached me to ask how the challenge was going. I had conversations that I would probably never have had without this challenge.

Work on projects, not tutorials

Every beginner will one day have to shift away from copying other people's code and following tutorials to start writing their own code. It's a difficult shift to make. The nature of the #100DaysOfCode challenge, and the fact that it spans a long period of time, will help you make this shift.

Opportunity to really commit to learning to code

I'd been wanting to code more for a while. This challenge forced me to not just think about it, but actually do it.

An active GitHub profile

You can clearly see how active my GitHub profile became when I started taking the challenge in January. It's also clear that I needed a bit of a break at the end of the challenge in April. But after that I kept on coding. The #100DaysOfCode challenge is a great way to build a new habit and consistency.

Keep track of your progress

It's easy to not realise how far you've come. Being able to look back at the Tweet thread that tracks what I did every day during these 100 days helps me see how much I learnt and achieved in that period of time.

It's a good excuse to Tweet every day!

Challenges

Finding the time

This was one of the hardest aspects for me. Work takes up a lot of my time, and I'm always busy when I'm not at work. I had to cut down the exercising and socialising to find an extra hour every day to code.

Not knowing exactly what technologies to learn

The diversity of technologies and programming languages that exist is one of the most exciting aspects of tech. And yet, as a beginner, it can also be daunting and it can be hard to know what technologies to focus on.

Not knowing what resources to use

The Internet is brimming with available resources (both free or for a fee). As much as this is amazing, it can also be overwhelming. How do I know which resources are good and trustworthy? How do I know what formats work well for me? Etc.

Feeling hopeless when progress is slow

100 days is a long time and there will naturally be some days that feel more difficult. It's normal to feel discouraged at times. On days like that, it's important to take a step back and appreciate how much you've done so far.

The imposter syndrome is real!

Tweeting about your progress daily can make you feel vulnerable. "What if people think I'm too slow or that what I'm learning is too easy?" "What if I'm not good enough?" It can be hard at times, but it's important to learn to block out these thoughts and ignore them.

How to succeed

Find a way to hold yourself accountable

This was really important for me. Knowing that my colleagues, my managers, my friends were following my progress is what ensured that I didn't give up in the harder moments.

Have a broad plan

You don't need a specific plan, but I would suggest having a list of things you want to do or learn about. My list included items such as using my Raspberry Pi or refactoring a Python project to be object-oriented. As I made my way through the challenge, I found out about new technologies and concepts and added these to my list. This meant that, when I was stuck for what to do next, I could simply revert back to my list and get started on the next item.

It's ok to have a bad day

You will have bad days. Embrace them and Tweet about it. You will likely get support from the community. Having a bad day is part of the journey. Talking about it will also help others feel less lonely when they have a bad day.

It's ok to take a few days off

Similarly, it's important to take a few days off if you need to. Midway through the challenge I took 3 days off, to enjoy some time with my family. I was feeling particularly tired then, because of a busy period at work and in my personal life. Taking a short break enabled me to feel energised and motivated enough to take on the second half of the challenge.

Consistency really matters

Consistency is what will help you develop a habit.

Persevere

It took me ~70 days before I started feeling the benefits of the challenge, before I started drawing connections, and before I started feeling that this challenge was worth it! Learning takes time. If I'd given up any earlier, I would never have realised how much I could learn during this period of time.

Don't be afraid to ask for help

I was lucky to do some code pairing sessions throughout this challenge. Sometimes, you just need someone to sit down with you and explain a concept or help you debug. Don't be afraid to ask for help: there will always be someone out there who is willing to help you.

Resources

The community is very active and there are many resources to help get you started and support you along the way. For example:

The #100DaysOfCode official website
The community on Twitter
The #100DaysOfCode Slack workspace
Podcast episodes: CodeNewbie and Learn To Code With Me

A snapshot into my learnings

I chose to get a broad exposure to different technologies, concepts and languages rather than focus on one big project during the 100 days. Here are some examples:

I used a Raspberry Pi for the first time

I learnt about object-oriented programming

I became more familiar with JavaScript and wrote code in Node.js

I looked into what happens when you deploy a project to Heroku

I explored asynchronous programming and promises

I dived into databases and used both MySQL and MongoDB

I honed my bash skills

I used Flask and Jinja to build the front-end for a Python project

And lots, lots more...

Getting started with Twitter's Account Activity API

Aurelia Specker — Fri, 31 May 2019 17:03:01 +0000

Have you ever wanted to follow account events on Twitter in realtime? This tutorial will help you get started with Twitter's webhook-based Account Activity API.

You can watch the tutorial here, or continue reading below.

// Detect dark theme var iframe = document.getElementById('tweet-1134499619759644672-96'); if (document.body.className.includes('dark-theme')) { iframe.src = "https://platform.twitter.com/embed/Tweet.html?id=1134499619759644672&theme=dark" }

Twitter's Account Activity API sends you events related to a Twitter user's account activities in realtime. For example, the Account Activity API delivers a JSON event payload every time a subscribed user creates or deletes a Tweet, Retweets, receives or sends a Direct Message, follows a new user, or likes a post. You can find a full list of the types of activities served by the Account Activity API in Twitter's developer documentation.

A webhook-based API

The Account Activity API is different to Twitter's REST and streaming APIs, because it uses a combination of REST endpoints for setup and management, and webhooks for data delivery. If you've never used them before, webhooks deliver data by "pushing" (aka POSTing) events to the webhook URL of a registered web application. This is done via an HTTP call in real-time when a specific event (aka an account activity) happens.

Twitter's more common APIs are based on a client-server approach, where you would create a client application that makes a connection to Twitter's servers to receive some information. This can be based on a streaming model, where your client connection to Twitter's servers always stays open to receive data in realtime, or, it can be a client-triggered request, for example if you wanted to retrieve a specific Tweet.

With the Account Activity API, this pattern is inverted. Instead of your client application making a connection request to Twitter's servers, it's Twitter's servers that make a connection back to your web application listening for data. In this case, Twitter acts as the client and opens a connection to your server to post activities to that webhook every time an event occurs.

There are many advantages to using webhook-based APIs:

From a computing-resourcing perspective, webhooks require less resources than if you have an ongoing open connection to a streaming API.
There is confirmation on both sides of an activity being delivered per POST with an HTTP 200 response code.
If an activity is not successfully delivered, there is a built-in retry process to re-POST, which reduces the chance of missed data.
With webhooks, your application no longer has to make periodic polling requests to an API to check for new data to be available.
New data will automatically be sent to your server as soon as it's available. Therefore, the data received is almost always the most up-to-date.

What you need to get started

You'll need the following:

A developer account is required to use any of Twitter's APIs. You can apply for one if you don't have one already.
A Twitter developer app with the right permissions. You can create a new one if needed, by selecting the "Create an app" button under the Apps section of your developer portal.
- Make sure to enable "Read, Write, and Direct Messages", on the permissions tab of your app page. Also, note that permission settings are not retroactive for authorized users with previous lower permission settings.
- On the "Keys and Access token" tab, take a note of your app's Consumer API Key and API Secret. This information will enable you to generate a bearer token.
- You will also have to generate your app's access token and access token secret. You will need these when you register your webhook URL.
A developer environment, which you can set up in the Dev Environments section of your developer portal. Select your app and choose an environment name. The name you choose will replace the :env_name token when you start using the different Account Activity API endpoints.
A specific, secured web application with URL for receiving data through webhooks. A tool like ngrok could be useful for creating a local web address to test your webhooks in development.

Securing your webhooks

In order to receive activities from Twitter via webhooks, you'll need to create and secure a web application that can receive HTTP (POST and GET) requests on a URL of your choice. This app needs to be secured with the same Twitter app credentials that you set up in your developer environment for the Account Activity API, and coded to create the crc_token using these credentials. Twitter will regularly send your webhook challenge response check (known as CRC) GET requests, for which your web app needs to return the crc_token generated using the Twitter application credentials that match those of your whitelisted developer environment app.

You can register a webhook URL for the Account Activity API using the POST account_activity/all/:env_name/webhooks endpoint.

Set up for the Account Activity API is done with API requests - there's no user interface built into the developer portal. We've provided an example dashboard tool on our TwitterDev GitHub page, which you might find useful for testing: https://github.com/twitterdev/account-activity-dashboard

Your registered webhook URL is where the Account Activity API will send account activities, in the form of HTTP POST requests containing the event data.

Authentication

Authentication can be tricky with the Account Activity API, because there are different auth methods for different functions of the API. Most of the webhook setup is done through application-only authentication with a bearer token. Check the docs for more information on how to build one.

Subscriptions are set up via application-user authentication for each user. If you want to receive events for your own Twitter account, you can use the access token generated with the API key and API secret key from your developer portal, and this is a great way to get started and test the API. If you want to receive events relating to another Twitter account, the owner of that account will have to grant you permission via the 3-legged OAuth process. This process is also something that you will need to implement in your server-side code, as this will enable you to receive a user's access tokens. In the example dashboard app, we use Sign in with Twitter to get the tokens for each user, so take a look if you want to learn more.

Subscriptions

Once you have a user's access token from the 3-legged OAuth process, you can use it with application-user authentication to create a new subscription for that user.

At this stage, you should have these things:

A running server-side web application ready to receive webhook events.
A registered webhook URL where Twitter will send events (also known as account activities) in the form of POST requests.
One or more authorised users in the form of registered subscriptions for whom you will receive account activities.

OK! You're ready to start receiving events!

Events

You'll start receiving events for a user as soon as they are subscribed to your webhook URL. So, what exactly is an event? In this case, an event corresponds with an activity related to the Twitter account of a subscribed user. Each event is delivered in the form of a JSON payload. In the example below, a "message create" event is delivered when the user receives a Direct Message. There is a "text" field that contains the content of the message, and other information.

{
    "for_user_id": "1102321381",
    "direct_message_events": [{
        "type": "message_create",
        "id": "1133674120380657669",
        "created_timestamp": "1559123947602",
        "message_create": {
            "target": {
                "recipient_id": "1102321381"
            },
            "sender_id": "1029741837916012544",
            "message_data": {
                "text": "Good morning ☀️☕",
                "entities": {
                    "hashtags": [],
                    "symbols": [],
                    "user_mentions": [],
                    "urls": []
                }
            }
        }
    }],
    "users": {
        "1102321381": {
            "id": "1102321381",
            "created_timestamp": "1358552743000",
            "name": "Aurelia Specker",
            "screen_name": "AureliaSpecker",
            "location": "London, England",
            "description": "partner engineer @TwitterUK 👩‍💻 • instructor @CodeFirstGirls • outdoor enthusiast 🌄",
            "protected": false,
            "verified": false,
            "followers_count": 589,
            "friends_count": 745,
            "statuses_count": 521,
            "profile_image_url": "http://pbs.twimg.com/profile_images/1025560966434443264/vsVPbXiT_normal.jpg",
            "profile_image_url_https": "https://pbs.twimg.com/profile_images/1025560966434443264/vsVPbXiT_normal.jpg"
        },
        "1029741837916012544": {
            "id": "1029741837916012544",
            "created_timestamp": "1534344560624",
            "name": "Aurelia",
            "screen_name": "re_testing",
            "location": "London, England",
            "description": "I like traveling 🐪🌴☀️",
            "protected": false,
            "verified": false,
            "followers_count": 4,
            "friends_count": 10,
            "statuses_count": 52,
            "profile_image_url": "http://pbs.twimg.com/profile_images/1029742489090121728/-83RqB5N_normal.jpg",
            "profile_image_url_https": "https://pbs.twimg.com/profile_images/1029742489090121728/-83RqB5N_normal.jpg"
        }
    }
} {
    "for_user_id": "1102321381",
    "direct_message_indicate_typing_events": [{
        "created_timestamp": "1559123947551",
        "sender_id": "1029741837916012544",
        "target": {
            "recipient_id": "1102321381"
        }
    }],
    "users": {
        "1102321381": {
            "id": "1102321381",
            "created_timestamp": "1358552743000",
            "name": "Aurelia Specker",
            "screen_name": "AureliaSpecker",
            "location": "London, England",
            "description": "partner engineer @TwitterUK 👩‍💻 • instructor @CodeFirstGirls • outdoor enthusiast 🌄",
            "protected": false,
            "verified": false,
            "followers_count": 589,
            "friends_count": 745,
            "statuses_count": 521,
            "profile_image_url": "http://pbs.twimg.com/profile_images/1025560966434443264/vsVPbXiT_normal.jpg",
            "profile_image_url_https": "https://pbs.twimg.com/profile_images/1025560966434443264/vsVPbXiT_normal.jpg"
        },
        "1029741837916012544": {
            "id": "1029741837916012544",
            "created_timestamp": "1534344560624",
            "name": "Aurelia",
            "screen_name": "re_testing",
            "location": "London, England",
            "description": "I like traveling 🐪🌴☀️",
            "protected": false,
            "verified": false,
            "followers_count": 4,
            "friends_count": 10,
            "statuses_count": 52,
            "profile_image_url": "http://pbs.twimg.com/profile_images/1029742489090121728/-83RqB5N_normal.jpg",
            "profile_image_url_https": "https://pbs.twimg.com/profile_images/1029742489090121728/-83RqB5N_normal.jpg"
        }
    }
}

It is not possible to choose to receive certain types of events and exclude others - you'll always receive all event types related to the subscribed user(s). A full list of the event types can be reviewed in the developer documentation.

If you need retries, you can sign up for a paid option to enable them. With retries, if the activity is not delivered and your webhook does not return a successful response, the Twitter server will try sending the activity again. Enterprise customers also have additional redundancy and recovery coverage with the Account Activity Replay API, providing a way to resend activities for a requested period of time in the previous five days.

The more subscriptions you have, the more important capacity planning becomes, to ensure that your systems are capable of receiving a large number of events.

Further resources

Twitter's developer docs contain further resources, including:

Guides
Frequently asked questions
API Reference
Account Activity API reference index
Quick start guide (for enterprise customers)

Join our Developer Community or Tweet us at @TwitterDev if this tutorial inspires you to build anything. Let us know what you think on our feedback channel.

Nevertheless, Aurelia Specker Coded

Aurelia Specker — Thu, 07 Mar 2019 18:31:37 +0000

I started to code because...

I didn't find my previous job challenging enough. I wanted to get a job where I could constantly learn new skills and push myself to achieve new goals.

Coding is challenging - there are so many things to learn and, when you eventually do learn them, they change again (for example, libraries and languages get updated all the time). But coding is also extremely empowering and rewarding. There are so many resources available out there and you can learn to build pretty much anything if you put in the right amount of effort and time. And that addictive feeling when you finally get something to work... priceless!

A year after I started coding, I got a job as a Partner Engineer. I love that my job stretches me and enables me to learn new things every day!

I'm currently working on...

Several things... In January, I committed to the #100DaysOfCode challenge. The idea is that for 100 days, you code for at least one hour a day. I've recently hit the halfway mark and I've already learnt so many things. Initially, I was worried that I wouldn't have the right inspiration to keep this going for 100 days, but I'm actually finding that I want to do too many things and don't have enough time.

I've started my own little projects (such as building an app that tracks Tweet engagement with the Twitter API) and I've taken online courses (for example a Udemy course on using Node.js).

I've also been code pairing with some more experienced developers and have learnt a huge amount from coding with them.

I am also co-teaching the Code First: Girls Python course in London. It's a challenge at times, because explaining something requires of you that you really understand it (and I'm still relatively new to Python myself!). But it's also incredibly fun and so gratifying to inspire other girls to get into coding.

My advice for others who are learning to code...

Get involved in the community of developers. First of all, it's fun and you'll meet some like-minded people that will help you stay motivated during times of self-doubt. You will also learn a lot faster, by talking to other developers about their own experiences. Being part of the community will also bring up new opportunities: finding a new mentor, hearing about a job opening, etc.
Code on a regular basis. We all have busy lives and, sometimes, it can be hard to continue learning when you're coding as a 'hobby on the side'. Make the time. Even if it's one hour every few days, make it a priority. The result is very fulfilling.
Finally, don't be too harsh on yourself. It's really easy to get imposter syndrome, especially when you're surrounded by coders who know so much more than you do. There will be ups and there will be downs. Take things one baby step at a time and never give up - it's worth it & you can totally do it!