DEV Community: Tony Vu

Top 5 things you can learn about remote developer job postings with the new Twitter API

Tony Vu — Thu, 08 Oct 2020 21:52:56 +0000

The code for this tutorial is available on GitHub.

In a previous tutorial, you learned how to get Tweets containing remote developer job listings in real-time using the Twitter API.

In this follow up tutorial, you will learn how to build an app to answer five must-know things about remote developer job openings posted on Twitter in the last seven days including:

How many Tweets about remote developer job openings were posted in the last seven days in total?
What day of the week had the most remote developer jobs Tweeted in the past seven days?
What are the most in-demand programming languages based on those Tweets?
Which Tweet received the most engagement via retweets, likes, replies, and quotes?
What do some of these Tweeted jobs look like?

To answer these questions, you will be building an app that uses the recent search endpoint, one of the first endpoints of the new Twitter API. Armed with answers to these questions, the aspiring job seeker can devise a strategy to optimize their job search and help land their next job!

Setup

To get started here’s what you will need:

You must have an developer account. If you don’t have one already, you can sign up for one. Access is available with active keys and tokens for a developer App that is attached to a Project created in the developer portal.
Node.js
Npm (This is automatically installed with Node. Make sure you have npm 5.2 or higher.)
Npx (Included with npm 5.2 or higher)

First, install Node.js. Check out the Downloads section from Node’s website and download the source code or installer of your choice. Alternatively, if you are running on a Mac you can install the Node package using the Brew package manager

Open a terminal window and bootstrap your React app using create-react-app by using npx.

npx create-react-app remote-dev-jobs-analytics

After create-react-app has finished executing, change to the newly created remote-dev-job-analytics directory and replace the scripts block in your package.json with the following script block in your package.json. These lines will provide a command shortcut to concurrently run your client and server backend code in development or production as needed.

cd remote-dev-jobs-analytics

package.json

  "scripts": {
    "start": "npm run development",
    "development": "NODE_ENV=development concurrently --kill-others \"npm run client\" \"npm run server\"",
    "client": "react-scripts start",
    "server": "nodemon server/server.js",
    "build": "react-scripts build",
    "test": "react-scripts test",
    "eject": "react-scripts eject"
  },

After updating the scripts section, your package.json should now look as follows.

Next, remove all files within the src/ subdirectory.

rm src/*

Then, create a new file within the src/ subdirectory called index.js. The code for this file will be as follows.

import React from "react";
import ReactDOM from "react-dom";
import App from "./components/App";

ReactDOM.render(<App />, document.querySelector("#root"));

Credentials

Connecting to the recent search endpoint requires you to authenticate using a bearer token from your app in the Twitter developer portal. To utilize your bearer token, you will need to have the following environment variable set. You can do so by issuing the following command in your terminal window assuming you are using bash as your shell. Replace , including the left and right angle brackets, with your bearer token.

export TWITTER_BEARER_TOKEN=<YOUR BEARER TOKEN HERE>

Server-Side Code

First, you will need to get started with implementing the Node server, which will be responsible for making the actual requests to the Twitter API. This Node server will serve as a proxy between your browser-based React client and the Twitter API. On your Node server, you will need to create API endpoints that connect to the recent search endpoint. In turn, requests from your React client will be proxied through to your local Node server.
Before you go any further, cd to the project root directory and install the following dependencies

npm install concurrently express body-parser util request http path http-proxy-middleware axios react-router-dom react-twitter-embed react-chartjs-2

Next, while still within your project root directory, create a new subdirectory called “server” and a new file within that subdirectory called “server.js”.

mkdir server
touch server/server.js

This source code file will contain all of your backend logic for connecting to and receiving Tweets from the recent search endpoint. The contents of your server.js file will be as follows.

server.js

const axios = require("axios");
const express = require("express");
const bodyParser = require("body-parser");
const moment = require("moment");

const app = express();
let port = process.env.PORT || 3000;

const BEARER_TOKEN = process.env.TWITTER_BEARER_TOKEN;

app.use(bodyParser.json());
app.use(bodyParser.urlencoded({ extended: true }));

const searchURL = "https://api.twitter.com/2/tweets/search/recent";
const query =
  '(developer OR software) remote (context:66.961961812492148736 OR context:66.850073441055133696) -is:retweet -"business developer"';
const maxResults = 100;

const requestConfig = {
  headers: {
    Authorization: `Bearer ${BEARER_TOKEN}`,
  },
  params: {
    max_results: maxResults,
    query: query,
    "tweet.fields": "context_annotations,created_at,public_metrics",
  },
};

const authMessage = {
  title: "Could not authenticate",
  detail: `Please make sure your bearer token is correct. 
      If using Glitch, remix this app and add it to the .env file`,
  type: "https://developer.twitter.com/en/docs/authentication",
};

app.get("/api/search/recent", async (req, res) => {
  if (!BEARER_TOKEN) {
    res.status(401).send(authMessage);
  }

  try {
    const response = await getSearchResults();
    res.send(response);
  } catch (e) {
    console.log(e);
  }
});

const getSearchResults = async (config = requestConfig) => {
  try {
    const response = await axios.get(searchURL, config);
    return response.data;
  } catch (e) {
    console.log(e);
  }
};

const getAllTweets = async () => {
  let response = await getSearchResults();
  let tweets = [];

  while (response.meta.next_token) {
    let config = {
      ...requestConfig,
      params: {
        ...requestConfig.params,
        next_token: response.meta.next_token,
      },
    };

    response = await getSearchResults(config);
    tweets = tweets.concat(response.data);
  }

  return tweets;
};

const getCount = async () => {
  let response = await getSearchResults();
  let resultCount = response.meta.result_count;

  while (response.meta.next_token) {
    let config = {
      ...requestConfig,
      params: {
        ...requestConfig.params,
        next_token: response.meta.next_token,
      },
    };

    response = await getSearchResults(config);
    resultCount = resultCount + response.meta.result_count;
  }

  return resultCount;
};

const countsByDay = async () => {
  let tweets = await getAllTweets();

  return tweets.reduce(
    (counts, tweet) => ({
      ...counts,
      [moment(tweet.created_at).format("ddd - MM/DD")]:
        (counts[moment(tweet.created_at).format("ddd - MM/DD")] || 0) + 1,
    }),
    {}
  );
};

const countsByLanguage = async () => {
  let counts = {};

  const languages = [
    "javascript",
    "JavaScript",
    "android",
    "frontend",
    "ios",
    "backend",
    "node",
    "nodejs",
    "python",
    "react",
    "scala",
    "c#",
    "rails",
    "ruby",
    "php",
    "java",
    "blockchain",
    ".net",
    "sql",
    "java",
    "php",
    "golang",
    "go",
    "wordpress",
  ];

  const tweets = await getAllTweets();

  for (tweet of tweets) {
    for (language of languages) {
      if (
        tweet.text.includes(language) ||
        tweet.text.includes(language.toUpperCase())
      ) {
        counts[language] = (counts[language] || 0) + 1;
      }
    }
  }

  if (counts["JavaScript"]) {
    counts["javascript"] += counts["JavaScript"];
    delete counts.JavaScript;
  }

  if (counts["node"]) {
    counts["nodejs"] += counts["node"];
    delete counts.node;
  }

  if (counts["golang"]) {
    counts["go"] += counts["golang"];
    delete counts.node;
  }

  return counts;
};

const sortCounts = (counts, keyName = "name") => {
  let sortedCounts = Object.keys(counts).map((language) => ({
    [keyName]: language,
    total: counts[language],
  }));

  sortedCounts.sort((a, b) => {
    return b.total - a.total;
  });

  return sortedCounts;
};

app.get("/api/search/recent/top", async (req, res) => {
  if (!BEARER_TOKEN) {
    res.status(401).send(authMessage);
  }

  const tweets = await getAllTweets();
  let tweetsByEngagement = {};
  for (tweet of tweets) {
    const total_engagement = Object.values(tweet.public_metrics).reduce(
      (total_engagement, public_metric) => total_engagement + public_metric
    );
    tweetsByEngagement[tweet.id] = total_engagement;
  }

  res.send({ result: sortCounts(tweetsByEngagement, "id")[0] });
});

app.get("/api/search/recent/count", async (req, res) => {
  if (!BEARER_TOKEN) {
    res.status(401).send(authMessage);
  }

  const results =
    req.query.group === "day" ? await countsByDay() : await getCount();

  res.send({ count: results });
});

app.get("/api/search/recent/language", async (req, res) => {
  if (!BEARER_TOKEN) {
    res.status(401).send(authMessage);
  }

  try {
    let results = await countsByLanguage();
    results = sortCounts(results);
    res.send({ count: results.slice(0, 10) });
  } catch (e) {
    console.log(e);
  }
});

if (process.env.NODE_ENV === "production") {
  app.use(express.static(path.join(__dirname, "../build")));
  app.get("*", (request, res) => {
    res.sendFile(path.join(__dirname, ".../build", "index.html"));
  });
} else {
  port = 3001;
}

app.listen(port, () => console.log(`Listening on port ${port}`));

In the server-side code the following endpoints are being built

The /api/search/recent/count endpoint, by default, returns the total number of jobs for the last seven days. Passing in the group query parameter, with one of the following values will display one of the following
- group=day will return the number of jobs broken down by day in the last seven days-
- group=language will return the number of jobs broken down by programming language mentioned in the Tweet text, if present, in the last seven days
The /api/search/recent/top endpoint returns the Tweet receiving the most engagement. This endpoint uses the public metrics field to return likes, favorites, retweets, and quotes in the Tweet payload. Using these statistics, you can determine which Tweets are receiving the most engagement or attention.
The /api/search/recent endpoint returns the Tweets matching the following search query

(developer OR software) remote (context:66.961961812492148736 OR context:66.850073441055133696) -is:retweet -"business developer”

This search query instructs the recent search endpoint to match on Tweets containing the keywords “developer” or “software” and with the keyword “remote” present in the Tweet text. Additionally, this search query uses the “context” operator to match on Tweets containing specific domain and entity names.

      "context_annotations": [
        {
          "domain": {
            "id": "65",
            "name": "Interests and Hobbies Vertical",
            "description": "Top level interests and hobbies groupings, like Food or Travel"
          },
          "entity": {
            "id": "847544972781826048",
            "name": "Careers",
            "description": "Careers"
          }
        },
        {
          "domain": {
            "id": "66",
            "name": "Interests and Hobbies Category",
            "description": "A grouping of interests and hobbies entities, like Novelty Food or Destinations"
          },
          "entity": {
            "id": "961961812492148736",
            "name": "Recruitment",
            "description": "Recruitment"
          }
        }

The context operator follows the format context:.. As seen in the example payload above, the domain ids 65 and 66 represent the “Interests and Hobbies Category”. The entity ID 961961812492148736 represents the “Recruitment” entity and the entity ID 847544972781826048 represents the “Career” entity. For a complete list of domains, the Tweet Annotations documenation contains a table with 50+ domain names.

Finally, the operators “-is:retweet” and ”-business developer” can be used to exclude retweets from the search results and to exclude any Tweets containing “business developer”. Retweets are excluded to avoid duplicates in the search results and Tweets containing the terms “business developer” are excluded since that is irrelevant.

Client-Side Code

The next step is to work on the following React components to display the information mentioned above.

App.js - The parent component that will, in turn, render all other components

Tweet.js - Displays a Tweet containing a job posting

Day.js - Displays a bar chart of the number of Tweets posted by day for the last seven days

Top.js - Renders the Tweet that has received the most engagement in the past seven days

Tweets.js - Placeholder component that displays the top ten programming languages posted, the Top.js component, the Day.js component, and renders multiple Tweet.js components

Spinner.js - Renders a loading indicator for any pending API calls

Now you will need to get started with creating the React components. Under your /src subdirectory, create a directory called “components”. The source code files above will be stored in this new directory. First, create the parent most component of the application. This component will be responsible for rendering all other components.

App.js

import React from "react";
import { BrowserRouter, Route } from "react-router-dom";

import Tweets from "./Tweets";

const App = () => {
  return (
    <div className="ui container">
      <div className="introduction"></div>

      <h1 className="ui header">
        <img
          className="ui image"
          src="/Twitter_Logo_Blue.png"
          alt="Twitter Logo"
        />
        <div className="content">
          Remote Developer Job Analytics
          <div className="sub header">Powered by Twitter data</div>
        </div>
      </h1>

      <div className="ui grid">
        <BrowserRouter>
          <Route exact path="/" component={Tweets} />
        </BrowserRouter>
      </div>
    </div>
  );
};

export default App;

Next, create the parent component for rendering a sample of Tweets containing job postings.

Tweets.js

import React, { useEffect, useState } from "react";
import axios from "axios";
import Tweet from "./Tweet";
import Top from "./Top";
import Day from "./Day";
import Spinner from "./Spinner";

const initialState = {
  tweets: [],
};

const Tweets = () => {
  const [tweets, setTweets] = useState([]);
  const [tweetCount, setTweetCount] = useState(0);
  const [topTweetId, setTopTweetId] = useState(null);
  const [error, setError] = useState(null);
  const [isLoading, setIsLoading] = useState(false);

  useEffect(() => {
    getTweets();
    getTweetCount();
    getTopTweetId();
  }, []);

  const getTweets = async () => {
    try {
      setIsLoading(true);
      const response = await axios.get("/api/search/recent");
      setTweets(response.data.data);
      setIsLoading(false);
    } catch (e) {
      setError(e.response.data);
      setIsLoading(false);
    }
  };
  const getTweetCount = async () => {
    try {
      const response = await axios.get("/api/search/recent/count");
      console.log(response);
      setTweetCount(response.data.count);
    } catch (e) {
      setError(e.response.data);
      setIsLoading(false);
    }
  };

  const getTopTweetId = async () => {
    const response = await axios.get("/api/search/recent/top");
    setTopTweetId(response.data.result.id);
  };

  const errors = () => {
    if (error) {
      return (
        <div className="sixteen wide column">
          <div className="ui message negative">
            <div className="header">{error.title}</div>
            <p key={error.detail}>{error.detail}</p>
            <em>
              See
              <a href={error.type} target="_blank" rel="noopener noreferrer">
                {" "}
                Twitter documentation{" "}
              </a>
              for further details.
            </em>
          </div>
        </div>
      );
    }
  };

  const dashboard = () => {
    if (!isLoading) {
      if (!error) {
        return (
          <React.Fragment>
            <div className="sixteen wide column">
              <div className="ui segment">
                <div className="ui header center aligned ">
                  Total number of Tweets
                </div>
                <div className="ui header center aligned ">{tweetCount}</div>
              </div>
            </div>
            <div className="eight wide column">
              <div className="ui segment">
                <Top />
              </div>
            </div>
            <div className="eight wide column">
              <div className="ui segment">
                <Day />
              </div>
            </div>
            <div className="eight wide column">
              <div className="ui header">Top Tweet</div>
              <Tweet key={topTweetId} id={topTweetId} />
            </div>
            <div className="eight wide column">
              <div className="ui basic segment">
                <div className="ui header">Recent Tweets</div>
                {tweets.map((tweet) => (
                  <Tweet key={tweet.id} id={tweet.id} />
                ))}
              </div>
            </div>
          </React.Fragment>
        );
      }
    } else {
      return <Spinner />;
    }
  };

  return (
    <React.Fragment>
      {errors()}
      {dashboard()}
    </React.Fragment>
  );
};

export default Tweets;

Next, create the component for rendering the Tweet receiving the most engagement.

Top.js

import React, { useEffect, useState } from "react";
import axios from "axios";
import Spinner from "./Spinner";

const Top = () => {
  const [countByLanguage, setCountByLanguage] = useState([]);
  const [isLoading, setIsLoading] = useState(false);

  const style = {
    fontSize: "17px",
  };

  useEffect(() => {
    getTopLanguages();
  }, []);

  const getTopLanguages = async () => {
    setIsLoading(true);
    const response = await axios.get("/api/search/recent/language");
    setCountByLanguage(response.data.count);
    setIsLoading(false);
  };

  const capitalize = (word) => {
    const first_letter = word.slice(0, 1).toUpperCase();
    return first_letter + word.slice(1);
  };

  const displayTopLanuguages = () => {
    {
      if (!isLoading) {
        return countByLanguage.map((count, i) => (
          <div style={style} className="item">
            {i + 1}. {capitalize(count.name)}
          </div>
        ));
      } else {
        return <Spinner />;
      }
    }
  };

  return (
    <React.Fragment>
      <div className="ui header">Top Programming Languages</div>
      <ul className="ui relaxed list"> {displayTopLanuguages()}</ul>
    </React.Fragment>
  );
};

export default Top;

Next, create the component for rendering an individual Tweet.

Tweet.js

import React from "react";
import { TwitterTweetEmbed } from "react-twitter-embed";

const Tweet = ({ id }) => {
  const options = {
    cards: "hidden",
    align: "left",
    width: "550",
    conversation: "none",
  };

  return <TwitterTweetEmbed options={options} tweetId={id} />;
};

export default Tweet;

Finally, create a component to display a loading indicator during any pending API calls.

import React from "react";

const Spinner = () => {
  return (
    <div>
      <div className="ui active centered large inline loader">
        <img
          className="ui image"
          src="/Twitter_Logo_Blue.png"
          alt="Twitter Logo"
        />
      </div>
    </div>
  );
};

export default Spinner;

Proxy Setup

The final step is to proxy requests from your client to your backend server. To do this, from within your src/ directory, create a new file called “setupProxy.js” and add the following code.

setupProxy.js

const { createProxyMiddleware } = require("http-proxy-middleware");

// This proxy redirects requests to /api endpoints to
// the Express server running on port 3001.
module.exports = function (app) {
  app.use(
    ["/api"],
    createProxyMiddleware({
      target: "http://localhost:3001",
    })
  );
};

You can now start up both the server and client by going to the project root directory and typing the following:

npm start

After this command completes, your default web browser should automatically launch and navigate to http://localhost:3000 where you can see a sample of Tweets containing job postings from the last seven days along with the information displayed to answer all of the questions raised in the introduction.

Conclusion

Using the recent search endpoint, you created an app to answer some questions about remote developer job postings from the last seven days. Answers to these questions could be very helpful to the aspiring developer or a developer that already has a job and wants to be more strategic about how they should approach their search for their next one.

Have you found interesting ways to extend this app? Follow me on Twitter and send me a Tweet to let me know. I used several libraries beyond the Twitter API to make this tutorial, but you may have different needs and requirements and should evaluate whether those tools are right for you.

Stream Tweets in real-time with v2 of the Twitter API

Tony Vu — Tue, 22 Sep 2020 16:34:34 +0000

Introduction

The Twitter API allows you to stream public Tweets from the platform in real-time so that you can display them and basic metrics about them.

In this tutorial, you will learn how to:

Set an objective for your work
Plan and prepare to process the needed data
Connect and authenticate to the appropriate API endpoint
Handle errors and disconnections
Display Tweets and basic metrics about them

Prerequisites

You must have a developer account. Access is available with active keys and tokens for a developer App that is attached to a Project created in the developer portal.
A bearer token from your App in the Twitter developer portal

Steps to consider

Step 1: Set an objective for your work

First, define what you want to accomplish and decide on the data you need to meet that objective.

Staying informed on a topic of interest: For example, you would like to stay current on updates, news, and events about Twitter’s API.
Detecting current trends: You have a hard time detecting current trends and would like to extract signals about what people are discussing in the world.

Step 2: Plan and prepare to process the needed data

Staying informed on a topic of interest

For the example of staying informed on a topic of interest, you will need to decide on what type of data you need and how to go about getting it in real time.

For example, in continuing with the example on keeping current with updates to the Twitter API, you will need to make sure you get Tweets from the @TwitterDev and the @TwitterAPI accounts the moment they are posted. You will also only want Tweets that contain links so that you can read up on any further context provided with a Tweet.

To get this kind of data in real time, you can use the filtered stream endpoints. The filtered stream endpoints requires you to define filtering criteria in order for it to know what kind of Tweets to send to you.

After defining your filtering criteria, you will need to apply this to the filtered stream endpoints.

Filtering criteria are applied to the filtered stream endpoints in the form of rules. Rules allow you to narrow down to only the Tweets you are looking for by using a set of operators.

While creating a filter it is often helpful to think about what type of data you don’t want to receive and work backward from there. You will need to make sure you have the right keyword and adjust your rule to prevent unwanted Tweets from entering into your stream. For examples of how you can build more complex rules, see our documentation on building a rule.

Based on the filtering criteria just defined, you can create a rule containing the following operators.

from:
has:links

The “from:” operator matches any Tweet from a specific user and the “has:links” operator matches Tweets containing links in the Tweet body. Together, these operators form the following rule which instructs the filtered stream endpoints to filter for Tweets from the accounts @twitterdev or @twitterapi that contain links.

(from:twitterdev OR from:twitterapi) has:links

To add this rule, issue a POST request to the filtered stream rules endpoint with an added payload as an array of rules and operators. The example below shows how you can use cURL to do so. To authenticate, replace $BEARER_TOKEN (including the dollar sign) with the Bearer Token from your App in the developer portal

curl -X POST 'https://api.twitter.com/2/tweets/search/stream/rules' \ -H "Content-type: application/json" \ -H "Authorization: Bearer $BEARER_TOKEN" -d \ '{  "add": [    {"value": "(from:twitterdev OR from:twitterapi) has:links"}  ] }'`

You can also POST to the rules endpoint using one of our code samples

Detecting current trends

For the example of detecting current trends, you will need to consider what kind of data you need and how much data you need in order to perform your analysis. For example, you may need to read and analyze a broad, but relatively manageable set of Tweets. This data should also be current or relatively current. While the text of a Tweet will likely be important for your analysis, there are other data elements you will need to consider. Within the text of a Tweet itself, you will also need to consider if you need hashtags, mentions, or if you are looking for certain keywords.

Beyond the text of a Tweet, a Tweet contains several different fields of data so you will want to decide which fields would best aid your trend detection needs. Based on the fields you decide on, you can then, for example, perform some basic frequency analysis on keywords, hashtags, mentions, or Tweet annotations made available in a Tweet payload. By default, you will only get back the id and text of each Tweet. If you would like additional data returned about each Tweet, you can add additional fields and expansions to your request URL.

After formulating your data requirements, you will need to figure out how to go about getting this data. To get the data for this analysis, the 1% random sample of public Tweets provided by the sampled stream endpoint can meet this need since it provides a small subset of data relative to the total amount of public Tweets. Additionally, the data is sent to you in real time as it happens, which will meet the requirement of the data being current.

Step 3: Connect and authenticate to the appropriate endpoint

Staying informed on a topic of interest

Once you have defined your data and set your filtering criteria using rules, you can connect to the filtered stream streaming endpoint to start getting data. The example below shows how you can use cURL to do so. To authenticate, replace $BEARER_TOKEN with the Bearer Token from your App in the developer portal.

curl -X GET -H "Authorization: Bearer $BEARER_TOKEN" "https://api.twitter.com/2/tweets/search/stream"

You can also POST to the streaming endpoint using one of our code samples

Detecting current trends

To connect to the sampled stream endpoint, issue a GET request to the endpoint. The example below shows how you can use cURL to do so. To authenticate, replace $BEARER_TOKEN with the Bearer Token from your App in the developer portal.

curl -X GET -H "Authorization: Bearer $BEARER_TOKEN" "https://api.twitter.com/2/tweets/sample/stream"

You can also connect to the sampled stream endpoint using one of our code samples

Step 4: Handle errors and disconnections

At any time while you are connected to either the filtered stream endpoint or the sampled stream endpoint, you may be disconnected either voluntarily or involuntarily. Voluntary disconnects occur when you independently terminate the connection to the streaming endpoint, whether because your code actively closes the connection, or where network settings terminate the connection. Involuntary disconnections occur when either streaming endpoint actively disconnects your App from the stream. These types of disconnections occur due to one of the following reasons.

Full Buffer: Your App is not reading the data fast enough, or a network bottleneck is slowing data flow.
Too many connections: Your App established too many simultaneous connections to the data stream. When this occurs, the filtered stream endpoint will wait 1 minute, and then disconnect the most recently established connection if the limit is still being exceeded.
Server maintenance: The Twitter team deployed a change or update to the system servers

You should build the proper logic into your code to automatically reconnect in the event any of these involuntary disconnections occur. Additionally, if you only need Tweets for a limited time, you may want to consider building in the proper timing in you code to disconnect from the streaming endpoint after your desired period of time has passed.

To avoid full buffer errors due to the high volume of Tweets that you will receive, your code should not do any real processing work as it reads the stream. Read the stream and then hand the activity to another thread or process to do your processing asynchronously. This processing can include performing frequency analysis or any type of heavy processing work.

For an example of how to build in reconnection logic to your application, check out the sample apps below that use the filtered stream endpoints.

Step 5: Display Tweets and basic metrics about them

Once you are reliably getting your data, you will then need to think about what you would like to do with them. One common option is to display some of them on a web page or dashboard, along with some basic metrics to describe what’s happening over time.

You should keep in mind that id and text will always be populated since they are the basic building blocks of a Tweet, but other fields may not always be present. For instance, not all Tweets will contain image attachments. You will want to consider what fields the data you are looking for is located in, which you would like to display, and which may not always be present, dependent upon the nature of the Tweet. Lastly, you will also want to reference and comply with Twitter’s Display Requirements when displaying Tweets. Using embedded Tweets can help meet these requirements.

If you would like to keep a running total of the number of Tweets read over time and display this in a chart, you should keep in mind that the amount of Tweets can fluctuate based on how much more or less people are Tweeting at a given time. You may want to create a visualization to display these fluctuations over time. Additionally, as you are ingesting and displaying these Tweets, you may want to temporarily save them to a data store and display the most recent ones or organize them on a page by page basis due to the amount of Tweets that can come in at once and your limited screen real estate. You can also consider displaying Tweets based on their organic performance metrics such as the ones receiving the most retweets, replies, or likes.

Finally, consider that Tweets can be related to one another either because someone retweets a Tweet or replies to a Tweet. If helpful towards your objective, you may want to display Tweets in a way to indicate this relationship, if any.

Next steps

Get started building with our filtered stream quick start guide.
Check out the filtered stream API reference to learn more about what’s available.
Get started building with our sampled stream quick start guide.
Learn more about the sampled stream endpoint.
Check out the sampled stream API reference to learn more about what’s available.
Get inspired by reading our other tutorials.

Building an app to stream Tweets in real-time using the Twitter API

Tony Vu — Wed, 12 Aug 2020 17:11:30 +0000

The code for this tutorial is available on GitHub. You can also check out the live demo of the app on Glitch

This tutorial will walk you through how to build your own real-time Tweet streaming app using the filtered stream endpoints and Tweet annotations to listen for Tweets based on your own topics of interest.

After building the app, you will learn about how it can be applied to some real-life examples to inspire you to get started such as.

Discovering new music videos: Imagine a dashboard that lets you see the music videos being shared across Twitter
Finding remote developer job openings: Imagine creating a remote developer job listings app with remote developer job openings being shared across Twitter
Learning about personal finance and savings: Surface public conversations about personal finance and savings happening on Twitter

Setup

To get started here’s what you will need

You must have a developer account. If you don’t have one already, you can apply for one. Access is available with active keys and tokens for a developer App that is attached to a Project created in the developer portal.
Node.js
Npm (This is automatically installed with Node. Make sure you have npm 5.2 or higher.)
Npx (Included with npm 5.2 or higher)

Open a terminal window and bootstrap your React app using create-react-app by using npx.

npx create-react-app real-time-tweet-streamer

After create-react-app has finished executing, change to the newly created real-time-tweet-streamer directory and replace the scripts block in your package.json with the following script block in your package.json. These lines will provide a command shortcut to concurrently run your client and server backend code in development or production as needed.

cd real-time-tweet-streamer

package.json

"scripts": {
  "start": "npm run development",
  "development": "NODE_ENV=development concurrently --kill-others \"npm run client\" \"npm run server\"",
  "production": "npm run build && NODE_ENV=production npm run server",
  "client": "react-scripts start",
  "server": "node server/server.js",
  "build": "react-scripts build",
  "test": "react-scripts test",
  "eject": "react-scripts eject"
},

After updating the scripts section, your package.json should now look as follows.

{
  "name": "real-time-tweet-streamer",
  "version": "0.1.0",
  "private": true,
  "dependencies": {
    "@testing-library/jest-dom": "^4.2.4",
    "@testing-library/react": "^9.3.2",
    "@testing-library/user-event": "^7.1.2",
    "react": "^16.13.1",
    "react-dom": "^16.13.1",
    "react-scripts": "3.4.1"
  },
  "scripts": {
    "start": "npm run development",
    "development": "NODE_ENV=development concurrently --kill-others \"npm run client\" \"npm run server\"",
    "production": "npm run build && NODE_ENV=production npm run server",
    "client": "react-scripts start",
    "server": "node server/server.js",
    "build": "react-scripts build",
    "test": "react-scripts test",
    "eject": "react-scripts eject"
  },
  "eslintConfig": {
    "extends": "react-app"
  },
  "browserslist": {
    "production": [
      ">0.2%",
      "not dead",
      "not op_mini all"
    ],
    "development": [
      "last 1 chrome version",
      "last 1 firefox version",
      "last 1 safari version"
    ]
  }
}

Next, remove all files within the src/ subdirectory.

rm src/*

Then, create a new file within the src/ subdirectory called index.js. The code for this file will be as follows.

import React from "react";
import ReactDOM from "react-dom";
import App from "./components/App";

ReactDOM.render(<App />, document.querySelector("#root"));

Credentials

Connecting to the filtered stream endpoints requires you to authenticate using a bearer token from your app in the Twitter developer portal. To utilize your bearer token, you will need to have the following environment variable set. You can do so by issuing the following command in your terminal window assuming you are using bash as your shell. Replace <YOUR BEARER TOKEN HERE>, including the left and right angle brackets, with your bearer token.

export TWITTER_BEARER_TOKEN=<YOUR BEARER TOKEN HERE>

Server Side Code

First, you will need to get started with implementing the Node server, which will be responsible for making the actual requests to the Twitter API. This Node server will serve as a proxy between your browser-based React client and the Twitter API. On your Node server, you will need to create API endpoints that connect to the filtered stream endpoints. In turn, requests from your React client will be proxied through to your local Node server.

Before you go any further, cd to the project root directory and install the following dependencies

npm install concurrently express body-parser util request http socket.io path http-proxy-middleware request react-router-dom axios socket.io-client react-twitter-embed

Next, while still within your project root directory, create a new subdirectory called “server” and a new file within that subdirectory called “server.js”.

mkdir server
touch server/server.js

This source code file will contain all of your backend logic for connecting to and receiving Tweets from the filtered stream endpoint. The contents of your server.js file will be as follows.

server.js

const express = require("express");
const bodyParser = require("body-parser");
const util = require("util");
const request = require("request");
const path = require("path");
const socketIo = require("socket.io");
const http = require("http");

const app = express();
let port = process.env.PORT || 3000;
const post = util.promisify(request.post);
const get = util.promisify(request.get);

app.use(bodyParser.json());
app.use(bodyParser.urlencoded({ extended: true }));

const server = http.createServer(app);
const io = socketIo(server);

const BEARER_TOKEN = process.env.TWITTER_BEARER_TOKEN;

let timeout = 0;

const streamURL = new URL(
  "https://api.twitter.com/2/tweets/search/stream?tweet.fields=context_annotations&expansions=author_id"
);

const rulesURL = new URL(
  "https://api.twitter.com/2/tweets/search/stream/rules"
);

const errorMessage = {
  title: "Please Wait",
  detail: "Waiting for new Tweets to be posted...",
};

const authMessage = {
  title: "Could not authenticate",
  details: [
    `Please make sure your bearer token is correct. 
      If using Glitch, remix this app and add it to the .env file`,
  ],
  type: "https://developer.twitter.com/en/docs/authentication",
};

const sleep = async (delay) => {
  return new Promise((resolve) => setTimeout(() => resolve(true), delay));
};

app.get("/api/rules", async (req, res) => {
  if (!BEARER_TOKEN) {
    res.status(400).send(authMessage);
  }

  const token = BEARER_TOKEN;
  const requestConfig = {
    url: rulesURL,
    auth: {
      bearer: token,
    },
    json: true,
  };

  try {
    const response = await get(requestConfig);

    if (response.statusCode !== 200) {
      if (response.statusCode === 403) {
        res.status(403).send(response.body);
      } else {
        throw new Error(response.body.error.message);
      }
    }

    res.send(response);
  } catch (e) {
    res.send(e);
  }
});

app.post("/api/rules", async (req, res) => {
  if (!BEARER_TOKEN) {
    res.status(400).send(authMessage);
  }

  const token = BEARER_TOKEN;
  const requestConfig = {
    url: rulesURL,
    auth: {
      bearer: token,
    },
    json: req.body,
  };

  try {
    const response = await post(requestConfig);

    if (response.statusCode === 200 || response.statusCode === 201) {
      res.send(response);
    } else {
      throw new Error(response);
    }
  } catch (e) {
    res.send(e);
  }
});

const streamTweets = (socket, token) => {
  let stream;

  const config = {
    url: streamURL,
    auth: {
      bearer: token,
    },
    timeout: 31000,
  };

  try {
    const stream = request.get(config);

    stream
      .on("data", (data) => {
        try {
          const json = JSON.parse(data);
          if (json.connection_issue) {
            socket.emit("error", json);
            reconnect(stream, socket, token);
          } else {
            if (json.data) {
              socket.emit("tweet", json);
            } else {
              socket.emit("authError", json);
            }
          }
        } catch (e) {
          socket.emit("heartbeat");
        }
      })
      .on("error", (error) => {
        // Connection timed out
        socket.emit("error", errorMessage);
        reconnect(stream, socket, token);
      });
  } catch (e) {
    socket.emit("authError", authMessage);
  }
};

const reconnect = async (stream, socket, token) => {
  timeout++;
  stream.abort();
  await sleep(2 ** timeout * 1000);
  streamTweets(socket, token);
};

io.on("connection", async (socket) => {
  try {
    const token = BEARER_TOKEN;
    io.emit("connect", "Client connected");
    const stream = streamTweets(io, token);
  } catch (e) {
    io.emit("authError", authMessage);
  }
});

console.log("NODE_ENV is", process.env.NODE_ENV);

if (process.env.NODE_ENV === "production") {
  app.use(express.static(path.join(__dirname, "../build")));
  app.get("*", (request, res) => {
    res.sendFile(path.join(__dirname, "../build", "index.html"));
  });
} else {
  port = 3001;
}

server.listen(port, () => console.log(`Listening on port ${port}`));

Filtering Tweets on the filtered stream endpoint using rules

Filtered stream has two endpoints, a streaming endpoint, to receive data and a rules endpoint that is used to create and delete rules. The filtered stream endpoints require you to define search queries called rules, in order for it to know what kind of Tweets to send to you. Rules allow you to narrow down to only the Tweets you are looking for by using a set of operators. You will see some example use cases and corresponding rules you can use later once you finish building the app.

Client-Side Code

The next step is to work on the following React components

App.js - The parent component that be will, in turn, render all other components
NavBar.js - Displays the navigation bar for navigating between the Tweet feed and managing rules
Tweet.js - Displays a Tweet on the page
TweetFeed.js - Renders multiple Tweet components at once in a “feed” like fashion
Rule.js - Renders an individual rule on your stream
RuleList.js - Renders multiple Rule components and displays an input field to add a rule
ErrorMessage.js - Renders any status or error messages to the screen
Spinner.js - Renders a loading indicator for any pending API calls

To style all of your components, you will be using Semantic UI. Include the CSS for Semantic UI in your project by adding the line below to the <head> tag in your index.html file located in the ~/real-time-tweet-streamer/public directory.

<link rel="stylesheet" href="https://cdnjs.cloudflare.com/ajax/libs/semantic-ui/2.4.1/semantic.min.css" />

App.js

import React from "react";
import { BrowserRouter, Route } from "react-router-dom";

import Navbar from "./Navbar";
import TweetFeed from "./TweetFeed";
import RuleList from "./RuleList";

class App extends React.Component {
  render() {
    return (
      <div className="ui container">
        <div className="introduction"></div>

        <h1 className="ui header">
          <div className="content">
            Real Time Tweet Streamer
            <div className="sub header">Powered by Twitter data</div>
          </div>
        </h1>

        <div className="ui container">
          <BrowserRouter>
            <Navbar />
            <Route exact path="/" component={RuleList} />
            <Route exact path="/rules" component={RuleList} />
            <Route exact path="/tweets" component={TweetFeed} />
          </BrowserRouter>
        </div>
      </div>
    );
  }
}

export default App;

Next, create the component for the navigation bar.

Navbar.js

import React from "react";
import { NavLink } from "react-router-dom";

const Navbar = () => {
  return (
    <div className="ui two item menu">
      <NavLink to="/tweets" className="item" target="_blank">
        New Tweets
      </NavLink>
      <NavLink to="/rules" className="item" target="_blank">
        Manage Rules
      </NavLink>
    </div>
  );
};

export default Navbar;

Next, create the parent component for rendering all job listings.

TweetFeed.js

import React, { useEffect, useReducer } from "react";
import Tweet from "./Tweet";
import socketIOClient from "socket.io-client";
import ErrorMessage from "./ErrorMessage";
import Spinner from "./Spinner";

const reducer = (state, action) => {
  switch (action.type) {
    case "add_tweet":
      return {
        ...state,
        tweets: [action.payload, ...state.tweets],
        error: null,
        isWaiting: false,
        errors: [],
      };
    case "show_error":
      return { ...state, error: action.payload, isWaiting: false };
    case "add_errors":
      return { ...state, errors: action.payload, isWaiting: false };
    case "update_waiting":
      return { ...state, error: null, isWaiting: true };
    default:
      return state;
  }
};

const TweetFeed = () => {
  const initialState = {
    tweets: [],
    error: {},
    isWaiting: true,
  };

  const [state, dispatch] = useReducer(reducer, initialState);
  const { tweets, error, isWaiting } = state;

  const streamTweets = () => {
    let socket;

    if (process.env.NODE_ENV === "development") {
      socket = socketIOClient("http://localhost:3001/");
    } else {
      socket = socketIOClient("/");
    }

    socket.on("connect", () => {});
    socket.on("tweet", (json) => {
      if (json.data) {
        dispatch({ type: "add_tweet", payload: json });
      }
    });
    socket.on("heartbeat", (data) => {
      dispatch({ type: "update_waiting" });
    });
    socket.on("error", (data) => {
      dispatch({ type: "show_error", payload: data });
    });
    socket.on("authError", (data) => {
      console.log("data =>", data);
      dispatch({ type: "add_errors", payload: [data] });
    });
  };

  const reconnectMessage = () => {
    const message = {
      title: "Reconnecting",
      detail: "Please wait while we reconnect to the stream.",
    };

    if (error && error.detail) {
      return (
        <div>
          <ErrorMessage key={error.title} error={error} styleType="warning" />
          <ErrorMessage
            key={message.title}
            error={message}
            styleType="success"
          />
          <Spinner />
        </div>
      );
    }
  };

  const errorMessage = () => {
    const { errors } = state;

    if (errors && errors.length > 0) {
      return errors.map((error) => (
        <ErrorMessage key={error.title} error={error} styleType="negative" />
      ));
    }
  };

  const waitingMessage = () => {
    const message = {
      title: "Still working",
      detail: "Waiting for new Tweets to be posted",
    };

    if (isWaiting) {
      return (
        <React.Fragment>
          <div>
            <ErrorMessage
              key={message.title}
              error={message}
              styleType="success"
            />
          </div>
          <Spinner />
        </React.Fragment>
      );
    }
  };

  useEffect(() => {
    streamTweets();
  }, []);

  const showTweets = () => {
    if (tweets.length > 0) {
      return (
        <React.Fragment>
          {tweets.map((tweet) => (
            <Tweet key={tweet.data.id} json={tweet} />
          ))}
        </React.Fragment>
      );
    }
  };

  return (
    <div>
      {reconnectMessage()}
      {errorMessage()}
      {waitingMessage()}
      {showTweets()}
    </div>
  );
};

export default TweetFeed;

Next, create the child component for the previous component that renders an individual Tweet.

Tweet.js

import React from "react";
import { TwitterTweetEmbed } from "react-twitter-embed";

const Tweet = ({ json }) => {
  const { id } = json.data;

  const options = {
    cards: "hidden",
    align: "center",
    width: "550",
    conversation: "none",
  };

  return <TwitterTweetEmbed options={options} tweetId={id} />;
};

export default Tweet;

Next, create the component responsible for rendering all the rules on our stream as well as displaying input controls for creating new rules. In this case, we will only be using one rule.

RuleList.js

import React, { useEffect, useReducer } from "react";
import axios from "axios";
import Rule from "./Rule";
import ErrorMessage from "./ErrorMessage";
import Spinner from "./Spinner";

const reducer = (state, action) => {
  switch (action.type) {
    case "show_rules":
      return { ...state, rules: action.payload, newRule: "" };
    case "add_rule":
      return {
        ...state,
        rules: [...state.rules, ...action.payload],
        newRule: "",
        errors: [],
      };
    case "add_errors":
      return { ...state, rules: state.rules, errors: action.payload };
    case "delete_rule":
      return {
        ...state,
        rules: [...state.rules.filter((rule) => rule.id !== action.payload)],
      };
    case "rule_changed":
      return { ...state, newRule: action.payload };
    case "change_loading_status":
      return { ...state, isLoading: action.payload };
    default:
      return state;
  }
};

const RuleList = () => {
  const initialState = { rules: [], newRule: "", isLoading: false, errors: [] };
  const [state, dispatch] = useReducer(reducer, initialState);
  const exampleRule = "from:twitterdev has:links";
  const ruleMeaning = `This example rule will match Tweets posted by 
     TwtterDev containing links`;
  const operatorsURL =
    "https://developer.twitter.com/en/docs/twitter-api/tweets/filtered-stream/integrate/build-a-rule";
  const rulesURL = "/api/rules";

  const createRule = async (e) => {
    e.preventDefault();
    const payload = { add: [{ value: state.newRule }] };

    dispatch({ type: "change_loading_status", payload: true });
    try {
      const response = await axios.post(rulesURL, payload);
      if (response.data.body.errors)
        dispatch({ type: "add_errors", payload: response.data.body.errors });
      else {
        dispatch({ type: "add_rule", payload: response.data.body.data });
      }
      dispatch({ type: "change_loading_status", payload: false });
    } catch (e) {
      dispatch({
        type: "add_errors",
        payload: [{ detail: e.message }],
      });
      dispatch({ type: "change_loading_status", payload: false });
    }
  };

  const deleteRule = async (id) => {
    const payload = { delete: { ids: [id] } };
    dispatch({ type: "change_loading_status", payload: true });
    await axios.post(rulesURL, payload);
    dispatch({ type: "delete_rule", payload: id });
    dispatch({ type: "change_loading_status", payload: false });
  };

  const errors = () => {
    const { errors } = state;

    if (errors && errors.length > 0) {
      return errors.map((error) => (
        <ErrorMessage key={error.title} error={error} styleType="negative" />
      ));
    }
  };

  const rules = () => {
    const { isLoading, rules } = state;

    const message = {
      title: "No rules present",
      details: [
        `There are currently no rules on this stream. Start by adding the rule 
        below.`,
        exampleRule,
        ruleMeaning,
      ],
      type: operatorsURL,
    };

    if (!isLoading) {
      if (rules && rules.length > 0) {
        return rules.map((rule) => (
          <Rule
            key={rule.id}
            data={rule}
            onRuleDelete={(id) => deleteRule(id)}
          />
        ));
      } else {
        return (
          <ErrorMessage
            key={message.title}
            error={message}
            styleType="warning"
          />
        );
      }
    } else {
      return <Spinner />;
    }
  };

  useEffect(() => {
    (async () => {
      dispatch({ type: "change_loading_status", payload: true });

      try {
        const response = await axios.get(rulesURL);
        const { data: payload = [] } = response.data.body;
        dispatch({
          type: "show_rules",
          payload,
        });
      } catch (e) {
        dispatch({ type: "add_errors", payload: [e.response.data] });
      }

      dispatch({ type: "change_loading_status", payload: false });
    })();
  }, []);

  return (
    <div>
      <form onSubmit={(e) => createRule(e)}>
        <div className="ui fluid action input">
          <input
            type="text"
            autoFocus={true}
            value={state.newRule}
            onChange={(e) =>
              dispatch({ type: "rule_changed", payload: e.target.value })
            }
          />
          <button type="submit" className="ui primary button">
            Add Rule
          </button>
        </div>
        {errors()}
        {rules()}
      </form>
    </div>
  );
};

export default RuleList;

Next, create the child component of RuleList.js responsible for displaying a single rule and deleting a rule.

Rule.js

 import React from "react";

 export const Rule = ({ data, onRuleDelete }) => {
   return (
     <div className="ui segment">
       <p>{data.value}</p>
       <div className="ui label">tag: {data.tag}</div>
       <button
         className="ui right floated negative button"
         onClick={() => onRuleDelete(data.id)}
       >
         Delete
       </button>
     </div>
   );
 };

 export default Rule;

Next, create a component for displaying any status or error messages.

ErrorMessage.js

import React from "react";

const ErrorMessage = ({ error, styleType }) => {
  const errorDetails = () => {
    if (error.details) {
      return error.details.map(detail => <p key={detail}>{detail}</p>);
    } else if (error.detail) {
      return <p key={error.detail}>{error.detail}</p>;
    }
  };

  const errorType = () => {
    if (error.type) {
      return (
        <em>
          See
          <a href={error.type} target="_blank" rel="noopener noreferrer">
            {" "}
            Twitter documentation{" "}
          </a>
          for further details.
        </em>
      );
    }
  };

  return (
    <div className={`ui message ${styleType}`}>
      <div className="header">{error.title}</div>
      {errorDetails()}
      {errorType()}
    </div>
  );
};

export default ErrorMessage;

Finally, create a component to display a loading indicator during any pending API calls.

Spinner.js

import React from "react";

const Spinner = () => {
  return (
    <div>
      <div className="ui active centered large inline loader"></div>
    </div>
  );
};

export default Spinner;

Proxy Setup

The final step is to proxy requests from your client to your backend server. To do this, from within your src/ directory, create a new file called “setupProxy.js” and add the following code.

setupProxy.js

const { createProxyMiddleware } = require("http-proxy-middleware");

// This proxy redirects requests to /api endpoints to
// the Express server running on port 3001.
module.exports = function (app) {
  app.use(
    ["/api"],
    createProxyMiddleware({
      target: "http://localhost:3001",
    })
  );
};

You can now start up both the server and client by going to the project root directory and typing the following.

npm start

After this command completes, your default web browser should automatically launch and navigate to http://localhost:3000. You will then be taken to the rules management section of the app.

Now that you have an app in place to listen for any kind of Tweet you want, let’s walk through some real-life examples of how this app can be used such as

Discovering new music videos
Finding remote developer job openings
Learning about personal finance and savings

For each of the examples and accompanying rules listed below, you can navigate to the rules section of the app and simply copy and paste the rule into the input field to add it to your stream. Once the rule has been added, it will take effect within seconds and only Tweets matching the criteria of that rule will be sent to you.

Remote developer job openings

In this first example, let’s say you are interested in finding remote developer job openings. To surface these kinds of Tweets, you can use the following rule.

(developer OR engineer) remote (context:66.961961812492148736 OR context:66.850073441055133696)

To understand what this rule is doing, you can break it down into two parts. The keywords part and the Tweet annotations part.

Using keyword operators

(developer OR engineer) remote

The keywords part of the rule will match Tweets containing the keywords “developer” or “engineer” and the keyword “remote”. These keywords alone will certainly match Tweets containing remote developer job postings, but it will also match on irrelevant Tweets. For example, the Tweet below will match this rule.
https://twitter.com/EraserFarm/status/1220013392766947332

Since this is not the desired Tweet, you need to take this rule a step further. Though this rule matches irrelevant Tweets, it is also successful in matching Tweets with actual job postings. For example, the rule you have so far will also result in matching the Tweet below.

https://twitter.com/plataformatec/status/1225460423761317888

Tweet annotations: Using context operators

The challenge you have now is, though you are receiving Tweets containing job postings you will still have to go through irrelevant Tweets. If only there was a way to only match on Tweets that contain job postings as best as possible. Are there operators you can use in your rule that only match these relevant Tweets?

Fortunately, this is where the power of Tweet annotations comes in. Take a closer look at an example Tweet object payload that was sent for this Tweet on your filtered stream endpoint. Within the payload, the nested “context_annotations” field is present. Context annotations are delivered as a “context_annotations” field in the payload. These annotations are inferred based on the Tweet text and result in the domain and/or entity labels, which can be used to discover Tweets on topics that may have been previously difficult to surface. Note, that these fields will only be included if data is present since not all Tweets will contain this data.
```
 "context_annotations": [
     {
       "domain": {
         "id": "66",
         "name": "Interests and Hobbies Category",
         "description": "A grouping of interests and hobbies entities, like Novelty Food or Destinations"
       },
       "entity": {
         "id": 961961812492148736,
         "name": "Recruitment",
         "description": "Recruitment"
       }
     },
     {
       "domain": {
         "id": "66",
         "name": "Interests and Hobbies Category",
         "description": "A grouping of interests and hobbies entities, like Novelty Food or Destinations"
       },
       "entity": {
         "id": "850073441055133696",
         "name": "Job search",
         "description": "Job search"
       }
     }
   ],
```
To match on the domain and entity ids within the context_annotations field, you can use the “context” operator. The “context” operator can be used to instruct your filtered stream endpoint to match on Tweets containing specific domain and entity names. Here’s what that would look like.

(context:66.961961812492148736 OR context:66.850073441055133696)

The operators above follow the format “context:.”. As seen in the example payload above, the domain id 66 represents the “Interests and Hobbies Category”. The entity ID 961961812492148736 represents the “Recruitment” entity and the entity ID 850073441055133696 represents the “Job search” entity. For a complete list of domains, the Tweet annotations docs contain a table with 50+ domain names.

With the operator explanation out of the way, this 2nd part of the rule matches Tweets containing the entity names “Recruitment” or “Jobs search”.

In summary, taking both parts of this rule together, it will match on Tweets that contain the keywords “developer” or “engineer” and the keyword “remote”, but only if those Tweets also contain the entity names “Recruitment” or “Jobs search”

Discovering new music videos

If you need new music video suggestions, you can start by using a simple rule that matches on Tweets containing the keywords “song” and “YouTube”. You will also want Tweets that actually link out to external videos

song youtube has:links

Taking a closer look at the payload of this Tweet, you see that it has some annotations on it that can help you match more relevant Tweets. Notice the Annotation with an entity label of “Pop” and a domain name of “Music Genre”

 "context_annotations": [
    {
      "domain": {
        "id": "10",
        "name": "Person",
        "description": "Named people in the world like Nelson Mandela"
      },
      "entity": {
        "id": "871815676998033408",
        "name": "Ally Brooke",
        "description": "Ally Brooke"
      }
    },
    {
      "domain": {
        "id": "54",
        "name": "Musician",
        "description": "A musician in the world, like Adele or Bob Dylan"
      },
      "entity": {
        "id": "871815676998033408",
        "name": "Ally Brooke",
        "description": "Ally Brooke"
      }
    },
    {
      "domain": {
        "id": "55",
        "name": "Music Genre",
        "description": "A category for a musical style, like Pop, Rock, or Rap"
      },
      "entity": {
        "id": "810938279801470977",
        "name": "Pop",
        "description": "Pop"
      }
    }
  ],

To make this rule better and narrow down your Tweets to be even more relevant, you can update your rule as follows.

song youtube has:links context:55.810938279801470977

This will take the original rule you used a step further by narrowing down to only the Tweets that are labeled with the Music Genre domain label and the Pop entity label.

Learning about personal finance and savings

As a final example, let’s say you are interested in learning about personal finance and you can be savvier about your spending and savings. You also only want original Tweets that contain links to online articles to learn more.

Going through a similar process as you did with earlier examples if you simply add the following rule, only Tweets containing the words “personal”, “finance” and “savings" will be sent to you.

“personal finance savings”

Taking a look at one of the Tweet payloads, the nested annotations contains an entity label about personal finance that will help you narrow down your Tweets to the most relevant ones.
```
"annotations": {
  "context": [
    {
      "context_domain_id": 66,
      "context_domain_id_str": "66",
      "context_domain_name": "Interests and Hobbies Category",
      "context_domain_description": "A grouping of interests and hobbies entities, like Novelty Food or Destinations",
      "context_entity_id": 847888632711061504,
      "context_entity_id_str": "847888632711061504",
      "context_entity_name": "Personal finance",
      "context_entity_description": "Personal finance"
    },
```
Using the context operator to match on Tweets containing this label, you can revise your rule to look as follows.

context:66.847888632711061504 has:links -is:retweet savings

Conclusion

Using the filtered stream endpoints to stream publicly available Tweets to your server and annotations, you created an app to more easily surface Tweets around a topic of interest. The filtered stream endpoint gives you the haystack of data in the form of Tweets and the Tweet annotations help you find the needle in that haystack.

Have you found other interesting ways to use this app? Follow me on Twitter and send me a Tweet to let me know. I used several libraries beyond the Twitter API to make this tutorial, but you may have different needs and requirements and should evaluate whether those tools are right for you.

Check out the live demo of the app on Glitch.

Building a remote developer job listings app with the Twitter API

Tony Vu — Tue, 03 Mar 2020 18:00:17 +0000

The code for this tutorial is available on GitHub.

With the start of the new year, I have had quite a few friends complain how difficult it is to find remote developer job openings. Not only that, but due to the location flexibility offered by these roles, it can be fiercely competitive to land a job due to the candidate pool being much larger than a comparable job opening in a defined geographic area. If only there was another way to surface these kinds of job postings and give candidates a leg up by being among the first to apply the moment a job opening is known…This is where I thought the Twitter API would be great for meeting this need!

With the recent release of the filtered stream endpoints and Tweet Annotations in Twitter Developer Labs, you can use these new capabilities to build an app that delivers you Tweets containing remote developer job openings the moment they are posted on Twitter.

Even if you are not interested in finding a job, this could be the beginning of a prototype for the remote developer job listing board you have always wanted to get going.

On the other hand, if you already have a developer job listings board, this can be another source of real time data to augment your job listings feed.

Before getting into building the app, here’s some terminology out of the way.

What is Twitter Developer Labs?

Twitter Developer Labs is where developers can go for early access to new API endpoints, features and versions. Labs is used to test out new ideas, and invite the developer community to share their feedback to help shape Twitter’s roadmap.

What are the filtered stream endpoints?

The filtered stream endpoints are endpoints released within Twitter Developer Labs that allow developers to filter the real-time stream of public Tweets. Developers can filter the real-time stream by applying a set of rules (specified using a combination of operators).

What are Tweet annotations?

Tweet annotations are metadata fields added to the Tweet object last December, currently on the filtered stream endpoint. Tweet annotations can help you explore new topics, and find out more information about the Tweets you care about. These two new fields, entity and context, each offer contextual information about the Tweet text itself. Tweets are analyzed and annotated based on the content of the Tweet. These newly added fields are referred to as "annotations".

Setup

To get started, here’s what you will need

Twitter Developer account: if you don’t have one already, you can apply for one.
Once you have an account, head over to the Labs section of the Twitter developer portal and click "Join Labs"
Select "Activate" next to Filtered Stream, then select a Twitter developer app
Consumer API keys from the details page of the app you created in the step above found in the Twitter developer portal.
Node.js
Npm (This is automatically installed with Node. Make sure you have npm 5.2+ or higher.)
Npx (Included with npm 5.2+ or higher)

You are going to be using the npm package, create-react-app, to create the scaffolding for a single-page React app. Open a terminal window and bootstrap your React app using create-react-app by using npx. Npx is a binary included with npm versions 5.2+ or higher, by default, that allows you to run executables from the npm registry, a database of JavaScript packages.

npx create-react-app remote-developer-jobs-streamer

After create-react-app has setup an initial directory structure for you, change to the newly created remote-developer-jobs-streamer directory and create a new subdirectory called “client”. Next, move all of the files and folders under the project root directory to this new client directory

cd remote-developer-jobs-streamer
mkdir client
mv * client

Going forward, your project root directory will contain all source code and files relevant to your Node server. The client directory, as you guessed it, will contain all source code files for your React code base.

The next step is to proxy requests from your client to your Node backend server. To do this, from within your client/ directory, add the following line to your package.json file.

“proxy”: “http://localhost:5000”

Credentials

Connecting to the filtered stream endpoints requires your app to authenticate using application-only authentication using a bearer token. The Node server you will be building shortly will include generating a bearer token using your app’s consumer API keys. To make that happen, you will need to have the following environment variables set. You can do so by issuing the following commands in your terminal window assuming you are using bash as your shell.

export TWITTER_CONSUMER_KEY=<YOUR API KEY HERE>
export TWITTER_CONSUMER_SECRET=<YOUR API SECRET KEY HERE>

Server Side Code

First, you will need to get started with implementing the Node server, which will be responsible for making the actual requests to the Twitter API. This Node server will serve as a proxy between your browser based React client and the Twitter API. On your Node server, you will need to create API endpoints that connect to the Filtered Stream endpoints. In turn, requests from your React client will be proxied through to your local Node server.

Before you go any any further, cd to the project root directory and install the following npm packages

npm install express body-parser util request http socket-io path

Next, while still within the project root directory, create a new file called “server.js”. This source code file will contain all of your backend logic for connecting to and receiving Tweets from the Filtered Stream endpoint. The contents of your server.js file will be as follows.

server.js

const express = require("express");
const bodyParser = require("body-parser");
const util = require("util");
const request = require("request");
const path = require("path");
const socketIo = require("socket.io");
const http = require("http");

const app = express();
const port = process.env.PORT || 5000;
const post = util.promisify(request.post);
const get = util.promisify(request.get);

app.use(bodyParser.json());
app.use(bodyParser.urlencoded({ extended: true }));

const server = http.createServer(app);
const io = socketIo(server);

const CONSUMER_KEY = process.env.TWITTER_CONSUMER_KEY;
const CONSUMER_SECRET = process.env.TWITTER_CONSUMER_SECRET;

let timeout = 0;

const bearerTokenURL = new URL("https://api.twitter.com/oauth2/token");

const streamURL = new URL(
  "https://api.twitter.com/labs/1/tweets/stream/filter?format=detailed&expansions=author_id"
);

const rulesURL = new URL(
  "https://api.twitter.com/labs/1/tweets/stream/filter/rules"
);

const errorMessage = {
  title: "Please Wait",
  detail: "Waiting for new jobs to be posted..."
};

const sleep = async delay => {
  return new Promise(resolve => setTimeout(() => resolve(true), delay));
};

async function bearerToken(auth) {
  const requestConfig = {
    url: bearerTokenURL,
    auth: {
      user: CONSUMER_KEY,
      pass: CONSUMER_SECRET
    },
    form: {
      grant_type: "client_credentials"
    }
  };

  const response = await post(requestConfig);
  return JSON.parse(response.body).access_token;
}

app.get("/rules", async (req, res) => {
  const token = await bearerToken({ CONSUMER_KEY, CONSUMER_SECRET });
  const requestConfig = {
    url: rulesURL,
    auth: {
      bearer: token
    },
    json: true
  };

  try {
    const response = await get(requestConfig);

    if (response.statusCode !== 200) {
      throw new Error(response.body.error.message);
    }

    res.send(response);
  } catch (e) {
    res.send(e);
  }
});

app.post("/rules", async (req, res) => {
  const token = await bearerToken({ CONSUMER_KEY, CONSUMER_SECRET });
  const requestConfig = {
    url: rulesURL,
    auth: {
      bearer: token
    },
    json: req.body
  };

  try {
    const response = await post(requestConfig);

    if (response.statusCode === 200 || response.statusCode === 201) {
      res.send(response);
    } else {
      throw new Error(response);
    }
  } catch (e) {
    res.send(e);
  }
});

const streamTweets = (socket, token) => {
  const config = {
    url: streamURL,
    auth: {
      bearer: token
    },
    timeout: 31000
  };

  const stream = request.get(config);

  stream
    .on("data", data => {
      try {
        const json = JSON.parse(data);
        if (json.connection_issue) {
          socket.emit("error", json);
          reconnect(stream, socket, token);
        } else {
          socket.emit("tweet", json);
        }
      } catch (e) {
        socket.emit("heartbeat");
      }
    })
    .on("error", error => {
      // Connection timed out
      socket.emit("error", errorMessage);
      reconnect(stream, socket, token);
    });
};

const reconnect = async (stream, socket, token) => {
  timeout++;
  stream.abort();
  await sleep(2 ** timeout * 1000);
  streamTweets(socket, token);
};

io.on("connection", async socket => {
  try {
    const token = await bearerToken({ CONSUMER_KEY, CONSUMER_SECRET });
    io.emit("connect", "Client connected");
    const stream = streamTweets(io, token);
  } catch (e) {}
});

if (process.env.NODE_ENV === "production") {
  app.use(express.static(path.join(__dirname, "client/build")));

  app.get("*", function(req, res) {
    res.sendFile(path.join(__dirname, "client/build", "index.html"));
  });
}

server.listen(port, () => console.log(`Listening on port ${port}`));

Filtering Tweets on the filtered stream endpoint using rules

Filtered Stream has two endpoints, a streaming endpoint and a rules and operators endpoint. The rules endpoint is used to create and delete rules, comprised of a series of one or more operators, that determine the criteria to match Tweets. For this app, based on referencing the Filtered Stream operators documentation, you will be using the following rule.

(developer OR engineer) remote (context:66.961961812492148736 OR context:66.850073441055133696)

To understand what this rule is doing, you can break it down into two parts. The keywords part and the Tweet annotations part.

Using keyword operators

(developer OR engineer) remote

The keywords part of the rule will match Tweets containing the keywords “developer” or “engineer” and the keyword “remote”. These keywords alone will certainly match Tweets containing remote developer job postings, but it will also match on irrelevant Tweets. For example, the Tweet below will match this rule.

https://twitter.com/EraserFarm/status/1220013392766947332

Since this is not the desired Tweet, you need to take this rule a step further. Though this rule matches irrelevant Tweets, it is also successful in matching Tweets with actual job postings. For example, the rule you have so far will also result in matching the Tweet below.

https://twitter.com/plataformatec/status/1225460423761317888

Tweet annotations: Using context operators

The challenge you have now is, though you are receiving Tweets containing job postings you will still have to go through irrelevant Tweets. If only there was a way to only match on Tweets that contain job postings as best as possible. Are there operators you can use in your rule that only match these relevant Tweets? Fortunately, this is where the power of Tweet annotations comes in. Take a closer look at an example Tweet object payload that was sent for this Tweet on your filtered stream endpoint. Within the payload, the nested “context_annotations” field is present. Context annotations are delivered as a “context_annotations” field in the payload. These annotations are inferred based on the Tweet text and result in domain and/or entity labels, which can be used to discover Tweets on topics that may have been previously difficult to surface. Note, that these fields will only be included if data is present since not all Tweets will contain this data.

      "context_annotations": [
        {
          "domain": {
            "id": "66",
            "name": "Interests and Hobbies Category",
            "description": "A grouping of interests and hobbies entities, like Novelty Food or Destinations"
          },
          "entity": {
            "id": 961961812492148736,
            "name": "Recruitment",
            "description": "Recruitment"
          }
        },
        {
          "domain": {
            "id": "66",
            "name": "Interests and Hobbies Category",
            "description": "A grouping of interests and hobbies entities, like Novelty Food or Destinations"
          },
          "entity": {
            "id": 850073441055133696,
            "name": "Job search",
            "description": "Job search"
          }
        }
      ],

To match on the domain and entity ids within the context_annotations field, you can use the “context” operator. The “context” operator can be used to instruct your filtered stream endpoint to match on Tweets containing specific domain and entity names. Here’s what that would look like.

(context:66.961961812492148736 OR context:66.850073441055133696)

The operators above follow the format context:<domain id>.<entity id>. As seen in the example payload above, the domain id 66 represents the “Interests and Hobbies Category”. The entity ID 961961812492148736 represents the “Recruitment” entity and the entity ID 850073441055133696 represents the “Job search” entity. For a complete list of domains, the Tweet Annotations docs contains a table with 50+ domain names.

With the operator explanation out of the way, this 2nd part of the rule matches Tweets containing the entity names “Recruitment” or “Jobs search”.

In summary, taking both parts of this rule together, it will match on Tweets that contain the keywords “developer” or “engineer” and the keyword “remote”, but only if those Tweets also contain the entity names “Recruitment” or “Jobs search”.

The other filtered stream endpoint is the streaming endpoint, which uses a simple GET connection. Once a connection is established, Tweets are delivered in JSON format through a persistent HTTP streaming connection. You will only receive content matching your rules while connected to the stream. In this case, you will only receive content matching the rule above. You will come back to this rule later as you build out the client side of our app.

Client Side Code

The first thing you will need to do is add a script line to your package.json file within the project root directory. These lines will provide a command shortcut to concurrently run your client and server backend code.

~/remote-dev-jobs-streamer/package.json

  "scripts": {
    "client": "cd client && yarn start",
    "server": "nodemon server.js",
    "dev": "concurrently --kill-others-on-fail \"yarn server\" \"yarn client\""
  }

Next, you will need to install some npm packages. You will need the “request” and “axios” modules to make http requests to your backend server and “react-moment” to reformat the the timestamp provided in a Tweet object into a human readable format for display in your app. You will also need the “react-router-dom” package to navigate between components. Again, make sure you are in the client/ subdirectory and not the project root directory.

npm install request react-moment react-router-dom axios socket.io-client

Following installation of those packages, install the rest of the packages. From the client/ subdirectory type the following

npm install

The next step is to work on the client source code. To display all the UI goodness in your web browser, you will be creating the following React components.

App.js - The parent component that be will in turn render all other components
NavBar.js - Displays the navigation bar on the left hand side for navigating between the jobs feed and managing rules
Job.js - Does the heavy lifting of extracting the relevant fields from a Tweet and turning it into a job posting
JobList.js - Renders multiple Job components at once in a “feed” like fashion
Rule.js - Renders an individual rule on your stream
RuleList.js - Renders multiple Rule components and displays an input field to add a rule
ErrorMessage.js - Renders any status or error messages to the screen
Spinner.js - Renders a loading indicator for any pending API calls

To style all of your components, you will be using Semantic UI. Include the CSS for Semantic UI in your project by adding the line between the <head> tags in your index.html file located in the ~/remote-dev-jobs-streamer/client/public directory.

<link rel="stylesheet" href="https://cdnjs.cloudflare.com/ajax/libs/semantic-ui/2.4.1/semantic.min.css" />

Now you will need to get started with creating the React components. Under your remote-dev-jobs-streamer/client/src directory, create a directory called “components”. The source code files above will be stored in this new directory. First, create the parent most component of our application. This component will be responsible for rendering all other components.

App.js

import React from "react";
import { BrowserRouter, Route } from "react-router-dom";

import Navbar from "./Navbar";
import JobList from "./JobList";
import RuleList from "./RuleList";

class App extends React.Component {
  render() {
    return (
      <div className="ui container">
        <div className="introduction"></div>

        <h1 className="ui header">
          <img
            className="ui image"
            src="/Twitter_Logo_Blue.png"
            alt="Twitter Logo"
          />
          <div className="content">
            Remote Developer Jobs
            <div className="sub header">Powered by Twitter data</div>
          </div>
        </h1>

        <div className="ui grid">
          <BrowserRouter>
            <Navbar />
            <Route exact path="/" component={RuleList} />
            <Route exact path="/rules" component={RuleList} />
            <Route exact path="/jobs" component={JobList} />
          </BrowserRouter>
        </div>
      </div>
    );
  }
}

export default App;

Next, create the component for the navigation bar you see on the left hand side in the screenshot at the top of this post.

Navbar.js

import React from "react";
import { NavLink } from "react-router-dom";

const Navbar = () => {
  return (
    <div className="three wide column">
      <div className="ui vertical pointing menu">
        <NavLink to="/jobs" className="item">
          New Jobs
        </NavLink>
        <NavLink to="/rules" className="item">
          Manage Rules
        </NavLink>
      </div>
    </div>
  );
};

export default Navbar;

Next, create the parent component for rendering all job listings.

JobList.js

import React, { useEffect, useReducer } from "react";
import Job from "./Job";
import socketIOClient from "socket.io-client";
import ErrorMessage from "./ErrorMessage";
import Spinner from "./Spinner";

const reducer = (state, action) => {
  switch (action.type) {
    case "add_job":
      return {
        ...state,
        jobs: [action.payload, ...state.jobs],
        error: null,
        isWaiting: false
      };
    case "show_error":
      return { ...state, error: action.payload, isWaiting: false };
    case "update_waiting":
      return { ...state, error: null, isWaiting: true };
    default:
      return state;
  }
};

const JobList = () => {
  const initialState = {
    jobs: [],
    error: {},
    isWaiting: true
  };

  const [state, dispatch] = useReducer(reducer, initialState);
  const { jobs, error, isWaiting } = state;

  const streamTweets = () => {
    const socket = socketIOClient("/");
    socket.on("connect", () => {});
    socket.on("tweet", json => {
      if (json.data) {
        dispatch({ type: "add_job", payload: json });
      }
    });
    socket.on("heartbeat", data => {
      dispatch({ type: "update_waiting" });
    });
    socket.on("error", data => {
      dispatch({ type: "show_error", payload: data });
    });
  };

  const errorMessage = () => {
    const message = {
      title: "Reconnecting",
      detail: "Please wait while we reconnect to the stream."
    };

    if (error && error.detail) {
      return (
        <div className="twelve wide column">
          <ErrorMessage key={error.title} error={error} styleType="warning" />
          <ErrorMessage
            key={message.title}
            error={message}
            styleType="success"
          />
          <Spinner />
        </div>
      );
    }
  };

  const waitingMessage = () => {
    const message = {
      title: "Still working",
      detail: "Waiting for new jobs to be Tweeted"
    };

    if (isWaiting) {
      return (
        <React.Fragment>
          <div className="twelve wide column">
            <ErrorMessage
              key={message.title}
              error={message}
              styleType="success"
            />
          </div>
          <Spinner />
        </React.Fragment>
      );
    }
  };

  useEffect(() => {
    streamTweets();
  }, []);

  const showJobs = () => {
    if (jobs.length > 0) {
      return (
        <div className="ui segments">
          {jobs.map(job => (
            <Job key={job.data.id} json={job} />
          ))}
        </div>
      );
    }
  };

  return (
    <div className="twelve wide stretched column">
      {errorMessage()}
      {waitingMessage()}
      {showJobs()}
    </div>
  );
};

export default JobList;

Next, create the child component for the previous component that renders an individual job listing. This component extracts some JSON fields from a Tweet object and displays information about the job listing such as the listing title, date it was posted, description, hashtags, and entity names.

Job.js

import React from "react";
import Moment from "react-moment";
import "../stylesheets/Job.css";

const Job = ({ json }) => {
  const { created_at, id } = json.data;
  const { name = "", username = "" } = json.includes.users[0];

  const hashtags = () => {
    if (json.data.entities && json.data.entities.hashtags) {
      return json.data.entities.hashtags.map(hashtag => (
        <span key={hashtag.tag} className="ui label">
          #{hashtag.tag}
        </span>
      ));
    }
  };

  const title = () => {
    const { entities } = json.data;

    if (
      entities &&
      entities.urls &&
      entities.urls[0] &&
      entities.urls[0].title
    ) {
      return entities.urls[0].title;
    } else {
      return json.data.text.substring(0, 32) + "...";
    }
  };

  const annotations = () => {
    if (json.data.context_annotations) {
      return json.data.context_annotations.map(context => (
        <span key={context.entity.id} className="ui small teal basic label">
          {context.entity.name}
        </span>
      ));
    }
  };

  return (
    <a
      href={`http://www.twitter.com/${username}/status/${id}`}
      target="_blank"
      rel="noopener noreferrer"
    >
      <div className="ui segment job">
        <h4 className="ui header">
          {title()}
          <div className="sub header">{name}</div>
          <Moment
            className="sub header"
            parse="YYYY-MM-DDTHH:mm:ss.ZZZZ"
            fromNow
          >
            {created_at}
          </Moment>
        </h4>
        <p>{json.data.text}</p>
        {hashtags()}
        {annotations()}
      </div>
    </a>
  );
};

export default Job;

Next, create the component responsible for rendering all the rules on our stream as well as displaying input controls for creating new rules. In this case, we will only be using one rule.

RuleList.js

import React, { useEffect, useReducer } from "react";
import axios from "axios";
import Rule from "./Rule";
import ErrorMessage from "./ErrorMessage";
import Spinner from "./Spinner";

const reducer = (state, action) => {
  switch (action.type) {
    case "show_rules":
      return { ...state, rules: action.payload, newRule: "" };
    case "add_rule":
      return {
        ...state,
        rules: [...state.rules, ...action.payload],
        newRule: "",
        errors: []
      };
    case "add_errors":
      return { ...state, rules: state.rules, errors: action.payload };
    case "delete_rule":
      return {
        ...state,
        rules: [...state.rules.filter(rule => rule.id !== action.payload)]
      };
    case "rule_changed":
      return { ...state, newRule: action.payload };
    case "change_loading_status":
      return { ...state, isLoading: action.payload };
    default:
      return state;
  }
};

const RuleList = () => {
  const initialState = { rules: [], newRule: "", isLoading: false, errors: [] };
  const [state, dispatch] = useReducer(reducer, initialState);
  const exampleRule = `(developer OR engineer) remote (context:66.961961812492148736
                OR context:66.850073441055133696)`;
  const ruleMeaning = `This example rule will match Tweets containing the  
     keywords "developer" or "engineer" and the keyword "remote", but only if 
      they contain the context entity labels "Careers" or "Recruiting"`;
  const operatorsURL =
    "https://developer.twitter.com/en/docs/labs/filtered-stream/operators";
  const rulesURL = "/rules";

  const createRule = async e => {
    e.preventDefault();
    const payload = { add: [{ value: state.newRule }] };

    dispatch({ type: "change_loading_status", payload: true });
    try {
      const response = await axios.post(rulesURL, payload);
      if (response.data.body.errors)
        dispatch({ type: "add_errors", payload: response.data.body.errors });
      else {
        dispatch({ type: "add_rule", payload: response.data.body.data });
      }
      dispatch({ type: "change_loading_status", payload: false });
    } catch (e) {
      dispatch({
        type: "add_errors",
        payload: [{ detail: e.message }]
      });
      dispatch({ type: "change_loading_status", payload: false });
    }
  };

  const deleteRule = async id => {
    const payload = { delete: { ids: [id] } };
    dispatch({ type: "change_loading_status", payload: true });
    await axios.post(rulesURL, payload);
    dispatch({ type: "delete_rule", payload: id });
    dispatch({ type: "change_loading_status", payload: false });
  };

  const errors = () => {
    const { errors } = state;

    if (errors && errors.length > 0) {
      return errors.map(error => (
        <ErrorMessage key={error.title} error={error} styleType="negative" />
      ));
    }
  };

  const rules = () => {
    const { isLoading, rules } = state;

    const message = {
      title: "No rules present",
      details: [
        `There are currently no rules on this stream. Start by adding the rule 
        below.`,
        exampleRule,
        ruleMeaning
      ],
      type: operatorsURL
    };

    if (!isLoading) {
      if (rules && rules.length > 0) {
        return rules.map(rule => (
          <Rule key={rule.id} data={rule} onRuleDelete={id => deleteRule(id)} />
        ));
      } else {
        return (
          <ErrorMessage
            key={message.title}
            error={message}
            styleType="warning"
          />
        );
      }
    } else {
      return <Spinner />;
    }
  };

  useEffect(() => {
    (async () => {
      dispatch({ type: "change_loading_status", payload: true });

      const response = await axios.get(rulesURL);
      const { data: payload = [] } = response.data.body;
      dispatch({
        type: "show_rules",
        payload
      });
      dispatch({ type: "change_loading_status", payload: false });
    })();
  }, []);

  return (
    <div className="twelve wide column">
      <form onSubmit={e => createRule(e)}>
        <div className="ui fluid action input">
          <input
            type="text"
            autoFocus={true}
            value={state.newRule}
            onChange={e =>
              dispatch({ type: "rule_changed", payload: e.target.value })
            }
          />
          <button type="submit" className="ui primary button">
            Add Rule
          </button>
        </div>
        {errors()}
        {rules()}
      </form>
    </div>
  );
};

export default RuleList;

Next, create the child component of RuleList.js responsible for displaying a single rule and deleting a rule.

Rule.js

import React from "react";

export const Rule = ({ data, onRuleDelete }) => {
  return (
    <div className="ui segment">
      <p>{data.value}</p>
      <div className="ui label">tag: {data.tag}</div>
      <button
        className="ui right floated negative button"
        onClick={() => onRuleDelete(data.id)}
      >
        Delete
      </button>
    </div>
  );
};

export default Rule;

Next, create a component for displaying any status or error messages.

ErrorMessage.js

import React from "react";

const ErrorMessage = ({ error, styleType }) => {
  const errorDetails = () => {
    if (error.details) {
      return error.details.map(detail => <p key={detail}>{detail}</p>);
    } else if (error.detail) {
      return <p key={error.detail}>{error.detail}</p>;
    }
  };

  const errorType = () => {
    if (error.type) {
      return (
        <em>
          See
          <a href={error.type} target="_blank" rel="noopener noreferrer">
            {" "}
            Twitter documentation{" "}
          </a>
          for further details.
        </em>
      );
    }
  };

  return (
    <div className={`ui message ${styleType}`}>
      <div className="header">{error.title}</div>
      {errorDetails()}
      {errorType()}
    </div>
  );
};

export default ErrorMessage;

Finally, create a component to display a loading indicator during any pending API calls.

Spinner.js

import React from "react";

const Spinner = () => {
  return (
    <div className="twelve wide column">
      <div className="ui active centered large inline loader">
      </div>
    </div>
  );
};

export default Spinner;

After you have finished writing code for both the client side and server side, start up both the server and client by going to the project root directory and typing the following.

[mbp ~/remote-developer-jobs-streamer (master)]$ yarn dev
yarn run v1.15.2
warning package.json: No license field
$ concurrently --kill-others-on-fail "yarn server" "yarn client"
warning package.json: No license field
warning package.json: No license field
$ nodemon server.js
$ cd client && yarn start
warning ../package.json: No license field
$ react-scripts start
[0] [nodemon] 1.19.1
[0] [nodemon] to restart at any time, enter `rs`
[0] [nodemon] watching: *.*
[0] [nodemon] starting `node server.js`
[0] Listening on port 5000
...

After this command completes, your default web browser should automatically launch and navigate to http://localhost:3000. You will then be taken to the rules management section of the app. Simply add the rule below to filter Tweets containing remote developer job postings.

(developer OR engineer) remote (context:66.961961812492148736 OR context:66.850073441055133696)

Congratulations! You have successfully built your very own remote developer job listings app.

Conclusion

Using the filtered stream endpoint to stream publicly available Tweets to your server and Tweet annotations, you were able to surface remote developer job listings. This helped solve a problem that not only I had at one point in my life, but also my friends experienced too. The filtered stream endpoint gives you the haystack of data in the form of Tweets and the Tweet annotations help you find the needle in that haystack.

Thanks to the real-time stream of Tweets provided by filtered stream and Tweet annotations, you are able to take your use of Twitter to another level. Hope this inspires you to continue building other solutions using the Twitter API and better understand how annotations can be used in practice. I used several libraries beyond the Twitter API to make this tutorial, but you may have different needs and requirements and should evaluate whether those tools are right for you.