Building an app to stream Tweets in real-time using the Twitter API

The code for this tutorial is available on GitHub. You can also check out the live demo of the app on Glitch

This tutorial will walk you through how to build your own real-time Tweet streaming app using the filtered stream endpoints and Tweet annotations to listen for Tweets based on your own topics of interest.

After building the app, you will learn about how it can be applied to some real-life examples to inspire you to get started such as.

• Discovering new music videos: Imagine a dashboard that lets you see the music videos being shared across Twitter

• Finding remote developer job openings: Imagine creating a remote developer job listings app with remote developer job openings being shared across Twitter

• Learning about personal finance and savings: Surface public conversations about personal finance and savings happening on Twitter

Setup

To get started here’s what you will need

• You must have a developer account. If you don’t have one already, you can apply for one. Access is available with active keys and tokens for a developer App that is attached to a Project created in the developer portal.
• Node.js
• Npm (This is automatically installed with Node. Make sure you have npm 5.2 or higher.)
• Npx (Included with npm 5.2 or higher)

First, install Node.js. Check out the Downloads section from Node’s website and download the source code or installer of your choice. Alternatively, if you are running on a Mac you can install the Node package using the Brew package manager

Open a terminal window and bootstrap your React app using create-react-app by using npx.

npx create-react-app real-time-tweet-streamer

After create-react-app has finished executing, change to the newly created real-time-tweet-streamer directory and replace the scripts block in your package.json with the following script block in your package.json. These lines will provide a command shortcut to concurrently run your client and server backend code in development or production as needed.

cd real-time-tweet-streamer

package.json

"scripts": {
  "start": "npm run development",
  "development": "NODE_ENV=development concurrently --kill-others \"npm run client\" \"npm run server\"",
  "production": "npm run build && NODE_ENV=production npm run server",
  "client": "react-scripts start",
  "server": "node server/server.js",
  "build": "react-scripts build",
  "test": "react-scripts test",
  "eject": "react-scripts eject"
},

After updating the scripts section, your package.json should now look as follows.

{
  "name": "real-time-tweet-streamer",
  "version": "0.1.0",
  "private": true,
  "dependencies": {
    "@testing-library/jest-dom": "^4.2.4",
    "@testing-library/react": "^9.3.2",
    "@testing-library/user-event": "^7.1.2",
    "react": "^16.13.1",
    "react-dom": "^16.13.1",
    "react-scripts": "3.4.1"
  },
  "scripts": {
    "start": "npm run development",
    "development": "NODE_ENV=development concurrently --kill-others \"npm run client\" \"npm run server\"",
    "production": "npm run build && NODE_ENV=production npm run server",
    "client": "react-scripts start",
    "server": "node server/server.js",
    "build": "react-scripts build",
    "test": "react-scripts test",
    "eject": "react-scripts eject"
  },
  "eslintConfig": {
    "extends": "react-app"
  },
  "browserslist": {
    "production": [
      ">0.2%",
      "not dead",
      "not op_mini all"
    ],
    "development": [
      "last 1 chrome version",
      "last 1 firefox version",
      "last 1 safari version"
    ]
  }
}

Next, remove all files within the src/ subdirectory.

rm src/*

Then, create a new file within the src/ subdirectory called index.js. The code for this file will be as follows.

import React from "react";
import ReactDOM from "react-dom";
import App from "./components/App";

ReactDOM.render(<App />, document.querySelector("#root"));

Credentials

Connecting to the filtered stream endpoints requires you to authenticate using a bearer token from your app in the Twitter developer portal. To utilize your bearer token, you will need to have the following environment variable set. You can do so by issuing the following command in your terminal window assuming you are using bash as your shell. Replace <YOUR BEARER TOKEN HERE>, including the left and right angle brackets, with your bearer token.

export TWITTER_BEARER_TOKEN=<YOUR BEARER TOKEN HERE>

Server Side Code

First, you will need to get started with implementing the Node server, which will be responsible for making the actual requests to the Twitter API. This Node server will serve as a proxy between your browser-based React client and the Twitter API. On your Node server, you will need to create API endpoints that connect to the filtered stream endpoints. In turn, requests from your React client will be proxied through to your local Node server.

Before you go any further, cd to the project root directory and install the following dependencies

npm install concurrently express body-parser util request http socket.io path http-proxy-middleware request react-router-dom axios socket.io-client react-twitter-embed

Next, while still within your project root directory, create a new subdirectory called “server” and a new file within that subdirectory called “server.js”.

mkdir server
touch server/server.js

This source code file will contain all of your backend logic for connecting to and receiving Tweets from the filtered stream endpoint. The contents of your server.js file will be as follows.

server.js

const express = require("express");
const bodyParser = require("body-parser");
const util = require("util");
const request = require("request");
const path = require("path");
const socketIo = require("socket.io");
const http = require("http");

const app = express();
let port = process.env.PORT || 3000;
const post = util.promisify(request.post);
const get = util.promisify(request.get);

app.use(bodyParser.json());
app.use(bodyParser.urlencoded({ extended: true }));

const server = http.createServer(app);
const io = socketIo(server);

const BEARER_TOKEN = process.env.TWITTER_BEARER_TOKEN;

let timeout = 0;

const streamURL = new URL(
  "https://api.twitter.com/2/tweets/search/stream?tweet.fields=context_annotations&expansions=author_id"
);

const rulesURL = new URL(
  "https://api.twitter.com/2/tweets/search/stream/rules"
);

const errorMessage = {
  title: "Please Wait",
  detail: "Waiting for new Tweets to be posted...",
};

const authMessage = {
  title: "Could not authenticate",
  details: [
    `Please make sure your bearer token is correct. 
      If using Glitch, remix this app and add it to the .env file`,
  ],
  type: "https://developer.twitter.com/en/docs/authentication",
};

const sleep = async (delay) => {
  return new Promise((resolve) => setTimeout(() => resolve(true), delay));
};

app.get("/api/rules", async (req, res) => {
  if (!BEARER_TOKEN) {
    res.status(400).send(authMessage);
  }

  const token = BEARER_TOKEN;
  const requestConfig = {
    url: rulesURL,
    auth: {
      bearer: token,
    },
    json: true,
  };

  try {
    const response = await get(requestConfig);

    if (response.statusCode !== 200) {
      if (response.statusCode === 403) {
        res.status(403).send(response.body);
      } else {
        throw new Error(response.body.error.message);
      }
    }

    res.send(response);
  } catch (e) {
    res.send(e);
  }
});

app.post("/api/rules", async (req, res) => {
  if (!BEARER_TOKEN) {
    res.status(400).send(authMessage);
  }

  const token = BEARER_TOKEN;
  const requestConfig = {
    url: rulesURL,
    auth: {
      bearer: token,
    },
    json: req.body,
  };

  try {
    const response = await post(requestConfig);

    if (response.statusCode === 200 || response.statusCode === 201) {
      res.send(response);
    } else {
      throw new Error(response);
    }
  } catch (e) {
    res.send(e);
  }
});

const streamTweets = (socket, token) => {
  let stream;

  const config = {
    url: streamURL,
    auth: {
      bearer: token,
    },
    timeout: 31000,
  };

  try {
    const stream = request.get(config);

    stream
      .on("data", (data) => {
        try {
          const json = JSON.parse(data);
          if (json.connection_issue) {
            socket.emit("error", json);
            reconnect(stream, socket, token);
          } else {
            if (json.data) {
              socket.emit("tweet", json);
            } else {
              socket.emit("authError", json);
            }
          }
        } catch (e) {
          socket.emit("heartbeat");
        }
      })
      .on("error", (error) => {
        // Connection timed out
        socket.emit("error", errorMessage);
        reconnect(stream, socket, token);
      });
  } catch (e) {
    socket.emit("authError", authMessage);
  }
};

const reconnect = async (stream, socket, token) => {
  timeout++;
  stream.abort();
  await sleep(2 ** timeout * 1000);
  streamTweets(socket, token);
};

io.on("connection", async (socket) => {
  try {
    const token = BEARER_TOKEN;
    io.emit("connect", "Client connected");
    const stream = streamTweets(io, token);
  } catch (e) {
    io.emit("authError", authMessage);
  }
});

console.log("NODE_ENV is", process.env.NODE_ENV);

if (process.env.NODE_ENV === "production") {
  app.use(express.static(path.join(__dirname, "../build")));
  app.get("*", (request, res) => {
    res.sendFile(path.join(__dirname, "../build", "index.html"));
  });
} else {
  port = 3001;
}

server.listen(port, () => console.log(`Listening on port ${port}`));

Filtering Tweets on the filtered stream endpoint using rules

Filtered stream has two endpoints, a streaming endpoint, to receive data and a rules endpoint that is used to create and delete rules. The filtered stream endpoints require you to define search queries called rules, in order for it to know what kind of Tweets to send to you. Rules allow you to narrow down to only the Tweets you are looking for by using a set of operators. You will see some example use cases and corresponding rules you can use later once you finish building the app.

The other filtered stream endpoint is the streaming endpoint, which uses a simple GET connection. Once a connection is established, Tweets are delivered in JSON format through a persistent HTTP streaming connection. You will only receive Tweets matching your rules while connected to the stream.

Client-Side Code

The next step is to work on the following React components

App.js - The parent component that be will, in turn, render all other components
NavBar.js - Displays the navigation bar for navigating between the Tweet feed and managing rules
Tweet.js - Displays a Tweet on the page
TweetFeed.js - Renders multiple Tweet components at once in a “feed” like fashion
Rule.js - Renders an individual rule on your stream
RuleList.js - Renders multiple Rule components and displays an input field to add a rule
ErrorMessage.js - Renders any status or error messages to the screen
Spinner.js - Renders a loading indicator for any pending API calls

To style all of your components, you will be using Semantic UI. Include the CSS for Semantic UI in your project by adding the line below to the <head> tag in your index.html file located in the ~/real-time-tweet-streamer/public directory.

<link rel="stylesheet" href="https://cdnjs.cloudflare.com/ajax/libs/semantic-ui/2.4.1/semantic.min.css" />

Now you will need to get started with creating the React components. Under your /src subdirectory, create a directory called “components”. The source code files above will be stored in this new directory. First, create the parent most component of the application. This component will be responsible for rendering all other components.

App.js

import React from "react";
import { BrowserRouter, Route } from "react-router-dom";

import Navbar from "./Navbar";
import TweetFeed from "./TweetFeed";
import RuleList from "./RuleList";

class App extends React.Component {
  render() {
    return (
      <div className="ui container">
        <div className="introduction"></div>

        <h1 className="ui header">
          <div className="content">
            Real Time Tweet Streamer
            <div className="sub header">Powered by Twitter data</div>
          </div>
        </h1>

        <div className="ui container">
          <BrowserRouter>
            <Navbar />
            <Route exact path="/" component={RuleList} />
            <Route exact path="/rules" component={RuleList} />
            <Route exact path="/tweets" component={TweetFeed} />
          </BrowserRouter>
        </div>
      </div>
    );
  }
}

export default App;

Next, create the component for the navigation bar.

Navbar.js

import React from "react";
import { NavLink } from "react-router-dom";

const Navbar = () => {
  return (
    <div className="ui two item menu">
      <NavLink to="/tweets" className="item" target="_blank">
        New Tweets
      </NavLink>
      <NavLink to="/rules" className="item" target="_blank">
        Manage Rules
      </NavLink>
    </div>
  );
};

export default Navbar;

Next, create the parent component for rendering all job listings.

TweetFeed.js

import React, { useEffect, useReducer } from "react";
import Tweet from "./Tweet";
import socketIOClient from "socket.io-client";
import ErrorMessage from "./ErrorMessage";
import Spinner from "./Spinner";

const reducer = (state, action) => {
  switch (action.type) {
    case "add_tweet":
      return {
        ...state,
        tweets: [action.payload, ...state.tweets],
        error: null,
        isWaiting: false,
        errors: [],
      };
    case "show_error":
      return { ...state, error: action.payload, isWaiting: false };
    case "add_errors":
      return { ...state, errors: action.payload, isWaiting: false };
    case "update_waiting":
      return { ...state, error: null, isWaiting: true };
    default:
      return state;
  }
};

const TweetFeed = () => {
  const initialState = {
    tweets: [],
    error: {},
    isWaiting: true,
  };

  const [state, dispatch] = useReducer(reducer, initialState);
  const { tweets, error, isWaiting } = state;

  const streamTweets = () => {
    let socket;

    if (process.env.NODE_ENV === "development") {
      socket = socketIOClient("http://localhost:3001/");
    } else {
      socket = socketIOClient("/");
    }

    socket.on("connect", () => {});
    socket.on("tweet", (json) => {
      if (json.data) {
        dispatch({ type: "add_tweet", payload: json });
      }
    });
    socket.on("heartbeat", (data) => {
      dispatch({ type: "update_waiting" });
    });
    socket.on("error", (data) => {
      dispatch({ type: "show_error", payload: data });
    });
    socket.on("authError", (data) => {
      console.log("data =>", data);
      dispatch({ type: "add_errors", payload: [data] });
    });
  };

  const reconnectMessage = () => {
    const message = {
      title: "Reconnecting",
      detail: "Please wait while we reconnect to the stream.",
    };

    if (error && error.detail) {
      return (
        <div>
          <ErrorMessage key={error.title} error={error} styleType="warning" />
          <ErrorMessage
            key={message.title}
            error={message}
            styleType="success"
          />
          <Spinner />
        </div>
      );
    }
  };

  const errorMessage = () => {
    const { errors } = state;

    if (errors && errors.length > 0) {
      return errors.map((error) => (
        <ErrorMessage key={error.title} error={error} styleType="negative" />
      ));
    }
  };

  const waitingMessage = () => {
    const message = {
      title: "Still working",
      detail: "Waiting for new Tweets to be posted",
    };

    if (isWaiting) {
      return (
        <React.Fragment>
          <div>
            <ErrorMessage
              key={message.title}
              error={message}
              styleType="success"
            />
          </div>
          <Spinner />
        </React.Fragment>
      );
    }
  };

  useEffect(() => {
    streamTweets();
  }, []);

  const showTweets = () => {
    if (tweets.length > 0) {
      return (
        <React.Fragment>
          {tweets.map((tweet) => (
            <Tweet key={tweet.data.id} json={tweet} />
          ))}
        </React.Fragment>
      );
    }
  };

  return (
    <div>
      {reconnectMessage()}
      {errorMessage()}
      {waitingMessage()}
      {showTweets()}
    </div>
  );
};

export default TweetFeed;

Next, create the child component for the previous component that renders an individual Tweet.

Tweet.js

import React from "react";
import { TwitterTweetEmbed } from "react-twitter-embed";

const Tweet = ({ json }) => {
  const { id } = json.data;

  const options = {
    cards: "hidden",
    align: "center",
    width: "550",
    conversation: "none",
  };

  return <TwitterTweetEmbed options={options} tweetId={id} />;
};

export default Tweet;

Next, create the component responsible for rendering all the rules on our stream as well as displaying input controls for creating new rules. In this case, we will only be using one rule.

RuleList.js

import React, { useEffect, useReducer } from "react";
import axios from "axios";
import Rule from "./Rule";
import ErrorMessage from "./ErrorMessage";
import Spinner from "./Spinner";

const reducer = (state, action) => {
  switch (action.type) {
    case "show_rules":
      return { ...state, rules: action.payload, newRule: "" };
    case "add_rule":
      return {
        ...state,
        rules: [...state.rules, ...action.payload],
        newRule: "",
        errors: [],
      };
    case "add_errors":
      return { ...state, rules: state.rules, errors: action.payload };
    case "delete_rule":
      return {
        ...state,
        rules: [...state.rules.filter((rule) => rule.id !== action.payload)],
      };
    case "rule_changed":
      return { ...state, newRule: action.payload };
    case "change_loading_status":
      return { ...state, isLoading: action.payload };
    default:
      return state;
  }
};

const RuleList = () => {
  const initialState = { rules: [], newRule: "", isLoading: false, errors: [] };
  const [state, dispatch] = useReducer(reducer, initialState);
  const exampleRule = "from:twitterdev has:links";
  const ruleMeaning = `This example rule will match Tweets posted by 
     TwtterDev containing links`;
  const operatorsURL =
    "https://developer.twitter.com/en/docs/twitter-api/tweets/filtered-stream/integrate/build-a-rule";
  const rulesURL = "/api/rules";

  const createRule = async (e) => {
    e.preventDefault();
    const payload = { add: [{ value: state.newRule }] };

    dispatch({ type: "change_loading_status", payload: true });
    try {
      const response = await axios.post(rulesURL, payload);
      if (response.data.body.errors)
        dispatch({ type: "add_errors", payload: response.data.body.errors });
      else {
        dispatch({ type: "add_rule", payload: response.data.body.data });
      }
      dispatch({ type: "change_loading_status", payload: false });
    } catch (e) {
      dispatch({
        type: "add_errors",
        payload: [{ detail: e.message }],
      });
      dispatch({ type: "change_loading_status", payload: false });
    }
  };

  const deleteRule = async (id) => {
    const payload = { delete: { ids: [id] } };
    dispatch({ type: "change_loading_status", payload: true });
    await axios.post(rulesURL, payload);
    dispatch({ type: "delete_rule", payload: id });
    dispatch({ type: "change_loading_status", payload: false });
  };

  const errors = () => {
    const { errors } = state;

    if (errors && errors.length > 0) {
      return errors.map((error) => (
        <ErrorMessage key={error.title} error={error} styleType="negative" />
      ));
    }
  };

  const rules = () => {
    const { isLoading, rules } = state;

    const message = {
      title: "No rules present",
      details: [
        `There are currently no rules on this stream. Start by adding the rule 
        below.`,
        exampleRule,
        ruleMeaning,
      ],
      type: operatorsURL,
    };

    if (!isLoading) {
      if (rules && rules.length > 0) {
        return rules.map((rule) => (
          <Rule
            key={rule.id}
            data={rule}
            onRuleDelete={(id) => deleteRule(id)}
          />
        ));
      } else {
        return (
          <ErrorMessage
            key={message.title}
            error={message}
            styleType="warning"
          />
        );
      }
    } else {
      return <Spinner />;
    }
  };

  useEffect(() => {
    (async () => {
      dispatch({ type: "change_loading_status", payload: true });

      try {
        const response = await axios.get(rulesURL);
        const { data: payload = [] } = response.data.body;
        dispatch({
          type: "show_rules",
          payload,
        });
      } catch (e) {
        dispatch({ type: "add_errors", payload: [e.response.data] });
      }

      dispatch({ type: "change_loading_status", payload: false });
    })();
  }, []);

  return (
    <div>
      <form onSubmit={(e) => createRule(e)}>
        <div className="ui fluid action input">
          <input
            type="text"
            autoFocus={true}
            value={state.newRule}
            onChange={(e) =>
              dispatch({ type: "rule_changed", payload: e.target.value })
            }
          />
          <button type="submit" className="ui primary button">
            Add Rule
          </button>
        </div>
        {errors()}
        {rules()}
      </form>
    </div>
  );
};

export default RuleList;

Next, create the child component of RuleList.js responsible for displaying a single rule and deleting a rule.

Rule.js

 import React from "react";

 export const Rule = ({ data, onRuleDelete }) => {
   return (
     <div className="ui segment">
       <p>{data.value}</p>
       <div className="ui label">tag: {data.tag}</div>
       <button
         className="ui right floated negative button"
         onClick={() => onRuleDelete(data.id)}
       >
         Delete
       </button>
     </div>
   );
 };

 export default Rule;

Next, create a component for displaying any status or error messages.

ErrorMessage.js

import React from "react";

const ErrorMessage = ({ error, styleType }) => {
  const errorDetails = () => {
    if (error.details) {
      return error.details.map(detail => <p key={detail}>{detail}</p>);
    } else if (error.detail) {
      return <p key={error.detail}>{error.detail}</p>;
    }
  };

  const errorType = () => {
    if (error.type) {
      return (
        <em>
          See
          <a href={error.type} target="_blank" rel="noopener noreferrer">
            {" "}
            Twitter documentation{" "}
          </a>
          for further details.
        </em>
      );
    }
  };

  return (
    <div className={`ui message ${styleType}`}>
      <div className="header">{error.title}</div>
      {errorDetails()}
      {errorType()}
    </div>
  );
};

export default ErrorMessage;

Finally, create a component to display a loading indicator during any pending API calls.

Spinner.js

import React from "react";

const Spinner = () => {
  return (
    <div>
      <div className="ui active centered large inline loader"></div>
    </div>
  );
};

export default Spinner;

Proxy Setup

The final step is to proxy requests from your client to your backend server. To do this, from within your src/ directory, create a new file called “setupProxy.js” and add the following code.

setupProxy.js

const { createProxyMiddleware } = require("http-proxy-middleware");

// This proxy redirects requests to /api endpoints to
// the Express server running on port 3001.
module.exports = function (app) {
  app.use(
    ["/api"],
    createProxyMiddleware({
      target: "http://localhost:3001",
    })
  );
};

You can now start up both the server and client by going to the project root directory and typing the following.

npm start

After this command completes, your default web browser should automatically launch and navigate to http://localhost:3000. You will then be taken to the rules management section of the app.

Now that you have an app in place to listen for any kind of Tweet you want, let’s walk through some real-life examples of how this app can be used such as

• Discovering new music videos
• Finding remote developer job openings
• Learning about personal finance and savings

For each of the examples and accompanying rules listed below, you can navigate to the rules section of the app and simply copy and paste the rule into the input field to add it to your stream. Once the rule has been added, it will take effect within seconds and only Tweets matching the criteria of that rule will be sent to you.

• Remote developer job openings
  
  In this first example, let’s say you are interested in finding remote developer job openings. To surface these kinds of Tweets, you can use the following rule.
  
  (developer OR engineer) remote (context:66.961961812492148736 OR context:66.850073441055133696)
  
  To understand what this rule is doing, you can break it down into two parts. The keywords part and the Tweet annotations part.
  
  Using keyword operators
  
  (developer OR engineer) remote
  
  The keywords part of the rule will match Tweets containing the keywords “developer” or “engineer” and the keyword “remote”. These keywords alone will certainly match Tweets containing remote developer job postings, but it will also match on irrelevant Tweets. For example, the Tweet below will match this rule.
  https://twitter.com/EraserFarm/status/1220013392766947332
  
  Since this is not the desired Tweet, you need to take this rule a step further. Though this rule matches irrelevant Tweets, it is also successful in matching Tweets with actual job postings. For example, the rule you have so far will also result in matching the Tweet below.
  
  https://twitter.com/plataformatec/status/1225460423761317888
  
  Tweet annotations: Using context operators
  
  The challenge you have now is, though you are receiving Tweets containing job postings you will still have to go through irrelevant Tweets. If only there was a way to only match on Tweets that contain job postings as best as possible. Are there operators you can use in your rule that only match these relevant Tweets?
  
  Fortunately, this is where the power of Tweet annotations comes in. Take a closer look at an example Tweet object payload that was sent for this Tweet on your filtered stream endpoint. Within the payload, the nested “context_annotations” field is present. Context annotations are delivered as a “context_annotations” field in the payload. These annotations are inferred based on the Tweet text and result in the domain and/or entity labels, which can be used to discover Tweets on topics that may have been previously difficult to surface. Note, that these fields will only be included if data is present since not all Tweets will contain this data.
  
  

   "context_annotations": [
       {
         "domain": {
           "id": "66",
           "name": "Interests and Hobbies Category",
           "description": "A grouping of interests and hobbies entities, like Novelty Food or Destinations"
         },
         "entity": {
           "id": 961961812492148736,
           "name": "Recruitment",
           "description": "Recruitment"
         }
       },
       {
         "domain": {
           "id": "66",
           "name": "Interests and Hobbies Category",
           "description": "A grouping of interests and hobbies entities, like Novelty Food or Destinations"
         },
         "entity": {
           "id": "850073441055133696",
           "name": "Job search",
           "description": "Job search"
         }
       }
     ],
  

  
  
  To match on the domain and entity ids within the context_annotations field, you can use the “context” operator. The “context” operator can be used to instruct your filtered stream endpoint to match on Tweets containing specific domain and entity names. Here’s what that would look like.
  
  (context:66.961961812492148736 OR context:66.850073441055133696)
  
  The operators above follow the format “context:.”. As seen in the example payload above, the domain id 66 represents the “Interests and Hobbies Category”. The entity ID 961961812492148736 represents the “Recruitment” entity and the entity ID 850073441055133696 represents the “Job search” entity. For a complete list of domains, the Tweet annotations docs contain a table with 50+ domain names.
  
  With the operator explanation out of the way, this 2nd part of the rule matches Tweets containing the entity names “Recruitment” or “Jobs search”.
  
  In summary, taking both parts of this rule together, it will match on Tweets that contain the keywords “developer” or “engineer” and the keyword “remote”, but only if those Tweets also contain the entity names “Recruitment” or “Jobs search”
  
  

• Discovering new music videos
  
  If you need new music video suggestions, you can start by using a simple rule that matches on Tweets containing the keywords “song” and “YouTube”. You will also want Tweets that actually link out to external videos
  
  song youtube has:links
  
  Taking a closer look at the payload of this Tweet, you see that it has some annotations on it that can help you match more relevant Tweets. Notice the Annotation with an entity label of “Pop” and a domain name of “Music Genre”
  
  

   "context_annotations": [
      {
        "domain": {
          "id": "10",
          "name": "Person",
          "description": "Named people in the world like Nelson Mandela"
        },
        "entity": {
          "id": "871815676998033408",
          "name": "Ally Brooke",
          "description": "Ally Brooke"
        }
      },
      {
        "domain": {
          "id": "54",
          "name": "Musician",
          "description": "A musician in the world, like Adele or Bob Dylan"
        },
        "entity": {
          "id": "871815676998033408",
          "name": "Ally Brooke",
          "description": "Ally Brooke"
        }
      },
      {
        "domain": {
          "id": "55",
          "name": "Music Genre",
          "description": "A category for a musical style, like Pop, Rock, or Rap"
        },
        "entity": {
          "id": "810938279801470977",
          "name": "Pop",
          "description": "Pop"
        }
      }
    ],
  

  
  
  To make this rule better and narrow down your Tweets to be even more relevant, you can update your rule as follows.
  

  song youtube has:links context:55.810938279801470977
  

  This will take the original rule you used a step further by narrowing down to only the Tweets that are labeled with the Music Genre domain label and the Pop entity label.
  
  

• Learning about personal finance and savings
  
  As a final example, let’s say you are interested in learning about personal finance and you can be savvier about your spending and savings. You also only want original Tweets that contain links to online articles to learn more.
  
  Going through a similar process as you did with earlier examples if you simply add the following rule, only Tweets containing the words “personal”, “finance” and “savings" will be sent to you.
  
  “personal finance savings”
  
  Taking a look at one of the Tweet payloads, the nested annotations contains an entity label about personal finance that will help you narrow down your Tweets to the most relevant ones.
  
  

  "annotations": {
    "context": [
      {
        "context_domain_id": 66,
        "context_domain_id_str": "66",
        "context_domain_name": "Interests and Hobbies Category",
        "context_domain_description": "A grouping of interests and hobbies entities, like Novelty Food or Destinations",
        "context_entity_id": 847888632711061504,
        "context_entity_id_str": "847888632711061504",
        "context_entity_name": "Personal finance",
        "context_entity_description": "Personal finance"
      },
  

  
  
  Using the context operator to match on Tweets containing this label, you can revise your rule to look as follows.
  

  context:66.847888632711061504 has:links -is:retweet savings
  

Conclusion

Using the filtered stream endpoints to stream publicly available Tweets to your server and annotations, you created an app to more easily surface Tweets around a topic of interest. The filtered stream endpoint gives you the haystack of data in the form of Tweets and the Tweet annotations help you find the needle in that haystack.

Have you found other interesting ways to use this app? Follow me on Twitter and send me a Tweet to let me know. I used several libraries beyond the Twitter API to make this tutorial, but you may have different needs and requirements and should evaluate whether those tools are right for you.

Check out the live demo of the app on Glitch.