DEV Community: Wale Ayandiran

Twitter historic data scrapping with tweepy.

Wale Ayandiran — Mon, 05 Dec 2022 15:12:45 +0000

Getting tweets and tweets data (dates, like_count, retweets) etc from Twitter is easy as logging into the micro-blogging platform and taking any tweet you want from any publicly available account. This makes sense if one is only interested in a tweet or at most less than 10 tweets from a single account. it becomes a hassle when you need to fetch tweets from multiple users or a significant number of tweets.

Twitter provides a way to do this and more with their set of APIs.

To get historic data dated way back, you would need an academic access to the twitter API with proof that you're actually an academician at the time and a brief overview of the work you're doing with the data gathered.

Getting access to the API service is the straightforward part. After the access has been granted, how to fetch and maybe store such data in a way that twitter doesn't see your requests as spam due to the limits they have on the number of results you get per requests can be quite challenging.

This article will focus on fetching large data continuously while avoiding API rate limit.

TLDR;
here's the code below.

https://github.com/waleCloud/me/blob/91baedc52e5f65002bba01cf2bc062e9e9f1ff88/content/blog/tweepy-fetch-historical-tweet-stream/fetch_tweets.ipynb

Setup Twitter Developer Account
Create an APP and choose Academic Research Continue to follow the twitter verification steps and approval.

Finally once approval has been granted, it times to begin fetching tweets as far back as you need and the volume, lets get started.

Before we begin, let setup Python and Tweepy installed with Pip.

pip install tweepy
Next is to get a bearer token from Twitter on the app you've been granted approval on.

Let's connect to Twitter service with the token provided;


import tweepy

def authenticate():
  BEARER_TOKEN = 'Put your bearer token here'
  # attempt authentication
  try:
    print("Authenticating...")
    # create Client object
    client = tweepy.Client(bearer_token=BEARER_TOKEN)
    print("API value:....   ", client)
    # return client object
    return client
  except Exception as e:
    print("Error: Authentication Failed", e)

The above function when called will return the API client if the token provided is correct or prints out an error message.

API = authenticate()

Assuming everything goes well, the API can now be used to access Twitter's data.

Lets create a function to search for tweets using the search_all_tweets method.


def get_tweets(query, start_time, end_time, next_token):

  # empty list to store parsed tweets
  tweets = []

  expansions = ['author_id,in_reply_to_user_id,geo.place_id']
  tweet_fields = ['id,text,author_id,in_reply_to_user_id,geo,conversation_id,created_at,lang,public_metrics,referenced_tweets,reply_settings,source']
  user_fields =  ['id,name,username,created_at,description,public_metrics,verified']
  place_fields = ['full_name,id,country,country_code,geo,name,place_type']
  try:
    # call twitter api to fetch tweets
    fetched_tweets = API.search_all_tweets(query,
      end_time=end_time,
      start_time=start_time,
      expansions=expansions,
      tweet_fields=tweet_fields,
      place_fields=place_fields,
      user_fields=user_fields,
      next_token=next_token
    )

    return fetched_tweets

  except Exception as e:
    print("Error getting tweets", e)

The get_tweets function takes 4 parameters which includes the search query, start_time date to begin searching from, end_time end date of search, next_token a token that will be sent from twitter when the result is large and can only return a distinct number per request, we will be using this token to make subsequent call to this function. Imagine the next_token as some form of pagination to the next page, in this case the next set of results until it gets to the end of the list.

test out the search to confirm it works


result = get_tweets('elon musk', '2022-10-01T00:00:01Z', '2022-11-01T00:00:01Z',next_token=None)

print(result)

Since we want to have records of the tweets stored for processing further down the line, let's create a method to store the tweets in a csv file.

For this, we will need the pandas library,csv and time, import them as required.


import pandas as pd
import csv
import time

def append_to_csv(result_set, file_name):

    # A counter variable
    counter = 0
    # Open OR create the target CSV file
    csv_file = open(file_name, "a", newline="", encoding='utf-8')
    csv_writer = csv.writer(csv_file)

    # Loop through each tweet
    for tweet in result_set.data:
        # We will create a variable for each item since some of the keys might not exist for some tweets
        # So we will account for that

        author_id = tweet['author_id']
        created_at = tweet['created_at']
        if ('geo' in tweet):   
            geo = tweet['geo']['place_id']
        else:
            geo = " "
        tweet_id = tweet['id']
        lang = tweet['lang']
        retweet_count = tweet['public_metrics']['retweet_count']
        reply_count = tweet['public_metrics']['reply_count']
        like_count = tweet['public_metrics']['like_count']
        quote_count = tweet['public_metrics']['quote_count']
        source = tweet['source']
        text = tweet['text']

        # Assemble all data in a list
        res = [author_id, created_at, geo, tweet_id, lang, like_count, quote_count, reply_count, retweet_count, source, text]        

        # Append the result to the CSV file
        csv_writer.writerow(res)
        counter += 1

    # When done, close the CSV file
    csv_file.close()

    # Print the number of tweets for this iteration
    print("# of Tweets added from this response: ", counter)

Things are beginning to take shape, now the final piece to joining all these functions to create a stream of tweets.


count = 0 # Counting tweets per time period
max_count = 100 # Max tweets per time period
flag = True
next_token = None
query = 'Parag Musk'
start_time = '2022-10-01T00:00:01Z'
end_time = '2022-11-01T00:00:01Z'
 # Total number of tweets we collected from the loop
# Check if flag is true
def run_fetch(flag=True, next_token=None, count=0, total_tweets=0 ):
  if (flag is not True):
    return
  while flag:
    print("-------------------")
    print("Token: ", next_token)
    result = get_tweets(query=query, start_time=start_time, end_time=end_time, next_token=next_token) # get_tweet function called
    result_count = result.meta['result_count']
    if result_count is not None and result_count > 0:
      # print("Start Date: ", start_list[i])
      append_to_csv(result, "data.csv")
      count += result_count
      total_tweets += result_count
      print("Total # of Tweets added: ", total_tweets)
      print("-------------------")
      if result.meta['next_token']:
        # Save the token to use for next call
        next_token = result.meta['next_token']
        print("Next Token: ", next_token)
        time.sleep(5)
        run_fetch(True, next_token=next_token, count=count, total_tweets=total_tweets)
      else:
        # Since this is the final request, turn flag to false to move to the next time period.
        flag = False
        next_token = None
    time.sleep(5)
  print("Total number of results: ", total_tweets)

The above code runs recursively until there's no Next_token which stops the search, a sleep for 5 milliseconds to allow guard against rate limit before the next run_check.

With that we can now begin the stream by running run_fetch().

With this approach, you can get as many tweets as you want continuously, one caveat is your machine needs to be running and not go into hibernate/sleep mode else the process breaks.

One way I can quickly think of to mitigate this is to run this over the cloud on something like google colab, that way you don't have to worry about your machine sleeping as truly its google's machine that carry's all the burden 😅.

I hope this helps someone.

Scaling Frontend Applications - Coding Guidelines

Wale Ayandiran — Sun, 03 Apr 2022 03:11:52 +0000

Structuring your frontend application can become tricky as your product and codebase grows. In this post, I will be sharing what I have found to a working solution for almost any kind of Frontend project size. This guideline is strongly inspired by a pattern termed as the duck pattern.

The entire idea of ducks is to group seemingly related files together in a way that makes it modular, modification easier, scalable, and easy to deconstruct whenever a need arises and move technologies around like state libraries etc.

Ducks are extroverted, gregarious animals who feel most at minimal fuss when they're in groups.
Duck pattern at its core is about colocating small files that work together as a unit to make working with them a breeze. Lets see guideline below;

Files and Folder Convention
Components
- Presentational Components
- Connected Components
- Styling Components
Interacting with Backend
State sharing

File and Folder Convention

Using the feature pattern to colocate feature related files rather than by functions, lets take a look at a login example

Feature-first ✅

Login/
  Login.tsx
  index.ts
  store/
    reducers.ts
    actions.ts

"Feature-first" refers to naming your top-level folders after the primary feature your app contains, Login in this case.

Because each new feature comes with its own folder, this technique scales considerably better.

You can have files that aren't associated with any functionality and call them common/shared/core e.t.c. because you want to reuse code across several functionalities in your product.

Function-first ❌

Components/
  Login.tsx
  Signup.tsx
Containers/
  Login.tsx
store/
  reducers.ts
  actions.ts

"Function-first" refers to naming your top-level folders after the purpose of the files they contain.
So far, you've got containers, components, actions, reducers, and so on.

This isn't going to scale at all.
Files are added to the same directories as your programme evolves and additional features are added.

The issue also involves tying the folders together.
A single flow in your programme will almost certainly necessitate editing files from all directories.

Using the "Feature-first" approach, we can generate a general Project struture like so;

src/
  pages/ ---> Contains top level files rendering as a page
    login {feature-folder}/ ---> Would contains components, api|hooks|actions files & folders related to login pages, if these components are going to be reused elsewhere aside login, move it into the core/components directory.
  core/ ---> Globally shared, reusable, components and files JSX related.
    components/ ---> Globally Shared React components, mostly dumb/presentational components
      {ComponentName}/
        ComponentName.tsx ---> Using named exports e.g `export const ComponentName = () => {}` Always keep this file as simple as possible
        Styles.tsx ---> A case for using styledComponents, all created elements will be stored here, exported using named exports
        index.ts ---> exports { ComponentName } from './Componentname'
        utils.ts ---> Optional when you need to move some functions out of the component file to keep things clean.
  utils/ ---> JS files that are globally needed, helper functions, etc.

Components

Your Frontend Components will most likely be grouped into 2 kinds, presentational and connected components.

Worthy to Remember

Endeavor to use functional components all through because, why not 🤷🏾? it saves you from dealing with class components and its numerous lifecycle methods.
Have an index file that exports all components from a feature folder, helps to organize your imports and exports.

Presentational Components

Have no dependencies on the rest of the application.
Values and callbacks are passed into these via props.

Example:

ComponentName/
  ComponentName.tsx ---> Using named exports e.g `export const ComponentName = () => {}` Always keep this file as simple as possible
  Styles.tsx ---> A case for using styledComponents, all created elements will be stored here, exported using named exports
  index.ts ---> exports { ComponentName } from './Componentname'
  utils.ts ---> Optional when you need to move some functions out of the component file to keep things clean.

export const PresentationComponent = ({ prop1, props2 ...propN }) => (
  <div>Show something</div>
);

Connected Components

are responsible for retrieving data.
are aware of the store and be connected to it.
provide data to other components.
are responsible for dispatching actions.
grab data from store and then passes that data down to its children as props.

Example:

ComponentName/
  ComponentName.tsx ---> Using named exports e.g `export const ComponentName = () => {}` Always keep this file as simple as possible
  Styles.jsx ---> A case for styledComponents, all created elements will be stored here, exported using named exports
  actions/ ---> handles all Async events, and certain Events that needs to be seperated from the components.
    store/ reducers/ etc
    api|hooks/
  index.js ---> exports { ComponentName } from './Componentname'
  utils.js ---> Optional when you need to move some functions out of the component file to keep things clean.

Styling Components

Because I've been a making a case for using styled components, we will like to keep these clean and away from jsx logic. All created styled components will be inside an Styles.js file inside the component folder.

Example:

// Styles.js
import styled from "styled-components";

export const Header = styled("div")`
  display: flex;
  flex-direction: column;
  width: 500px;
  height: 200px;
`;

export const Footer = styled("div")`
  display: flex;
  flex-direction: column;
  width: 500px;
  height: 200px;
`;
export const LeftMenu = styled("div")`
  display: flex;
  flex-direction: column;
  width: 500px;
  height: 200px;
`;

Interacting with Backend

All backend related actions should be in the actions folder within each components directory. see Connected Components above.

State sharing

There's a couple of options for this and I see most teams now are leaning towards React Context for React applications,
other worthy mentions include:
Redux, VueX, Mobx.

Same philosophy applies regardless of the state library employed.

Actions/
  reducers/

This writeup is highly opinionated on my experience but a lot of teams both small and large have similar approach to handling their frontend applications.

Let me know if you find this useful or have questions or share how you've been able to structure your frontend applications at work.

Photo by Simone Hutsch on Unsplash

Using React useRef Hook to access immediate past props or state.

Wale Ayandiran — Thu, 10 Feb 2022 05:05:51 +0000

originally posted on my personal blog walecloud.me

Using React useRef Hook to access immediate past props or state

Do you want to know what the previous state of a value is before it got updated in React?
You can easily leverage useRef to track the previous value of state or props in React.

Recently, while working on a project built with React and Firebase, we had a use-case for knowing what the previous state of a value was. This need came to be when a state item needed to be reset if an ID from firebase changes.

The useRef hook in react is ideal for things like this, you probably thought its sole purpose is for DOM manipulation but it can be more and almost anything you want it to be.

TL;DR

// usePrevious hook React official documentation

import { useEffect, useRef } from 'react';

export const usePrevious = (value) => {
  const ref = useRef();
  useEffect(() => {
    ref.current = value;
  }, [value]);
  return ref.current;
};

// use returned value like so;

  const prevModuleId = usePrevious(moduleId);

Hopefully Reacts make the usePrevious into an official hook soon as it seems like a no-brainer.

How does the usePrevious hook works?

Short answer:

useRef: A container that is useful to keep a mutable (changeable) value in its .current property
useEffect: Allows for monitoring changes and performing side effects in functional components.

You can read more about both hooks on the React official site

First, we create an instance of Ref whenever the hook is called.
The useEffect only runs when the value parameter changes and then assign that to the ref's .current property
Finally, we return the ref.current.

The first time the hook is called, ref.current will be undefined until a state or props value changes until then before the useEffect hook is executed to reflect the latest previous value of the parameter.

Find this helpful? Kindly share so others can too.
cheers 🥂

Azure App slots - (Making releases seamless)

Wale Ayandiran — Tue, 01 Oct 2019 22:06:25 +0000

Hello! So I was going through a talk I gave late 2017 about Continous delivery using Azure App slots, presentation link => https://speakerdeck.com/walecloud/continuous-delivery-microsoft-azure-app-slots.

I will be highlighting the steps needed to create an application in Azure using the APP Service, deploying varying versions with App slots.

Disclaimer: Some things have probably changed since then so thread with caution;

Step 1: Create the project source

Step 2: Create a web app in Azure

Step 3: Config Azure web App

Step 4: Choose source code repo for deployment

Step 5: Create a feature branch in source code

Step 6: Add app slot

Step 7: deploy a new app feature

Step 8: Swap slots for new feature added

Well, I hope it helps someone deploying with Azure.

Link to the actual work that inspired this => https://speakerdeck.com/walecloud/continuous-delivery-microsoft-azure-app-slots

Updating react nested state properties

Wale Ayandiran — Sun, 06 Jan 2019 17:49:47 +0000

Working with react inevitable requires you to manage the state of your application at some point, bearest minimum the state of a component you're currently working with.

Recently, I was working on a side project, and I felt like trying something different from the regular

this.state = { email: '', password: '' };

I was trying to login a user with email and password credential, so I felt like wrapping that up into a user object for human readability or better still no reason sake :).

I did this

this.state = {
  user: {
    email: '',
    password: ''
  }
};

Then i noticed my form fields were not taking in the inputs, more like its on readOnly mode.

After googling, it occured to me that the state wasn't updating using the regular

 onChange(e) {
   const { name, value } = e.target;
   this.setState({ [name]: value });

Why because the properties (email and password) are nested under the user state property. At first, I was like that shouldn't be much of a problem, calling user.email, user.password should do the trick right? You guessed right, it didn't and there i was thinking straight i just wanted things to work so I can move on wiht my code.

The Solution

Quick answer:

onChange(e) { 
  const { user } = { ...this.state };
  const currentState = user;
  const { name, value } = e.target;
  currentState[name] = value;

  this.setState({ user: currentState });
}

Long Version:

react's setState doesn't take care of nested properties, in this case email and password. So the only way to make changes is to access the parent state object user whenever a new change occurs to either email or password.

What this means is that everytime you type a new character on any of the field, the user state gets to be re-created. This is not optimal if your user object has lots of fields say like a bulky registration form dealing with over 10 fields is not considered ideal.

The above code uses some es6 features, such as the spread operator & Destructuring.

It's a general bad practise to mutate (add/remove elements of the state directly) state in react. states should be recreated if it must change.

The line const { user } = { ...this.state } does just that using the spread operator(...) to get the current state. Assigned the value returned to currentState variable const currentState = user;, destructure the emmitted events const { name, value } = e.target; coming from the input fields to get the name and thier value. Update the currentState with the value of the input currentState[name] = value; where name is the property of the user state object email and password.

Finally, we transpose the user with the currentState whenever an update has been successfully made this.setState({ user: currentState });.

My recommendation

Some people will argue about using nested properties because React is not oriented to work with nested state. Maybe this is correct or not but here's what I think;

I would suggest you avoid using nested state, if you must then endeavor to make it as light as possible because the downside is every tiny piece of change will recreate the parent object, which is not good for performance.