Explore Geopolitical data from GDELT

Shreyas Soni — Wed, 06 Oct 2021 18:15:11 +0000

In this blog, we will explore the geopolitical data from GDELT and see how that data can be used in the analysis.

What is GDELT?

The GDELT Project created by Kalev H. Leetaru monitors the world's news from every country in over 100 languages and identifies the people, locations, organizations, themes, sources, emotions, counts, quotes, images, and events driving our global society.

In this blog, we will have a look at the Events database of GDELT and how this data can be used for analysis.

Event Database

The GDELT Event Database catalog over 20 main categories and more than 300 subcategories. Each category is given a particular cameo code. We will be looking into the 20 main cameo codes. That includes

Make Public Statement
Appeal
Express intent to cooperate
Consult
Engage in diplomatic cooperation
Engage in material cooperation
Provide aid
Yield
Investigate
Demand
Disapprove
Reject
Threaten
Protest
Exhibit military posture
Reduce relations
Coerce
Assault
Fight
Use unconventional mass violence

Let's see how we can get the data for these events for all countries.

How to get the data?

BigQuery You can query any data you want according to your need. Here is an example of a query.

select SQLDATE,EventRootCode,Actor1CountryCode,NumMentions from gdeltv2.events;

Using gdelt python package
- Installation: pip install gdelt
- Call the gdelt version 2 database.

gd2 = gdelt.gdelt(version=2)

Use gd2 object to search for the data of a given date and set table to events.

results = gd2.Search(['2020-01-01'],table='events',coverage=True)

Processing the data to get Timeseries data for all countries

Load the data into the notebook.

df = pd.read_csv("gdelt.csv");

The data output of the gdelt object has all the columns present in the events database. Now filter it to the columns necessary, i.e., SQLDATE, EventRootCode, Actor1CountryCode, NumMentions

results = results[['SQLDATE','EventRootCode','NumMentions','Actor1CountryCode']]

Convert the SQLDATE format from 'YYYYMMDD' to 'YYYY-MM-DD'.

results['SQLDATE'] = results['SQLDATE'].apply(lambda x: pd.to_datetime(str(x), format='%Y-%m-%d'))

Aggregate the data based on SQLDATE, EventRootCode, and Actor1CountryCode.

results = results.groupby(['SQLDATE','EventRootCode','Actor1CountryCode']).agg('sum').reset_index()

Data Analysis and Visualization

Mapping a Line Chart of a particular Cameo code for the country over time.
Example: Protest in USA (Aggregated to Weekly basis)
Mapping Top Cameo codes in a country based on the Number of Mentions of the particular cameo code.
Example: Top Trends in USA (Last Week)

Mapping Top Countries in a particular cameo code based on the Number of Mentions of the particular cameo code in the country.
Example: Top Countries in Protest (Last Week)
Plot a choropleth map for a particular cameo code.
Example: Protest (Today)

Technology Used

Python
Pandas
Plotly

Code: Link

Co-author: @ashishsalunkhe