DEV Community: konkon

Analysis of ECS 236th meeting abstracts(2) - word embedding by Word2Vec and SCDV

konkon — Mon, 07 Oct 2019 02:58:58 +0000

Introduction

This is an serial article about language analysis of ECS 236th meeting abstracts.
In this series, I've been explaining the technique used in my webapp ECS Meeting Explorer. The introduction of this app is available in an article below,

ECS Meeting Explorer - webapp for scientific conference

My previous article about data scraping is in following link,

Analysis of ECS 236th meeting abstracts(1) - data scraping with BeautifulSoup4

In this atricle, I will give an explanation of word embedding, vectorization of words used in all abstract text.

Preparation

In the series of article, I will use Python. Please install these libraries.

numpy > 1.14.5
pandas > 0.23.1
matplotlib > 2.2.2
beautifulsoup4 > 4.6.0
gensim > 3.4.0
scikit-learn > 0.19.1
scipy > 1.1.0

Before the analysis, please download all ECS 236th meeting abstracts from official site. Unzip and place it in same directory as jupyter-notebook.

Data scraping by BeautifulSoup4 was explained in my previous article, please check it before!

Word embedding by Word2Vec

Word2Vec (W2V) is a machine learning model used to produce word embedding, which is words mapping to vector space.

Word2Vec is a kind of unsupervised learning, therefore we don't have to label training data. It is precious to me because it is a hard job at any time.

In this experiments, we use Word2Vec implemented in Gensim. So, we don't have to make models by ourselves. Further information about Word2Vec are below,

models.word2vec – Word2vec embeddings(Gensim documentation)
Word2vec Tutorial | RARE Technologies

The original paper of word2vec.

Distributed Representations of Words and Phrases and their Compositionality

Now, we have a list contains detail of all abstract, title, authors, affiliations, session name, and contents as follows,

> dic_all
[{'num': '0001',
 'title': 'The Impact of Coal Mineral Matter (alumina and silica) on Carbon Electrooxidation in the Direct Carbon Fuel Cell',
 'author': ['Simin Moradmanda', 'Jessica A Allena', 'Scott W Donnea'],
 'affiliation': 'University of Newcastle',
 'session': 'A01',
 'session_name': 'Battery and Energy Technology Joint General Session',
 'contents': 'Direct carbon fuel cell DCFC as an electrochemical device...',
 'mod_contents': ['direct','carbon','fuel','cell', ... ,'melting'],
 'vector': 0,
 'url': '1.html'}, ... ]

Then, Let's get lists of words modified for language analysis.

# make word list for W2V learning
docs = [i['mod_contents'] for i in dic_all]

This is a code for learning Word2Vec model. Only few lines!

#Word2Vec model learning and save it.
from gensim.models.word2vec import Word2Vec
import logging
logging.basicConfig(format='%(asctime)s : %(levelname)s : %(message)s', level=logging.INFO)
model = Word2Vec(docs, sg=1, size=200, window=5, min_count=30, workers=4, sample=1e-6, negative=5, iter=1000)
print('corpus = ',model.corpus_count)

The line 5 'model = Word2Vec(docs, ...)' corresponds to Word2Vec learning. The parameter 'size' sets the dimension of word vectors, in this case, 200. Please see documents for the other parameters of this function.

After the learning, make a vocabulary and word vectors from Word2Vec model.
This vocabulary is saved as .npy file in same directory.

#make word dictionary
vocab = [i for i in model.wv.vocab]
dictionary = {}
for n,j in enumerate(vocab):
    dictionary[j] = n
np.save('dictionary.npy', np.array([dictionary]))

#make word vectors from the model.
word_vectors = [model.wv[i] for i in model.wv.vocab]
word_vectors = np.array(word_vectors)

Now, we gets the words list and corresponding vectors.
In this vector space, similarity between words is expressed as a distance. We usually uses cosine distance for such a high-dimentional vector spaces.

The function for calculate a word similarity is below,

def CalcSim(target, vectors, dictionary):
    target_vec = vectors[dictionary[target]]

    search_results = []
    for n,vector in enumerate(vectors):
        sim = cos_sim(target_vec,vector)
        result = {'num': n, 'value': list(dictionary.keys())[n], 'similarity': sim}
        search_results.append(result)

    summary_pd = pd.io.json.json_normalize(search_results)
    summary_sorted = summary_pd.sort_values('similarity', ascending=False)
    return summary_sorted

Okay, Let's search the similar words to 'sustainable', recent buzz-words.

target='sustainable'
summary_sorted = CalcSim(target, word_vectors, dictionary)
summary_sorted[:10]

The result, top 10 words are shown like this,

num	similarity	value
588	1	sustainable
105	0.648442	renewable
100	0.552662	energy
1625	0.54727	fuels
862	0.541807	efficient
1624	0.53353	fossil
13	0.521877	electricity
607	0.480525	technologies
138	0.472065	production
108	0.471985	wind

The word most similar to 'sustainable' is 'renewable'.
It's satisfactory result, isn't it?

2-dimentional visualization of word vectors

As I mentioned, the size of word vectors is 200.
It is impossible for human beings to imagine such a high dimensional data. Dimension reduction is needed for the visualization.

In this case, we will use Principal Component Analysis (PCA) from 200 to 100, and t-distributed Stochastic Neighbor Embedding (t-SNE) from 100 to 2. These methods are implemented in scikit-learn.

The function for dimension reduction is this,

from sklearn.decomposition import IncrementalPCA
from sklearn.manifold import TSNE

def tsne_reduction(dataset):

    n = dataset.shape[0]
    batch_size = 500

    ipca = IncrementalPCA(n_components=100)
    for i in tqdm(range(n//batch_size)):
        r_dataset = ipca.partial_fit(dataset[i*batch_size:(i+1)*batch_size])
    r_dataset = ipca.transform(dataset)

    r_tsne = TSNE(n_components=2, random_state=0, perplexity=50.0, n_iter=3000).fit_transform(r_dataset)

    return(r_tsne)

w2v_tsne = tsne_reduction(word_vectors)

Now, we can plot 2-dimensional word vectors.

Left shows the scatter plots of all word vectors, Right shows some highlighted points with corresponding words.

Precomputation of word-topic vectors by SCDV

We can estimate document vectors of abstract by averaging this word vectors with certain weight (such as tf-idf). But in this case, I will apply a method named SCDV: Sparse Composite Document Vectors to modify word vectors.

There are 2 steps for SCDV to build a document vector.

Precomputation of word-topics vectors.
Build sparse document vectors using word-topics vectors.

In this section, I will explain the former process.

This is a flow chart of computing word-topic vectors (image from here). It is divided into 3 process.

Word vectors are classified into several clusters with soft clustering algorithms, which allows words to belong to every cluster with certain probability.
Word-cluster vectors are made by multiplying vectors with the probability of belonging for each cluster.
Concatenate all word-cluster vectors with idf (inverse document frequency) weighting to form word-topic vector.

This is a function to transform word vectors to word-topic vectors.


def WordTopicVectors(word_vectors)

    #Gaussian Mixture Modelling
    num_clusters = 30
    clf =  GaussianMixture(n_components=num_clusters,covariance_type="full")
    z_gmm = clf.fit(word_vectors)
    idx = clf.predict(word_vectors)
    idx_proba = clf.predict_proba(word_vectors)

    #Calculate word idf
    words = list(dictionary.keys())
    words = np.array(words)
    word_idf = np.zeros_like(words, dtype=np.uint32)

    for doc in tqdm(docs):
        lim = len(doc)
        for w in doc:
            if(lim == 0):
                break
            else:
                idx = np.where(w == words)
                word_idf[idx] += 1
                lim -= 1

    word_counts = word_idf
    word_idf = np.log(len(docs) / word_idf) + 1

    #Concatenate word vector with GMM cluster
    gmm_word_vectors = np.empty((word_vectors.shape[0], word_vectors.shape[1] * num_clusters))
    n = 0
    for vector,proba,idf in zip(word_vectors,idx_proba,word_idf):
        for m,p in enumerate(proba):
            if(m == 0):
                cluster_vector = vector * p
            else:
                cluster_vector = np.hstack((cluster_vector,vector * p))
        gmm_word_vectors[n] = idf * cluster_vector
        n += 1

    return(gmm_word_vectors)

#Calculate word-topic vectors
gmm_word_vectors = WordTopicVectors(word_vectors)

In this function, we used gaussian mixture model for clustering. The number of cluster is recommended as 60 or higher in original paper, but now I choose 30 (because of memory issue for webapp).

The dimension of word-topic vectors will be 200(original word vector)×30(number of cluster) = 6000.
Then, visualize it with t-SNE dimension reduction!

Comparing to the word vectors by Word2Vec, The clusters for each words are separated clearly.
This means that these vectors well represent the relationship between words and topics.

Let's see the details of each cluster and corresponding words.

This figure clearly shows that words of same topic belongs to the same cluster.

Conclusion

In this article, I demonstrated the word embedding by W2V and modification by SCDV.
I will explain about building document vector with this word-topic vectors!

Analysis of ECS 236th meeting abstracts(1) - data scraping with BeautifulSoup4

konkon — Sun, 29 Sep 2019 12:09:58 +0000

Introduction

Recently, I've published an article about introduction of my webapp, ECS Meeting Explorer.

ECS Meeting Explorer - Explore ECS meeting easily and deeply

I will give an explanation of key technique used in this app.
Main topic of this article is data scraping from .html file with BeautifulSoup4.

Environment

In the series of article, I will use Python. Please install these libraries.

numpy > 1.14.5
pandas > 0.23.1
matplotlib > 2.2.2
beautifulsoup4 > 4.6.0
gensim > 3.4.0
scikit-learn > 0.19.1
scipy > 1.1.0

Jupyter notebook is also available on my GitHub repository. Check it!

Download meeting abstract

First of all, we need to get whole data of meeting abstract.
As I mentioned in before article, we can download it as .zip from below.

https://www.electrochem.org/236/download-abstracts

After unzipped, please place it in same directory as jupyter-notebook.
Please keep in mind that it will expire in 9/30/2020.

Data Scraping from table of contents

As I mentioned, We use beautifulsoup4 to scraping data.
In the ECS meeting, all presentations are categorized as several research fields.

# Load Table of Contents
soup = bs4.BeautifulSoup(open('./ECS MA2019-02.003/Abstracts/tos.html', encoding='utf-8'),'lxml')

# Function to remove html tags
px = re.compile(r"<[^>]*?>")

#Load Categories
p_len = len(soup('h2'))
categorys = []
for i in range(p_len):
    c = px.sub("", str(soup('h2')[i]))
    categorys.append(c)
print("categoly_num ="+str(p_len))
print(categorys)

This is a result,

'A—Batteries and Energy Storage'
'B—Carbon Nanostructures and Devices'
'C—Corrosion Science and Technology'
'D—Dielectric Science and Materials'
'E—Electrochemical/Electroless Deposition'
'F—Electrochemical Engineering'
'G—Electronic Materials and Processing'
'H—Electronic and Photnonic Devices and Systems'
'I—Fuel Cells, Electrolyzers, and Energy Conversion'
'J—Luminescence and Display Materials, Devices, and Processing'
'K—Organic and Bioelectrochemistry'
'L—Physical and Analytical Electrochemistry
Electrocatalysis, and Photoelectrochemistry'
'M—Sensors'
'Z—General Topics'

There are 14 major categories in ECS 236th meeting.
Next, there are sub-categories for each. These are scraped from toc.html as follows,

#Load Sessions
p_len = len(soup('h3'))
sessions = []
for i in range(p_len):
    c = px.sub("", str(soup('h3')[i]))
    sessions.append(c)
print("session_num ="+str(p_len))
print(sessions)

There are 58 sub-categories. Too much to write it down!

Scrape abstract texts and properties of presentation

Now, let's begin to scrape abstract texts of presentation.
Before that, please load a stop word list.

#Load stopwords
with open('stopwords.txt', 'r', encoding='utf-8') as f:
    stops = f.readlines()
stopwords = []
for s in stops:
    s = s.replace("\n", "")
    stopwords.append(s)

This is a list of words filtered out before processing, such as pronoun, ordinary verbs and so on. It is necessary for improving word embedding and building a document vectors (will be explained in another article).

You can find stop word list as .txt format on a GitHub repository.
If you clone my GitHub repositry, it already contains a stopwords.txt.

Code for meeting abstract scraping as follows,

dic_all = []

# make session number and name dictionary
dic_sessions = {}
for s in sessions:
    id_ = s.split('-',1)
    dic_sessions[id_[0]]=id_[1]

#Get all pathes of abstract .html file
path = os.listdir("./ECS MA2019-02.003/Abstracts/Html")

#Scraping html documents
for n,i in enumerate(tqdm(path)):
    soup = bs4.BeautifulSoup(open("./ECS MA2019-02.003/Abstracts/Html/"+i, encoding='utf-8'), "html5lib")

    #Get title, authors, affiliation
    title =px.sub("", soup.title.string)
    author = px.sub("", str(soup('p')[1]))
    author = author.split(', ')
    affiliation = px.sub("", str(soup('p')[2]))
    affiliation = affiliation.replace("a", "", 1)

    #Get presentation number, session
    number = re.search('-[0-9]{4}', str(title))
    number = number.group().replace("-", "")
    session = re.match('.*[0-9]{2}', str(title))
    session = session.group()

    #Remove session name in title
    title = re.sub('\w[0-9]{2}-[0-9]{4}', "", title)
    title = title.replace(u"\xa0",u"")
    title = title.replace(session, '')
    title = title.replace('(Invited)', '')

    session = session.replace(number, '')
    session = session.replace('-', '')
    try:
        session_name = dic_sessions[session]
    except:
        session = session.split(u'\xa0',1)[0]
        session_name = dic_sessions[session]

    session_name = session_name.replace(u'&amp;', '&')

    #Get main text and reshape it
    content = px.sub("", str(soup.find(class_=re.compile('contaner_content'))))
    stoplist = [",", ".", ":", ";", "(", ")", "{", "}", "[", "]", "\'", "\""]
    for s in stoplist:
        content = content.replace(s, "")
    content = content.replace("\t", "")

    #Get word list for Word2Vec Learning
    c_list = content.split(" ")
    c_list = [c.lower() for c in c_list]
    for s in stopwords:
        c_list = [c for c in c_list if i != s]
    c_list = [c for c in c_list if c != ""]

    #Summarize it as dictionary
    dic ={'num':number,'title':title,'author':author,
          'affiliation':affiliation,'session':session, 'session_name':session_name,
          'contents':content, 'mod_contents': c_list, 'vector':n, 'url':i}

    dic_all.append(dic)

I'm sorry for these unsophisticated codes, but it works, anyway.

The details of all presentation including abstract text are contained as dictionary format, this is an example,

{'num': '0001',
 'title': 'The Impact of Coal Mineral Matter (alumina and silica) on Carbon Electrooxidation in the Direct Carbon Fuel Cell',
 'author': ['Simin Moradmanda', 'Jessica A Allena', 'Scott W Donnea'],
 'affiliation': 'University of Newcastle',
 'session': 'A01',
 'session_name': 'Battery and Energy Technology Joint General Session',
 'contents': 'Direct carbon fuel cell DCFC as an electrochemical device...',
 'mod_contents': ['direct','carbon','fuel','cell', ... ,'melting'],
 'vector': 0,
 'url': '1.html'}

'contents' and 'mod_text' are abstract text without/with preprocessing.

Data analysis of presentation in ECS meeting

We got ECS 236th meeting abstracts as a format easy to analyze.
Then, before language analysis, how about a statistical analysis?

Now, Lets calculate three metrics, number of presentation, number of authors, and number of affiliations for each 14 categories.

charactors = [i[0] for i in categorys]
cat_counts = np.zeros(len(charactors))
authors_counts = {}
words_counts = {}

for c in charactors:
    authors_counts[c] = []
    words_counts[c] = []

for d in dic_all:
    char = d['session'][0]
    cat_counts[charactors.index(char)] += 1
    authors_counts[char].append(len(d['author']))
    words_counts[char].append(len(d['contents'].split(' ')))

Firstly, I will show the number of presentation of each categories.


fig = plt.figure(figsize=(10,4),dpi=200)
plt.bar(charactors, cat_counts, label=categorys)
plt.xlabel('Categories',fontsize=14)
plt.title('Number of presentation',fontsize=16)
plt.show()

This plot shows that presentations are not distributed equally.
The most popular category is A - 'Batteries and Energy Storage', 711 counts. The second one is I - 'Fuel Cells, Electrolyzers, and Energy Conversion', 655 counts. The counts of these two categories reach more than half of total amounts(2461) of presentations.
This fact is reasonable because these two topics are related to the energy problem. It means many budgets injected to this field.

points = tuple(authors_counts[i] for i in charactors)
fig = plt.figure(figsize=(10,4),dpi=200)
plt.boxplot(points)
plt.gca().set_xticklabels(charactors)
plt.xlabel('Categories',fontsize=14)
plt.title('Number of author',fontsize=16)
plt.show()

This is a boxplot shows how many authors named on. Actially, it seems no trends exist for each categories.
The presentation with the largest number of authors is No. 0701, 'Comparison of Correlated Li⁺ Diffusion from Molecular Dynamics of Promising Solid Electrolytes' listed 18 member from 9 different affiliation!

Finally, I will show the top 20 list of author's affiliation.

import collections
affiliation_list = []
for a in dic_all:
    affiliation_list.extend(a['affiliation'])   
a_count = collections.Counter(affiliation_list)
a_sorted = sorted(a_count.items(), key=lambda x:x[1], reverse=True)

Rank	Affiliation	Counts
1	Georgia Institute of Technology	103
2	Argonne National Laboratory	73
3	National Renewable Energy Laboratory	56
4	Lawrence Berkeley National Laboratory	54
5	Oak Ridge National Laboratory	48
6	University of South Carolina	41
7	Sandia National Laboratories	33
8	Massachusetts Institute of Technology	31
9	Los Alamos National Laboratory	31
10	Colorado School of Mines	29
11	Pacific Northwest National Laboratory	28
12	Purdue University	27
13	Tokyo Institute of Technology	25
14	Lawrence Livermore National Laboratory	22
15	Faraday Technology, Inc.	21
16	U.S. Naval Research Laboratory	21
17	Tohoku University	21
18	Auburn University	20
19	Kyushu University	19
20	University of Michigan	18

Top 10 affiliations are monopolized by institute in United States. Tokyo Institute of Technology, Rank 13, has the largest counts among institutes outside the US.

Conclusion

Well, I can't reach the topic of natural language processing yet.
But it is interesting to collect basic data and analyze statistically.

In next article, I will start language processing, word embedding by Word2Vec and modification by SCDV.

ECS Meeting Explorer - webapp for scientific conference

konkon — Fri, 27 Sep 2019 20:13:18 +0000

Introduction

I'm Kohei Ishikawa (a.k.a. konkon). Ph.D student in Nagoya-University, Japan.

My research topic is Li battery, especially, Li metal anodes.
If space permits, I'd like to introduce my research, but it's not a main topic in this article.

This article aims for introduction of webapp I made, ECS Meeting Explorer!

Motivation

ECS meeting is biannual scientific conference organized by the electrochemical society.
Before the conference open, we should read huge amounts of meeting abstracts (usually in a plane or train) and make a schedule.
In the ECS 236th meeting held in October 13-17, 2019 in Atlanta, there are 2461 abstracts in all. It is almost impossible for us to read all of them!

The ECS provides online program and mobile apps and they can search abstracts with keywords.
However, even if we use this, making a meeting schedule is hard work. Sometimes, we lose an opportunity to attend interesting presentation.

This is my motivation to make original search app!

Overview

Search with number

This is a main page of ECS Meeting Explorer.
If you are a participant, please enter your presentation number in upper form and submit it!

This is a sample result for No. 0213, presentation titled 'Tripling the Energy Density of Insertion-Type Electrode Materials for Rechargeable Alkali-Ion Batteries By Introducing Carefully Selected Dopants'.
There are 70 presentation displayed in descending similarities.

If you want to save the result, please push 'Download CSV'!

On the top of search result, there is an interactive scatter plot.
This is a visualization of all presentation in the vector space. PCA and t-sne were used for dimensional reduction of document vectors.
Magenta colored and lime colored points are represents target and extracted similar presentation.
If you click the plot, you can move to corresponding abstract web page in ECS digital library.

Search with text

You are not a presenter?, no worries!
Let's search with free text in lower form. For example, search with 'Suppression of Lithium dendrite' is like below.

Presentations about Li metal anodes seems to be successfully extracted.
Now, you can search with any text related to your research interest!

Technical details

This apps based on natural language processing (vector space model) and machine learning, works with Python.
I used Word2vec implemented in Gensim to obtain vector representation of words, and sparse composite document vector (SCDV) to build document vectors.
For further instruction, below articles are very useful!

Flask, Jinja2 and Bootstrap are used for web application frameworks, design template and CSS framework.

I chose Heroku as a web platform. It works with Hobby dyno (about $7.00 per month) and average memory usage is less than 256MB, so far.

I'm planning to write an article for technical instruction, later date...

Closing talk

Fortunately, we can download all meeting abstracts of 236th meeting as zip format from official page.
Recently, ECS initiates Free the Science to propagate an open science. I have a great respect for this action. I hope they enjoy it!

Actually, This is my first apps for English speaker.
If you have any comments or question, feel free to contact me!