DEV Community: MariaZentsova

Creating trigger functions with Supabase

MariaZentsova — Mon, 18 Apr 2022 13:11:56 +0000

One of the really interesting products, I've tried recently is supabase. It allows to create a backend for a react app with Postgresql as a database in a few minutes.

One of the most powerful features of Supabase is that it allows to use advanced Postgresql capabilities in a very easy way.

For instance, I needed to produce a table with aggregated statistics from another table, and update the aggregated stats on every data change. We often need this aggregated state of the data for analytics and charts.

To do this, we will use postgresql triggers in order to subscribe for particular events, like records update, insert and delete.

This capability is available in a Supabase -> Database section.

First, we need to create a function, that will fire off on any changes in the table. To do so, go to Database -> Functions -> Create a new function.

The return type of the function would be trigger.

The function I use creates an aggregated statistics from the funding table, resulting in amount of startup funding generated by country and industry type.



begin
  insert into public.investment_industry_country(industry_id, country_id, total_usd)
  SELECT 
  s.industry, 
  s.startup_hq, 
  SUM(f.amount_usd) AS ku
FROM public.startups s 
LEFT JOIN public.funding f 
  ON s.id = f.startup_id
GROUP BY 
  s.industry, 
  s.startup_hq

on conflict(industry_id, country_id) 
do update set total_usd = EXCLUDED.total_usd;
return null;
end;

Also, as this function should have access to our tables, we need to set it up as a security definer.

To make use of this function, we need to set up an actual trigger via Database -> Triggers -> Create a new trigger.

After we gave a name to the trigger, and assigned a table it will keep track of, we can select what types of events we are interested in. As I want an aggregated view, I selected all types of data changes - create, update and delete.

Then, we could select a function we've created earlier, which would run after table changes.

Once we click "confirm" the trigger is ready and the work of creating table for analytics is automated.

Here is how using extended supabase backend, we've created an aggregated table, that allows to produce some charts and statistics. Supabase makes it quite easy to utilise advanced Supabase capabilities and extend the database functionality.

Extracting company names from news using AWS Comprehend

MariaZentsova — Tue, 22 Mar 2022 17:32:32 +0000

Many business use cases, such as competitor research, finding new customers or partners require a list of companies, that are operating in a particular field.

In my case, I needed a list of startups and investors, that work in clean tech. I decided to do this work in Sagemaker notebook, as it is a good environment for data science work, and allows easy access to all AWS resources.

AWS Comprehend, which I've used to extract the companies, is an out of the box Natural Language Processing service, that allows to perform Named Entity Recognition, text classification and topic modelling.

1. Setting IAM role permissions

AWS controls access to its services, using IAM roles. To use AWS Comprehend in a Sagemaker, we need to assign an access to this service to our role.

We could get a current role from a Sagemaker session.

sagemaker_session = sagemaker.Session()
role = sagemaker.get_execution_role()

Once we know, what role is used, we can attach AWS Comprehend permissions to it.

2. Fetching data from S3 bucket

For this analysis we need just a few libraries, such as boto3, sagemaker, pandas and seaborn.

import boto3
import pandas as pd
import sagemaker

from tqdm.notebook import tqdm
tqdm.pandas()

import seaborn as sns
sns.set_theme(style="whitegrid")

The source data for analysis are TechCrunch articles from 2012 to 2021, filtered by clean tech category with machine learning. Let's fetch the data from s3 bucket. We are particularly interested in clean_text field, which contains a cleaned body of the article.

# Load the dataset
s3 = boto3.client('s3')
obj = s3.get_object(Bucket = 'mysustinero',Key = 'cleantechs_predicted.csv')

techcrunch_data = pd.read_csv(obj['Body'])

3. Running AWS Comprehend and investigating the response

Let's run AWS Comprehend for a single article and investigate the result.

# launch comprehend client 
client = boto3.client('comprehend')

response = client.batch_detect_entities(
    TextList=[
       techcrunch_data['clean_text'][819],
    ],
    LanguageCode='en'
)

Response is a JSON object, which contains extracted entities with the accuracy and offsets. There are a number of entities available, including ORGANIZATION, EVENT, PERSON and LOCATION.

response['ResultList'][0]['Entities']

4. Applying entity extraction to the whole dataset

Once I understood the structure of the response, it was pretty easy to write a function that extracts a list of companies from the text.

def get_organizations(text):
    response = client.batch_detect_entities(TextList=[text[0:4900]],LanguageCode='en')
    # getting all entities
    data = response['ResultList'][0]['Entities']

    # get all dictionaries that contain orgs
    orgs = [d for d in data if d['Type'] == 'ORGANIZATION']

    # extract unique orgs, using set comprehension
    unique_orgs = {d['Text'] for d in orgs}

    return unique_orgs

Then it's time to apply it to the whole dataset.

df['orgs_list'] = df['clean_text'].progress_apply(get_organizations)

To normalise a dataset, so there is a link between article url and company name, I use pandas explode() function.

# reshape a dataset
result_final = result.explode('orgs_list')

Let's calculate how many unique companies we extracted using this method.

orgs = result_final['orgs_list'].unique()
len(orgs)
# 5507

5. Analyse what Clean Tech companies Techcrunch writes about

After extracting company names, we could run different kinds of analysis. I wanted to extract data what companies in this field TechCrunch writes most about.

#calculate companies mentioned
dff = result_final.groupby(['orgs_list']).size().reset_index()
dff = dff.rename(columns= {0: 'count', 'orgs': 'company'})

# Generating most popular cleantech companies
dff_chart=dff[dff['count']>15]
categories_chart = sns.barplot(
    y="company", 
    x="count", 
    data= dff_chart, 
    order=dff_chart
    .sort_values('count', ascending=False).company).set(title='Most Mentioned Companies in TechCrunch CleanTech articles')

We see that Tesla mentioned quite a lot of time in articles about clean tech amongst other companies. Interestingly Monsanto is also mentioned quite a lot, probably in a comparison to clean tech start ups.

Using AWS Comprehend allowed me to quickly generate a list of companies in clean tech ecosystem. This saves a lot of time versus a traditional method, when such datasets are created manually by an analyst.

As the list is generated by machine learning there are bound to be mistakes, and companies not in clean tech area, that are just mentioned in the articles. However, quality checking a generated list is still much more simple and fast.

How to interact with AWS EC2 instance via git bash

MariaZentsova — Sun, 20 Feb 2022 14:47:27 +0000

AWS EC2 is a service, which could be used for almost anything from website hosting to training machine learning models.

AWS allows encrypted access to EC2 using key pairs, where a user stores its own key.

One of the easiest ways to connect to running EC2 instance is by using git bash.

1. Check that Git is installed

Open the terminal and run git --version, which would show the git version currently installed on the system. If it is not installed, the download git from the official website: https://git-scm.com/downloads

2. Set the permission on your .pem file

To use this .pem file with ssh, we need to set a permission on it with chmod command, where feb19.pem is a file with our key. We need to do it, so the key won't be publicly visible.



chmod 400 feb19.pem

3. Access EC2 from the terminal

Select your instance and click "Connect".

Once we have an encryption file and EC2 instance public DNS, then we can connect.



ssh -i feb19.pem ec2-user@ec2-44-201-115-163.compute-1.amazonaws.com

One thing to keep in mind, when we restart an instance, public IP gets reassigned, and often due to permissions we won't be able to reuse the same encryption file with the new dns. To avoid this, we could use AWS Elastic IP service.

When this is useful? Let imagine, we have to retrain a machine learning model weekly, to save the costs we will stop the instance, when it's not utilised. To avoid modifying our scripts, we could allocate an Elastic IP, so we won't have any problems that arise from the constantly changing address. AWS has a good tutorial, how to allocate static IP to an instance.

4. Copy files to the EC2 instance

To perform some tasks, often we would copy local file on the instance, using the secure copy command scp, where:

- i - identity file flag
feb19.pem - file with our key
hello_world.py - python file we want to copy on a server
ec2-user - a standard username
ec2-44-201-115-163.compute-1.amazonaws.com - the dns address of ec2 instance
:~/ - indicates a root folder, where we want to copy the file



scp -i feb19.pem hello_world.py ec2-user@ec2-44-201-115-163.compute-1.amazonaws.com:.

To copy entire local folder, we need to use -r flag for recursive option to copy all files in a directory.



scp -i feb19.pem -r to_test ec2-user@ec2-44-201-115-163.compute-1.amazonaws.com:~/

Once we're connected to the instance, we can delete a file



rm hello_world.py

As well as see all files currently available on the instance by running



ls -la

5. Installing libraries on the instance

To install any particular library on the instance run



sudo yum install <package_name>

List command will allow to see all packages that are already installed.



yum list installed

6. Run python script

To run python scripts on EC2 instance, first we need to set up python environment.



sudo yum install python36

Once python is installed, let's run our script



Python hello_world.py

Download files from the instance

Once the work is done and we generated an output, we could copy it back to our machine. Here I copy all files from the instance source directory to my local machine.



scp -i feb20.pem -r ec2-user@ec2-3-86-190-91.compute-1.amazonaws.com:~/ <localmachine>/Downloads

Disconnect from the EC2 instance

To halt the connection with the instance, run simple Ctrl + Dcommand.

Resources:

How to load data from S3 to AWS SageMaker

MariaZentsova — Mon, 24 Jan 2022 17:11:44 +0000

AWS Sagemaker is a great way to analyse data in the cloud and train machine learning models. Most convenient way to store data for machine learning abd analysis is S3 bucket, which could contain any types of data, like csv, pickle, zip or photos and videos.

Here I want to review how to load different data formats from S3.

AWS has created a great boto3 library, which allows for easy access to aws ecosystem of tools and products. SageMaker is a part of aws ecosystem of tools, so it allows easy access to S3.

One of the key concepts in boto3 is a resource, an abstraction that provides access to AWS API and resources. Each AWS resource instance has a unique identifier, which we can use to call it.

Let's get a resource and call a bucket.

import boto3

s3 = boto3.resource('s3')
bucket = s3.Bucket('bucket_id')

If you don't know your bucket id and can't view it in a console, we can list all buckets available.

for bucket in s3.buckets.all():
    print(bucket)

Let's print keys for all files we have in our bucket.

for file in bucket.objects.all():
    print(file.key)

1. Loading csv or txt files

Most often data come in csv or txt format, and this one is quite easy to load.

# Getting data from AWS S3 bucket
s3 = boto3.client('s3')
obj = s3.get_object(Bucket = 'process-news',Key = 'techcrunch_data.csv')

techcrunch_data = pd.read_csv(obj['Body'])

If you don't want to use boto3, you can load csv data from s3 using just pandas.

news_df = pd.read_csv('s3://folder_name/your_data.csv')

2. Loading Pickle

Pickle is a data format that uses very compact binary representation. Python module Pickle allows us to read these type of files from the s3.Object.

import pickle

data = pickle.loads(bucket.Object("your_file.pickle").get()['Body'].read())

Machine Learning models can also be saved, as a pickle file.

3. Loading JSON

JSON format is very popular and APIs most often return data in this format.

import json 

obj = s3.Object('test', 'sample_json.txt')
content = content_object.get()['Body'].read().decode('utf-8')
json = json.loads(file_content)

4. Loading zip archives

Often the data we're working with are quite big and stored in zip archive.


# Import required libraries
from io import BytesIO
from zipfile import ZipFile
import json

# Get a zip object

obj = bucket.Object('corpus-webis-tldr-17.zip')

bytes_ = BytesIO(obj.get()["Body"].read())

z = zipfile.ZipFile(bytes_)

# check what files we have in our archive
for file in z.namelist():
    print(file)

# here we have a json file in our archive - example.json
# load file from the archive
file = z.open('example.json')
data = json.load(file)

5. Loading an images from a folder

Image can be loaded as any other file by key.

file = bucket.download_file(KEY, 'my_local_image.jpg')

Topic modelling with Gensim and SpaCy on startup news

MariaZentsova — Mon, 17 Jan 2022 17:05:24 +0000

This winter I'm embarking on a new NLP project with the goal to analyse global investment trends in clean energy start ups.

One of the best sources of start up news globally is TechCrunch, so to have a high-level overview of all possible start up news topics, I've extracted the articles for the last ten years.

For the topic modelling itself, I am going to use Gensim library by Radim Rehurek, which is very developer friendly and easy to use.

1. Text preprocessing

The TechCrunch collection of startup news is absolutely amazing. I've extracted those by using the api, so data would require some cleaning.

Let's start by fetching the data from AWS S3 bucket.

# Getting data from AWS S3 bucket
s3 = boto3.client('s3')
obj = s3.get_object(Bucket = 'process-news',Key = 'techcrunch_data.csv')

techcrunch_data = pd.read_csv(obj['Body'])

techcrunch_data.shape

#(32634, 54)

The dataset contains more than 30,000 news articles, however the text for title and body needs to be extracted from the 'rendered' key.

"{'rendered': 'Irish virtual sports giant’s new startup bets big on rugby in bid for US market share'}"

# Getting the text from 'rendered' key
techcrunch_data["clean_title"] =  [ast.literal_eval(x)['rendered'] for x in techcrunch_data["title"]]
techcrunch_data["clean_content"] =  [ast.literal_eval(x)['rendered'] for x in techcrunch_data["content"]]

Now, to get the complete data for our model, let's concatenate title and body of the article.

# Concatenate title and content in one string
techcrunch_data['clean_text'] = techcrunch_data["clean_title"] + ' ' + techcrunch_data["clean_content"]

The text is still far from being ready for the model though.

techcrunch_data['clean_text'][25675]

The article needs to be stripped from html markup and unicode characters, I also want to replace TechCrunch specific quotation marks and /n, /t and /r symbols.

# getting a text from the markup
techcrunch_data["clean_text"] = [ BeautifulSoup(text, 'lxml').text for text in techcrunch_data["clean_text"]]

# remove unicode characters
techcrunch_data["clean_text"] = [unicodedata.normalize('NFKD', x) for x in techcrunch_data["clean_text"]]

# remove quotation marks
techcrunch_data["clean_text"] = [re.sub(r'[“”@()–-]+',' ', x) for x in techcrunch_data["clean_text"]]

# remove repeating spaces
techcrunch_data["clean_text"] = [re.sub(r'\s+',' ', x) for x in techcrunch_data["clean_text"]]

As the result the text looks much cleaner, and we can proceed with further processing and lemmatisation using SpaCy.

2. Cleaning data with SpaCy

SpaCy is one of the most popular NLP libraries, and is very fast and flexible.

I will use it for lemmatisation and to extract only nouns for my topics.

To speed up the processing, I will disable a Named Entity Recognition part of a SpaCy pipeline. The corpus is quite large, so I will use nlp.pipe to speed up the processing.

Additionally SpaCy will handle stop words removal, which are just most common words that don't have a lot of information value.

# loading the model
nlp = spacy.load("en_core_web_sm")

# remove ner function to speed up the processing
nlp.disable_pipes('ner')

# pipeline is a series of functions that is applied to a text
print(nlp.pipe_names)

# ['tok2vec', 'tagger', 'parser', 'attribute_ruler', 'lemmatizer'] 

# defining the dataset
dataset = techcrunch_data['clean_text']

docs = []

# nlp.pipe makes processing more efficient
for text in tqdm(nlp.pipe(dataset), total=len(dataset)):
    doc = nlp(text) 
    pt = [token.lemma_.lower() for token in doc if
           (len(token.lemma_) > 1 and token.pos_ == "NOUN" and 
          not token.is_stop)]
    docs.append(pt)

3. Words removal

We already removed stop words with SpaCy, however there are further processing I decided to do to improve the accuracy of the topics model.

I removed words that appear just once in the whole corpora, as they won't have any significance.

# count how many times a word occurs in our corpus
counts_word_occurence = Counter(chain(*[x for x in data]))

# get a list of words that appeared only once in the whole corpus
low_freq_words = {key:value for (key,value) in counts_word_occurence.items() if value==1}

len(low_freq_words)
# 14272

# drop words that appear only once in the whole dataset
docs = [[lemma for lemma in text if counts_word_occurence[lemma]>1] for text in docs]

# drop words that appear only once in the whole dataset
docs = [[lemma for lemma in text if counts_word_occurence[lemma]>1] for text in docs]

Additionally, I've removed words that appear in the vast majority of the articles, like startup, founder and so on. All these news are about start ups, and I wanted to dive into more details.

# Dropping words that occur in more then 40% of the articles

# Getting the corpus length 
docs_length=len(docs)

# calculate in how many documents a word appeared
counts_word_percentage = Counter(chain(*[set(x) for x in docs]))

# calculate in what % of all articles a word appears
counts_word_percentage = {key:(value/docs_length)*100 for (key,value) in counts_word_percentage.items()}


# get words with high frequency
high_freq_words = {key:value for (key,value) in counts_word_percentage.items() if value>40}

high_freq_words

Now, when we have only noun lemmas for our corpus and removed all the words I decided not to use, let's look how our article looks like.

# our article transformed into a list of lemmas
pp = pprint.PrettyPrinter(compact=True)
pp.pprint(docs[25675])

Once we removed some words, the articles length has changed and it would be quite interesting to see its distribution.

lengths =  [len(x) for x in docs]
# Build the histogram, distribution of lemmas in the texts
sns.histplot(lengths)

The list with cleaned lemmas is available on my GitHub.

4. Topic modelling with the best number of topics.

One of the most popular algorithms for topic modelling is Latent Dirichlet allocation. It's a generative probabilistic model, based on Bayesian theorem.

In its essence, words that occur together repeatedly in a text corpora will be used to generate topics.

However, topic models are hard to interpret. Once of measures that give us some insight to our model is a Coherence score, which measures how subset of words fits together.

Gensim has a CoherenceModel method, so we could use it to evaluate the best number of topics for our corpus. I will try two methods, available in Gensim: 'u_mass', 'c_v'.

Let's write a function, that will calculate both 'u_mass' and 'c_v' coherence measures on our model for different number of topics.

# Defining dictionary and corpus with Gensim
dictionary = corpora.Dictionary(docs)
corpus = [dictionary.doc2bow(text) for text in docs]

def calculate_coherence(dictionary, corpus, docs, start, stop):
    scores = []
    for topics in range(start, stop):

        # defining the model
        lda_model = ldamodel.LdaModel(corpus, id2word=dictionary, num_topics=26, alpha='auto', eval_every=5)

        # U_mass coherence score
        cm_u_mass = CoherenceModel(model=lda_model, corpus=corpus, dictionary=dictionary, coherence='u_mass')
        u_mass_coherence = cm_u_mass.get_coherence()

        # C_v coherence score
        cm_c_v = CoherenceModel(model=lda_model, texts=docs, dictionary=dictionary, coherence='c_v')
        c_v_coherence = cm_c_v.get_coherence()

        values = [topics, u_mass_coherence, c_v_coherence]

        scores.append(values)

    return scores

# calculare scores
scores = calculate_coherence(dictionary, corpus, docs, 10, 30)

Having the scores will allow us to assess what number of topics that would give the best coherence.

# scores to df
df = pd.DataFrame(scores, columns = ['number_of_topics','u_mass_coherence','c_v_coherence'])

# tidying the df
df = df.melt(id_vars=['number_of_topics'], value_vars=['u_mass_coherence','c_v_coherence'])

# Plotting u_mass_coherence
sns.lineplot(data=df.loc[df['variable'] == 'u_mass_coherence'], x="number_of_topics", y="value").set_title('u_mass coherence')

# Plotting c_v_coherence
sns.lineplot(data=df.loc[df['variable'] == 'c_v_coherence'], x="number_of_topics", y="value").set_title('c_v coherence')

6. Visualising the results

According to our data the model that gives best coherence score has 22 topics. Previously, we already defined a corpus and a dictionary, so it's time to build the final model.

lda_model = ldamodel.LdaModel(corpus, id2word=dictionary, num_topics=22, alpha='auto', eval_every=5)

# print topics
lda_model.print_topics(-1)

pyLDAvis library allows to visualise topics as bubbles with keywords, which is super cool if you want to explore them in more details.

pyLDAvis.enable_notebook()
viz = pyLDAvis.gensim.prepare(lda_model, corpus, dictionary)
viz

7. Saving the model

Once the model is built, we could save it on disk for the future use on unseen data.

# save the model to the disk
temp_file = datapath('model')
lda_model.save(temp_file)

Link to GitHub: https://github.com/MariaZentsova/gensim-topics-startup-news

Useful resources:

Evaluate Topic Models: Latent Dirichlet Allocation (LDA)

Latent Dirichlet Allocation

Exploring the Space of Topic Coherence Measures

Evaluation of topic modeling topic coherence

How to share Jupyter Notebook on GitHub with Poetry

MariaZentsova — Wed, 05 Jan 2022 10:02:28 +0000

Poetry is a dependency management and packaging tool for python. It makes it really easy to create and share data science work with others without worrying about version conflicts.

Poetry creates a new virtual environment for each project, allows to add and track packages and even publish your work to PyPi. It can be used to make it easy for fellow data scientists to reproduce a jupyter notebook just by running poetry install to create the same virtual environment.

1. Why use Poetry?

You might wonder, why use poetry at all? For me it was several reasons.

Great dependency resolver
Sometimes with conda I run into some weird dependency issues, which forced me to spend time on StackOverflow looking through various posts to resolve the issue. Poetry has a dependency resolver, where the solutions are already implemented if they exist.
Build for both dependancy management and packaging
Poetry allows not only track packages and resolve dependency conflicts, but it also helps to publish packages to PiPy. So it's very helpful to learn it if you plan to release your own package at some point.
Lots of people are using it
Many great data science notebooks are packaged with poetry, so it is useful to understand how it works and to have it installed.

2. Installing poetry

The recommended way of installing poetry is via a custom installer, which will isolate the package from the rest of the system.

On OSX:

curl -sSL https://install.python-poetry.org | python3 -

On Windows:

(Invoke-WebRequest -Uri https://install.python-poetry.org -UseBasicParsing).Content | python -

To make sure it's installed correctly, let's check the poetry version.

$ poetry --V

3. Creating a new project

Poetry project creates a new virtual environment.

$ poetry new jupyter-demo

The project consists of several files, most important one is pyproject.toml, which contains a list of all project dependencies.

4. Installing Jupyter

To add packages to poetry project, we just need to use a simple poetry add command.

$ poetry add jupyter ipykernel

Once Jupyter is installed, let's add a couple of packages often used in data science.

$ poetry add pandas tensorflow

To uninstall any package, we can use poetry remove command.

$ poetry remove tensorflow

Poetry allows us to see details of a particular package installed in our virtual environment by using poetry show with the package name.

$ poetry show pandas

5. Track your packages

While we could always look up all the packages in pyproject.toml file, we could see packages in a command line by running poetry show --tree to see all packages and their dependencies.

To see the latest updates for packages on PyPi to check if we're using latest versions, just run poetry show --latest.

For compatibility, poetry also allows to export dependencies in requirements.txt

$ poetry export -f requirements.txt --output requirements.txt

6. Run the Jypyter notebook

To run any executable from our newly created environment, we need to use poetry run command. Let's spin up jupyter server.

$ poetry run jupyter notebook

Copy and paste one of the URLS from your command line in the browser.

So now we have a jupyter environment running and can create a new notebook. I've just created a Test_notebook.ipynb in jupyter_demo folder.

7. Publish Notebook on GitHub

Once notebook is ready and environment with all packages is created, it's time to share your work on GitHub!

First, we need to lock the package versions we use with poetry lock command, so people who will work with our code can create the same virtual environment.

$ poetry lock

Once the packages are locked, you can initialise a local git repository, add all files and push initial commit.

$ git init -b main
$ git add .
$ git commit -m "My first commit"

Then, create a git repo with GitHub UI, don't initialise it, then copy the repo url.

$ git remote add origin  <YOU_REPO_URL> 
$ git push -u origin main

This pushes all your files to Github, so other people can easily access your work and set up similar environment.

8. Run notebook from existing poetry project

To recreate the notebook published by somebody else, we just need to clone the repo and set up virtual environment by running poetry install.

$ git clone <YOU_REPO_URL> 
$ cd <your_repo>
$ poetry install

Once the files are copied and virtual environment is created, we can spin up the notebook, just like before.

$ poetry run jupyter notebook

This is how, in a few easy step we can share and reproduce jupyter notebooks with poetry. Hope, this will inspire you to give poetry a try!

Building a Line Chart in React with VisX

MariaZentsova — Wed, 22 Dec 2021 10:01:39 +0000

VisX is a low-level charting library by AirBnB, which uses d3 for math and calculations. It has a steep learning curve, but on the upside, it's possible to create performant and customisable charts in react.

To build this chart, I will work with IEA data on R&D energy technology spend by country. The data are very comprehensive and go back to 1974. I've preprocessed the data and filtered out those for Denmark for Total and Renewable energy spend to compare both.

This is the resulted data structure:



export const data = 
[{"country":"DENMARK","currency":"RDDUSD","type":"RENEWABLE","year":1975,"amount":0.804},
{"country":"DENMARK","currency":"RDDUSD","type":"RENEWABLE","year":1976,"amount":1.350},
{"country":"DENMARK","currency":"RDDUSD","type":"RENEWABLE","year":1977,"amount":7.928},
{"country":"DENMARK","currency":"RDDUSD","type":"RENEWABLE","year":1978,"amount":15.357}]

What we're going to build:

App layout with Chakra UI
Line Chart with VisX
Tooltip and crosshair for interactivity

1. Creating NextJS App



npx create-next-app@latest visx-demo
cd visx-demo

2. Installing libraries

Chakra UI is an awesome UI components library, which has easy styling capabilities.



npm i @chakra-ui/react @emotion/react@^11 @emotion/styled@^11 framer-motion@^4

VisX is a modular library, and allows us to use only components we need and to keep the bundle size low.



npm install @visx/axis @visx/event @visx/glyph @visx/gradient @visx/grid @visx/group @visx/responsive @visx/scale @visx/shape @visx/tooltip

3. Building layout with Chakra UI

For our app to work with Chakra UI, we need to modify pages/_app.js component, to include chakra styling.



import { ChakraProvider, CSSReset } from "@chakra-ui/react";
import Head from "next/head";

const GlobalStyle = ({ children }) => {
  return (
    <>
      <Head>
        <meta content="width=device-width, initial-scale=1" 
         name="viewport" />
      </Head>
      <CSSReset />
      {children}
    </>
  );
};

function MyApp({ Component, pageProps }) {
  return (
    <ChakraProvider>
      <GlobalStyle />
    <Component {...pageProps} />
    </ChakraProvider>
  ) 
}

export default MyApp

Lets build a simple header in components/Header.js:



import { Box, Flex,  Heading } from "@chakra-ui/react";

const Header = () => {
    return (
        <Box
            pos="fixed"
            as="header"
            top="0"
            bg='#242730'
            left="0"
            right="0"
            borderBottomWidth="1px"
            width="full"
            height="4rem"
        >
            <Box width="full" mx="auto" px={6} pr={[1, 6]} height="100%" >
                <Flex size="100%" p={[0, 3]} pl={[0, 4]} align="center" justify="space-between">
                    <Box as="a" d="block" href="/" aria-label="VisX Area Chart">
                        <Heading color="gray.100" as="h4" size="md">VizX Area Chart</Heading>
                    </Box>
                </Flex>
            </Box>
        </Box>
    );
}

export default Header;

Now we can import header in pages/index.js file and build simple layout with chakra ui components.



import Header from "../components/Header"
import { Container, Heading, Box, Text, Link } from '@chakra-ui/react'
import ParentSize from '@visx/responsive/lib/components/ParentSize';
import LineChart from '../components/LineChart';
import { data } from '../data/stats_for_Denmark'

export default function Home() {
  return (
    <>
      <Header />
      <Box height='100vh' bg="#242730">
        <Container maxW='4xl' height='85vh' mt="4rem" >
          <Heading ml='40px' as='i' size='md' color={'gray.100'}>Denmark R&D Spend on Renewable Energy vs Total</Heading>
         // Chart will go here
          <Link ml='40px' fontSize='sm' color={'gray.100'} href='https://www.iea.org/data-and-statistics/data-product/energy-technology-rd-and-d-budget-database-2' isExternal>
            Data by IEA, 2021
          </Link></Container>
      </Box>
    </>
  )
}

4. Making Viz responsive

To make the chart responsive, we need to be aware of width and height of a parent container. VisX library makes it really easy with its <ParentSize /> component, which could be imported from @visx/responsive/lib/components/ParentSize. We will wrap our chart component in <ParentSize /> and will get width and height as props.



 <ParentSize>
{({ width, height }) => 
    <LineChart data={data} width={width} height={height} />}</ParentSize>

Now let's build two svg rectangles to understand <Group /> and visualise margins. VisX <Group /> components is just a container for all children <g /> elements, it allows to simplify the code and pass the margins.

In components/LineChart.js:



import { Group } from "@visx/group";

function LineChart({ data, width, height }) {
    // define margins from where to start drawing the chart
    const margin = { top: 40, right: 40, bottom: 50, left: 40 };
    // defining inner measurements
    const innerWidth = width - margin.left - margin.right;
    const innerHeight = height - margin.top - margin.bottom;
    return (
        <svg width={width} height={height} >
              <rect x={0} y={0} width={width} height={height} fill={'#718096'} rx={14} />
            <Group left={margin.left} top={margin.top}>
            <rect x={0} y={0} width={innerWidth} height={innerHeight} fill={'#A0AEC0'} />
            </Group>
        </svg>
    )
}

export default LineChart

Layout with and margins:

Browser starts drawing <svg /> elements in a coordinate system from top left corner (0,0), and the margins are defined clockwise with this initial coordinate, i.e. point of origin.

5. Building Axes

Let's start small and build axes. To build axes, we need to define scales first. Under the hood VisX uses d3, so we it's really useful to review d3 scales documentation. D3 will use math to help translate our numbers to pixel locations inside the container.

To get a range for our scales, I use d3.extent()function, which returns a min and max values from the array.

Ticks on axes can be formatted, using any functions, in this example I transform date label into the string, to remove a comma from the default number formatting.

const formatDate = (year) => year.toString()

In components/Chart.js:



 // Defining selector functions
 const getRD = (d) => d.amount;
 const getDate = (d) => d.year;

// Defining scales

// horizontal, x scale
const timeScale = scaleLinear({
    range: [0, innerWidth],
    domain: extent(data, getDate),
    nice: true
    })

// vertical, y scale
const rdScale = scaleLinear({
     range: [innerHeight, 0],
     domain: extent(data, getRD),
     nice: true,
});

Defining Axes inside the <svg/> container:



<AxisLeft
    tickTextFill={'#EDF2F7'}
    stroke={'#EDF2F7'}
    tickStroke={'#EDF2F7'}
    scale={rdScale}
    tickLabelProps={() => ({
        fill: '#EDF2F7',
        fontSize: 11,
        textAnchor: 'end',
      })} 
/>
<text x="-125" y="20" transform="rotate(-90)" fontSize={12} fill='#EDF2F7'>
     R&D Spend, RDDUSD
</text>
<AxisBottom
    scale={timeScale}
    stroke={'#EDF2F7'}
    tickFormat={formatDate}
    tickStroke={'#EDF2F7'}
    tickTextFill={'#EDF2F7'}
    top={innerHeight}
    tickLabelProps={() => ({
        fill: '#EDF2F7',
        fontSize: 11,
        textAnchor: 'middle',
    })} 
/>

I also want to add <GridRows /> and GridColumns /> for our chart, they will use the same scales as the axes.



<GridRows 
    scale={rdScale} 
    width={innerWidth} 
    height={innerHeight - margin.top} 
    stroke='#EDF2F7' 
    strokeOpacity={0.2} 
/>
<GridColumns 
    scale={timeScale} 
    width={innerWidth} 
    height={innerHeight} 
    stroke='#EDF2F7' 
    strokeOpacity={0.2} 
/>

This is how the result looks like. I usually remove inner and outer rectangles only in the end, and allow them to guide placing the elements during the work on the chart.

6. Building Line Charts

For line charts, we are going to use <LinePath /> component. I want to build two lines to compare Denmark Total R&D investments versus investment in renewable energy. For that, I will filter the data from the original array and will define a series, which will contain both outputs to build lines.

Color scheme for lines I took from this awesome MetBrewer R color palettes package.



//colours for lines
const colors = ['#43b284', '#fab255']

// data for lines
const data1 = data.filter(function (el) {
    return el.type === "RENEWABLE"
});

const data2 = data.filter(function (el) {
    return el.type === "TOTAL"
});

const series = [data1, data2]

Mapping over the data to place <LinePath /> inside the<svg /> container:



{series.map((sData, i) => (
    <LinePath
         key={i}
         stroke={colors[i]}
         strokeWidth={3}
         data={sData}
         x={(d) => timeScale(getDate(d)) ?? 0}
         y={(d) => rdScale(getRD(d)) ?? 0}
    />
))}

Now we can see lines for each of the data series:

7. Adding interactivity

For this chart I wanted to add crosshairs, which are just thin line that centers on a datapoint, and a tooltip with a year and R&D Spend value.

To add a tooltip, a whole component must be wrapped in a

with relative position, according to docs.



<div position = 'relative'>
/// Your whole component ///
</div>

VisX has a handy hook, which does a tons of work for us.



// tooltip parameters
const { 
    tooltipData, 
    tooltipLeft = 0, 
    tooltipTop = 0, 
    showTooltip, 
    hideTooltip 
} = useTooltip();

However, positioning the tooltip is a but tricky. In a nutshell, we need to get a X and Y coordinates from the mouse hover, and translate them into the data values.

For that, most examples I've seen use d3.bisect() to find the position of the value in the dataset. As I have two lines, for which I need the d values, I've also filtered the array by the date value I got from the bisect function.



 // function get data from a year
 const getD = (year) => {
     const output = data.filter(function (el) {
         return el.year === year
     })
     return output
    }

Now, lets define a function, that would handle the tooltip:



const handleTooltip = useCallback((event) => {
const { x } = localPoint(event) || { x: 0 };
const x0 = timeScale.invert(x - margin.left); // get Date from the scale

const index = bisectDate(data, x0, 1); // get index of this date from the array
const d0 = data[index - 1];
const d1 = data[index];
let d = d0;
// is previous data point available?
if (d1 && getDate(d1)) {
    d = x0.valueOf() - getDate(d0).valueOf() > 
         getDate(d1).valueOf() - x0.valueOf() ? d1 : d0;
     }
showTooltip({
    tooltipData: getD(d.year),
    tooltipLeft: x,
    tooltipTop: rdScale(getRD(d))
})
})

This is how Tooltip data looks like in my case for two lines:



[
    {
        "country": "DENMARK",
        "currency": "RDDUSD",
        "type": "RENEWABLE",
        "year": 2006,
        "amount": 41.657
    },
    {
        "country": "DENMARK",
        "currency": "RDDUSD",
        "type": "TOTAL",
        "year": 2006,
        "amount": 112.857
    }
]

Now let's define the tooltip, we need to place it outside of the



{/* render a tooltip */}
{tooltipData ? (
     <TooltipWithBounds 
         key={Math.random()}
         top={tooltipTop}
         left={tooltipLeft}
         style={tooltipStyles}
      > 
      <p>{`Total Spend: $${getRD(tooltipData[1])}`}</p>
      <p>{`Renewable Spend: $${getRD(tooltipData[0])}`}</p>
      <p>{`Year: ${getDate(tooltipData[1])}`}</p>
      </TooltipWithBounds>
      ) 
: null}

Defining a crosshair, as a simple , all <g/> elements must go inside the<svg />:



{tooltipData && (
    <g>
         <Line
             from={{ x: tooltipLeft - margin.left, y: 0 }}
             to={{ x: tooltipLeft - margin.left, y:innerHeight 
             }}
              stroke={'#EDF2F7'}
              strokeWidth={2}
              pointerEvents="none"
              strokeDasharray="4,2"
           />
    </g>
)}

Defining dots to show on hover using @visx/glyph:



{tooltipData && tooltipData.map((d, i) => (
    <g>
        <GlyphCircle 
            left={tooltipLeft - margin.left}
            top={rdScale(d.amount) + 2}
            size={110}
            fill={colors[i]}
            stroke={'white'}
            strokeWidth={2} />
     </g>
))}

Now we need to call our handleTooltip function to pass positioning values to the tooltip, crosshair and glyphs.

I wanted to show crosshair if a user will hover at any point inside the chart, for that I changed the fill of the first to transparent and call handleTooltip on any user interaction:



<rect 
    x={0} 
    y={0} 
    width={innerWidth} 
    height={innerHeight} 
    fill={'transparent'}
    onTouchStart={handleTooltip} 
    onTouchMove={handleTooltip}
    onMouseMove={handleTooltip}
    onMouseLeave={() => hideTooltip()}
/>

I've also placed this <rect /> after all my elements, because they are stacked one onto the other, and making this the top element will enable interactivity for all of chart.

This how the final result looks like:

To be completely honest, even thought I've worked with d3.js before, building the chart with VisX was a bit tedious and hard at the time. But as I was playing with the code, I've learned to love its concise API and the power and flexibility it gives to the developer.

If you want to try it too, here is a link to GitHub: https://github.com/MariaZentsova/visx-linechart

Examples, I've used for study and inspiration:
How to Make Beautiful Graphs With vx and React-Motion by Dylan Mozlowski
VisX Area Chart
VisX lines with glyphs
VisX Area Difference Chart