DEV Community: WinkJS

How to visualize timeline of a Wiki article?

Sanjaya Kumar Saxena — Tue, 18 Apr 2023 16:28:24 +0000

Automatic generation of the timeline — a graphical representation of a time period, on which important events are marked — from a Wikipedia article is a fascinating idea and very useful in quickly grasping the historical perspective. This post outlines the approach to create a well formatted timeline from any Wikipedia article using WinkNLP’s API and Named Entity Recognition (NER) feature:

Fetch the article's contents and convert them into a WinkNLP document.
Iterate through detected entities and filter only DATEs.
Use shapes of dates to convert them into standard Unix time.
Using parentSentence() API, extract the sentence containing the date; also markup() the date to highlight it in the corresponding sentence.
Collect each Unix time and sentence pair in an array and sort them on Unix time.
Converts this array into a well formatted timeline using Observable capabilities along with some CSS.

The above approach is realized in about 30 lines of code:

timeLine = {
  const response = await fetch( `https://en.wikipedia.org/w/api.php?action=query&prop=extracts&titles=${WikiArticleTitle || '2022 United Nations Climate Change Conference'}&explaintext=1&formatversion=2&format=json&origin=*` );
  const body = await response.json();
  const text = body.query.pages[ 0 ].extract;

  var doc = nlp.readDoc( text || '' );
        var timeline = [];
        doc
          .entities()
          .filter( ( e ) => {
            var shapes = e.tokens().out( its.shape );
            // We only want dates that can be converted to an actual
            // time using new Date()
            return (
              e.out( its.type ) === 'DATE' &&
              (
                shapes[ 0 ] === 'dddd' ||
                ( shapes[ 0 ] === 'Xxxxx' && shapes[ 1 ] === 'dddd' ) ||
                ( shapes[ 0 ] === 'Xxxx' && shapes[ 1 ] === 'dddd' ) ||
                ( shapes[ 0 ] === 'dd' && shapes[ 1 ] === 'Xxxxx' && shapes[ 2 ] === 'dddd' ) ||
                ( shapes[ 0 ] === 'dd' && shapes[ 1 ] === 'Xxxx' &&  shapes[ 2 ] === 'dddd' ) ||
                ( shapes[ 0 ] === 'd' &&  shapes[ 1 ] === 'Xxxxx' && shapes[ 2 ] === 'dddd' ) ||
                ( shapes[ 0 ] === 'd' &&  shapes[ 1 ] === 'Xxxx' &&  shapes[ 2 ] === 'dddd' )
              )
            );
          })
          .each( ( e ) => {
            e.markup();
            let eventDate = e.out();
            if ( isNaN( eventDate[ 0 ] ) ) eventDate = '1 ' + eventDate;
            timeline.push({
              date: e.out(),
              unixTime: new Date( eventDate ).getTime() / 1000,
              sentence: e.parentSentence().out( its.markedUpText )
            })
          });

        return timeline.sort( ( a, b ) => a.unixTime - b.unixTime )
}

You can see it in action on an interactive Observable notebook — "How to visualize timeline of a Wiki article?".

About winkNLP

WinkNLP is a developer friendly JavaScript library for Natural Language Processing (NLP). It can easily process large amount of raw text at speeds over 650,000 tokens/second on a M1 Macbook Pro in both browser and Node.js environments. It even runs smoothly on a low-end smartphone's browser.

It is built ground up with a lean code base that has no external dependency. A test coverage of ~100% and compliance with the Open Source Security Foundation best practices make winkNLP the ideal tool for building production grade systems with confidence.

How to tokenize a string?

Pallavi Ratra — Tue, 09 Feb 2021 15:59:21 +0000

To tokenize a string using winkNLP, read the text using readDoc. Then use the tokens method to extract a collection of tokens from the string. Follow this with the out method to get this collection as a JavaScript array. This is how you can tokenize a string:

// Load wink-nlp package  & helpers.
const winkNLP = require( 'wink-nlp' );
// Load "its" helper to extract item properties.
const its = require( 'wink-nlp/src/its.js' );
// Load english language model — light version.
const model = require( 'wink-eng-lite-model' );
// Instantiate winkNLP.
const nlp = winkNLP( model );

// Input string
const text = '#Breaking:D Can’t get over this #Oscars selfie from 
@TheEllenShow🤩https://pic.twitter.com/C9U5NOtGap';
// Read text
const doc = nlp.readDoc( text );
// Tokenize the string
const tokens = doc.tokens();
console.log( tokens.out() );

This returns an array of tokens:

[
  '#Breaking', ':D', 'Ca', 'n’t', 'get', 'over', 'this', '#Oscars', 
'selfie','from', '@TheEllenShow', '🤩', 
'https://pic.twitter.com/C9U5NOtGap'
]

winkNLP has a lossless tokenizer which preserves and reproduces the original text. The tokenizer intelligently handles hyphenation, contractions and abbreviations. It also detects token types like ‘word’, ‘number’, ‘punctuation’, ‘symbol’, etc.

How to run NLP in the browser?

Pallavi Ratra — Tue, 02 Feb 2021 18:13:57 +0000

While winkNLP was initially made to run in NodeJS, it can just as easily be used in the browser. To do this, we need to use the web version of the English lite model — wink-eng-lite-web-model. We'll also need a tool that can bundle all the required modules, something like Webpack or Browserify. For the purpose of this tutorial we'll use Browserify. First, lets install the required packages:

We'll be installing Broswerify globally, refer to its documentation for more details.

npm install wink-nlp --save
npm install wink-eng-lite-web-model --save
npm install -g browserify

Next, we'll write our JavaScript as we normally would. Here, we'll make a new file called token-counter.js and require winkNLP, some helpers, and the web model:

const winkNLP = require( 'wink-nlp' );
const its = require( 'wink-nlp/src/its.js' );
const as = require( 'wink-nlp/src/as.js' );
const model = require( 'wink-eng-lite-web-model' );
const nlp = winkNLP( model )

const text = `Its quarterly profits jumped 76% to $1.13 billion 
for the three months to December, from $639 million of previous 
year.`;
const doc = nlp.readDoc( text );

doc.entities().each((e) => e.markup());
document.getElementById("result").innerHTML = doc.out(its.markedUpText);

Now, we'll use Broswerify to bundle all the required modules into a single file:

browserify token-counter.js -o bundle.js

This will create a new file called bundle.js which you can include in your HTML as you would any other:

<div id="result"></div>
<script src="bundle.js" charset="utf-8"></script>

This will create the following output:

Its quarterly profits jumped 76% to $1.13 billion for the three months to December, from $639 million of previous year.

Performance considerations

It is important to note that this is a fully-featured English language model. Make sure to use gzip when you serve it on the web. This will reduce its size to under 1MB (from the uncompressed 3.5MB). Also, setting an appropriate cache header will ensure that the client doesn't have to download it multiple times.

How to split text into sentences in NLP?

Pallavi Ratra — Tue, 26 Jan 2021 10:53:25 +0000

To split any text into sentences using winkNLP, read the text using readDoc. Then use the sentences method to get a collection of sentences from the text. Follow this with the out method to get this collection as a JavaScript array. This is how you can split a text into sentences:

// Load wink-nlp package  & helpers.
const winkNLP = require( 'wink-nlp' );
// Load "its" helper to extract item properties.
const its = require( 'wink-nlp/src/its.js' );
// Load english language model — light version.
const model = require( 'wink-eng-lite-model' );
// Instantiate winkNLP.
const nlp = winkNLP( model );

// Input text
const text = 'AI Inc. is focussing on AI. It is based in 
              the U.S.A. It was started on 06.12.2007.';
// Read text
const doc = nlp.readDoc( text );
// Extract sentences from the data
const sentences = doc.sentences().out();
console.log( sentences );

This returns an array of sentences:

[
  'AI Inc. is focussing on AI.',
  'It is based in the U.S.A.',
  'It was started on 06.12.2007.'
]

If no sentence-break is found in the input text, the output is the complete text as an array with a single member.

A sentence is usually split at a full-stop, question mark or exclamation mark. Even in the presence of abbreviations, honorifics, etc., winkNLP attempts to intelligently identify the sentence boundary.

How to do sentiment analysis?

Pallavi Ratra — Mon, 18 Jan 2021 15:13:15 +0000

Sentiment analysis is used to check if a phrase has positive or negative sentiment. To get the sentiment score, start by reading the phrase using readDoc. Then using the its.sentiment property of the out() method, get the sentiment score of the phrase. This is how you can check if a phrase is positive or negative:

// Load wink-nlp package & helpers.
const winkNLP = require( 'wink-nlp' );
const its = require( 'wink-nlp/src/its.js' );
const model = require( 'wink-eng-lite-model' );
const nlp = winkNLP( model );

const text = 'Very excited to be part of the @winkjs_org team:D!';
const doc = nlp.readDoc( text );

// Calculate sentiment score
const sentiment = doc.out( its.sentiment );
console.log( 'Sentiment Score:', sentiment );

if( sentiment > 0 ) {
  console.log( 'Sentiment: Positive' );
} else {
  console.log( 'Sentiment: Negative' );
}

// -> Sentiment Score: 0.6
// -> Sentiment: Positive

The sentiment score will be between -1 and 1, with 0 representing an overall neutral sentiment. You can get the sentiment of the entire document, or a single sentence in a text.

Sentiment analysis can be used to judge the overall response to an event, product etc. In chatbots, it can be used to change the response based on the sentiment of the message it has received. See our Twitter Hashtag showcase to see this in action.

winkNLP's English language lite model uses ML-SentiCon as a base with further training. For emojis it uses the Emoji Sentiment Ranking. Together, they deliver an f-score of about 84.5%.

How to find date and time in text?

Prateek Saxena — Mon, 04 Jan 2021 12:54:52 +0000

Date and time, and other named entities can be extracted from a document using winkNLP. To do this, we’ll load a document and ask for its entities. We’ll then filter down to just the ones we need. If we want to get the Unix time from the text we can look at the shape and check if the text can directly be sent to the Date object to be parsed. Here is how we would find the date and time entities in a text using winkNLP:

// Load wink-nlp package & helpers.
const winkNLP = require( 'wink-nlp' );
const its = require( 'wink-nlp/src/its.js' );
const model = require( 'wink-eng-lite-model' );
const nlp = winkNLP( model );

const text = `The release happened on 21 August 2020 at 4:32 pm`;
var doc = nlp.readDoc(text);

doc.entities().filter( e => {
  if ( e.out( its.type ) === 'DATE' ) {
    console.log( e.out(), new Date( e.out() ) );
    // -> 21 August 2020
    // -> Fri Aug 21 2020 00:00:00 GMT+0530 (India Standard Time)
  }

  if ( e.out( its.type ) === 'TIME' ) {
    console.log( 'Time:', e.out() );
    // -> Time:
    // -> 4:32pm
  }
});

This would give you all the strings that contain dates or times, and the Unix time corresponding to it if the format allows it. You can now use the usual JavaScript functions like sort on this data structure to get the insights you need. For example our Wikipedia Timeline showcase uses it to create visualizations of articles.

Raw texts may contain many named entities like time, money, and hashtags. The English language lite model for winkNLP finds entities spanning multiple tokens by employing pre-trained finite state machine.

Todo applications that automatically add due dates based on the text that was entered, or email clients that add events to your calendar based on time and location do this by using this form of named entity extraction. It can also be used to create a timeline of events based on raw text.

How to run NLP on a PDF file?

Sanjaya Kumar Saxena — Tue, 15 Dec 2020 11:26:54 +0000

On several occasions we need to extract information from PDF documents. The first step is to convert the PDF document into raw text using a PDF parser. In the following example, we use the pdf-parse NPM package to achieve this. Once we have the raw text, it can be passed to winkNLP's readDoc method to process the text. The doc object returned by this API can be used to access a variety of information such as named entities, sentences containing negation, overall or sentence wise sentiment score and many more. We have illustrated extraction of named entities here — also known as NER.

// Load wink-nlp package & helpers.
const winkNLP = require( 'wink-nlp' );
const its = require( 'wink-nlp/src/its.js' );
const model = require( 'wink-eng-lite-model' );
const nlp = winkNLP( model );

const fs = require('fs');
const pdf = require( 'pdf-parse' );

// Read PDF file.
let dataBuffer = fs.readFileSync( './sample.pdf' );

// Parse & extract entities from the dataBuffer.
pdf( dataBuffer ).then( function( data ) {
  const doc = nlp.readDoc(data.text);
  console.log( doc.entities().out( its.detail ) );
});

The above code will read the PDF file located in the current directory and print all the named entities detected along with their type i.e. DATE, TIME, MONEY, EMAIL and many more. Each entity is in the form of a Javascript object containing two properties — value and type; for example, {value: 'March 15, 1972', type: 'DATE' }.

winkNLP’s English language lite model uses a pre-trained state machine to recognize named entities.

This could be useful in extraction of meaningful information from a resume, financial document or a complete book.

Photo by Annie Spratt on Unsplash