DEV Community: code2k13

Create real time language visualization of tweets in minutes

code2k13 — Sun, 01 Aug 2021 14:48:52 +0000

In this article I will show you how to perform language identification on tweets and how to stream the results to a webpage and display a real time visualization. We will use my open source project nlphose and C3.js to create this visualization in minutes and without writing any Python code !

To run this example you need following software

Docker
ngrok (optional, not required if your OS has GUI)
Internet Browser

Starting the nlphose docker container

Run the below code at shell/command prompt. It pulls the latest nlphose docker image from Docker hub. After running the command, it should start 'bash' inside the container

docker run --rm -it -p 3000:3000 code2k13/nlphose:latest

Running the nlphose pipeline inside the container

Copy paste the below command inside the container's shell prompt. It will start the nlphose pipeline

twint -s "netflix" |\
./twint2json.py |\
./lang.py |\
jq -c '[.id,.lang]' |\
./ws.js

The above code collects tweets from twitter containing the term "netflix" using the 'twint' command and performs language identification on it. Then it streams the output using a socket.io server. For more details, please refer to wiki of my project. You can also create these commands graphically, as shown below using the NlpHose Pipeline Builder tool

Exposing local port 3000 on the internet (optional)

If you are running this pipeline on a headless server (no browser), you can expose port 3000 of your host machine over the internet using ngrok.

./ngrok http 3000

Running the demo

Download this html file from my GitHub repo. Edit the file and update the following line in the file with ngrok url

var endpointUrl = "https://your_ngrok_url"

If you are not using ngrok, and have browser installed on your system (which is running the docker container), simply change the line to:

var endpointUrl = "http://localhost:3000"

You will need to run this file from a local webserver (example http-server or python -m http.server 8080).Once you run it, you should see a webpage like the one shown below:

That's it !! Hope you enjoyed this article !

NlphoseBuilder : A tool to create NLP pipelines via drag and drop

code2k13 — Sat, 17 Jul 2021 19:05:23 +0000

✨Checkout the live demo here

Recently I completed work on a tool called nlphoseGUIBuilder that allows creation of complex NLP pipelines visually, without writing a single line of code ! It uses Blockly to enable creation of NLP pipelines using drag and drop.

Currently following operations are supported:

Sentiment Analysis (AFINN)
NER (Spacy)
Language Identification (FastText)
Chunking (NLTK)
Sentiment Analysis (Transformers)
Question Answering (Transformers)
Zero shot Classification (Transformers)

The tool generates a nlphose command that can be executed in a docker container to run the pipeline. These pipelines can process streaming text like tweets or static data like files. They can be executed just like normal shell command using nlphose. Let me show you what I mean !

Below is pipeline that searches Twitter for tweets containing 'netflix' and performs named entity recognition on it.

It generates a nlphose command which looks like this

twint -s netflix |\ 
./twint2json.py |\ 
./entity  |\ 
./senti

When the above pipeline is run using nlphose, you can expect to see stream of JSON output similar to the one shown below:

....
{
  "id": "6a5fe972-e2e6-11eb-9efa-42b45ace4426",
  "text": "Wickham were returned, and to lament over his absence from the Netherfield ball. He joined them on their entering the town, and attended them to their aunt’s where his regret and vexation, and the concern of everybody, was well talked over. To Elizabeth, however, he voluntarily acknowledged that the necessity of his absence _had_ been self-imposed.",
  "afinn_score": -1.0,
  "entities": [
    {
      "label": "PERSON",
      "entity": "Wickham"
    },
    {
      "label": "ORG",
      "entity": "Netherfield"
    },
    {
      "label": "PERSON",
      "entity": "Elizabeth"
    }
  ]
}
...

Lets try out something more, the below pipeline searches for tweets containing the word 'rainfall' and then finds the location where it rained using 'extractive question answering'. It also filters out answers with lower scores.

Here is the nlphose command it generates:

twint -s rainfall |\ 
./twint2json.py |\ 
./xformer.py --pipeline question-answering --param 'where did it rain' |\ 
jq 'if (.xfrmr_question_answering.score) > 0.80 then . else empty end'

It is also possible to create a pipeline that processes multiple files from a folder :

The above pipeline generates this command:

./files2json.py -n 3  data/*.txt |\ 
./xformer.py --pipeline question-answering --param 'who gave the speech ?' |\ 
jq 'if (.xfrmr_question_answering.score) > 0.80 then . else empty end'

Play with the tool here: https://ashishware.com/static/nlphose.html

Here is the link to the projects git repository: https://github.com/code2k13/nlphoseGUI

Here is a YouTube link of the tool in action:

Don't forget to checkout the repository of the companion project nlphose: https://github.com/code2k13/nlphose

Create NLP pipelines with drag and drop

code2k13 — Tue, 06 Jul 2021 20:00:12 +0000

Recently I have started work on query builder GUI for my open source project nlphose.

I am using Blockly to implement this feature.

Still in initial phase , I have uploaded video on YouTube which you guys can check out !

Motion detection in microscope images using Python and OpenCV

code2k13 — Mon, 05 Jul 2021 11:54:10 +0000

There are three types of organisms one can see in above video, the round one, the long one and couple of amobea

Last year, I was introduced to a wonderful scientific instrument called the ‘Foldscope’. I have spent hours observing things with it. One of my favorite past times is to observe ciliates using the Foldscope. Ciliates are very simple single cell organisms which are easy to find and come in numerous shapes and sizes. Most ciliates move very fast, and you need some skill with a microscope to follow them on the slide. This inspired me to write some code that could detect moving objects in a video and draw rectangles around them. Amazingly, I believe I was able to do a decent job with under 60 lines of python code
(https://github.com/code2k13/motiondetection )

In this post I will discuss concepts which I used for detecting moving objects and how the work together to come up with the end results.

Reading video with Python and OpenCV

The first thing we need to do is be able to load frames one by one from a video. OpenCV makes this task very easy. OpenCV has a very convenient function called ‘cv2.VideoCapture’ which allows returns an object that can be used to find out information about the video (like width, height,frame rate). The same object allows us to read a single frame from the video by calling ‘read()’ method on it. The ‘read()’ method returns two values, a boolean indicating success of the operation and the frame as an image.

cap = cv2.VideoCapture("video_input.mp4") 
if cap.isOpened(): 
    width  = cap.get(3) # float
    height = cap.get(4) # float
    fps = cap.get(cv2.CAP_PROP_FPS)

The full video can be read frame by frame using following code:

success = True
while success:
    success,im = cap.read() 
    if not success:
        break

Writing videos using OpenCV

Writing videos with OpenCV is also very easy. Similar to ‘VideoCapture’ function, the ‘VideoWriter’ function can be used to write video , frame by frame. This function expects path of output image, code information,frames per second, width and height of output image as parameters

fourcc = cv2.VideoWriter_fourcc('m','p','4','v')
out = cv2.VideoWriter('video_output.mp4',fourcc , int(fps), (int(width),int(height)))

Write a frame to the video is as easy as calling

out.write(image_obj)

Finding frame difference

The above video was generated out of frame diffs from the original video

Images are represented as matrices in the memory. OpenCV has a function called ‘cv2.absdiff()’ which can be used to calculated absolute difference of two images. This is the basis of our motion detection. We are relying on the fact that when something in the video moves it's absdiff will be non zero for those pixels. However if something stationary and has not moved in two consequitve frames, the absdiff will be zero. So, as we read the video frame by frame, we compare current frame with older frame and calculate absdiff matrix. The dimensions of this matrix are same as of the images to be compared.

Sounds easy , right ? But there are some problems with this approach. Firstly, cameras and software produce artifacts when then capture and encode videos. Such artifacts give us non-zero diff even when the object is stationary . Uneven lighting and focussing can also cause non-zero diffs for stationary portions of images.

After experimenting with some approaches, I found out that thresholding the diff image using mean values works very well.

frame_diff[frame_diff < frame_diff[np.nonzero(frame_diff)].mean()] = 0
frame_diff[frame_diff > frame_diff[np.nonzero(frame_diff)].mean()]

Using edge detection to improve accuracy

Edge detection using 'Sobel' filter performed on the frame diff video

As microorganisms move, they push matter around them, which gives positive pixels after diffing. But we want to differentiate micro-organisms from other things. Also focussing plays an important part here. Generally a lot of out of focus moving objects will also give positive frame differences. Mostly these are blurred objects which we simply want to ignore. This is where edge detection comes into play on edges and borders in the image. To find those we use edge detection. This can be easily achieved by using 'sobel' filter from scikit learn package.

output = filters.sobel(frame_diff)

Using contour detection to detect objects

Video generated after performing contour detection on the Sobel filter output

Most protozoans like ciliates will not always show a clear border (because they are mostly transparent). So when we use edge detection to detect shapes/outlines of moving objects, we get broken edges. In my experience contour detection works very well to group such broken edges and generated a more continuous border.

contours, hierarchy = cv2.findContours(thresh,cv2.RETR_TREE,cv2.CHAIN_APPROX_SIMPLE)

OpenCV has built-in functions to find contours. The best part is, the function is able to find nested contour structures and return a hirearchy. The 'cv2.findCountours' function returns heirarchy of contours. We only consider top level contours (who dont have a parent). If for a contour 'idx' , if 'hierarchy[0][idx][3]' returns -1, it means that it is a top level contour and it does not have any parent. Everything else we ignore.

Creating bounding boxes

Video showing bounding boxes drawn over contours

Creating boxes around countours can require a bit of math. Luckily OpenCV has a convinient function 'cv2.boundingRect' which returns center coordinates, with and height of bounding rect around a given contour.Once we have that, drawing a rectangle on our frame can simply be down using cv2.rectangle function. We can pass the color and border-width when drawing the rectangle to this function.

if hierarchy[0][idx][3]== -1 :
    x,y,w,h = cv2.boundingRect(contour)
    if (w*h <= (q)**2):
        continue
    image = cv2.rectangle(image, (x,y),(x+w,y+h), color, 1)

The concept of ‘q’

Like I explained earlier, videos taken using a microscope can be messy. There can a lot going on. We may be only interested in detecting objects of a certain size. This where I have introduced a parameter called 'q'. This parameter was used for altering settings for various filters I experimented with. Currently this is only used to filter out bounding rects which are smaller than q^2 in area. You should experiment with different values of 'q' , depending the resolution of your video and size of objects you are interested in.

Things to improve

I want to make this approach fast enough so that it can run in realtime. Also it would be nice if I can get this ported to android or a mobile phone. I also plan to experiment with ML based segmentation techniques for better detection.

The full code

The full code , along with sample video is available on Github:
https://github.com/code2k13/motiondetection