DEV Community: Rituraj Borpujari

Building Adda: A high performance group chat application in C

Rituraj Borpujari — Fri, 14 Mar 2025 16:41:35 +0000

Introduction

I don't use C at my work but I do occasionally get back to this fantastic language. Some time back I developed a Terminal alias to manage my directory bookmarks and allow faster navigation using fzf. I was intrigued by the Unix Philosophy and its overall design.So I began looking at different syscalls Linux provides and understanding some of them. I even tried out building C programs to replicate behaviours of everyday programs like cat, cp etc.

Then I came to a domain which I have never touched in C before: Network Programming. I was kind of scared of that to be honest. But nevertheless I loved Network programming. I am, after all, a Web Developer who builds and maintains Web APIs.

Building it out

But I wanted to try it out now. I have a copy of Unix Network Programming and I referenced that for most parts. But I frequently used Man pages as it helped me look up things faster once I know the name of the function

Working my way forward with the book and man pages, I was able to create a socket, bind it to address 0.0.0.0 (any IP on my local machine) with a specified port and listen to it. The network ordering of addresses and ports required some read up and I implemented the order conversion functionality myself before I actually stumbled upon htonl, and htons macros.

With that out of the way, I was able to create a server which listens on a port for TCP connections, accepts the first one and echoes whatever data the client sends to it. For client program, I still relied on NodeJs.

Concurrency and IO Multiplexing

One thing I really like about the man pages is that every page had a little SEE ALSO section at the end. Without that I wouldn't have found select function. select led me to poll and soon I enhanced my echo server to use IO Multiplexing to support multiple clients concurrently, all with a single thread. I later saw this in the book which explained concepts in detail.

Now I started forwarding messages from one client to all others and I've got a bare-bones chat server instead of a concurrent echo server. Whenever one client is ready with some data to be read, (as poll tells me this) I can then loop over all the clients again and forward the message (write) it to all

The poll method only polls on the valid file descriptors of the client sockets provided. Any invalid descriptor (negative value) would be ignored. This helped me immensely. Since I could have a list of fds stored on the stack itself and when a client disconnects, I just simply set the relevant fd in the list as -2. The message writing logic then can simply skip over all the client fds in the list which have value as -2

Performance and Threading

Now all of this is great, but the performance is a bit slower. Handling 2000 concurrent connections takes a bit more time than expected. Time to employ threading.

I referred back to the book and found Posix Threads. After trying it out, I laid out my design like this

main thread to poll for new connections
message_reader thread to poll for messages from clients
message_writer thread to write available messages to all clients (except the author)

I reused the same list of fds I used to poll in both message_reader as well message_writer to let both the threads know the list of clients. main thread would accept new connections and add it in this list itself.

The remaining part was to find a way to transfer messages from reader to writer. I used another file to act as a channel for the messages. That way I don't need to handle locks and can also poll on the channel file in message_writer

So the message_reader thread would poll on the clients for data to be read, and once available it would read and write it just to the channel file. A single write operation per read message while iterating over messages to be read from all clients meant that the server could now read faster.

On the other hand, message_writer would poll only on the channel file, and once data is available it would read it and write it to all clients in the list. The ordering of messages is also preserved this way. The message read first is the message that is forwarded first to all clients

This is great. The message throughput increased significantly and I was happy. But I soon saw a peculiar thing. Even for zero clients, the cpu usage was pretty high. I had not enabled Non-Blocking IO on any file descriptor so poll should block until data is available on the provided fds. And since reader has a list of zero fds, it is out of the question. It is configured to use poll with a timeout of 500 ms so as it uses the latest list of fds everytime, but that should not bring the CPU usage to much higher numbers.

Its obvious that message_writer is to blame. And I found out the poll function is returning immediately for the channel file saying it has data to be read (POLLIN event is being raised). But the subsequent read returned zero bytes. So it looped over the poll-read logic continuously. This is causing the high CPU usage. I looked into man page of poll for info and I found out the thing I skipped earlier.

Being "ready" means that the requested operation will not block; thus, poll()ing regular files, block devices, and other files with no reasonable polling semantic always returns instantly as ready to read and write

Okay so using a channel file is not possible. After reading about Unix Sockets (sockets for Interprocess communication) I came to this amazing function socketpair. It creates a pair of sockets which are connected to each other. So, data written to any of them is available on the other. Perfect!

I updated my channel file logic to use channel socket pairs and soon everything came back to normal. CPU Usage is Zero for zero clients and it also increased predictively now as the number of clients increased

Benchmarks

The program acheives the following benchmark result on my local machine

CPU: Ryzen 3700x (8 cores 8 threads)
RAM: 16GB
Message length: 211 bytes

Note: CPU usage is divided by the number of cores (top - Iris Mode off)

Number of Clients	CPU Usage	RAM Usage	1M messages write
0	0	1.6m	NA
500	1.3%	1.6m	~4 seconds
1000	4.9%	1.6m	~1.3 seconds
2000	5.6%	1.6m	~0.9 seconds
4000	5.3%	1.7m	~1.5 seconds

Conclusion

The server is working fine now. Obviously this is not ready for production usage as it would have to be inspected for error scenarios, edge cases and security considerations. And its feature list is still at the beginning.

I named it Adda a word popular in Assamese and many other languages in India, which roughly translates to hang out / chat with a group of friends

This was built as a fun project and I enjoyed building this and learned a lot about Network Programming and also C in general.

In future I would try to enhance this to allow command parsing and User authentication to help better suit the use case. For now, I am happy with what it is!

Project repository

The project repository is available on Github here at Adda

References

Building a Downloader in Go

Rituraj Borpujari — Wed, 12 Mar 2025 06:16:03 +0000

People say Go is well suited for Network applications. With its light
Goroutines and huge standard library it can get most of the job done
without even needing a third-party library.

Great, let me build a terminal program to download files

Accio

The summoning charm from Harry Potter
seems like a fit name for my downloader. Like when Harry Potter says
"Accio Dittany" and out pops the bottle of Dittany potion from
Hermione's bag with an "Undectable Extension Charm", our version of
Accio would do something similar, albeit with URLs.

So running accio url would get you the file from the url.

Simple enough! Let's start then.

The process

The module net/http gives us this amazing method for making a GET request -

http.Get(url string) (resp *httpResponse, err error)

With this, we can run a GET request to url and we get a response object
(of type httpResponse) and a possible error (of type error). We check for any error that might have occured while making the request and then
move on to reading the data bytes from httpResponse.Body.

But before reading the body, we would need to create and open a file for
writing with the following method. And os module has the capability we need

os.OpenFile(name string, flag int, perm fs.FileMode) (*os.File, error)

we also need os to get the CLI argument for URL being passed into our program
by the user while running command accio URL. We need io module because it has defined the io.EOF error value that
we need to test to know when we've reached the end of reading the response
body. With the file opened and the response body ready for reading, we would simply
need to loop this following

Copy Data process

Initialize a buffer
Read n bytes from response body into a buffer
If n is non-zero we write n bytes to the file
If error occured is EOF (End of file - indicates response data end)

We break the loop (go to step 6)
Go to step 2 (repeat)
We close the file and the response body

Here's the actual code

response, err := http.Get(downloadUrl)
// handle error
defer response.Body.Close()

file, err := os.OpenFile(
    "download",
    os.O_WRONLY | os.O_CREATE,
    os.ModePerm,
)
// handle error
defer file.Close()

buffer := make([]byte, 512)
for {
    buffer = buffer[0:512]
    n, err := response.Body.Read(buffer)

    if n > 0 {
        file.Write(buffer[0:n])
    }

    if err != nil {
        if err != io.EOF {
            fmt.Fprintf(
                os.Stderr,
                "download error: %s\n",
                err,
            )
            os.Exit(-1)
        }
        break
    }
}

We open the file with flags as os.O_WRONLY | os.O_CREATE which causes it to
open a file for writing and the perm as os.ModePerm so that the opened file
gets the same permissions as our executable program accio

We reset the buffer slice's capacity to 512 each time in the loop to
utilize the full capacity and let the Read method of response.Body to fill
it up with n bytes.

We then proceed to write only n bytes to the file because that's how much we
read in this iteration from response.Body

If error has occured, we need to stop everything. If the error is not EOF
we also log this onto the terminal. Otherwise, we silently break the loop

Showing progress

The next part is to show progress of the download. For this I'm going to use the
channel feature of golang.

Basically the download (more specifically the copy of bytes from response body
to file) would be happening on another go routine and it will send progress
stats over a channel. The main routine can then read messages from this channel
to display progress.

Here's the structure I used for the download progress status message

type DownloadStatus struct {
    Error           error
    IsComplete      bool
    BytesDownloaded int64
}

This will suffice. The copying logic will post a message of type
DownloadStatus on the statusChannel channel so that the main routine can
display the progress

here's the updated function to copy data from src to dest while posting
updates of the progress

func copyVerbose(dest io.Writer,
                 src io.Reader,
                 statusChannel chan DownloadStatus) {
    buf := make([]byte, COPY_BUFFER_SIZE)
    status := DownloadStatus{}

    for {
        nread, err := src.Read(buf)
        if nread > 0 {
            nwritten, err := dest.Write(buf[0:nread])
            if err != nil {
                status.Error = err
                break
            }
            status.BytesDownloaded += int64(nwritten)
        }
        if err != nil {
            if err == io.EOF {
                status.IsComplete = true
            } else {
                status.Error = err
            }
        }
        statusChannel <- status
    }
}

Showing progress in

Now to show progress on the main routine, we need to do this

create a ticker (which ticks periodically at say 500 ms)
use select to read from ticker and the statusChannel
if read from statusChannel, store it in a local variable
if read from ticker channel, show the progress
go to step 2 and repeat

of course, we would close this repeat loop once we find the status message
contains IsComplete as true

here's the code for that (partial main function)

lastStatus := DownloadStatus{}
ticker:= time.NewTicker(time.Millisecond * 500)
go downloadUrl(url, &DownloadOptions{Filepath: filename}, statusChannel)

done := false
for done != true {
    select {
    case <-ticker.C:
        n, unit := getFormattedSize(lastStatus.BytesDownloaded)
        if lastStatus.IsComplete == true {
            fmt.Printf("\x1B[1K\rcompleted: %.2f %s\n", n, unit)
            done = true
            break
        }
        if lastStatus.Error != nil {
            fmt.Fprintf(
                os.Stderr,
                "download failed: %s\n",
                lastStatus.Error)
            done = true
            break
        }
        fmt.Printf("\x1B[1K\r%.2f %s", n, unit)
    case status := <-statusChannel:
        lastStatus = status
    }
}

The escape sequence in printf (\x1B[1K) deletes the current line and \r
moves the cursor to the beginning of the line. See more

This effectively shows us the updated progress by deleting the old progress
every 500 ms

Conclusion

We've got ourselves a http(s) downloader using Golang's standard libraries only.
The downloader shows progress as it downloads and saves the file with name as
found looking at the url (or download if the url doesn't contain a
/resource_name.extension part at the end

There are many more things that can be done here. Some of the basic ones are

Add multi connections to speed up downloads
Add pause / resume capability

Both of them requires the usage of response header Accept-Range which denotes
whether the server supports requesting file data by a specific range. The
corresponding request header is Range which can be added in the request to let
the server know which range of data we need

I'll pick this up in the next iteration of Accio our very own http downloader.

Full code

https://github.com/riturajborpujari/accio

References

Competing with JSON.stringify - by building a custom one

Rituraj Borpujari — Wed, 14 Aug 2024 05:38:03 +0000

This came up during a discussion with my friend about Recursion. Why not build
a Javascript JSON.stringify method as a recursive programming exercise? Seems like a great
idea.

I quickly drafted out the first version. And it performed horribly! The
time required was about 4 times that of the standard JSON.stringify.

The first draft

function json_stringify(obj) {
  if (typeof obj == "number" || typeof obj == "boolean") {
    return String(obj);
  }

  if (typeof obj == "string") {
    return `"${obj}"`;
  }

  if (Array.isArray(obj)) {
    return "[" + obj.map(json_stringify).join(",") + "]";
  }

  if (typeof obj === "object") {
    const properties_str = Object.entries(obj)
      .map(([key, val]) => {
        return `"${key}":${json_stringify(val)}`;
      })
      .join(",");
    return "{" + properties_str + "}";
  }
}

By running the following, we can see that our json_stringify works as
expected.

const { assert } = require("console");
const test_obj = {
  name: "John Doe",
  age: 23,
  hobbies: ["football", "comet study"]
};

assert(json_stringify(test_obj) === JSON.stringify(test_obj))

To test more scenarios, and multiple runs to get an average idea of how our
script runs, we made a simple testing script!

A simple testing script

function validity_test(fn1, fn2, test_values) {
  for (const test_value of test_values) {
    assert(fn1(test_value) == fn2(test_value));
  }
}

function time(fn, num_runs = 1, ...args) {
  const start_time = Date.now()

  for (let i = 0; i < num_runs; i++) {
    fn(...args);
  }

  const end_time = Date.now()
  return end_time - start_time
}


function performance_test(counts) {
  console.log("Starting performance test with", test_obj);

  for (const count of counts) {
    console.log("Testing", count, "times");

    const duration_std_json = time(JSON.stringify.bind(JSON), count, test_obj);
    console.log("\tStd lib JSON.stringify() took", duration_std_json, "ms");

    const duration_custom_json = time(json_stringify, count, test_obj);
    console.log("\tCustom json_stringify() took", duration_custom_json, "ms");
  }
}

const test_obj = {} // a deeply nested JS object, ommitted here for brevity 
const test_values = [
  12,
  "string test",
  [12, 34, 1],
  [12, true, 1, false],
  test_obj
];

validity_test(JSON.stringify, json_stringify, test_values);
performance_test([1000, 10_000, 100_000, 1000_000]);

Running this we get the timings like the following.

Testing 1000 times
    Std lib JSON.stringify() took 5 ms
    Custom json_stringify() took 20 ms
Testing 10000 times
    Std lib JSON.stringify() took 40 ms
    Custom json_stringify() took 129 ms
Testing 100000 times
    Std lib JSON.stringify() took 388 ms
    Custom json_stringify() took 1241 ms
Testing 1000000 times
    Std lib JSON.stringify() took 3823 ms
    Custom json_stringify() took 12275 ms

It might run differently on different systems but the ratio of the time taken
by std JSON.strngify to that of our custom json_stringify should be about
1:3 - 1:4

It could be different too in an interesting case. Read on to know more about
that!

Improving performance

The first thing that could be fixed is the use of map function. It creates
new array from the old one. In our case of objects, it is creating an array of
JSON stringified object properties out of the array containing object entries.

Similar thing is also happening with stringification of the array elements too.

We have to loop over the elements in an array, or the entries of an object! But
we can skip creating another array just to join the JSON stringified parts.

Here's the updated version (only the changed parts shown for brevity)

function json_stringify(val) {
  if (typeof val === "number" || typeof val === "boolean") {
    return String(val);
  }

  if (typeof val === "string") {
    return `"${val}"`;
  }

  if (Array.isArray(val)) {
    let elements_str = "["

    let sep = ""
    for (const element of val) {
      elements_str += sep + json_stringify(element)
      sep = ","
    }
    elements_str += "]"

    return elements_str
  }

  if (typeof val === "object") {
    let properties_str = "{"

    let sep = ""
    for (const key in val) {
      properties_str += sep + `"${key}":${json_stringify(val[key])}`
      sep = ","
    }
    properties_str += "}"

    return properties_str;
  }
}

And here's the output of the test script now

Testing 1000 times
        Std lib JSON.stringify() took 5 ms
        Custom json_stringify() took 6 ms
Testing 10000 times
        Std lib JSON.stringify() took 40 ms
        Custom json_stringify() took 43 ms
Testing 100000 times
        Std lib JSON.stringify() took 393 ms
        Custom json_stringify() took 405 ms
Testing 1000000 times
        Std lib JSON.stringify() took 3888 ms
        Custom json_stringify() took 3966 ms

This looks a lot better now. Our custom json_stringify is taking only 3 ms
more than JSON.stringify to stringify a deep nested object 10,000 times.
Although this is not perfect, it is an acceptable delay.

Squeezing out more??

The current delay could be due to all the string creation and concatenation
that's happening. Every time we run elements_str += sep + json_stringify(element)
we are concatenating 3 strings.

Concatenating strings is costly because it requires

creating a new string buffer to fit the whole combined string
copy individual strings to the newly created buffer

By using a Buffer ourselves and writing the data directly there might give us
a performance improvement. Since we can create a large buffer (say 80 characters)
and then create new buffers to fit 80 characters more when it runs out.

We won't be avoiding the reallocation / copying of data altogether, but we will
reducing those operations.

Another possible delay is the recursive process itself! Specifically the
function call which takes up time. Consider our function call json_stringify(val)
which just has one parameter.

Understanding Function calls

The steps would be

Push the return address to the stack
push the argument reference to the stack
In the called function
1. Pop the parameter reference from the stack
2. Pop the return address from the stack
3. push the return value (the stringified part) onto the stack
In the calling function
1. Pop off the value returned by the function from the stack

All these operations happen to ensure function calls happen and this adds CPU
costs.

If we create a non-recursive algorithm of json_stringify all these operations
listed above for function call (times the number of such calls) would be
reduced to none.

This can be a future attempt.

NodeJs version differences

One last thing to note here. Consider the following output of the test script

Testing 1000 times
        Std lib JSON.stringify() took 8 ms
        Custom json_stringify() took 8 ms
Testing 10000 times
        Std lib JSON.stringify() took 64 ms
        Custom json_stringify() took 51 ms
Testing 100000 times
        Std lib JSON.stringify() took 636 ms
        Custom json_stringify() took 467 ms
Testing 1000000 times
        Std lib JSON.stringify() took 6282 ms
        Custom json_stringify() took 4526 ms

Did our custom json_stringify just perform better than the NodeJs standard
JSON.stringify???

Well yes! But this is an older version of NodeJs (v18.20.3). Turns out, for
this version (and lower also perhaps) our custom made json_stringify works
faster than the standard library one!

All the tests for this article (except this last one) has been done with
Node v22.6.0

The performance of JSON.stringify has increased from v18 to v22. This is so great

It is also important to note that, our script performed better in NodeJs v22.
So, it means, NodeJs has increased the overall performance of the runtime too.
Possibly an update has happened to the underlying V8 engine itself.

Well, this has been an enjoyable experience for me. And I hope it will be for
you too. And in the midst of all this enjoyment, we learnt a thing or two!

Keep building, keep testing!

Terminal Tools, Shell Aliases, and directory bookmarks

Rituraj Borpujari — Fri, 03 May 2024 17:10:52 +0000

I like using terminal tools and programs. They are so handy that you can't ignore once you get used to them. There are many of them at your disposal, each doing something which is unique in their own.

They do their own thing best, and they work with each other. Take a look at Unix Philosophy to understand the guiding principles that these tools follow to make them so easy to work with!

As I begin using these tools and the many ways in which they allow composability, I found myself using such composed commands and even scripts. Here's one that allows me to move to frequently used directories without typing the path everytime, from any directory I'm in!

Directory bookmarks

We use cd a lot in terminal. It allows us to change to the directory we want to work on. As I started using fzf, finding a specific subdirectory became much more easier than before. I can type in cd $(fzf) and it will open up a search window where I can fuzzy-search for a specific directory. I can write a shell alias like alias cdf="cd $(fzf)" and then I can just type in cdf and press return to activate the command. Its great!

But still, when I'm deep in a particular directory and I wanted to move to another directory which is not a sub dir this gives me a problem. Because the required directory that I want to move into is not a subdirectory, I can't just cdf any more (atleast not from the current dir). I have to move up the directory structure untill the required directory becomes a subdir to the present working directory. This is inadequate!

What if I can store certain directories that I frequently cd into, somewhere in a file, and then use fzf to match and cd into whichever one I want. That would be awesome!

That's what I built with dmk (directory bookmark). Its a set of two shell aliases to help me navigate to any directory with ease. From anywhere!!!

Marking directories as bookmarks

First things first, I need a way to mark certain directories as important so that I can come back to them when needed. For this, I just need to save the directory path into a file.

If I run the command pwd >> ~/.dmks, the present working directory path gets appended to a file called .dmks in my home directory ~. Here, pwd command prints the present working directory to terminal, and >> causes that output to get appended to the filename provided after.

For simpler frequent usage, I created an alias alias dmk="pwd >> ~/.dmks". I added this command into my shell init rc file ~/.bashrc and the alias is available from any shell window I open next.

Moving into a bookmarked directory

Okay, now that we have a way of marking directories by saving their path into a file, its time to use fzf to move into one.

For this I run the command cd $(cat ~/.dmks | fzf). This moves into the directory which is given by the sub-shell command cat ~/.dmks | fzf. Whatever is the output of this sub-shell command gets used by cd and it will cause the change in working directory. It's key to note that, cat ~/.dmks prints the content of the file ~/.dmks into the shell and because of the pipe operator | it gets passed on to fzf. This file has all of our marked directories which we marked with dmk and this list is passed on to FZF. FZF then shows an interactive search window, from which we can search the required directory from the list of marked directories and select the one we need.

Again, to help simplify running this command regularly, I have created a shell alias for this too! alias cdmk="cd \$(cat ~/.dmks | fzf)" creates an alias and I write this command in the shell rc file ~/.bashrc to make it available in any new shell window as well.

Note the escape character \ is required before $ when writing into the shell rc file! Otherwise, the shell will execute the command itself when initializing the alias! We don't want that! We want us to be able to use the alias, and when we do that, we need the command to run!

That is that, and I get directory bookmark system available in any of my shell window!!! dmk and cdmk!!!

P.S. here is the link to the gist containing the aliases.

Cheers!

Twitter API: Searching tweets, replies

Rituraj Borpujari — Sat, 09 May 2020 14:49:01 +0000

Get started with Twitter's API, searching tweets, loading replies

This project is on Github repo as an API server. Clone this to use twitter API immediately(after you've created your twitter developer app), to search for tweets, load replies to tweets.

What you need to get started?

You will need a twitter developer app to access the Twitter API. The App will provide you with two keys: API key, and API secret which we will use for requesting data from the API endpoints. Follow the following steps to get yourself the necessary keys.

Apply for a twitter developer account
Create a twitter developer app
Generate your app's keys on the app's details page under Keys and tokens tab
Open terminal and run the following command to generate your app's Bearer token. curl -u '<API key>:<API secret key>' --data 'grant_type=client_credentials' 'https://api.twitter.com/oauth2/token'

Replace <API key> and <API secret key> with your app's key and secret.

Note: With Bearer token, your app can only search for public data. It can not post tweet, read/write messages from your account. For such purposes, you'll need the Access token and secret too. You can generate them on the same page where you generated your keys. For our use case this will be sufficient.

Project setup

Nodejs Typescript boilerplate here.

You are now ready to use Twitter API. We are using a Node.js application and writing in Typescript, but you can use any language you prefer. But depending on your language of choice, you may find a twitter library or may have to build your own. For Node.js, we have a library called twitter, which we will use to make the requests

Add the dependencies

Twitter library npm install twitter
Types npm install @types/twitter

Modify environment variables

Open up the src/.env file and add your app's keys and token.

TWITTER_APP_API_KEY=<your-api-key>
TWITTER_APP_API_SECRET=<your-api-secret>
TWITTER_APP_BEARER_TOKEN=<your-bearer-token>

Add this to the config file src/config.js to load these env variables

export const twitter = {
    consumer_key: process.env.TWITTER_APP_API_KEY,
    consumer_secret: process.env.TWITTER_APP_API_SECRET,
    bearer_token: process.env.TWITTER_APP_BEARER_TOKEN
}

Let's search for some tweets. Shall we?

In src/app.ts file write the following code.

import Twitter from 'twitter';
import {twitter} from './config';

let tw = new Twitter(twitter);

tw.get('search/tweets', {
    q: '#webdevelopment',
    count: 2
})
    .then(res => {
        console.log('Response: ',res);
    })
    .catch(err => {
        console.error(err);
    })

Run this code by npm run start and you should see Twitter respond you with two tweets as an array, and some metadata about the search itself

Search parameters

Here, we are only using two properties q and count. we are searching for tweets that contain the hashtag #webdevelopment, and we are retrieving only 2 tweets.
See the full list of properties available here.

Search operators

For standard search(free with 7 days of data), your search query q can use a list of standard search operators. In our example we are searching by hashtag #webdevelopment.

Pagination

For obvious reasons, Twitter allow only a maximum of 100 tweets to load at a time. You can load the next tweets(older), or refresh the search(newer tweets) using two parameters max_id and since_id. When you get response from twitter, you get one search_metadata field which contains next_results and refresh_url. From these two query strings, you can filter out the max_id and since_id respectively. Then if you want to load the older tweets simply add the max_id to your parameters.(alongside q).
In my repo, there is a PaginationController class which automatically filters out the max_id and since_id if you just supply next_results and refresh_url respectively.

Schema for tweets

While twitter gives a whole lot of detail with tweets, we do not need all that. For example tweets contain both id_str and id, but we will only use id_str as id as it contains the original value. Also what's good is typescript, if we don't use a type for tweet. So, I have created a schema for handling tweets, and also a filter function to filter out tweets in our schema from twitter responses. Check them out at my repo. They are in file src/api/twitter/schema.ts and src/api/twitter/filter/tweetFilter.ts

Loading replies

Loading replies is a bit tricky. Because, twitter does not provide us API to load replies for a tweet directly. That's sad. But thankfully, twitter standard search operators can be used to fulfill our need. It provides us a to:screen_name operator which allows us to load replies to a twitter user. The screen name is the twitter handle like @thisisrituraj, without the @ sign. So to search for replies to my account, I would search using q: "to:thisisrituraj".

This will load replies to my account. But I wanted replies for a particular tweet of mine. Not all. That's what we have to filter out by looking a field in the responded tweets. The in_reply_to_status_id_str field which holds the id of the tweet for which this tweet is a reply.

We have to load all the replies to my account, using pagination, and then filter out replies to a particular tweet using its id. But do we have to load all replies to filter out replies to just one tweet. Unfortunately, twitter does not provide us with a tweet's reply count to know how many replies are there for a tweet. BUT..., we it provides us with the timestamp when the tweet was created(created_at field inside a tweet). And we know that, replies to a tweet can not predate the tweet itself.

So, the steps to load replies are

Get the id of the tweet(id_str) for which we wan't to load replies to.
Get the user's screen name who posted this tweet.
Load replies to this user using to operator
Filter out replies by comparing in_reply_to_status_id_str with id_str
Use pagination to keep on loading replies until one of the following occurs
- We have run out of replies. This means no more replies are there, or we have loaded replies for past 7 days
- We find a reply whose timestamp predates the timestamp of the tweet

Here's the code for loading replies. (This is an excerpt from my repo)

export default class TwitterClient {
    // code omitted for brevity

    private async loadRepliesForTweet(tweet: ITweet): Promise<ITweetsResponse> {
        let { user, id_str } = tweet;

        // save tweet's post datestamp
        let tweet_at = new Date(tweet.created_at);

        // load all replies to this user
        let allRepliesToTweet: ITweet[] = [];
        let replies: ITweetsResponse;
        let max_id = null;

        do {
            // load all replies to user
            replies = await this.searchTweets({
                q: `to:${user.screen_name}`,
                max_id,
                include_entities: false,
                count: 100
            });

            // select only replies done to THIS tweet
            replies.tweets.forEach(tweet => {
                if (tweet.in_reply_to.status_id_str === id_str) {
                    allRepliesToTweet.push(tweet);
                }
            })

            // Get max_id from next_results
            if (replies.search_metadata.next_results) {
                let next = <string>replies.search_metadata.next_results;

                // Use PaginationController to get max_id
                max_id = PaginationController.getParameterFromQueryString(next, 'max_id');
            }
            else {
                // BREAK loop if no more results exist
                break;
            }

            // get last reply tweet's post datestamp
            let last_reply_at = new Date(replies.tweets[replies.tweets.length - 1].created_at);

            // BREAK loop if last reply is earlier than tweet itself
            if (last_reply_at.valueOf() < tweet_at.valueOf()) {
                break;
            }

        } while (true);

        return { tweets: allRepliesToTweet, search_metadata: replies.search_metadata };
    }
};

That's all folks!

That was all about using the Twitter API for searching tweets and loading their replies. We saw, that although Twitter does not allow us to load replies to a tweet directly, we can use its Standard Search Operators and some coding to solve that.

How do you like this post. Do you have any idea about building something with Twitter API. Let me know. I develop software applications and am very much into problem solving. Connect with me on linkedin

Node.js, its awesome!

Rituraj Borpujari — Fri, 01 May 2020 06:43:51 +0000

checkout what makes Nodejs so popular

What is Node.js?

Node.js is an open-source and cross-platform JavaScript runtime environment. It runs the V8 JavaScript engine, the core of Google Chrome, outside of the browser.
Node.js uses event-driven non-blocking I/O design, which makes it very performant while reducing CPU and memory usage. A Node.js app runs on a single process, and it doesn't create new thread for every request like other server side languages. There is only one OS process running the event-loop, the core of Node.js application. This event-loop runs your code, and registers callbacks for asynchronous non-blocking I/O. This makes it a suitable candidate for highly scalable containerized applications and microservices.

What is V8 Javascript Engine

V8 Javascript engine is a javascript runtime developed in C++ for compiling and executing javascript code in Google Chrome. This is what runs the javascript code from the webpage you're viewing with Chrome. V8 is always evolving, just like the other JavaScript engines around, to speed up the Web and the Node.js ecosystem.

Why is non-blocking I/O important?

There is one key understanding here. The speed of an application mostly depends on the speed of the I/O devices. Read and writes from the main memory(RAM) is the fastest(leave aside CPU registers and caches for simplicity), followed by Disk reads and writes(reading from a file or local database server), followed by Network level I/O(like a HTTP request, database server requests). This is why applications are loaded into main memory, rather than keeping it in disk before executing. Extensive graphical applications loads the data into the GPU memory for faster access by the GPU, as GPU memory(VRAM) provides more than 10X speed in sequential access than RAM.

But a web application needs to access Network I/O even though its extremely slower than RAM I/O. So in a threaded model like PHP, Python the thread waits for the I/O method to complete. That is why they need multiple threads to handle concurrent requests, where one thread is allocated for every request. But with increase in number of threads comes three main issues

Increase in memory usage(RAM)
Possibility of deadlocks
Possible issues in thread concurrency The second and third one is prevented using careful designing and deadlock detection and breaking methodologies, while the first one needs machines with more RAM, and/ or clustering.

Specialties of Node.js

Because of its event-driven non-blocking I/O nature, it doesn't wait for the I/O to finish but registers a callback to happen when the I/O completes. The event-loop then continues performing other tasks. Node.js internally uses thread pool to manage callbacks and event notification. It will be notified by the callback with an event when I/O method finishes, and it can continue from there. Due to this single threaded nature it provides the ability to

Handle thousands of concurrent requests without increasing burden on the main memory
No possibility of a deadlock
No issues with thread concurrency

Because of its Javascript(one of the most popular language for the web) based nature, it provides the following advantages

Large collection of user libraries
One language in both server and client side

Threads with Node.js?

Although Node.js is designed with single threading in mind, you can also leverage the power of multiple cores when you want. You can create child processes, manage clusters with simple interfaces. For the extreme cases(like machine learning, complex computations), you can use multi-threaded libraries in C++(the fastest language) as an addon library. The C++ addon libraries provide interface for Node.js applications to use them like regular Node.js libraries. Much like how Tensorflow(written in C++) exposes interface for Python to use it like a python library.

When to use Node.js?

Because of its simple asynchronous nature and its event-driven model it has preffered choice for applications of different types and scale.

Highly scalable network applications, microservices
Reducing number of servers while maintaining network throughput
Increasing the ability to handle large number of concurrent requests

From this post you can see why Node.js is becoming the first choice for large scale network applications and services. Many companies have ported over their codebase to Node.js from other languages and have significantly reduced the number of servers, and increased ability to handle concurrent requests while doing so.

Wondering if your application idea is suitable for Node.js?

I am a software developer and a skilled problem solver wih good experience in web technologies. I build web applications, scalable APIs, business management systems. Let's discuss your project and we can understand specific needs of your system. Before crafting your idea into application code, I can also help you get a good overview of your system.

Regards,
Rituraj Borpujari
https://in.linkedin.com/in/riturajborpujari