DEV Community: Pankaj Tanwar

Challenge of day #21 - Split a String in Balanced Strings

Pankaj Tanwar — Sat, 09 Oct 2021 06:53:31 +0000

Hello!

It's day 21 of my coding diary. Yesterday, vercel was having some issues with their build system. Well, it's up and working smooth now. Let's directly move to the problem of the day.

Problem of the day - Split a String in Balanced Strings

Tag - Easy

Balanced strings are those that have an equal quantity of 'L' and 'R' characters.

Given a balanced string s, split it in the maximum amount of balanced strings.

Return the maximum amount of split balanced strings.

Example 1:

Input: s = "RLRRLLRLRL"
Output: 4

This is the worst problem statement, I have ever seen. It's really confusing man, all the example test cases are not clear. I spent time unnecessarily, building some dynamic programming logic for this dumb question.

So, the problem says, to find the maximum amount of split balanced strings. It does not clearly talk about the order of L & R.

Mentioning `splitting gives an idea of considering one sub-string having multiple balanced sub-strings as one only. but still, initially, I got confused with logic for considering all corner cases.

It turned out to be the simplest problem with just tracking count of L & R, that's it.

Here is my solution

`cpp
class Solution {
public:
int balancedStringSplit(string s) {
int res = 0;
int balance = 0;

    for(int i=0;i<s.length();i++) {
        balance += s[i] == 'L' ? 1 : -1;
        if(balance == 0) res++;
    }
    return res;
}

};
`

Not to mention, run time 0ms and faster than 100%. Honestly, I am not happy. Such uncleared problems make me less excited about solving more problems.

Well, I will try some good medium problems tomorrow!

Thanks for being part of my daily-code-workout journey. As always, if you have any thoughts about anything shared above, don't hesitate to reach out.

You might like previous editions of my coding diary

Challenge of the day #20 - To Lower Case.

Pankaj Tanwar — Fri, 08 Oct 2021 05:28:23 +0000

Hey-yo people!

Welcome to yet another edition (day 20) of my coding diary. I am sure, you might be having some thoughts about what I do here. So, every day I solve a challenge, sharing some interesting approaches + my learning.

Well, let's jump straight to the problem statement for the day!

Problem of the day - To Lower Case

Tag - Easy

Given a string s, return the string after replacing every uppercase letter with the same lowercase letter.

Example 1:

Input: s = "Hello"
Output: "hello"

I don't even need to re-iterate the problem. It's the simple challenge one can ever get. Hold on, I know it can be solved within seconds but trust me, we are going to discuss some really cool insights.

I have a couple of solutions for this problem -

The very first, simplest solution

class Solution {
public:
    string toLowerCase(string s) {
        for(int i=0;i<s.length();i++) {
            s[i] = tolower(s[i]);
        }
        return s;
    }
};

I am sorry, don't punish me for using the built-in function.

ASCII comes to rescue

class Solution {
public:
    string toLowerCase(string s) {
        for(int i=0;i<s.length();i++) {
            if(int(s[i]) >= 65 && int(s[i]) <= 90) s[i] += 32;
        }
        return s;
    }
};

Pretty straightforward!

Tip for C++ beginners: Use int() to convert char to decimal and use char() to convert, a number to ASCII Char.

I usually, struggle to remember the exact ASCII code of A, Z,a,z. So, I did this -

class Solution {
public:
    string toLowerCase(string s) {
        for(int i=0;i<s.length();i++) {
            if(s[i] >= 'A' && s[i] <= 'Z') s[i] += 32;
        }
        return s;
    }
};

Ahh, Some times I don't even remember the difference is 32. So,

if(s[i] >= 'A' && s[i] <= 'Z') s[i] = s[i] - 'A' + 'a';

Here is the hilarious approach to optimize it further.

As, to optimize the comparison, instead of using -

if(s[i] >= 'A' && s[i] <= 'Z')

we can do,

if(s[i] <= 'Z' && s[i] >= 'A')

So, the thought is, for every lower case letter, it will get rejected in the first check itself, of the if condition. One comparison saved! (drum rolls please)

I know I am being too picky here but that's how I optimized it further. Not to mention, my runtime beats 100% of the submissions.

One guy solved it using bit manipulation. Insane!

You might like previous editions of my coding diary

Challenge #19 - N-Repeated Element in Size 2N Array.

Pankaj Tanwar — Thu, 07 Oct 2021 06:19:47 +0000

Howdy!

This is day 19 of my coding diary and I am pretty excited to share today's challenge with you guys!

Spoiler Alert - This is one of the interesting problems I have ever encountered. I feel every easy problem has a really smart hidden approach.

Problem of the day - N-Repeated Element in Size 2N Array

Tag - Easy

You are given an integer array nums with the following properties:

nums.length == 2 * n.
nums contains n + 1 unique elements.
Exactly one element of nums is repeated n times.

Return the element that is repeated n times.

Example 1:

Input: nums = [1,2,3,3]
Output: 3

Ahh, very basic problem. Just use a hashmap, store the count and return a result that has more than one reputation.

But wait, I could smell the trap here. Leetcode didn't seem happy with my old school hashmap approach (O(n) time and O(n) space).

class Solution {
public:
    int repeatedNTimes(vector<int>& nums) {
        map<int,int> list;

        for(int num: nums) {
            list[num]++;
        }

        for(auto val: list) {
            if(val.second == nums.size()/2) return val.first;
        }
        return -1;
    }
};

Is it even possible to optimize it further?

Let's find out!

So, if we observe more closely, we can see that, if a number is repeated n times, in an array of length 2 * n, the repeated number will have to appear either next to each other (nums[i] and nums[i+1]) or after another (nums[i] and nums[i+2]).

I know it was tough, read it again. It took even, me some time to digest.

I was literally, blown away with the intuition. It comes with practice and solving problems of different - different patterns.

The edge case is [1,2,3,1]. In this case, we just return the last element.

Here is the code

class Solution {
public:
    int repeatedNTimes(vector<int>& nums) {
        for(int i=0;i<nums.size()-2;i++) {
            if(nums[i] == nums[i+1] || nums[i] == nums[i+2]) return nums[i];
        }
        return nums[nums.size()-1];
    }
};

This solution has the O(n) runtime.

Learning

Never settle for the first approach getting accepted, check out how others have done it. It's a great learning experience.

Thanks for being part of my daily-code-workout journey. As always, if you have any thoughts about anything shared above, don't hesitate to reach out.

You might like previous editions of my coding diary

Challenge #18 - Count Negative Numbers in a Sorted Matrix

Pankaj Tanwar — Wed, 06 Oct 2021 06:13:44 +0000

Hey-yo!

Welcome to day #18 of my coding diary. It has been a great journey so far, a really good daily-code-workout. It's too much fun, trust me.

TLDR;

Problem seemed pretty easy to me at first but it turned out be a gem when I tried to optimize the solution.
This is one of the cleverest problems I have ever seen.
One can just slap on the brute force approach and get on with his life BUT I would recommend everyone to just solve this problem once.

Problem of the day - Count Negative Numbers in a Sorted Matrix

Tag - Easy

Given a m x n matrix grid which is sorted in non-increasing order both row-wise and column-wise, return the number of negative numbers in grid.

Example 1:

Input: grid = [[4,3,2,-1],[3,2,1,-1],[1,1,-1,-2],[-1,-1,-2,-3]]
Output: 8
Explanation: There are 8 negatives number in the matrix.

Ahhmmm, calm down, calm down! I know this seems pretty easy. I could not stop my hands, coding the brute force O(n2)solution for it.

class Solution {
public:
    int countNegatives(vector<vector<int>>& grid) {
        int res = 0;
        for(auto row: grid) {
            for(int val: row) {
                if(val < 0) res++;
            }
        }
        return res;
    }
};

Quite straightforward. 30 seconds of code! haha. But pretty bad from an optimization point of view. Let's think of a good approach.

When I see something sorted, I call my dearest friend, binary search. Let's reduce the time complexity from O(n2) to O(n log m).

Here is the code

class Solution {
public:
    int getIndex(vector<int> row) {
        int low = 0;
        int high = row.size() - 1;
        int mid = (low+high)/2;

        while(low != high) {
            mid = (low+high)/2;
            if(row[mid] >= 0) {
                low = mid +1; 
            } else {
                high = mid;
            }
        }
        return row[low] >= 0 ? -1 : low;
    }

    int countNegatives(vector<vector<int>>& grid) {
        int res = 0;
        int index;
        for(auto row: grid) {
            index = getIndex(row);
            res += (index == -1 ? 0 : row.size() - index);
        }
        return res;
    }
};

Pardon my ugly code style.

I hit submit and was about to celebrate my small achievement BUT a small text below the problem statement shattered all my hopes.

Follow up: Could you find an O(n + m) solution?

Oh man, I thought I cracked the best solution for this. Ah shit, here we go again.

No worries, let's try to find it can be done in O(n + m).

I am not gonna lie. I tried my best but could not find any approach to solving it in the given time complexity. I can just think of something, checking for negative element, if found, based on that, just tick mark the index for new rows also.

After checking out the discussion tab, I was blown away by the approach. It was so clever.

So, We start from the upper right, check if one positive element is found, we just jump to the next row, for the same index (as we know columns are also in a sorted manner).

I know, it's complex to understand. This guy has written an excellent explanation. Check it out.

Here is the smartest approach

class Solution {
public:
    int countNegatives(vector<vector<int>>& grid) {
        int res = 0;

        int row = grid.size();
        int col = grid[0].size();
        int r = 0;
        int n = col - 1;

        while(r < row) {
            while(n >= 0 && grid[r][n] < 0) n--;
            res += col - n - 1;
            r++;
        }
        return res;
    }
};

I am in love with this problem. The way, it's designed is commendable. It was fun solving it.

As always, if you have any thoughts about anything shared above, don't hesitate to reach out.

Do let me know if you are also solving problems.

You might like previous editions of my coding diary

Challenge #17 - Sum of Unique Elements.

Pankaj Tanwar — Tue, 05 Oct 2021 07:04:31 +0000

Helllllooooo!

Pretty much excited about today's problem (No, 17 is not my lucky number.) Cool, let's directly jump to the challenge for Day #17 of my coding diary.

Problem of the day - Sum of Unique Elements

Tag - Easy

You are given an integer array nums. The unique elements of an array are the elements that appear exactly once in the array.

Return the sum of all the unique elements of nums.

Example 1:

Input: nums = [1,2,3,2]
Output: 4
Explanation: The unique elements are [1,3], and the sum is 4.

The problem is quite easy to understand. Now, after solving some good number of problems, I feel, multiple approaches are striking to my mind instantly, after reading the problem statement (or may be the problems are easy!)

So, we need to find the sum of only unique elements in the array. Ummm, as of now I can think of three approaches.

1. My all-time savior best friend, Hashmap

Loop over the list, save the count of each element in the hashmap
Loop over the list again, if the count is 1, add else tata-bye-bye
just return the final result

Here is the code -

class Solution {
public:
    int sumOfUnique(vector<int>& nums) {
        int res = 0;
        map<int,int> hash;        
        for(int num: nums) {
            hash[num]++;
        }

        for(int num: nums) {
            if(hash[num] == 1) res += num;
        }
        return res;
    }
};

Pretty nice solution.

2. Using sort

Sort the list
Loop over the list and just check if the previous element is the same (means element is duplicated), skip that

Ahh, please don't be angry with me for not coding the solution for this. It would be a too slow approach, O(n log n). And I don't like leetcode rewarding me for the slowest submission ever!

3. Using a constant array

Whenever I see the constraints are too small, my mind automatically starts thinking of having a constant array. We can replace the hashmap with a constant array here.

Ummm, can I do it one pass only? let's try!

class Solution {
public:
    int sumOfUnique(vector<int>& nums) {
        int res = 0;
        int arr[101] = {0};

        for(int num: nums) {
            if(arr[num] == 0) {
                res += num;
                arr[num]++;
            } else if(arr[num] == 1) {
                res -= num;
                arr[num] = -1;
            }
        }
        return res;
    }
};

So, what I am doing here?

Keep a constant array of length 101
If, number is not repeated, add it to final result and increase the count
If number is repeated, I deduct that number and assign the count of that number to -1 (means I don't want to see this again in my life)

Pretty simple right?

Oh man, I just checked this guy has 7 solutions for this problem.

As always, if you have any thoughts about anything shared above, don't hesitate to reach out.

You might like previous editions of my coding diary

Challenge #16 - Best Time to Buy and Sell Stock

Pankaj Tanwar — Mon, 04 Oct 2021 07:18:23 +0000

Hello!

I hope you had a lovely weekend. Here is my today's journal (btw it's day 16) of my coding diary.

Problem of the day - Best Time to Buy and Sell Stock

Tag - Easy

You are given an array prices where prices[i] is the price of a given stock on the ith day.

You want to maximize your profit by choosing a single day to buy one stock and choosing a different day in the future to sell that stock.

Return the maximum profit you can achieve from this transaction. If you cannot achieve any profit, return 0.

Example 1:

Input: prices = [7,1,5,3,6,4]

Output: 5

Explanation: Buy on day 2 (price = 1) and sell on day 5 (price = 6), profit = 6-1 = 5.

Note that buying on day 2 and selling on day 1 is not allowed because you must buy before you sell.

Ahh, quite a famous problem. It's the most favorite interview challenge of every interviewer for an entry-level job. Let's try and see if I am a good fit for an entry level job🤭.

So, the problem is pretty straightforward. I just need to find the maximum possible difference between two numbers (in increasing order only).

Well, I welcome my dearest friend, brute force approach. Directly coded the solution within seconds.

class Solution {
public:
    int maxProfit(vector<int>& prices) {
        int ans = 0;
        for(int i=0;i<prices.size();i++) {
            for(int j=i+1;j<prices.size();j++) {
                ans = max(ans, prices[j]-prices[i]);
            }
        }
        return ans;
    }
};

Hit submit and ....... Leetcode shattered all my hopes, with a beautiful Time Limit Exceeded message.

Well, I need to find some optimized way, brute force doesn't work always.

So, I need to find the maximum difference between a number with all numbers next to him. For that, I just need to know what is the maximum number among all those numbers.

Oh man, pretty easy peasy. I will iterate from backward and keep on tracking the maximum number till date and with each number, I will calculate the difference and see if it's the maximum or not.

Don't worry, here is the code -

class Solution {
public:
    int maxProfit(vector<int>& prices) {
        int ans = 0;
        int max_val = INT_MIN;
        for(int i=prices.size()-1;i>=0;i--) {
            max_val = max(max_val, prices[i]);
            ans = max(ans, max_val - prices[i]);
        }
        return ans;
    }
};

Woohoo, all ticks green. No pain full TLE this time. Ummm, it was a nice problem. Interesting thing was, test cases were designed so beautiful that one can't celebrate his brute approach submission.

As always, if you have any thoughts about anything shared above, don't hesitate to reach out to me.

You might like previous editions of my coding diary

Challenge #15 - Count Number of Pairs With Absolute Difference K

Pankaj Tanwar — Sun, 03 Oct 2021 06:36:40 +0000

Bonjour!

Thanks for sticking along, it's day 15 of my coding diary. Well, I started this journey to discipline my self and I feel, now I am literally enjoying it. Let's jump to the problem statement for today.

TLDR;

Never settle down by just seeing green tick with brute force approach. Think harder to reach the optimized one. (I found 3 approaches for the problem)

Easy problems might have interesting hidden challenges

Problem of the day - Count Number of Pairs With Absolute Difference K

Tag - Easy

Given an integer array nums and an integer k, return the number of pairs (i, j) where i < j such that |nums[i] - nums[j]| == k.

The value of |x| is defined as:

x if x >= 0.
-x if x < 0.

Example 1:

Input: nums = [1,2,2,1], k = 1
Output: 4

Just after reading the problem statement carefully, like any other dev, a brute force, O(n2), the slowest approach came to my mind and I just started typing without wasting a second.

class Solution {
public:
    int countKDifference(vector<int>& nums, int k) {
        int res = 0;
        for(int i=0;i<nums.size();i++) {
            for(int j=i+1;j<nums.size();j++) {
                if(abs(nums[i]- nums[j]) == k) res++;
            }
        }
        return res;
    }
};

As expected, the worst approach. It took 39ms, faster than 7%, Arghhhh. I knew it.

I again read the problem statement. A quick thought came to my mind, why not store count for each value and check count of val + k and val - k.

class Solution {
public:
    int countKDifference(vector<int>& nums, int k) {
        map<int,int> list;
        int res = 0;
        for(auto val: nums) list[val]++;

        for(auto val: nums) {
            list[val]--;
            res += list[val+k] + list[val-k];
        }
        return res;
    }
};

Approach -

Store count of each value
Iterate over the nums array
For each element, reduce the count for the current value first, and check the count of val - k and val + k
return the final value, that's the answer

I hit submit in the excitement of O(n) approach, BUT leetcode said, Ummm, it's a good try but still slower than 60% submission, think harder. WTH, I thought I cracked it.

I kept on digging more. I again read the problem statement, no luck! Suddenly, I looked at the constraints. It was a light bulb moment.....

Constraints:

1 <= nums.length <= 200
1 <= nums[i] <= 100
1 <= k <= 99

Let's remove the sluggish hashmap and use an array of length 200.

class Solution {
public:
    int countKDifference(vector<int>& nums, int k) {
        int list[201] = {0};
        int res = 0;

        for(auto val: nums) {
            res += (val-k >= 0 ? list[val-k] : 0) + list[val+k];
            list[val]++;
        }
        return res;
    }
}

Hit submit, and boom! It's 9ms, faster than 90% of solutions. Oh man, that was fun. I am gradually recognizing the patterns.

You might like previous editions of my coding diary

Prevent duplicate cron job running.

Pankaj Tanwar — Fri, 03 Sep 2021 15:23:04 +0000

Today, while working on an in-house project, I encountered a really interesting problem. I needed a python script, running every 30 minutes, pulling some information from a third party, processing the data, updating on my local database & take a rest till the next round. I wrote the script and set-up the cron.

Smooth right!

But, my happiness didn't last long. Sometimes, my script took more than 30 minutes to execute. This presented me with a beautiful issue of cron jobs overlapping & data duplication. I didn't want the jobs to start stacking up over each other.

Ahh. Cute concurrency problem.

To fix this, like any other developer, a couple of thoughts popped up in my mind.

Modify my python script, use some internal package to list down all running processes & grep if the same cron is already running. If yes, maybe it's not a good time to run it.
Why not look for the existence of a particular file mylock.txt and exit if it exists or create it if it doesn't?

Both solutions seemed pretty lousy & unsafe. And touching a working code is my biggest nightmare.

Our internal discussion, headed me over to a beautiful tool, Flock.

So, What's flock?

Flock is a very easy & simple tool. This tiny utility comes by default with the util-linux package.

Its mechanism is pretty neat and simple. For execution, it takes a lock file & command to run as input. It puts a lock on a given lock file and releases the lock when the script is executed. Lock on the file helps the tool decide, whether to run the script or not, in the next round.

Just to add here, file locking is a mechanism to restrict access to a file among multiple processes. It allows only one process to access the file at a specific time.

How to setup a cron job, using flock ?

Setting up a cron using flock is pretty simple.

Step 1 - Install flock, if not available in your system

yum install -y util-linux

You can verify if flock has been installed by whereis flock in linux system. It should show /usr/bin/flock as a path.

Step 2 - Open up cron tab

crontab -e

Step 3 - Install your new cron

 */30 * * * * /usr/bin/flock -w 0 /home/myfolder/my-file.lock python my_script.py

And you are done.

The moment flock starts, it locks the my-file.lock file & if in next round, the previous cron is already running, it will not the script again.

Don't worry about my-file.lock , flock will create it for you if it doesn't exist.

To verify the lock, try -

fuser -v /home/myfolder/my-file.lock

So, my crontab entry looked like this -

 */30 * * * * /usr/bin/flock -w 0 /home/myfolder/my-file.lock python my_script.py > /home/myfolder/mylog.log 2>&1

Well, calm down. I know, I have added some random texts to my cron. Let's decode the meaning of >/home/myfolder/mylog.log 2>&1 one by one.

> is standard I/O redirection.
/home/myfolder/mylog.log is a black hole where any data is sent
2 is the file descriptor for standard error (STDERR)
> , again for redirect
& symbol for file descriptor
1 is file descriptor for standard output (STDOUT)

2>&1 means a redirection of channel 2 (STDERR) to channel 1 (STDOUT) so both outputs are now on the same channel 1. >/home/myfolder/mylog.log means, output from channel 1 will be sent to this black hole.

To sum it up, output & errors are generated while the execution of your script will go to this file.

The unfortunate truths of flock

How to run multiple commands with flock

I had an interesting use case. Due to some system absolute path-related stuff inside my python script, I had to run the script as a combination of two commands.

Instead of -

python /home/myfolder/script.py

I needed to do -

cd /home/myfolder/ && python script.py

Running multiple commands with the help of flock, is a bit tricky. After a bit of struggle, this one worked for me.

*/30 * * * * cd /home/myfolder/ /usr/bin/flock -w 0 /home/myfolder/my-file.lock && python my_script.py > /home/myfolder/mylog.log 2>&1

flock doesn't always seem to be working.

Flock does advisory locking, which is a cooperative locking scheme which means you will be able to override the lock if you don't cooperate.

It has been raised many times that if, the flock is used to invoke a command in a subshell, other programs seem to be able to read/write to the locked file. This issue on stackoverflow talks about this in detail.

What is the sorting algorithm behind ORDER BY query in MySQL?

Pankaj Tanwar — Wed, 21 Jul 2021 17:15:55 +0000

Since the last couple of weeks, I have been working on MySQL more closely. MySQL is a brilliant piece of software. I remember reading about all the sorting algorithms in college so I was curious to know which algorithm MySQL uses and how ORDER BY query works internally in such an efficient manner.

I started exploring & digging deeper into its open source code and I was amazed with its implementation.

MySQL is canny. It’s sorting algorithm depends on a couple of factors -

Available Indexes
Expected size of result
MySQL version

MySQL has two methods to produce sorted/ordered streams of data.

1. Smart use of Indexes

Firstly MySQL optimiser analyses the query and figures out if it can just take advantage of sorted indexes available. If yes, it naturally returns records in index order. (The exception is NDB engine, which needs to perform a merge sort once it gets data from all storage nodes)

Hands down to the MySQL optimiser, who smartly figures out if the index access method is cheaper than other access methods.

Really interesting thing to see here is-

The index may also be used even if ORDER BY doesn’t match the index exactly, as long as other columns in ORDER BY are constant.
Sometimes, the optimizer probably may not use Index if it finds indexes expensive as compared to scanning through the table.

For example -

Assume, we have an index on userId and mobileNumber in the user table and we run below query -

SELECT * FROM USER
ORDER BY userId , mobileNumber;

Here, you might feel Indexes on userId and mobileNumber enables optimizer to use index BUT this query has “ *SELECT ** ”, which is selecting more columns than just userId and mobileNumber.

In this case, scanning through an entire index, to find columns which are not in the index, is more expensive than scanning the table and sorting the results. Here, the optimizer probably does not use the index.

2. Filesort Algorithm

If Indexes can not be used to satisfy an ORDER BY clause, MySQL utilises filesort algorithm. This is a really interesting algorithm. In a nutshell, It works like this -

It scans through the table and finds the rows which matches the WHERE condition
It maintains a buffer and stores a couple of values (sort key value, row pointer and columns required in the query) from each row in it. The size of this chunk is system variable sort_buffer_size.
When, buffer is full, it runs a quick sort on it based on the sort key and stores it safely to the temp file on disk and remembers a pointer to it.
It will repeat the same step on chunks of data until there are no more rows left.
Now, it has a couple of chunks which are sorted.
Finally, it applies merge sort on all sorted chunks and puts it in one result file.
In the end, it will fetch the rows from the sorted result

If the expected result fits in one chunk, the data never hits disk, but remains in RAM.

To your surprise, suppose, if we have 1 Billion rows in our user table and if we run below two queries -

SELECT * FROM users ORDER BY userId ;

SELECT * FROM users ORDER BY userId LIMIT 5;

MySQL optimizer is smart enough to understand, it's not worth using merge sort. So, It uses heap sort if we just want a set of results, not the entire data.

Ordering by multiple columns does not require scanning the database twice! In MySQL (<v4.1), it used to read the data twice, once to match rows in WHERE clause and one, in the end to prepare the result from row pointers.

TL;DR

Smart algorithm behind MySQL ORDER BY -

External merge sort (quick sort + merge sort) if data doesn’t fits into the memory
Quick sort, if data fits into the memory and we want all of it
Heap sort, if data fits into the memory but we are using LIMIT to fetch only some results
Index lookup (not exactly a sorting algorithm, just a pre-calculated binary tree)

With the help of EXPLAIN statement, we can see if query is making use of filesort or indexes.

References -

MySQL Official Website

MySQL filesort : Github

If you enjoyed reading it, you might like -

@the2ndfloorguy on twitter!

🎉 Launching Bestresources.co : Share & explore personal resources at one place!

Pankaj Tanwar — Wed, 30 Jun 2021 13:43:42 +0000

Namaste 🙏

I am Pankaj, a self taught software developer passionate about exploring tech. Today, I feel extremely proud to launch https://bestresources.co

Motivation

One fine day, I was having a technical discussion on a Linkedin comment thread. I had written a blog post on Apache Kafka so I was asked if I could share some good resources available on the same.

This was my moment of inspiration to build a solution for the tiring, cumbersome job of finding the relevant resources. How do you actually learn something new in today’s life?

Suppose you want to learn JavaScript. There are plenty of resources available on the web. You hit a search on google, and you start reading. You waste a plenty of time to find some perfect resources for you.

But here is the catch! When someone else starts learning JavaScript, he too repeats the same cycle. How can the process be expedited?

What if you could share your personal resources with the guy? What if you had access to the resources used by seniors engineers at the first place? It would have saved a tons of time.

Well, there are a couple of ways, some of the devs post a blog sharing your resources. BUT this is not scalable. Not everyone can find their blogs on the internet.

One, will have to dig deeper. It takes a lot of time to find something worthy.

I felt the need of a dedicated platform, where people can share their resources, personally used by them.

Introducing Bestresources.co - a free platform for the community to share & get direct access to the personal resources of the community.

https://bestresources.co is now live with curated resources by senior developers from Google, Amazon, Uber etc and so many self taught developers & freelancers.

Our Story

Story of https://bestresources.co started with a comment. Initially, I didn’t pay much attention to this idea and put it aside (like every other dev). But on 19 June, I got to know about the HarperDB hackathon by Hashnode & it motivated me to work on it.

I had a very limited time but I decided to finish the development and launch it anyhow.

I had to manage my regular job, my freelancing projects and other work. I enjoyed working late at night . I continued to ship features as much as possible. I had already made up my mind that I will not keep it as a demo project, whatever it takes, I will launch it.

And within 1.5 week, I was able to launch bestresoures.co 🎉

Features -

I am very very sensitive to code quality. 😛 While maintaining the code quality, (as much as possible), https://bestresources.co is live with below features -

*1. Listing of interesting resources *

As you head over to bestresources.co, it brings you interesting resources. Explore the hidden gem of the web.

2. Create & Manage

Add & share your resources with the community at a single click. Easy and beautiful UI will make it easier for you to do so.

3. Express gratitude to the author

Who does not like appreciation? Express your gratitude to the author author for taking out time and sharing his resources.

4. Bookmark

Save the resource for later use at just one tap. We always remember what you save!

5. Social Share

Loved the resource? Share socially and spread some love at a single click.

6. Search

Just type what you are looking to learn, we bring the best curated resources to you.

7. Sort

Sort the resources by interest, trend and time.

8. Magic Link - One tap authentication

Arrghhh. Remembering passwords is tough. Just type in your email and we will send you a secure magic link to login on a click.

9. Featured

Every week, we feature one resource to the home page, loved by the community.

10. The world of tags

Create and search & filter resources by popular tags. Suggest new tags!

11. Weekly newsletter

Every week, we send you some really exciting & interesting resources direct to your inbox.

12. Profile

We love author recognition. Update your profile and let your audience find you on social media.

13. Notifications

We notify you when something interesting comes up related to you.

14. Responsive and mobile friendly

Access https://bestresources.co from any device and spread love.

Technology Stack

Bestresources.co is baked by the latest tech stack. Here is the list -

NodeJs - Best backend JavaScript run time environment
HarperDB - Oh gosh! Fastest and the most flexible data storage. Single node, handling upto 100000 requests per second seamlessly.
Express - Super simple, backend application framework for NodeJs
Angular - JavaScript based frontend framework built by Google.
Nginx - Used as high performance load balancer, web server and reverse proxy. Very handy when it comes to caching.
AWS EC2 - Secure and resizable compute capacity in the cloud, deployed the application over there
Cloudflare - For SSL, client side caching & internet security
Mailgun - For email delivery

P.S. I personally send a thank you email to everyone who signs up, for showing trust!

What's next?

https://bestresources.co is ready to roll out some really exciting features in a couple of weeks.

Dark mode, of-course [first priority]
Personalised content in a more optimised manner
Mobile App
Auto detection of bad content and links
Comment thread for the resource
Thank the author with a note.
Hall of Fame (a separate page including most loved authors with their social media handles)
Idea Box (Request community for resources - inspired from hashnode)
Public APIs for developers to play with
Google, Github authentication
A more improved notifications
Improve trending section
High performance server side caching and a lot more

Would love to hear your valuable feedback or new feature requests.

Well, there were a lot of challenges on the way to bringing this project to life. It was fun, I will cover it in a separate post in detail.

Github

The complete source code is available on github. Feel free to DM me on twitter if you need more details. Would love to help.

Bestresources.co Github Link

If you feel like contributing to the platform, feel free to raise a PR. Let's give something back to the community.

Try Bestresources.co

Bestresources.co

There are many others, who have guided and helped this project come to life - friends, family, @Hashnode, internet strangers!

I am most grateful for everyone’s kindness ❤️

Support

If you feel, my little initiative https://bestresourcse.co adds value to you, please share it with your friends. Drop me a message on twitter, It's the real motivation for me.

What is latency? Let’s deep dive & understand possible ways to optimise it.

Pankaj Tanwar — Fri, 18 Jun 2021 07:00:49 +0000

Hi 👋,

Latency is yet another, a very important topic when we talk about backend engineering or networking. In this article, we will be discussing latency, it's importance and ways to optimise it in order to improve application performance.

What is latency?
Why is it important?
What causes latency?
How to measure latency?
How to optimise the latency?
Conclusion

1. What is latency?

Latency is the total time between a client’s action and the server’s response to that action. It’s simply a round trip time between browser & server.

Network latency is measured in milliseconds (ms).

2. Why is it important?

Latency is directly related to the performance of the application. High latency means a slow network and no one likes to be on a slow website. At a large scale, latency plays a very critical role.

Let’s understand with a simple example by emulating a GET request in a browser.

Blocking : The time for which request was in queue. (A chrome browser can only make at max 6 HTTP request at a time to the server)
DNS Resolution : Time taken in DNS lookup
Connecting : Time taken in a TCP handshake
TLS Setup : Time taken in establishing a secure TLS connection
Sending : Time taken to send the HTTP request to the server
Waiting : Time taken by the server to prepare the response
Receiving : Time taken to receive the response from server

For the very first request, for the first 14KB data, latency is longer due to DNS lookup, a TCP handshake and the secure TLS negotiation.

3. What causes latency?

Latency plays a vital role in system performance. It depends on various factors -

Distance

One of the principal causes of network latency is distance between client (who is making request) and the server (responding to the request).

For example - Suppose, my website (https://pankajtanwar.in/) is hosted in a data center in Delhi. So, for a user accessing from Jaipur (~200KM from Delhi), it is likely to respond within 20-30ms but for a user accessing from New York (~11000 KM from Delhi), it might face a latency close to 70ms.

Transmission Medium

The type of transmission medium being used for data packet transmission also impacts the latency. Modern optic fiber is ~200 times faster than old copper cable-based networks.

Multiple Routers

We should not ignore the fact that routers play a very important role in latency. Routers take some time and analyse the headers of each data package passing thought, which adds to network latency.

Poorly optimized server

Several factors on the backend server, such as slow database queries, low memory space, slow data processing and un-optimized code also impacts the latency.

4. How to measure latency?

There are several standard metrics to measure latency.

Time to First Byte (TTFB)

This is one of the widely accepted matrices to measure latency. As the name itself explains, TTFB is the time (in milliseconds) for a browser to receive the first byte of response from the server.

Ping

Ping is the most common utility to measure latency. It sends a 32 bytes packet to the server & measures the time it took to reach the destination and come back with response to the client.

Round Trip Time (RTT)

It’s a pretty common and simple matrix. It is the total time taken by the data packet, travelling from source to the destination and back.

5. How to optimize the latency?

In terms of application performance optimization, It is very important to reduce the causes of high latency. Here are major methods, which can help us reduce the latency.

CDN

Using CDN (Content Delivery Network) is a major step towards reducing the latency. CDN, caches content, serves it from the nearest data centre & provides an efficient path for data packets to travel on, which drastically reduces the round trip time and so latency.

HTTP/2

HTTP/2 is a highly efficient protocol which reduces the latency by enabling parallelised data transfers, response multiplexing, requests prioritisation, minimised protocol overhead by efficient compression of HTTP headers, reduced round trips and many more.

Client Side Caching

Browsers can cache some of the resources which reduces the calls to the server and improves the latency.

Server Side Optimisations

Server side optimisations such as less disk I/O, caching, efficient algorithms, smart database layer & asynchronous programming can help in optimising the latency.

6. Conclusion

Latency might seem a very simple concept but it plays a very crucial role when we build high scale systems like trading or gaming software. For such systems, even a double digit millisecond latency affects the performance.

In the next article, we will discuss how to system design a real time multiplayer game counter-strike (CS-Go) which is highly sensitive to latency (or ‘lag’).

Let’s connect https://twitter.com/the2ndfloorguy

Originally published at : https://blog.pankajtanwar.in/what-is-latency-lets-deep-dive-and-understand-possible-ways-to-optimise-it

References

Have you ever thought, how ‘nodemon’ works internally? Let’s build our own ‘nodemon’ in under 10 minutes!

Pankaj Tanwar — Wed, 02 Jun 2021 12:04:45 +0000

Hey 👋,

If you have ever worked with Node.Js, you must have used a package called nodemon for development.

For those who are not aware of it, nodemon is a tool that helps develop NodeJs based applications by automatically restarting the node application when file changes in the directory are detected. We can simply install it globally and use it throughout our system without making any additional changes to the code.

But, have you ever thought about how it works internally? How would you proceed if you are asked to build a nodemon clone?

This is a really interesting Node and JavaScript developer interview question. It helps the interviewer to test your basics such as NodeJs stream, child process, events, debouncing etc.

Github Repo Link - Nodekeeper

In this article, we will create a simple Node.Js CLI (command line application) tool named as nodekeeper, similar to nodemon. So, let’s get started.

Requirements

We should be able to run any JS file using nodekeeper <filename> command
Automatically restart the node application when changes are detected in files
Manually restart the server when user enters rs

On a high level, the problem might seem very difficult to implement but it’s not. The main idea behind it is to create a CLI tool which will create a node child process for the given file and keep eyes on the files in the repo. If new changes are detected, just kill the child process and again create a new process.

Ok, some of the terms might seem very technical. Let’s get more into details.

Let’s first understand how to create a NodeJs CLI tool.

So first, what is a CLI tool? CLI stands for ‘command line application’. It helps us run any command on the terminal which will do some magic with our system. For example - to run any JavaScript file NodeJs provides us with node CLI. We just have node index.js from the command line(terminal) and it executes the file. We can give commands just from the terminal.

In our use case also, we want to run a JavaScript file using nodekeeper index.js.

Let’s start. First, we create a new folder named nodekeeper and do npm init inside it to set up the node project.

mkdir nodekeeper 
cd nodekeeper
npm init

After it, a package.json file would be generated. Which will look something like this -

{
    "name": "nodekeeper",
    "version": "1.0.0",
    "description": "",
    "main": "index.js",
    "scripts": {
        "test": "echo \"Error: no test specified\" && exit 1"
    },
    "keywords": [],
    "author": "Pankaj Tanwar",
    "license": "ISC",
    "dependencies": {
    }
}

Let’s create a new javascript file index.js and paste the following code.

#!/usr/bin/env node
console.log(‘Hey! Welcome to nodekeeper’);

Here the first line which starts with #! is called ‘shebang’. Its main purpose is to inform the system what type of script is included in the rest of the file. Here, we have included a path to node binary which tells the system that our file is a file which can we executed by node.

To run a JavaScript file using nodekeeper index.js instead of node index.js we need to create a duplicate node binary.

For this, we add a new key “bin” inside our package.json file.

{
    "name": "nodekeeper",
    "version": "1.0.0",
    "description": "A lightweight alertnative to nodemon.",
    "main": "index.js",
    "scripts": {
        "test": "echo \"Error: no test specified\" && exit 1"
    },
    "bin": {
        "nodekeeper": "./index.js"
    },
    "keywords": [],
    "author": "Pankaj Tanwar",
    "license": "ISC",
    "dependencies": {
    }
}

Now, to install this tool to run globally in our system we run -

npm link

Now, anywhere in our system we can run any JavaScript file like this ‘nodekeeper ’. Really cool right?

Let’s understand now, what child processes are. As we all know, NodeJs is single threaded but still we can take advantage of the child processes with the help of the child_process module. To scale our node app, It helps us to leverage parallel processing on multi-core CPUs.

In simple terms, a child process allows us to run any system command.

Let’s understand child process with an analogy

Today, my father was working on something and he wanted to drink water. As I was sitting doing nothing (as usual), so, he asked me to bring a glass of water for him. Here, my father is the main process who is performing some work. He could also go and get the glass of water but it would impact his work so he has a child process (which is me), and assigned a task to it. This is called parallel computing. Now, my father can continue working on his task and when I (child process) completes my task, I will let the main process know.

So, when we want to run a JavaScript, in the terminal we run node index.js and we get the output. In a similar way, we can create a child process and tell it to run the node index.js command, and give us the output.

There are 4 ways to create a child process in Node.Js, spawn(), fork(), exec() and execFile(). For running a system command, spawn() and exec() are useful.

Syntax for spawn()

const spawn = require(‘child_process’).spawn;
let nodeServer = spawn(‘node’ , [ ‘index.js‘ ])

Syntax for exec()

const exec = require(‘child_process’).exec;
let nodeServer = exec(‘node index.js’, function(data) {
    console.log(data);
})

Both of them will run node index.js command on the terminal. To view the output, we need to pipe this child process to the main process. To do so,

let nodeServer = spawn(‘node’ , [ ‘index.js’ ], { stdio: [ process.stdin, process.stdout, process.stderr ]})

It has piped the child process to the main process. So, we can log it’s output on the terminal.

*BUT here is a catch in the working of spawn() and exec(). *

spawn() gives output in streams but exec() gives out after the whole data is received. Suppose, in the file index.js we have -

console.log(‘Hey Hashnode’)
setTimeout(function() {
    console.log(‘Timer completed’);
}, 5000)

If we run this file using both methods. We see that spawn child process, immediately logs ‘Hey Hashnode’ and after 5 seconds it logs ‘Timer completed’ but exec child process will log both lines after 5 seconds. So, it explains spawn gives output in streams, it does not wait for the file to execute completely.

For our use case, we need to spawn the child process.

For watching files to new changes, we can make use of NodeJs inbuilt module, fs. It exposes a function called fs.watchFile but there have been a lot of issues reported by the community saying it’s not reliable. It fires multiple events sometimes for a single file change which results in high CPU utilization. So, to overcome this problem we can use the chokidar package.

We can pass in watch paths and other paths, we want to be ignored and we can listen to it’s events to get notified when there is a new change.

const chokidar = require(‘chokidar’);

chokidar.watch([
    "/**/*/*js"
], {
    ignored : “**/node_modules/**”
}).on(‘all’, () => {
    console.log(‘File changes detected’);
)

So, whenever we detect changes, we can kill the current node child process and start a new process again.

To kill a process -

nodeServer.kill(‘SIGTERM’)

SIGTERM is just one of the signals which is used to kill any process. There are many types of signals. More Info can we found here

If we structure our code a bit, our final index.js for this would look like this -

#!/usr/bin/env node
const spawn         = require('child_process').spawn;
const chokidar      = require('chokidar');
const path          = require('path');

class Nodekeeper {
    constructor() {
        this.__init__();
    }

    __init__        = () => {
        this.args               = process.argv;
        this.fileName           = this.args[2];
        this.cwd                = process.cwd();
        this.watchPaths         = [
            path.join(this.cwd, "/**/*.js")
        ];
        this.ignoredPaths       = "**/node_modules/*";

        this.reload();
        this.startWatching();
        this.listeningEvents();
    }

    reload          = () => {
        if(this.nodeServer) this.nodeServer.kill('SIGTERM');

        this.nodeServer     = spawn('node', [ this.fileName ], { stdio: [ process.stdin, process.stdout, process.stderr ]});
    }

    startWatching   = () => {
        chokidar.watch(this.watchPaths, {
            ignored         : this.ignoredPaths,
            ignoreInitial   : true
        }).on('all', (event, path) => {
            this.reload();
        });
    }

    listeningEvents    = () => {
        // listening on CLI input
        process.stdin.on("data", (chunk) => {
            let cliInput = chunk.toString();

            switch(cliInput) {
                case 'rs\n':
                    this.reload();
                    break
            }
        });
    }
}

new Nodekeeper();

Now, we can see, if we create a new express server, go to that folder and run it using nodekeeper server.js , upon file changes, it will automatically restart the server.

We put everything in a nodekeeper class and export it as a module.

We have one more requirement which is when a user enters rs, we need to manually restart the server. It’s very simple as we have already implemented logic for restarting the server. To capture what user entered we just need to put an event on the main process.

process.stdin.on("data", (chunk) => {
    let cliInput = chunk.toString();

   switch(cliInput) {
        case 'rs\n':
             this.reload();
             break;
   }
});

It works great but there are still some issues.

What if we save multiple files at the same time or we press Ctrl+S multiple times. Chokidar would fire change events multiple times. So, it will kill the on-going node process and start a new one which is CPU extensive. To overcome this problem, we use a concept called ‘debounce’. We delay the execution for a time period and execute it once the user stops saving. (This concept is used in search bar suggestions, it will not fetch data on every keystroke, it will affect the performance, Instead of it, we typically fetch the data when user stops typing)

Hope you have a got a decent idea of what how to proceed to build a nodemon clone.

I have published with a modified version nodekeeper - a lightweight nodemon alternative. Package can be found here. If you are willing to contribute, pull requests are welcome.

References -

https://nodejs.org/

Show some love. Originally Published at https://www.pankajtanwar.in/blog/have-you-ever-thought-how-nodemon-works-internally-lets-build-our-own-nodemon-in-under-10-minutes

Let's connect - https://twitter.com/the2ndfloorguy

DEV Community: Pankaj Tanwar

Challenge of day #21 - Split a String in Balanced Strings

Challenge of the day #20 - To Lower Case.

Challenge #19 - N-Repeated Element in Size 2N Array.

Challenge #18 - Count Negative Numbers in a Sorted Matrix

TLDR;

Challenge #17 - Sum of Unique Elements.

1. My all-time savior best friend, Hashmap

2. Using sort

3. Using a constant array

Challenge #16 - Best Time to Buy and Sell Stock

Challenge #15 - Count Number of Pairs With Absolute Difference K

Prevent duplicate cron job running.

So, What's flock?

How to setup a cron job, using flock ?

Step 1 - Install flock, if not available in your system

Step 2 - Open up cron tab

Step 3 - Install your new cron

The unfortunate truths of flock

How to run multiple commands with flock

flock doesn't always seem to be working.

What is the sorting algorithm behind ORDER BY query in MySQL?

1. Smart use of Indexes

2. Filesort Algorithm

TL;DR

Smart algorithm behind MySQL ORDER BY -

References -

🎉 Launching Bestresources.co : Share & explore personal resources at one place!

Motivation

Our Story

Features -

Technology Stack

What's next?

Github

Try Bestresources.co

Support

What is latency? Let’s deep dive & understand possible ways to optimise it.

Table of Contents

1. What is latency?

2. Why is it important?

3. What causes latency?

4. How to measure latency?

5. How to optimize the latency?

6. Conclusion

Have you ever thought, how ‘nodemon’ works internally? Let’s build our own ‘nodemon’ in under 10 minutes!

Requirements

Let’s first understand how to create a NodeJs CLI tool.

Let’s understand child process with an analogy

Syntax for spawn()

Syntax for exec()