DEV Community: suvhotta

Understanding Object Copying in Python

suvhotta — Fri, 17 Nov 2023 11:54:53 +0000

My Initial Struggle with List Copying

When I began my programming journey in Python, I encountered a scenario that many new programmers face: copying a list and passing it to a function for further modification. Here's the approach I initially used and why it didn't work as expected.


def square_nums(nums_list):
  ...

original_list = [1, 2, 3, 4, 5]

copied_list = original_list

squared_nums = square_nums(copied_list)

However, I soon realized a critical issue: any modifications made to copied_list within the function were also reflected in original_list. This was puzzling and led me to delve deeper into Python's documentation.

The Revelation: Aliases vs. Actual Copies

Upon reading the official documentation, I figured out that the way I was copying the contents of the original_list was fundamentally flawed. In fact, as per Python semantics, I wasn't creating a copy, instead I was just creating an alias using the assignment operator. Any number of aliases we create, they will all point to the same object. That's why when the object itself was getting modified, I was getting altered values in the original_list in my code.

The Correct Approach to Copying in Python

Python provides 2 ways of doing so:
1) Shallow Copy
2) Deep Copy
Both of these are available in the copy module provided as part of Python's standard library.

Shallow Copy

In shallow copy, a new Container/Object is created but the containing items are still a reference to the original Container/Object.

This can be further verified by checking the memory of the containers and their individual items using the id() function.

This way of copying works best if we've immutable items.

This is achieved by using the copy.copy() function of the copy module.


import copy

original_list = [1, 2, 3]

# Creates a shallow copy
copied_list = copy.copy(original_list)

# Will print True
print(id(original_list[0]) == id(copied_list[0]))

original_list[0] = 0

# Will print False
print(id(original_list[0]) == id(copied_list[0]))

In the above code, the last print statement will still be False because integers are immutable. So once original_list[0] is re-assigned a new value, the reference is updated to the new integer value i.e. 0. But copied_list[0] continues to reference the original integer object 1. This behaviour would've been different if original_list contained mutable objects (like other lists, dictionaries, etc.), where a change in the mutable object inside original_list would be reflected in original_list as well due to the shared references.

Due to potential issues that could arise when mutable objects are involved, copy module has another offering: deepcopy.

Deep copy

In shallow copy, a new Container/Object is created and it recursively inserts copies of the objects(It doesn't always copy but depending on the type of the object) found in the original.


import copy

original_list = [1, 2, 3, [2,3,4]]

copied_list = copy.deepcopy(original_list)

# Will print False
print(id(original_list)==id(copied_list))

# Will print True
print(id(original_list[0])==id(copied_list[0]))

# Will print True
print(id(original_list[1])==id(copied_list[1]))

# Will print True
print(id(original_list[2])==id(copied_list[2]))

# Will print False
print(id(original_list[3])==id(copied_list[3]))

In the code above, we see an interesting thing i.e. when the objects are immutable like those at indexes 0, 1, 2 in original_list then those in copied_list also reference to the same object. But when the element is a mutable one like a list then a separate copy is maintained in the copied_list

The Role of the memo Dictionary in Deepcopy

Avoiding Redundant Copies:
When an object is copied using deepcopy, the memo dictionary keeps track of objects that have already been copied in that copy cycle. This is particularly important for complex objects that may refer to the same sub-object multiple times. Without memo, each reference to the same sub-object would result in a new and separate copy, potentially leading to an inefficient duplication of objects.
Handling Recursive Structures:
Recursive structures are objects that, directly or indirectly, refer to themselves. For example, a list that contains a reference to itself. deepcopy uses the memo dictionary to keep track of objects that have already been encountered in the copying process. If it encounters an object that’s already in the memo dictionary, it uses the existing copy from the memo rather than entering an infinite loop trying to copy the recursive reference.

Implementation in User defined Classes

In order to implement the functionalities of copy and deepcopy, we can use the magic methods copy() and deepcopy() within the class.
Internally even the list data structure is a class and that's how we get the list.copy() feature.

With this deeper insight into Python's object copying, you're now better prepared to handle the nuances of mutable and immutable objects in your coding adventures. Happy coding!

Multi-Threading in Python: Deep-dive

suvhotta — Sat, 19 Mar 2022 15:54:11 +0000

One universal truth about cPython: One thread runs python, while 'N' others sleep or wait for I/O

The above holds true thanks to the Global Interpreter Lock (GIL). I won't dig deep into GIL here as that would be taking us off track from the current discussion.
So if anybody is telling u that 2 or more threads can simultaneously do something with your cPython program, then its time to call their bluff. But I don't deny that Python isn't a multi-threaded programming language, its just that the multiple threads can't access your program at a time. Then comes the logical question, what's the use of such a multi-threading then? We'll be getting to the answer of that in a bit.

Python supports 2 kinds of Multi-Tasking:

Co-operative
Preemptive

Co-operative Multi-Tasking in Python:

When Python knows that its trying to do some which might take some time like performing some I/O or network operation, it voluntarily drops the GIL. In other words, the Python thread knows that it won't be doing any python work in a while and co-operates with other threads by voluntarily submitting the GIL access.

Further explained through the following example:

import socket

messages = []

def socket_connect():
    # This is an example of cooperative multi-tasking
    s = socket.socket()
    messages.append('connecting')
    s.connect(('python.org', 80))
    # Python thread drops the GIL while doing the socket connect operation.
    messages.append('connected')

threads = []

for i in range(3):
    t = threading.Thread(target=socket_connect)
    threads.append(t)
    t.start()

for t in threads:
    t.join()

print('\n'.join(messages))

Output:

connecting
connecting
connecting
connected
connected
connected

So why do we get 3 'connecting' strings followed by 3 'connected' strings instead of 3 pairs of 'connecting connected'?

The answer lies in the cooperative multi-tasking performed by Python thread.

Let's dig a bit deeper into what happens when we write the following line of code:

s.connect(('python.org', 80))

Internally it calls the following internal_connect function written in C: (https://github.com/python/cpython/blob/main/Modules/socketmodule.c)

Py_BEGIN_ALLOW_THREADS
res = connect(s->sock_fd, addr, addrlen);
Py_END_ALLOW_THREADS

The above follows a common paradigm of how GIL is released and re-acquired:

Save the thread state in a local variable.
Release the global interpreter lock.
Do some blocking I/O operation.
Reacquire the global interpreter lock.
Restore the thread state from the local variable.

Py_BEGIN_ALLOW_THREADS, and Py_END_ALLOW_THREADS are 2 macros
used to simplify the above processes.

So we can see that the GIL is released prior to establishing connect method being called and the same is acquired by another thread which again releases the GIL at the same point in the code. So by the time the 3rd thread has released the GIL, the 1st thread might have been successfully established the socket connection. If so, then the 1st thread is now ready to re-acquire the GIL and continue executing python code.

To see what difference it would make, if we didn't have the network operation, lets run the following code:

import socket

messages = []

def calculate_squares():
    messages.append('calculating')
    for i in range(5):
        i*i
    messages.append('calculated')

threads = []

for i in range(3):
    t = threading.Thread(target=calculate_squares)
    threads.append(t)
    t.start()

for t in threads:
    t.join()

print('\n'.join(messages))

Output:

calculating
calculated
calculating
calculated
calculating
calculated

As expected, there isn't any network operation involved, hence the GIL isn't released co-operatively.

Pre-emptive Multi-Tasking in Python:

When some thread has held the GIL for a certain specific amount of time(Default being 5ms), the interpreter pitches in to force the current thread to drop the GIL and pass it on to some other thread for execution. Here, the verb preempt means to halt the execution of a task with a view to resuming it later.

The same can be understood better in the following example:

messages = []

def run(thread_name):
    for i in range(50):
        for j in range(50):
            pass
        messages.append(f'Thread: {thread_name}')

threads = []

for i in range(2):
    t = threading.Thread(target=run, args=(i,))
    threads.append(t)
    t.start()

for t in threads:
    t.join()

print('\n'.join(messages))

The output of the above code will be mostly 50 occurrences of 'Thread: 0' followed by 50 occurrences of 'Thread: 1'. If you change however the ranges inside the run function to 5000, you'll notice that there will be some occurrences of 'Thread: 0' and somewhat similar number of occurrences of 'Thread:1'. So in short, the GIL is being switched pre-emptively as per some default set interval.

The following code can tell us what the default interval for switching the GIL between threads is:

import sys

print(sys.getswitchinterval())

By default for python3.10 it is 5ms (https://github.com/python/cpython/blob/main/Python/ceval_gil.h)
However the same can be changed as follows:

import sys

sys.setswitchinterval(0.0001)

The above sets the switch interval to 0.1 ms.

Let's say that there are 2 threads: Thread A, and Thread B. If Thread A holds a lock and gets preempted, then maybe Thread B could run instead of Thread A.
If Thread B is waiting for the lock that Thread A is holding, then Thread B is not waiting for the GIL. In that case Thread A reacquires the GIL immediately after dropping it, and Thread A continues.

Purpose:

Purpose of Co-operative multi-tasking in python is to finish tasks faster, in case they're waiting for I/O.
The purpose of pre-emptive multi-tasking isn't to go faster but rather to simulate parallelism. So instead of one thread running till the end, there can be many threads that appear to run parallel.

Inspired from: https://opensource.com/article/17/4/grok-gil#:~:text=By%20default%20the%20check%20interval,of%20bytecodes%2C%20but%2015%20milliseconds.

AWS Exception: PolicyLengthExceededException

suvhotta — Mon, 14 Mar 2022 09:36:48 +0000

The other day we'd an issue at work, where we were getting a PolicyLengthExceededException for our lambda permissions, which we were attaching programmatically during the runtime, every-time we were creating a resource. This wasn't a recurring issue but rather an intermittent one.

Upon further debugging, I came to know that the resource based policy for a lambda function is limited at 20KB.
The policy we were creating was overshooting the limit by 20 bytes. It's not much, but we've to abide by the limits set.

To my astonishment, the parameters being passed on to the policy were more or less same as compared to those from the other policies that we'd earlier attached to the same lambda.

So I'd now 2 genuine concerns:

If there was really any limit breach, this wouldn't have been an intermittent issue.
How did the earlier policies get added?

Upon some further R&D, found out that the policy size limit wasn't something that was being checked very rigorously. Instead the same was only being checked once you'd a large chunk of policies with similar names, performing similar jobs and which could have otherwise been clubbed into one using wildcards.

Solution:

Added an aws_lambda_permission component to our terraform script which creates the corresponding permission every-time the lambda function is created.

Note that I've generalised the source_arn using wildcard so that we don't have to create a policy every-time we create a resource and thus solving the problem of too many duplicate/similar policies.


resource "aws_lambda_permission" "allow_event_bridge_rule_permission" {
  statement_id  = "AllowLambdaInvocationFromEventBridge"
  action        = "lambda:InvokeFunction"
  function_name = "${var.func_name}"
  principal     = "events.amazonaws.com"
  source_arn    = "arn:aws:events:${var.aws_region}:${var.aws_account_id}:rule/model_*_name"
}

Note: Even after adding the above code to the tf script, still we were getting the same exception while executing the script. I removed all the existing duplicate policies and re-ran the script and it went fine.

Even if you're manually creating the policies from the console/CLI its the best practice to use wildcards in the source_arn so as to minimise the number of policies attached to a resource.

Hope this helps you!

Mocking using pytest

suvhotta — Sat, 11 Sep 2021 16:19:21 +0000

Pytest is a unit test module in python.
Mocking is a way of mimicking the working of a service in
a way to substitute for the real one.

In this post I'll be focussing on mocking different types of methods present in a class.

Class ABC has been defined in a python file python_class.py

class ABC:

    def call(self):
        self.static_method()
        ABC.class_method()
        self.instance_method()
        self._ABC__private_instance_method()
        ABC._ABC__private_class_method()
        ABC._ABC__private_static_method()

    @staticmethod
    def static_method():
        print("static_method")

    @classmethod
    def class_method(cls):
        print("class_method")

    def instance_method(self):
        print("instance method")

    def __private_instance_method(self):
        print("private instance method")

    @classmethod
    def __private_class_method(cls):
        print("private class method")

    @staticmethod
    def __private_static_method():
        print("private static method")

Yeah I know this isn't how class methods and static methods should be used, but that's not on today's agenda.

In the call method, we can see that there are 6 other methods being called. That's where we need to mock the methods - the place where they're being called/being used instead of where they've been defined.

But before that we need to install 2 packages: pytest, pytest-mock

from unittest.mock import Mock
from python_class import ABC


class TestPythonClass():
    def test_all_methods(self, mocker):
        directory = "python_class.ABC"
        instance_mock = mocker.patch(f"{directory}.instance_method", return_value=Mock())
        static_mock = mocker.patch(f"{directory}.static_method", return_value=Mock())
        class_method_mock = mocker.patch(f"{directory}.class_method", return_value=Mock())
        pvt_instance_mock = mocker.patch(f"{directory}._ABC__private_instance_method", return_value=Mock())
        pvt_class_mock = mocker.patch(f"{directory}._ABC__private_class_method", return_value=Mock())
        pvt_static_mock = mocker.patch(f"{directory}._ABC__private_static_method", return_value=Mock())

        ABC().call()

        assert instance_mock.call_count == 1
        assert static_mock.call_count == 1
        assert class_method_mock.call_count == 1
        assert pvt_instance_mock.call_count == 1
        assert pvt_class_mock.call_count == 1

mocker is a fixture that is shipped with the pytest-mock module. We can simply pass it on as an argument during the test method definition without importing.

In this way I've mocked 6 different types of methods:

instance method
class method
static method
private instance method
private class method
private static method

Lemme know in case I missed something or I need to add anything.

Git For Beginners - Part 2

suvhotta — Wed, 19 Aug 2020 19:13:36 +0000

This series is aimed to simplify the concept of git for beginners. In the first Part we've discussed about the basics of different working areas, different scenarios involving adding and committing changes to our repo.

We will continue with our approach of looking into various used cases, instead of just memorizing git commands.

Before going into that there was something about the git objects that I'd missed in my earlier post. We saw how the commit object stores info about the previous commits, so every child commit has a pointer to the parent commit and thus a linkage is established.

But can't this be the other way round? There could be a possibility that the parent commit stores pointer for the child commit. However this isn't something that is being followed because the git objects are immutable. So they can't be edited once they've been created. Hence any chance of parent storing child commit pointers get nullified.

Creating Branches:

Branch is just a pointer to any commit. Branches are required to work in parallel towards multiple issues related to a project. The default branch that gets created with our repo is called master.

Every time we commit, the master branch pointer moves forward automatically.

git branch <branch_name>

This command can be used to create a new branch.

However the above command just created a new branch, but it won't automatically get switched to this branch.

git checkout <branch_name>

This command checks out the newly created branch and we can begin our adding our files and commits to this branch.

git checkout -b <branch_name>

Most often than not, we want to create a branch and switch to it simultaneously i.e. combining the work done by the first and the second commands.

Merging Branches:

Suppose we've created a new branch to work on any enhancement of the project. Once the development of the same is completed, we would want to merge it with our initial branch i.e. master.

But before that we can check what all are the differences between the master and our new branch by the following command:

git diff master..<new_branch>

There are 2 types of merges namely Fast forward merge and Three way merge. Fast forward merge is when we try to merge one commit with a commit that can directly be reached by following the first commit's history, thus git directly moves the pointer forward as there isn't any diversion in the path of merging.

In case of Three way merge, the development has diverged somewhere from some past commits in either of the branches and thus a Fast forward merge is no longer possible. In this case Git performs a three way merge using the two snapshots provided by the branch tips and common ancestor of the two branches. Instead of just moving the pointer ahead as was the case in fast forward merge, git creates a new snapshot (commit) and this is called a merge commit.

The MERGE_HEAD pointer points to the head of the other branch. (The one getting merged)

steps for merging:

Checkout the branch where you wish to merge into:

git checkout branch1

merge the branch

git merge branch2

Squash and merge:

At times, there might be multiple commits in a branch but we wish to merge them as a single commit in our master, in such a case we opt to squash the commits.

Will group all feature branch commits into one commit then append it in the front of the master branch. Will add an extra dummy commit.

steps for squash and merge:

Checkout the branch where you wish to merge:

git checkout branch1

merge the other branch

git merge --squash new_branch

Note: Squashing doesn't delete the original commits that are squashed to form the new commit.

Re-base:

The strategy used so far to merge branches works absolutely fine. However the commit history is all messed up, esp. if there are multiple people working.

To avoid this issue, there is a cleaner way to merge the branches.

git re-base will append all commits history of the feature branch in the front of the master branch and won't add any extra dummy commit.

git forms a copy of the feature branch on top of the master branch, hence the name re-base.
steps:

git checkout feature

git rebase master

this would rebase the feature branch on to the latest commits of the master branch. So, this sort of changes the commit history for the feature branch.

git checkout master

git rebase feature

Note: Since git rebase changes the chronology of the commits, it isn't advisable to use it on commits which others depend on.

Deleting Branches

Most of the times after merging a branch, we don't need the branch anymore and that branch can be deleted.

git branch -d <branch_name>

Git For Beginners - Part 1

suvhotta — Mon, 10 Aug 2020 15:56:21 +0000

I've watched numerous videos on how to use git as a version control system. But I used to forget the stuffs every once in a while. That's when I decided to jump into the git internals and learn git from a different aspect this time. I didn't memorize the git commands, rather studied extensively on various practical used cases of those commands. Surely such a mental mapping has helped me in learning the basics of git faster and retain them in a better way.

Let the fun begin!

Working Areas:

Current Working Directory:
This refers to the physical area where we as developers actually write our code. It is here that we make our changes and expect our code to reach out to our colleague who might be working remotely thousands of KMs away.
Staging Area/Index:
Staging area is the intermediate area where the files reside before getting saved into the repository. Staging area is the penultimate stage where modifications are still possible before data gets stored into the repository.
.git repository:
.git is the repo where our transferable code is stored. Whenever we clone a repo using git clone, it is this .git directory which actually gets downloaded as a local copy and development starts on top of this.

The basic Git workflow goes something like this:

You modify files in your working tree.
You selectively stage just those changes you want to be part of your next commit, which adds only those changes to the staging area.
You do a commit, which takes the files as they are in the staging area and stores that snapshot permanently to your Git directory.

Starting a new Repo:

git init

Inside any directory if we type the above commands then it would create a .git hidden directory and would track all the changes to the files and folders of the parent directory.

Staging our files:

Suppose we initialized our directory and are about to add the first file to the staging area, this can be done by the following command for a single file:

git add <filename>

for multiple files to be staged:

git add .

Now that we've added our first files to the staging area, let's see how git stores them under the hood.

git actually stores objects in a key store format.

For files, these are stored as blob(Binary Large Objects) SHA-1 key is generated by passing the contents of the file through the encryption algorithm and the 40-character checksum hash now corresponds to the file.
Note: There is no file name attached to the blob object.

8f94139338f9404f26296befa88755fc2598c289 is an example of 40 digit checksum hash.

Now you must be wondering how does the file name gets mapped on to the blob contents?

This is done by another git object - tree.

Tree is basically used to store the filenames as well as the directories. A single tree object contains one or more entries, each of which is the SHA-1 hash of a blob or sub-tree with its associated mode, type, and filename.

100644 blob a906cb2a4a904a152e80877d4088654daad0c859 README
100644 blob 8f94139338f9404f26296befa88755fc2598c289 Rakefile
040000 tree 99f1a6d12cb4b6f19c8655fca46c3ecf317074e0 lib

The above is an example of a simple tree.

Git normally creates a tree by taking the state of your staging area or index and writing a series of tree objects from it. So, to create a tree object, you first have to set up an index by staging some files.

Commit our files:

Once we've staged our files in the staging area/index we can then commit those changes to the local Repo using the following command:

git commit -m "some sample message"

if we wish to write multi-line commits, we can simply write:

git commit

and it would open a text editor to write the multi-line commits.

Commits
The format for a commit object is simple: it specifies the top-level tree for the snapshot of the project at that point; the parent commits if any (the commit object described above does not have any parents); the author/committer information (which uses your user.name and user.email configuration settings and a timestamp); a blank line, and then the commit message.

Checking the differences between various areas :

git diff

this command shows the differences between the staged files and their corresponding versions in the working directory. It however doesn't check for those files which haven't been added to the staging area.

git diff --staged

this command marks the contrast between the committed and the staged files.

git status

Git status does 2 things:
it compares the index file with:

the current working directory changes are reported as Changes not staged for commit
the HEAD commit — changes are reported as Changes to be committed

Undoing any changes:

If suppose we've made some changes which we later feel aren't of much use. Then we can undo them before actually committing.

git reset --hard HEAD

This resets the file of the working directory and of the staging area to the commit pointed in HEAD.

Bringing back a version of file:

git checkout <file_name>

This command replaces the working directory version of the file with the prev version that has been staged.

git checkout <commit_hash> --<file_name>

This command replaces the working directory and staged versions of the file with the prev version that has been committed.

Please note that both the above commands, replace the current working directory content and this change can't be undone.

Un-staging a file:

git rm --cached <filepath>

This command removes the file from the staging area and keeps an untracked copy in the current working directory. If a commit is made now, it would remove that from the repo as well.

git reset HEAD <filepath>

In the case where the file is already in the repo, it replaces the index version of the file with the one from repo (HEAD), effectively unstaging the modifications to it.

Remove a file:

git rm <filepath>

The above command removes a file from both the working directory and staging area.
You might be thinking that we can simply delete the file from our system and that would work equally good. But no that isn't the case, git rm removes the file from both working directory and staging area while the rm command of operating system only deletes that from the working directory and we need to add the change again to get reflected in the staging area. Something kind of adding a negation to reflect the change.

Checking the commit graph:

git log provides us a log of all the commits that have been made to the repo. However to get a better representation use the following command:

git log --all --decorate --oneline --graph

There is a shortcut that I use to remember the above command - git log A DOG.

Under the hood : Python Dictionary

suvhotta — Fri, 17 Jul 2020 16:28:46 +0000

Dictionary has always been a developer's go to tool because of the unique relation between the key and value pairs. So whenever there is any need of capturing frequencies, unique objects etc. in any algorithm, dictionary is the first thing that pops up to our head.

Let's dig deep into how python dictionary works under the hood.

To explain it better I'll try to relate it with a phone-book.

Dictionary is implemented by using a data structure - hash table. The hash table again is a combination of two arrays - one for storing the indexes and the other for storing the key, values.

Adding a new key:

Initially a hash-table is always provided with a size of 8, which increases with the increase in num of keys.

To calculate the index of the key, a hash function is used. However if we calculate the hash of 'a' in a 64 bit system, it would return a value of 12416037344. So to avoid creation of a hash-table with such a gigantic size, a mask is used in a bit-wise & operation along with the hash result.

For a hash table of size 'n' the mask used is (2^n - 1).
This ensures that for the key, the index is never exceeding the size of the existing hash table.

Conflict:

Let's take our phone-book example and think of a hash table with a size of 26 each for each letter of the alphabet.

Let's save a contact by the name 'Ajay' and the hashing algorithm assigns it to the index of 5.

After a few month, a new intern 'Ashok' joins the company, and we've to save his number. The algorithm again assigns the same index 5 to Ashok, but Ajay's contact info is already present there in key, value form.

This kind of a situation is called a conflict.

Now let's think of different ways to resolve this conflict. One approach would be to use linked lists at each index of the hash table and the each individual nodes of the linked list represent a key, value pair. Such an approach would be a good one when we have a limited amount of data, however if the data size is huge, then we won't be able to offer a read operation within O[1].

The solution that CPython uses for conflict resolution is re-sizing.

Resizing:

Whenever a new key is added to a dict, CPython checks if the 2/3^rd of the existing size is filled or not. This ratio is called the load factor, and it is an indication of whether re-sizing is required or not. If the load factor is 2/3 then hash table needs a re-sizing

How much new space is to be added is calculated by the num_of_occupied spaces * 3. However the size should be a power of 2, hence the next highest power of 2 is chosen.

Initaly it starts with a size of 8, let's say 6 out of 8 spaces get occupied, then it should allocate a further more of 18 (6*3) but it increases the total size of the hash table to 32.

Accordingly the indexes for the hash table are adjusted.

Deleting a key:

Similarly when an element is deleted or say multiple elements are deleted, the hash-table size undergoes a shrinkage. However delete item operation doesn’t trigger an array resize. If the number of used slots is much less that the total number of slots. However, when a key/value pair is added, the need for resize is based on the number of used slots + dummy slots so it can shrink the array too.

This was an overview on the processes happening during the lifetime of a python dictionary. For more details on various edge cases and optimizations, I suggest you reading the source code.

Hope it was informative for you!

How Javascript works

suvhotta — Fri, 27 Mar 2020 11:04:05 +0000

Javascript is a single threaded language, at least the majority of it. Things are executed in a certain order. But at times it does work in a non-blocking manner which sort of gives it an asynchronous appearance. But how does the Javascript engine handle all of the script, Browser APIs everything at once ? I can't do all of it at once, even I got like 10 hands.

Before explaining how things work I'll give a briefing of different parts involved in running Javascript.

Call Stack:

Javascript is single threaded, it means it has one call stack. Call Stack is a data structure which keeps track of function calls in our program. When ever we call a function for its execution, we are pushing it to the stack. It is popped out of the stack when the execution is completed.

Heap:

This is where all the memory allocation happens for your variables, that you have defined in your program.

Tasks

Tasks ensure that actions are happening sequentially. A task can be as simple as any function scheduled in the Javascript. Between the tasks, the browser may render updates. All the things mentioned here are generally Macro Tasks. Callbacks of such tasks are usually dealt by a data structure called Task Queue.

There is also something called Micro Task. Micro Tasks are useful to make something async without taking the penalty of a whole new task.
As per MDN:

A microtask is a short function which is executed after the function or program which created it exits and only if the JavaScript execution stack is empty, but before returning control to the event loop being used by the user agent to drive the script's execution environment.

With ES6, Promises have been introduced into Javascript, which along with Mutation Observer, use the microtask queue to run their callbacks.

Task Queue:

Task Queue is a JavaScript run-time messaging queue which handles task that is allocated by different Web APIs. This queue is dedicated to handle the Web API callbacks. The messages are processed once the call stack is clear.

Microtask Queue:

As mentioned above, Microtask queue handles callbacks with respect to Promises and Mutation Observer.

Event Loop:

It has responsibility to see weather the call-stack is empty and does the task queue contains pending task to process. If the call-stack is empty, it will push the task to the call-stack from the queue and the task gets processed.

Here is how the Event Loop handles different tasks:

First, each time a task exits, the event loop checks to see if the task is returning control to other JavaScript code. If not, it runs all of the microtasks in the microtask queue. The microtask queue is, then, processed multiple times per iteration of the event loop, including after handling events and other callbacks.
Second, if a microtask adds more microtasks to the queue by calling queueMicrotask(), those newly-added microtasks execute before the next task is run. That's because the event loop will keep calling microtasks until there are none left in the queue, even if more keep getting added.

All of it can be better illustrated by the code below:

console.log('start')
setTimeout(()=>console.log('Time-out 1'),0)
setTimeout(()=>console.log('Time-out 2'),0)
Promise.resolve().then(()=>console.log('Promise 1'))
Promise.resolve().then(()=>console.log('Promise 2'))
console.log('end')

The output on Chrome would be:

start
end
Promise 1
Promise 2
Time-out 1
Time-out 2

Now why is that so?

Firstly, the two script jobs are scheduled on to the Call Stack as the first task. The logging start is picked up from the task queue and executed.
Set Timeout is handled by the Browser as it is a Browser window object. The timer and everything is calculated by the Browser as waits for a given delay then schedules a new task for its callback in the Task Queue. This is same for both the Timeouts. So there are two Timeout tasks waiting in the Task queue. As logging script end is part of the first task, so setTimeout is logged in a separate task.
After that, the Promises are run and their callbacks are pushed to the Microtask queue. They're pushed to the queue because the first task isn't yet completed and microtasks callbacks can't be run when one task is in mid-execution.
Then the logging end as it is also part of the first task which is yet to be completed. Note that we have already two Promise callbacks pending in the microtask queue, thus as soon as there is no task in middle of completion, the event loop prioritizes the completion of the microtasks callbacks(over the timeouts) and pushes them to the Call Stack. This process continues until the Mirco Task queue is empty.
Then the rest of the task queue is pushed to the Call Stack.

I Hope the above process was clear.

However the output will vary from Browser to Browser. This is sort-of excusable, as promises come from ECMAScript rather than HTML. ECMAScript has the concept of "jobs" which are similar to microtasks, but the relationship isn't explicit aside from vague mailing list discussions. However, the general consensus is that promises should be part of the microtask queue, and for good reason.

Before concluding this write-up, a brief intro on Web-Workers.
Web-workers help to handle the intensive tasks, which would have blocked the UI. Web-workers help in making the best use of multi-threading.

But wait, Wasn’t Javascript a single-threaded language? The very first sentence of this writeup says so?

Well, I stand true to my words. This is because, the Web workers aren’t part of the Javascript, rather they’re a built in features of the browsers which are being accessed through Javascript. Web Workers are in-browser threads that can be used to execute JavaScript code without blocking the event loop.

Will be covering them in a separate article shortly.

Till then Happy Quarantine, Stay Safe and Keep Learning!

Processes and Threads - Part 1

suvhotta — Mon, 16 Mar 2020 04:48:48 +0000

Ask anyone What is the difference between a Process and a Thread?. Most of them would first take a short pause and reply you "Ummmn they're actually the same thing."(Meanwhile deep-down there would be this inner battle to find out the exact answer)

As a beginner, I was also using both the terms interchangeably, till I realized it was a Cardinal Sin to do that.
Hence decided to make a two part series on what both of them do in our OS.

Process

Whenever any program is running on our computer, it can have one or multiple processes associated with it.

Thread

Thread can be thought of as the basic unit of execution of any process. Thus one process can consist of one or many threads.

Threads within the same Process run within a shared memory space while different Processes are allocated separate memory spaces.

Process Management and scheduling:

When a process is being scheduled, it may appear due to the Multi-Programming aspect that we are able to switch between typing a document, simultaneously while playing some music. This is something that happens by the help of the time sharing. Objective of time sharing is to switch the CPU between the programs so fast that the users can interact with each program while it is running and it would feel as if everything is happening at the same time. But if we dig deep into it, its the CPU which is actually switching at a very high rate between all these programs, to give a real time feel and run all of them in parallel without any lag.

This luxury isn't available in a Single Processor System. There will never be more than one running process in such systems.

In a Multi Processor System, there is a process scheduler, which takes care of the end to end scheduling process of various programs. If there are multiple processes queued up, then the job of the process/job scheduler is to allot the CPU to the processes depending on the priority. If there is a low priority process running and the OS gets a command to run a higher priority task, then the lower priority one is pushed to the waiting queue and the CPU resources are prioritized towards the higher priority process in hand.

So, Kudos to your OS,arranging the execution of applications so that you believe that there are several things happening at once. This is complicated because the CPU can only do one thing at a time. Today's multi-core processors and multi-processor machines can handle more work, but each processor core is still capable of managing one task at a time.

In order to give the appearance of lots of things happening at the same time, the operating system has to switch between different processes thousands of times a second. Here's how it happens:

A process occupies a certain amount of RAM. It also makes use of registers, stacks and queues within the CPU and operating-system memory space.
When two processes are multi-tasking, the operating system allots a certain number of CPU execution cycles to one program.
After that number of cycles, the operating system makes copies of all the registers, stacks and queues used by the processes, and notes the point at which the process paused in its execution.
It then loads all the registers, stacks and queues used by the second process and allows it a certain number of CPU cycles.
When those are complete, it makes copies of all the registers, stacks and queues used by the second program, and loads the first program.

Interprocess Communication:

When there are several dependent processes which need to pass data among themselves, then there is a clear need for internal communication. This is done by the Inter process Communication.

Two Methods are widely used:

Shared-Memory:
There would be a portion of memory shared by the processes and in that space, all the values to be passed on are written and communicated.
Message-Passing:
In the message passing model, the processes communicate with each other through messages.

There are also other options to distribute resources among different processes and help in inter-process communication - Semaphores, signals etc.

What is HTTP?

suvhotta — Mon, 24 Feb 2020 15:25:28 +0000

HTTP is Hyper Text Transfer Protocol which involves protocols involving communication between web servers and clients. It can be thought as the messenger of the web. Client is the one who makes the request(HTTP Message) and the server responds(HTTP Message) to that request.

HTTP is state-less i.e. it doesn't remember anything about the previous page when we go from one webpage to another(Cookies and Sessions are additional stuffs, and aren't part of HTTP core). Each request can be thought of as a single transaction.

HTTP is connection-less i.e. there is no fixed 24*7 connection between the client and the server, rather the connection happens with each request and response cycle. For further requests, there will be more new connections.

A HTTP Message has 3 parts:
Start line, Headers and Body. The message body is optional
The info present in the 3 section varies depending if that is a request or a response.

For Request:

startline has the method name telling the server what to do, URL and the version number.
Headers can be thought as a few key-value pairs like host, accept-language etc.
A message body is the one which carries actual HTTP request data (including form data and uploaded etc.)

For Response:

No method is required in the startline of a response. However it contains the http version and the status code.
A message body may contain HTTP response data from the server ( including files, images etc)

Some methods used in HTTP:

Method is a command instructing the server what to do.Method names are always in uppercase.
GET: Every time we visit a webpage, we make a GET request to the server via HTTP.

POST: When adding some data to the server like - submitting a form,creating a blog post, uploading an image. In a POST request, we send data to the server which is then stored in a database.

PUT: PUT request is used to update some already existing data.

DELETE: DELETE request deletes data from the server.

Each Request and Response has a header and a body.
The Response body is the html or json or whatever is being sent from the server end.

HTTP STATUS CODES:

As the name refers to, these are codes referring to different statuses.
It has a range from 100 to 500.

1XX : Informational
Request received/processing.

2XX : Success
Successfully received, understood and accepted.

3XX : Redirect
Further action/Redirect.

4XX : Client Error
Request doesn't have what it needs.

5XX : Server Error
Server couldn't fulfill some valid request.

How does the Internet Work?

suvhotta — Mon, 24 Feb 2020 11:31:38 +0000

Let's deep dive into a journey of what happens in the backend when a client(user) accesses the internet and how the server renders the data.

Whatever data we fetch over the internet are stored in some server inside a data center. Server can be thought of as a computer whose job is to provide us the content whenever requested.

Now you must be pondering as how this data is then transmitted from the data-centers located at different corners of the world to the users requesting them. One solution is to use Satellite communication, but since the distance involved is high, it would lead to users facing severe latency. So one more solution is to use a wired medium - one of the best wired medium is infact used to move the data from the data-centers to the users, drumrolls 'Optical Fibers'.

Optical fibers are a form of wired media, where in the data is transmitted in the forms of bits(zeros and ones) as light. This complex network of optical fibers can be generalized as the Internet. These optical fibers are scattered and placed throughout the world,covering all kinds of terrains.So from this we can figure out that we as clients are not usually directly connected to the internet, rather indirectly through an Internet service provider, while the datacenters' servers are directly connected to the internet.

How does the optical fibers identify the sender and receiver of the data?

This is done by a bunch of unique number and dot combinations known as IP Address which forms the shipping address in the Internet world. IP Address can be analogous to a home address but the stark contrast being that it is mostly a fixed value and depends on the Internet Service Provider(ISP) or the network where our system is connected. These IP Addresses can be static or dynamic. Most of the IP Addresses assigned to devices of the end users are dynamic in nature. For getting a static IP, some premium is to be paid. If you want your device use the same IP Address both at home as well as when you visit your grandma in the country-side having a different ISP, it would cost you a hefty sum.

Static IP Address is mostly used for dedicated Web,Database or other servers and also for accurate geological tracking purposes.

But now that the problem of identifying the sender receiver has been resolved, we come across a new issue- how do we ip addresses. The solution is pretty straight forward - needn't remember them at all, use domain names like facebook.com,google.com,youtube.com instead, which correspond to IP Addresses.

How does a domain name gets converted into an IP Address?

For this conversion, a phone directory like stuff called DNS is used. The DNS Server provides the ip address corresponding to the domain name that the user enters in the client computer.

Summarizing the process:

User enters domain name in browser -> browser looks up for the ip address corresponding to that domain name in the DNS Server -> this request is then forwarded to the data-center, or rather the respective server in the data-center -> Once the request reaches the server, data starts flowing out of it in the form of light-pulses through optical fibers(in the form of well defined packets) -> The router converts this light signals to EM signals and our laptop/mobile phones receive them.

Python: Logging Overview

suvhotta — Thu, 13 Feb 2020 11:21:59 +0000

The first question popping up with every developer noob is my print function is working perfectly fine, and why the hell do I need a log file?

Well your 50 line code's debugging can be easily managed by the simplest of all functions - print() but when it comes to debugging those 1000 lines of code-work, most of which is a result of many sleepless night efforts, surely you need something more than print() to suffice your needs. Drum-rolls: enter Log Files.

Python has an inbuilt logging module that fulfills all the logging needs of any program.

First Things First:

There are basically 5 log levels in Python associated with numeric values which are incremented in multiples of 10.
Critical(50) - Highest Priority which needs immediate attention.
Error(40) - Minute errors which can be resolved and program can re-run.
Warning(30) - Note of caution like space issues.
Info(20) - Confirmation that something is working as expected.
Debug(10) - Detailed information, mostly used while debugging any issue.

logging.basicConfig():

The function takes a keyword of arguments as parameters like filename, level,format.
The default level in logging is Warning or 30, however that can be changed as passing the level as:
level=logging.DEBUG or level=logging.INFO
The format would specify what format of message do we want to store in the logfile.

The Default logger is the root logger.

However when a new file is imported and it has its own logger, the root logger of the parent file gets replaced by that of the child logger and all the logging info for the parent file gets lost.

This problem can be resolved by creating separate loggers for each file and then configuring them according to our needs.

Creating a Logger:

logger = logging.getLogger(name)

After creating a logger, we need to create a filehandler.
fileHandler = logging.FileHandler('employee.log')

logger.addHandler(fileHandler)

sample formatting can be added to the filehandler and can be set by passing on the formatter parameter to fileHandler.setFormatter()

There are also handlers like StreamHandler,NullHandler etc.