DEV Community: Arka Bhowmik

API-Log Management | ETL -Vector | Searching and Debugging in Production :

Arka Bhowmik — Wed, 03 Mar 2021 06:46:31 +0000

Liquid syntax error: 'raw' tag was never closed

HyperCubed !! Easy State Management simplifying 2^n conditions

Arka Bhowmik — Wed, 24 Jun 2020 09:13:42 +0000

Do you have backend code looking like this ? or something similar?

if condition1:
    if condition1.1:
        do this
        if condition1.1.1:
            do that
        if condition1.1.2:
            do this
            if condition 1.1.2.1:
                do that
    else if condtion2:
        do that
    else:
        do that

This arises when you have unplanned state management arising from unstructured conditions. Having too many nested conditions will affect readability and mantainability of your code. So lets simplify this. Lets have no branching at all .

NOTE:

The approach followed below assumes that the states we are working with are somewhat related to one another.
Also, this approach is only useful for cases where the nested branching level is >=3
The approach focuses on identifying the core components of a condition and minimize redundant checks and transitions

The simplicity of Bit Logic:

We formulate each condition as a state. Let's try with an example where you have 8 different outcomes of your problem which means that there are 8 different output states. Let's number them from State0, State1, State2 to State7.
So the binary representation of every state, based on its index will be from [000 to 111]. For our simple example these states are the human states of mind:
[HAPPY ,IRRITATED ,BUSY ,ANGRY, LONELY, BAD, SAD ,EVIL]
Based on our mood, we will perform some action. But do we need 8 branching conditions for this ? Also, what happens if our mental state changes to a different mood, we write new code for that ? NEVER AGAIN !!!!

Finding a relation between the states by Filtering CORE conditions/states:

Our motive is simple, we need to plan our states in such a way that every bit corresponding to the state's index, has some information associated with it. For this we map all states in a visualization space and see the interdependencies the states have between eachother. The interdependencies will be such that if one state leads to another state by changing a core condition/state, we can say that they are separated by one bit or 1 hamming distance.
Lets clear this up by looking at the diagram of the hypercube with 3 dimensions where we have mapped these states in an open space, with arrows denoting state transitions/ dependencies.

( number of dimension in hypercube = number of bits = number of CORE conditional elements in your state)

Now imagine

For simplicity of understanding, we work with the human emotional state, where one mood can be altered by another mood state . Lets assign the bits at each of the 3 positions from MSB to LSB to denote the states :

IRRITATED = 1 LSB set for irritated (001)
BUSY = 2 MiddleSB set for busy. (010)
LONELY = 4 MSB set for lonely. (100)

these are our core conditional elements in our state transition diagram whose occurence can alter any state. This basically means that between all possible states. All different combinations of these 3 core states, will give us all the different states we have defined below:

HAPPY = 0          # 000 NO BAD FLAGS SET [ INITIAL STATE]
IRRITATED = 1      # 001 [CORE STATE LSB]
BUSY = 2           # 010 [CORE STATE mid-bit]
ANGRY = 3          # 011 BUSY   | IRRITATED
LONELY = 4         # 100 [CORE STATE MSB]
BAD = 5            # 101 LONELY | IRRITATED
SAD = 6            # 110 LONELY | BUSY
EVIL = 7           # 111 SAD    | IRRITATED

And we have a planned dependency between all our states !. We can now map what we intend to do when we are in that state, by mapping a function to that state. Check out the code snippet below. Each index [i] in the to_do list defines what the person wants to do when he in the [i]th state . If your states are not of 2^n , consider having a hashmap with :
{STATE_NAME: corr_function_name}

# DEMO HYPERCUBE STYLE PROGRAM DESIGN
class Moods:

    # seggregating basic states which can alter mood state logically
    #IRRITATED = 1      LSB  set for irritated
    #BUSY = 2           MiddleSB set for busy
    #LONELY = 4         MSB set for lonely

    HAPPY = 0          # 000 NO BAD FLAGS SET
    IRRITATED = 1      # 001
    BUSY = 2           # 010
    ANGRY = 3          # 011 BUSY   | IRRITATED
    LONELY = 4         # 100
    BAD = 5            # 101 LONELY | IRRITATED
    SAD = 6            # 110 LONELY | BUSY
    EVIL = 7           # 111 SAD  | IRRITATED

    # a list or a dict mapped corresponding to  above states. can be a list of function-pointers,
    # responses, emails and what not , based on your functional requirements!
    things_to_do = ["I'll go on a trip", "I need to break a vase or something" ,
     "I'll post an article on dev.to", "I'll eat an entire chocolate cake ",
      "I'll go to sleep", "I'll jump on my best friend",
       "I'll cry in a corner", "I kill some people in GTA V"]

    def __init__(self):
        self.mood = Moods.HAPPY

    def get_action(self):
        # performs a function based on current state. (customize this as required)
        return self.things_to_do[self.mood]

    def alter_state(self, added_mood):
        # alters current state based on external state or intervention
        # you can XOR it for neutralizing current state too, if you want instead
        self.mood = self.mood | added_mood

Since we know our state dependencies, lets run our code now and try to change our state to another state without having to touch another branching condtion.
Lets make Timm'ys mood BAD.

timmy = Moods()
print timmy.get_action()
timmy.alter_state(Moods.BAD)
print timmy.get_action()

output:

I'll go on a trip
I'll jump on my best friend

Looks like Timmy responded to his mood swings ! That too without getting our hands dirty.

Some possible use-cases:

API for Sending emails to people based on -> Their ROLE + ACTION_TYPE + SENDER
API responses
State based games

Some related articles for better coding approaches in API programming:

Avoiding Boilerplates

Also I came across this amazing video by Computerphile ! which helped me coin a Terminology for this programming style of mine:

The Perfect Code - Computerphile

SOLID design pattern principles in OOPS

FUNCTIONAL DESIGN PATTERNS

Zero Boilerplate , Zero Runtime errors : Coding with Monads

Arka Bhowmik — Sat, 13 Jun 2020 08:44:55 +0000

Note: this post will be updated with time

API backend programming can be stressful and repetitive. For the past one year, having refactored and rewritten over 10k lines of backend code, I understood the importance of following a proper structure/ discipline while coding backend APIs.
Thankfully, the concept of Monadic programming already existed in languages like Haskell and AngularJS.

This post will show you a template to implement a coding discipline like monads in Python ,which if followed will let you have zero runtime issues and minimal to no boilerplate in your code. For me, this helped me reduce my backend code volume to 1/3rd of its size, enhancing logging and stability, with zero performance latencies added.

Create the monad class
Monads are pure functions. This implicitly means that the type of input it takes in and type of output it produces will be the same. In this case, that state is a dict{}. This also means, that you cannot have exceptions or None type returns in your functions.

Our Monad class is defined to have :
state: This is the current state we have in our monad which represents the data we have currently processed. This is also the argument which we pass to the functions in out monads. We create a mondaic class which takes in a dict as the initial state. For API backend code, this state can be your request parameters. Think of it as if your'e passing the same variables in the function, just that they are wrapped in a dict and the variables names are the keys in the dict.

fns: It has a list of functions, that are function pointers to functions to be executed in order. for example, you can design your sign_up flow as:
[process_request_params, check_username_password, create_token, establish_session]

The .execute will call these functions in a loop till till the final state is reached. The final state should effectively be your response output. Here's what the flow looks like from one state to another state

efn: suppose your code breaks in-between function in the pipeline. The error function efn will process the current state of the monad and output the error as you require in the response. This saves you from getting 500 server errors from backend.

Our monadic code relies on the "status" parameter in status dict, to get "error" reports in-between functions.

class MaybeErrorMonad(object):

    def __init__(self, state, fns, efn):

        self.state = state  # current state (dict) in monadic pipeline
        self.fns = fns      # a list of functions in order of execution
        self.error_fn = efn # error function that reports any error, based on self.state
        self.finish = False # a flag to indicate that the self.state is final state

    def value(self):
        return self.state

    def execute(self):
        # a wrapper class that does execution of functions on current self.state in order
        def _fun():
            if self.finish:
                return self.state
            computation = self.state
            for i in self.fns:
                computation = i(computation)
                if computation.get("status") == "error":
                    return self.error_fn(computation)
            self.state = computation
            self.finish = True
            return self.state

        return _fun()

Next, we define some functions for testing our monad. I have added an error_handler_wrapper wrapper function which will ensure that our functions always return a proper response in a dict , if anything breaks mid-way. You can design this function to include logs with the function name and arguments, with current parameters for better debugging in production code. We also define our custom error function to output our error response , when we get error status midway.

def err_handler_wrapper(fun):
    # a error handler wrapper function to report any execution errors in dict
    def wrapper(*args):
        try:
            state = fun(*args)
        except:
            return {"status": "error", "reason": "Girlfriend not found"}
        return state
    return wrapper

def efn(state):
    return {"status": "Rejected :'(", "reason":state.get('reason') }

@err_handler_wrapper
def get_flowers(state):
    print "got flowers for %s"%state['girlfriend']
    state.update({"flowers": True})
    return state

@err_handler_wrapper
def get_chocolates(state):
    print "got chocolates for %s"%state['girlfriend']
    state.update({"chocolates": True})
    return state

@err_handler_wrapper
def propose(state):
    if state.get('flowers') and state.get('chocolates'):
        return {"status": "Success! she said yes"}
    return {"status": "she said no"}

Lets run our example where a guy tries to propose his crush.
if he buys just flowers, he will get rejected.
if he buys chocolates and flowers, he gets accepted.
if there is no 'girlfriend' in initial state, theres an error.

# CASE 1
to_do = [get_flowers ,get_chocolates, propose]
initial_state = {'girlfriend': 'Alice'}
executor = MaybeErrorMonad(initial_state, to_do, efn)
result = executor.execute()
print "CASE 1: %s \n"%str(result)

 #CASE 2
initial_state = {} # no girlfriend
executor = MaybeErrorMonad(initial_state, to_do, efn)
result = executor.execute()
print "CASE 2: %s \n"%str(result)

# CASE 3
to_do = [get_flowers, propose] # rejection beacuse of not enough efforts :P
initial_state = {'girlfriend': 'Alice'}
executor = MaybeErrorMonad(initial_state, to_do, efn)
result = executor.execute()
print "CASE 3: %s \n"%str(result)

And ...
Here's our output as expected.

This was a fairly simple example. Imagine the power of now being able to execute huge functions in just a simple sequential todo list with minimal branching. TRY IT NOW !!!

In the next update, we will learn how to structure our dict states and how to handle exceptions , and get parameters from proper context of execution.

For more ideas of elimination boilerplate, read this:
Hypercubed!!
Eliminate nested loops

Read more about monads here:-
Monadic Programming
Monads In python: PyMonad

Handy Linux hacks for Freshers at Work

Arka Bhowmik — Sun, 09 Feb 2020 11:01:00 +0000

Are you a freshser feeling lost in the world of servers, devboxes, debugging realtime systems etc?. We've been sailing in the same boat. Here are a few handy stuff that will be your lifesaver till you are a fresher.
We can all do a man now and understand what the command actually does, but here's a cheatsheet to get you started quick.

Returning complete Exceptions/Trace from logs using grep
You can use the -B and -A to print lines before and after the match.
grep -i -A 10 'some.java.Exception' my-server.log
This will print 10 lines after the Exception, including the line with exception. similarly -A will print lines before the exception

Print all files , with ports, parent processes etc
list the files/processes: sudo lsof -i -P -n
[For Debugging] LISTENING to: sudo lsof -i -P -n | grep LISTEN

Want more detail?: ps aux

a:- This option prints the running processes from all users.
u:- This option shows user or owner column in output.
x:- This option prints the processes those have not been executed from the terminal.
When you know the PID of the process, you can use kill PID_OF_THAT_PROCESS to kill it

Tracking/ Debugging Network connections
netstat pntl

Digging deeper into networking | fixing 502s, 504s
Did you get a Gateway Timeout or a Bad Gateway response? Do you know if the packet is even reaching that server ? Try to do a tcpdump to see what packets are going in and coming into your system.
tcpdump -nnvvS src 172.31.1.237 and dst port 80
this command will track all packets coming to port 80 of your system from 172.31.1.237. If we do the same on 172.31.1.237 and use our system's public IP on that device, we can see the packets arriving to that system from our server.

If you want to analyse network dum on private cloud cluster you can generate a pcap file and analyse it on Wireshark

`sudo ip -s -s link show

sudo nstat `

Use curl instead of POSTMAN for request/response debugging
curl -k -i -X POST your_url_of_concern.com -d"{your data in json or urlencoded text}" -H -H 'set cookie: some_cookie_expected_in_request=someid'

The -k is for avoiding checks for ssl, which might be unnecessary inside your own dev-box or organization's internal APIs.

the -i option will print the headers, cookies information, which is highly useful while debugging server failures, session tracking

replace edit delete occurence of text in file
this one replaces unix with linux in geekfile.txt
sed 's/unix/linux/' geekfile.txt

Allowing an app through firewall
First list all your apps requiring the feature
sudo ufw app list
Now allow the app through firewall
sudo ufw allow 'APP_NAME'

Transferring files to-from remote host over SCTP
This can come in handy when you want to get some downloaded or config files from/to servers to run some stuff locally

without keys: scp -r source_host_full_dir_path destination_host_full_dir_path

In this example, we transfer s.py from local to 172.31.1.237 remote machine

if using ssh key: scp -i your_ssh_key_path -r source_host_full_dir_path destination_host_full_dir_path
The -r is for including subfolders and files inside that folder. you wont need that for single files

If ever you get permission denied during transfer to or from a server, go to the source's directory and change ownership of the file to the current logged in user

Trouble sending certificates in json body ?*
certificates are in PEM format and line breaks are a hassle to pass through in json format. If you have to pass the plaintext of a cert in json, better convert those linebreaks into '\n'

awk 'NF {sub(/\r/, ""); printf "%s\\n",$0;}' your-cert.pem

Shortcut links to long directory paths
Do you have very long directorypaths, which you frequently need to access from current directory? why not have a shortcut to that file in your current directory. A symlink will create a pointer to that actual file from the current directory. All changes will be reflected on the original link.
ln sourcefilepath destinationfilepath

Executing commands directly in remote box
example : ssh -t username@host 'pwd'
*The -t is required if you are executing multiple commands or if you want the shell to be interactive. Not required if you are not doing any of that. However if you want to execute a shell script from your local to your remote machine , you have to scp file transfer the script to that machine and then execute it. credits to this *link.

Follwing up .logs using tail
tail -10f /location/of/your/.log

the "10" denotes that it will tail/print the last 10 lines, in that file. You can set it to whatever value you want. This will update in real-time as your log file updates.
You can use *.log instead of specific filename to track all log files in that directory

Setting up ENVIRONMENT VARIABLES :
export variablename=dont_add_spaces_here_whateveryouwant

use printenv variablename to print them

Comparing Floats in a shell script
Yes, it's a pain to compare floating point numbers in shell. This is not an issue in bash though ( yes bash and sh are different, in case you didn't know that). Here is an example to do so

num1=18.04 num2=18.04 OUTPUT=$(echo "$num1 == $num2" |bc -l)
will produce output 1 since they are equal.
Now, to use this OUTPUT, you again have to compare it using
[ $OUTPUT -ne 0]
this checks if the value of OUTPUT is not equal to zero.

Debugging | Logging stdouts from shell scripts
There might be occassions where you have to debug stuff inside a shell script, but there is no elegant way of logging inside of shell ... or is it ?

We can dynamically open, close, and copy stdout to achieve logging operations. Let’s redirect stdout (FD 1) to the log file:
#! /bin/bash script_log="/tmp/log_date +%F.log" exec 1>>$script_log echo "This will be written into log file rather than terminal.." echo "This too.."
Learn more

commits between 2 git hashes for better gitblame

git log <older commit hash>..<newer commit hash> --oneline | cut -d " " -f 1

easy git update between branches without rebase
Is rebasing causing conflicts for you even though you know your diff is perfectly fine and conflicts have already been resolved in upcoming commits ? In this case forget rebasing and do patching. take a diff between your current branch and the branch you wanna update (say master)

git diff origin/master --no-color > ~/Documents/your_patch_file.patch
now checkout to a fresh master branch and
git apply ~/Documents/your_patch_file.patch

you will have all the changes of your current branch, in this one now, without any conflicts.

Local git branch messed up? fix to latest working remote branch
sudo git clean -f -d; sudo git reset HEAD —hard

remove untracked files from git commits
git status | grep 'modified:' | cut -d' ' -f4|xargs sudo git rm -f

if you want to untrack deleted files too:
git status | grep 'deleted:' | cut -d' ' -f5|xargs sudo git rm -f

that cut -d' ' -f_number means you are splitting the current line by spaces and picking up the string corresponding to your file. so that f_number should be the position your file occurs in. It might vary in your case

clean unused old libraries from .m2 repository
cleans all files more than an hour of last access from m2. They will be accessed if you had built within the past hour

find ~/.m2 -amin +60 -iname '*.pom' | while read pom; do parent=`dirname "$pom"`; rm -Rf "$parent"; done

Using anaconda? Do you clear the unused files?
if you are thinking where all my disk space is going, go and check your anaconda package directory. Mine was 17.6GB. its crazy. Anaconda does not automatically clean files after installation.
conda clean -a will remove all tarballs, and setup packages of already installed packages. My directory size is now 10GB.

A few stuff for precautions:
Here are a few things to avoid when you are using sudo.

1.NEVER sudo in the root folder. This will deadlock your system.
2.To avoid using sudo in every place else, all the time, use sudo -s. and from there onwards, you need not type sudo again.
3.To avoid sudo in some situations,( especially when you are just testing some stuff), maybe consider changing ownership of files in that folder to your name so you can liberally access them.

chown -R your_username /foldername will change ownership of all files subfolders, in that directory to you

Learning regex and VIM
This will save you hours of typing time and your reputation for faster prod-issue debugging, once you get used to it. I recommend the below places to learn these skills in the fastest way possible:
Regexone
thoughtbot

Working with JAVA ? Read about tweaking JVM with these
Java Options