DEV Community: Saoni Deb

A decade in computer science (2014 - 2024)- A brief review.

Saoni Deb — Fri, 10 Jan 2025 23:09:40 +0000

When I first started leet-coding(2017) there were just 600 questions in total, today there are over 1800 questions in just arrays section!
The first level OA today for Amazon is much above the level of questions I was asked in the last round of interviews in Amazon 2016!
Earlier ML positions were filled based on projects that you worked on, now ML positions require DSA, System Design, Statistics, ML Basics, LLM knowledge, Security overview, DevOps and of-course ML specific hardware knowledge.
Networking can help you skip interview rounds(one of my friends just had 1 round of interview)!
Everyone has a different journey, so be patient, but keep moving because time does the same!
Take advise from people who are on the same road-map, that keeps you sane and motivated.

So please remember, as days go, everything is going to be tougher, and the longer you are paying attention to things that don't work for you, the farther you are from where you want to be.

Keep learning things, Move in the Right direction, Because Time and Tech waits for none!

Why do you need to learn and understand Algorithms?

Saoni Deb — Thu, 25 Apr 2024 08:11:47 +0000

Learning is a bit daunting when books don't explain why are you learning this particular thing.
In my early years I had been fortunate to have teachers who actually took out time to feed that curiosity, and explain why we were learning anything.
Be it calculus (which we use in reality to calculate how infinitesimal small force--additive or otherwise, can impact anything) or statistic which we need to visualize details about customers in real life.
If you can visualize one real time use case, learning becomes easier, it makes us feel that us learning something would actually help us contribute towards something in future.
Let's move towards algorithms now, specifically arrays today.
We know arrays are used to store contiguous memory but why do need the related algorithms?
Few of the must needed algorithms that you must have come across are --

Kadane's algorithm
Linear Search
Binary Search
Dutch National Flag
Prefix and Suffix sum
Sliding Window
Two Pointer Technique
Precomputation Covering corner cases, and the other types of arrays like sparse, jagged, 2D, frequency, etc are also needed. in our day to day solutions. We will start with where Kadane's algorithm is used in real life - its applications include buying and selling stocks, in genomic sequences analysis, as well as image processing etc. The intuition behind it is to find the maximum sum of a contiguous subarray in an array. It gives us an optimal solution in O(n) therefore it is one of the most efficient algorithms. Below are few of the questions that can be solved using Kadane's algorithm :

The Era of Big Companies

Saoni Deb — Fri, 20 Jan 2023 05:20:04 +0000

It seems that the Era of big companies has finally ended this year.
With Dream companies like Microsoft, Google & Amazon starting mass layoffs and sources mentioning some of the impending targeted layoffs, we can no longer feel that even attempting to switch right now to these companies will be a good move.
Coming back to start-ups or product that are just being built seem more stable than these companies now-a-days and would perhaps be a better choice to move to.
What do you think?

Create a self signed Certificate

Saoni Deb — Thu, 13 Oct 2022 09:12:57 +0000

Below is the link for creating a self signed certificate online:

Generate SAML Self-Signed X.509 Certificates - Create Self Signed Certs | SAMLTool.com

Generate Self-Signed X.509 Certificate. CSR + Public Certificate + private key

samltool.com

Minio Server Deployment

Saoni Deb — Wed, 12 Oct 2022 10:58:20 +0000

Steps to Deploy your minio server:

1> wget https://dl.min.io/server/minio/release/linux-amd64/archive/minio-20221008201100.0.0.x86_64.rpm -O minio.rpm sudo dnf install minio.rpm

2>mkdir ~/minio
minio server ~/minio --console-address :9090

3>Following output will be displayed:

API: http://192.0.2.10:9000  http://127.0.0.1:9000
RootUser: minioadmin
RootPass: minioadmin

Console: http://192.0.2.10:9090 http://127.0.0.1:9090
RootUser: minioadmin
RootPass: minioadmin

Command-line: https://min.io/docs/minio/linux/reference/minio-mc.html
   $ mc alias set myminio http://192.0.2.10:9000 minioadmin minioadmin

Documentation: https://min.io/docs/minio/linux/index.html

WARNING: Detected default credentials 'minioadmin:minioadmin', we recommend that you change these values with 'MINIO_ROOT_USER' and 'MINIO_ROOT_PASSWORD' environment variables.

4> Open http://127.0.0.1:9000 in a web browser to access the MinIO Console.

5> Install the MinIO Client
wget https://dl.min.io/client/mc/release/linux-amd64/mc chmod +x mc sudo mv mc /usr/local/bin/mc
mc alias set local http://127.0.0.1:9000 minioadmin minioadmin mc admin info local

Kats: Forecasting

Saoni Deb — Wed, 18 May 2022 20:23:19 +0000

Kats is a toolkit to analyze time series data, a lightweight, easy-to-use, and generalizable framework to perform time series analysis. Time series analysis is an essential component of Data Science and Engineering work at industry, from understanding the key statistics and characteristics, detecting regressions and anomalies, to forecasting future trends. Kats aims to provide the one-stop shop for time series analysis, including detection, forecasting, feature extraction/embedding, multivariate analysis, etc. Kats is released by Facebook's Infrastructure Data Science team. It is available for download on PyPI.

Here's, the sample example notebook with metrics expanded for better understanding:

https://colab.research.google.com/drive/1kElLOB0KArrU8hzpHh-bu_VG17yFWrhA?usp=sharing

Linear Regression With Time Series

Saoni Deb — Tue, 17 May 2022 08:33:32 +0000

Documentation: https://www.kaggle.com/code/ryanholbrook/linear-regression-with-time-series

This post aspires to show the application of modern machine learning methods to time series data with the goal of producing the most accurate predictions.

Once we check these steps, we'll know how to:

engineer features to model the major time series components (trends, seasons, and cycles),
visualize time series with many kinds of time series plots,
create forecasting hybrids that combine the strengths of complementary models, and
adapt machine learning methods to a variety of forecasting tasks.

What is a Time Series?
The basic object of forecasting is the time series, which is a set of observations recorded over time. In forecasting applications, the observations are typically recorded with a regular frequency, like daily or monthly.

This series records the number of hardcover book sales at a retail store over 30 days. Notice that we have a single column of observations Hardcover with a time index Date.

Linear Regression with Time Series
Linear regression adapts naturally to even complex forecasting tasks. The linear regression algorithm learns how to make a weighted sum from its input features.
For two features, we would have:

target = weight_1 * feature_1 + weight_2 * feature_2 + bias

During training, the regression algorithm learns values for the parameters weight_1, weight_2, and bias that best fit the target. (This algorithm is often called ordinary least squares since it chooses values that minimize the squared error between the target and the predictions.) The weights are also called regression coefficients and the bias is also called the intercept because it tells you where the graph of this function crosses the y-axis.

Time-step features
There are two kinds of features unique to time series: time-step features and lag features.

Time-step features are features we can derive directly from the time index. The most basic time-step feature is the time dummy, which counts off time steps in the series from beginning to end.

Linear regression with the time dummy produces the model:

target = weight * time + bias
The time dummy then lets us fit curves to time series in a time plot, where Time forms the x-axis.

Time-step features let you model time dependence.
A series is time dependent if its values can be predicted from the time they occured.
In the Hardcover Sales series, we can predict that sales later in the month are generally higher than sales earlier in the month.

Lag features
To make a lag feature we shift the observations of the target series so that they appear to have occured later in time. Here we've created a 1-step lag feature, though shifting by multiple steps is possible too.

Linear regression with a lag feature produces the model:

target = weight * lag + bias
So lag features let us fit curves to lag plots where each observation in a series is plotted against the previous observation.

You can see from the lag plot that sales on one day (Hardcover) are correlated with sales from the previous day (Lag_1). When you see a relationship like this, you know a lag feature will be useful.

More generally, lag features let you model serial dependence.
A time series has serial dependence when an observation can be predicted from previous observations. In Hardcover Sales, we can predict that high sales on one day usually mean high sales the next day.

Adapting machine learning algorithms to time series problems is largely about feature engineering with the time index and lags. For most of the course, we use linear regression for its simplicity, but these features will be useful whichever algorithm you choose for your forecasting task.

Tunnel Traffic is a time series describing the number of vehicles traveling through the Baregg Tunnel in Switzerland each day from November 2003 to November 2005. In this example, we'll get some practice applying linear regression to time-step features and lag features.

Notebook 1: https://www.kaggle.com/code/saonideb/linear-regression-with-time-series-17-05-22/

FastAPI

Saoni Deb — Wed, 06 Apr 2022 09:46:46 +0000

Documentation: https://fastapi.tiangolo.com

Source Code: https://github.com/tiangolo/fastapi

FastAPI is a modern, fast (high-performance), web framework for building APIs with Python 3.6+ based on standard Python type hints.

Key Features:

Fast
Fewer bugs
Intuitive
Easy
Short
Robust
Standards-based

Python Type Code below:

from datetime import date

from pydantic import BaseModel

# Declare a variable as a str
# and get editor support inside the function
def main(user_id: str):
    return user_id


# A Pydantic model
class User(BaseModel):
    id: int
    name: str
    joined: date

This can be used as:

my_user: User = User(id=3, name="John Doe", joined="2018-07-19")

second_user_data = {
    "id": 4,
    "name": "Mary",
    "joined": "2018-11-30",
}

my_second_user: User = User(**second_user_data)

Validation Types:

Validation for most (or all?) Python data types, including:

JSON objects (dict).
JSON array (list) defining item types.
String (str) fields, defining min and max lengths.
Numbers (int, float) with min and max values, etc.

Validation for more exotic types, like:

URL.
Email.
UUID.
...and others.

Asynchronous Code
Asynchronous code means that the language has a way to tell the computer / program that at some point in the code, it will have to wait for something else to finish somewhere else.

That "wait for something else" normally refers to I/O operations that are relatively "slow":

the data from the client to be sent through the network
the data sent by your program to be received by the client through the network
the contents of a file in the disk to be read by the system and given to your program
the contents your program gave to the system to be written to disk
a remote API operation
a database operation to finish
a database query to return the results

As the execution time is consumed mostly by waiting for I/O operations, they call them "I/O bound" operations.

It's called "asynchronous" because the computer / program doesn't have to be "synchronized" with the slow task, waiting for the exact moment that the task finishes, while doing nothing, to be able to take the task result and continue the work.

For "synchronous" (contrary to "asynchronous") they commonly also use the term "sequential", because the computer / program follows all the steps in sequence before switching to a different task, even if those steps involve waiting.

Difference between Concurrent & Parallelism -
This has been explained quite well using the burger example here:
https://fastapi.tiangolo.com/async/

JWT means "JSON Web Tokens"

Uvicorn is an ASGI (Asynchronous Server Gateway Interface) compatible server that will be used for standing up the backend API.

Steps to create a virtual environment in Ananconda

Saoni Deb — Wed, 23 Mar 2022 05:15:04 +0000

Create and activate virtual environment:

conda -V
conda update conda
conda create -n <venvname> python=<x.x> anaconda
conda activate <venvname>
conda deactivate

Install package in your venv:

conda install -n <venvname> <package>

Recursive Feature Elimination

Saoni Deb — Wed, 16 Mar 2022 10:17:34 +0000

Recursive Feature Elimination or RFE is primarily used for Feature ranking.

Given an external estimator that assigns weights to features (e.g., the coefficients of a linear model), the goal of recursive feature elimination (RFE) is to select features by recursively considering smaller and smaller sets of features.
First, the estimator is trained on the initial set of features and the importance of each feature is obtained either through any specific attribute or callable.
Then, the least important features are pruned from current set of features.
That procedure is recursively repeated on the pruned set until the desired number of features to select is eventually reached.

RFECV performs RFE in a cross-validation loop to find the optimal number of features.

First lets get to know how many types of feature selection is provided in sklearn.
The classes in the sklearn.feature_selection module can be used for feature selection/dimensionality reduction on sample sets, either to improve estimators’ accuracy scores or to boost their performance on very high-dimensional datasets.

Removing features with low variance_(Not applicable for dynamic data)_: >VarianceThreshold is a simple baseline approach to feature selection. >It removes all features whose variance doesn’t meet some threshold.
Univariate feature selection_(Not applicable for multivariate data): >Univariate feature selection works by selecting the best features based on univariate statistical tests. > It can be seen as a preprocessing step to an estimator. > _SelectKBest removes all but the highest scoring features > SelectPercentile removes all but a user-specified highest scoring percentage of features using common univariate statistical tests for each feature: false positive rate SelectFpr, false discovery rate SelectFdr, or family wise error SelectFwe. > GenericUnivariateSelect allows to perform univariate feature selection with a configurable strategy. This allows to select the best univariate selection strategy with hyper-parameter search estimator.
Recursive feature elimination > Recursive feature elimination with cross-validation is also available
Feature selection using SelectFromModel
Sequential Feature Selection
Feature selection as part of a pipeline

Prometheus

Saoni Deb — Fri, 21 Jan 2022 05:37:03 +0000

A few links to go through:
1 Red Hat's View
2 Why is monitoring important
3 Prometheus & Grafana
4 Journey
5 On Tensorflow
6 Anomaly Detection
7 Site Link
8 Demo

Mouse Auto Click Script

Saoni Deb — Fri, 21 Jan 2022 04:54:36 +0000

Last year I faced this Google Colab issue wherein during my model training time(which took over 6 hours), my notebook kept disconnecting due to inactivity.
Wrote this small script to keep my session and in turn my system active.

from pynput.mouse import Button, Controller
import time

mouse = Controller()

while True:
    mouse.click(Button.left, 1)
    time.sleep(30)