DEV Community: Shaurya Lalwani

Covariance VS Correlation

Shaurya Lalwani — Fri, 24 Jul 2020 16:09:06 +0000

Covariance:

Defines 3 types of relationships: Positive Trend, Negative Trend, and No Relationship
Covariance cannot tell whether the slope of the line representing the mentioned relationship between variables, is steep or not, instead, it will only express whether the slope is positive or negative.
Covariance can change even when the relationship does not, because covariance values are dependent on the scale and that also makes them difficult to interpret.

Correlation:

Correlation shows the strength of the relationship between variables
Correlation is standardized covariance. It does not depend on the scale of the data.
The more data we have, the more confidence we can have in the value of correlation that we obtain with the best fit line/plane

Thanks for reading! Happy learning!
If you'd like to support my writing, you can do that here:
https://www.buymeacoffee.com/shauryalalwani

Assumptions of Linear Regression (Tests to Perform)

Shaurya Lalwani — Wed, 15 Jul 2020 12:27:49 +0000

The tests to be performed to check the assumptions of linear regression:

No Autocorrelation: Durbin Watson Test
No Heteroscedacity: Goldfeld Test, Residual VS Fitted Plot
No Multi-Collinearity: VIF & Correlation Matrix
Normality of Residuals: Jarque Bera Test, Shapiro Test
Linearity of Residuals: Rainbow Test

Happy Learning!

How much time do you spend debugging your code? 🥺

Shaurya Lalwani — Tue, 14 Jul 2020 12:52:41 +0000

This is a common problem all of us might have faced at some point in time. I have personally spent quite a lot of time debugging when I was working on C and C++. When I dove into Python, that debugging reduced to almost nothing!

I'd like to understand how it goes on in other languages (front-end and back-end both). Let's discuss it!

4 Important Parts Of An Analytics Interview

Shaurya Lalwani — Tue, 14 Jul 2020 08:03:52 +0000

Key components of an Analytics Interview:

1. 👶 Introduce Yourself

You have to ace this section. You can speak about anything and everything. From the beginning of your life, until today, whatever you have done, try to explain how it helped you reach your decision of diving into analytics.

2. 👾 🤖 The "What Is Analytics" Section

Data Science, ML Engineer, BI Developer, Product Analyst, etc. have to usually face this part. It revolves around concepts related to data science in general, and this is where they get to understand your profile, and dive deeper into your previous roles, academic achievements, and extra-curricular achievements that are relevant to the role. They can dive deeper into an algorithm of your choice, and/or discuss your most recent project.

3. 🤯 Statistics And Its Applications

This is the make or break section if you're going for a highly technical role. This is where pure statistical concepts are asked to be explained, and they try to understand how you'll apply that knowledge to their business. This is usually where it starts getting highly technical in the interview, and a mix-up of concepts also happens, so make sure to keep track of everything being asked, and answer accordingly.

4. 👨‍🎓 👩‍🎓 Business Case

This is another important section, where your approach needs to be domain-based, i.e. before your interview, you have to thoroughly research the domain in which the company is working, and what are their most productive projects. You have to answer in this section, using your inherent knowledge and the research that you do before the interview.

Growing is a slow process sometimes. Do not get disappointed if you don't crack an interview. You'll always have more chances.

Just keep learning from experiences. And adapt accordingly.

Why I Code. And What I Learned From A Coding Life.

Shaurya Lalwani — Mon, 13 Jul 2020 14:53:00 +0000

This is a story about how I understood that coding is an important part of not only the tech culture but also life.

The Back Story 😎

I was in my tenth grade when I had decided that I'll be pursuing Electronics Engineering. Two years later, I was doing the same. Why? Because I found physics interesting: The Physics which was related to Electronics, such as light, electricity, circuits...you get the point. If you are still in school, I'll explain it to you in an even simpler way. Physics that we study in schools is "mostly" related to Mechanical or Electrical/Electronics Engineering.

Since I liked Physics (the electrical part of it), I decided to go for an Electronics Engineering, rather than a Mechanical Engineering course. It was only towards the end of my second year there, that I realized I was into coding. I wasn't into Electronics anymore, but I loved the course. I had time to build amazing projects, and I did.

What My Learning Was 😄

Meanwhile, my peers in computer science were designing websites and working on databases. I didn't get the chance to explore databases and/or web dev in a lecture, so I went where any kid goes to explore: the internet. And trust me, the internet is a boon to this World. You can learn anything and everything here. While working on my Electronics degree, I started diving into code, first C and C++, the masters of software engineering, and then Python, the new kid in town. 😃

What came to me as a shock was that my Electronics degree was restricting me, from getting a job where I code intensely. And I did want that job, really bad. The logic was flowing out of me, and I didn't want to design circuits anymore. It so happens, that in my country, it was pretty hard to even get a job, if you didn't know how to code. Close save, since I had worked a bit on my skills to learn code.

Why Do I Say Code Is Important? 🤖

Over the years, I worked on personal projects, and kept on learning new languages, because I had understood that code is omnipresent:

Why do you see a fulfillment robot in a warehouse?
Why do you see those automated vending machines?
Why do you see marketing companies targeting the right people these days?
Why do you see robots serving in restaurants these days?

Coding is the answer to all these questions.

To give you an example, consider the fourth point above. The construction and enhancement part of robotics is mostly automated these days. A few years ago, there were a lot of discussions based on airplane construction being automated, which caused a stir in people's minds that jobs will be lost. Some people lost their jobs, while others didn't.

A Deeper Explanation 👀

Recently, AutoML has been introduced in machine learning. Many people think that AutoML can replace data scientists and machine learning engineers. Let me tell you why that is impossible. Data scientists clean, wrangle, and mine the data, to make sense of it first. Then they do important modifications such as feature engineering and feature selection, which are extremely important as they need a human factor for their analysis. This is where AutoML misses out. It can give good results, but what about the explanations that one needs to give to the clients? And what about the data cleaning process, that takes days at times?
So, you get the point.
Coding is irreplaceable.

Now, I've given you a back story into my life as well. My interest lies in data and front-end. So, data is a sub-field of back-end dev, which implies that I am interested in full-stack.

An electronics engineer is playing with data today. The way I can bring back my love for electronics is to dive into computer vision. And I just might do that someday. :)

The Way Forward For Beginners 👨‍🎓 👩‍🎓

Computer Science is a very interesting and vast field, and I'd suggest you take it up and explore it. But, don't do it for the sake of it. It's okay if you want to be a mechanical, or electrical, or biomedical engineer; or a statistician, a product manager, a financial analyst, etc. What I'm trying to tell you, is that eventually, you might code, and there's a big probability hanging around that.

So, while pursuing your course, keep trying out new languages, to understand what interests you, and what keeps your mind fresh and active. Ten years down the line, you want to be working on something that keeps you active, and interested.

This is a personal opinion. Let me know yours as well! 😁

Thank you for reading!
If you'd like to support my writing, you can do so by getting me some coffee here:
https://www.buymeacoffee.com/shauryalalwani

What are the questions to be asked in a Linear Regression problem?

Shaurya Lalwani — Mon, 13 Jul 2020 12:37:57 +0000

Hello All!

Linear Regression is the key to understanding regression in machine learning. It is the algorithm where everything is born, and everything ends. It may not be as powerful as the Boosting algorithms are, but regression is the way forward since it has great interpretability.

While performing regression, two important metrics have to be noted and kept handy, for understanding the importance of variables, and how they affect the target:

How important is the predictor, for predicting the Target?
(R-squared will denote this)
Was this predictor's importance by chance?
(P-value obtained on running the algorithm, will denote this)

Thanks for reading! Let's discuss it!

What is Stacked Generalization in ML?

Shaurya Lalwani — Mon, 13 Jul 2020 05:37:11 +0000

The way to test out classification algorithms is generally by looking at the AUC-ROC curve, and by understanding the Precision and Recall. For assumption, think that we are dealing with a decision tree. This decision tree is built on all of the data and can have various hyper-parameters.

One way to achieve a better model, or to obtain required predictions, is to tune the tree. There is a method to fine-tune this technique to another level, i.e. make lots of decision trees (based on different hyper-parameters), and then add predictions from all these trees, and average them out. This is the random forest algorithm.

An important thing to note here is that it depends on the data at times, which algorithm will perform better on it. What if our data doesn't perform well on a decision tree?
In that case, we need to look at other algorithms, and if possible, an even better method would be to combine the best instances of all possible algorithms.

As described above, the random forest algorithm is a homogeneous ensemble, meaning that it is an average prediction of various instances of the same algorithm.
Now, we come to a heterogeneous ensemble. A heterogeneous ensemble is an average prediction based on outcomes of different algorithms applied to different samples of the data. This is widely used in the industry, as it removes the possibility of bias to a very high extent.

Now, the topic of the day: Stacked Generalization, is a method that comes under advanced machine learning, because this a step further from heterogeneous assembling. After performing the heterogeneous ensemble, the predicted values which we obtain, are used to build the model using a Neural Net, etc. such that a model based on predicted values is built, at a point where predicted values are representative of many algorithms (ensemble).

Happy learning!

Learn Python Based Data Science

Shaurya Lalwani — Sat, 11 Jul 2020 12:29:32 +0000

Hello All!

I have been learning data science and machine learning for quite a while now, and I wanted to share this:
There is a curve that needs to be followed while learning data science. I have seen many people learning through online resources, and that is great! My experience with online resources was such that I had been unable to find a structured method to learn data science and machine learning in a flow. For this reason, I decided to create notes, for anyone who wanted to learn/revise what they have worked on or is planning to work on, and I uploaded these notes in a Day-wise format, to my GitHub repository.

I chose a day-wise format, so that people who want to learn/revise Python, EDA, and ML Algorithms, can easily spend about a day on a particular uploaded notebook, and can grasp knowledge in a day-wise manner, so everything stays fresh and the learning is in a step-wise manner.

My repository is based on the following:

Basic Python
Advanced Python
Exploratory Data Analysis
Machine Learning Algorithms (Supervised And Unsupervised)

Here's the link: https://github.com/shauryaa117/Learn-Python-Based-Data-Science

Happy learning! Stay safe!