DEV Community: Machine Learning Tech Stories

What are the various machine learning usecases for companies/enterprises ?

Machine Learning Tech Stories — Thu, 24 Jun 2021 04:44:05 +0000

What are the machine learning usecases which companies/ enterprise focus on ?

(This is a placeholder for my exploration in the context of the machine learning problems solved by various companies/ enterprises, for research purposes.)

Aside:
Any user facing website/webapp/mobile app / edge app could perform data collection of the users. This is in the context of data collection.
Since there are limitations with amount of data, for certain usecases, people have also come up with data synthesis, where you can create data. Image synthesis/ generation or text synthesis.

As this is a collection, a template would be helpful to follow, for standard format.

Company
Datasets which the company might possess
Machine Learning problems/usecases

Data - is primarily in the form of text, images, videos ( sequence of images ), audio ( speech )

Google

Datasets:

user data,
web pages accessible across internet,
images in webpages,
text content in webpages,
any video uploaded in youtube,
any video practically accessible in the internet (which has allowed crawling)

Machine learning problems/usecases:

google meet - how to blur the background, inorder to focus on the subject in the video call ?
google meet - how to reduce background noise, inorder to focus on the speech of the subject ?
youtube - how to recommend a particular video among a collection of videos, which is potentially watchable by that user, for a long period of time ?
google search - which webpage or set of webpages matches the user query, in the closest possible way ?
google search - what is the user trying to search for ?

CRED

Datasets:

user data
user credit history
user purchase patterns
startups to be showcased in CRED catalogue
user investment history

Machine learning problems/usecases:

product recommendations - What product/ set of products could be recommended to a given user, which could potentially be converted to a purchase ?
credit worthiness - What is the credit worthiness score for a user , based on various parameters ?
purchase predictions - What is the likelihood of a product being purchased by a user in the future based on purchase history ?

Paypal

Tesla

Microsoft

Amazon

Facebook

NetFlix

Learning continues...

What does a machine learning toolbox consist of ?

Machine Learning Tech Stories — Thu, 24 Jun 2021 04:11:18 +0000

(This is a placeholder for my exploration of various tools used in the machine learning ecosystem)

As we consider machine learning projects, from end to end fashion, following are some of the phases and based on each phase, we have different toolsets which enable the engineer to perform the data processing in each phase.

Phases in Machine Learning project

Data ingestion
Data exploration
Data analysis
Data visualization
Model training
Model testing
Model deployment
Model serving
Model logging/healthchecks etc

Tools used in each phase

(unordered by its usage)

Python
numpy
Tensorflow
Apache Spark
pandas
R
SAS
jupyter notebooks
kubeflow

Learning continues...

What does it take to gain expertise in the field of Machine Learning ?

Machine Learning Tech Stories — Tue, 15 Jun 2021 19:09:43 +0000

( This is a placeholder for my learnings in the context of MLExpert , updated almost daily, until I complete the course)

Foundational Knowledge in Machine Learning

Supervised Learning

When machine is learning from the provided data, it is supervised, in the sense that, for it to make sense of data, some labeling information of the data is provided to it, so that, it can map labels with the data.
For example: label all images of cats with 'Cat' / label all images of dogs with 'Dog'

Unsupervised Learning

In this also, machine is provided with data, but it is not supervised, in the sense, that, there is no hand holding here. Somehow, machine has to figure out grouping of data and has to create cluster of data points having some sort of similarity.
For example: give a stream of tweets from twitter and let the machine try to find patterns from the textual data. It would clusterize the tweets according to the category of topic associated with the tweet. It could be related to sports or tech or politics etc

Deep learning

In this also, data is provided to the machine, but the data would not be clearly structured, for example: image data. This is where neural networks are involved, specifically they are set of functions which try to determine pattern from data. There are lot of hidden layers which try to decode the pattern associated with the data.

Recommendation Systems

In this, for the end user, something needs to be recommended. For example, in case of ecommerce platform, we need to recommend products which could be potentially bought by the end user. Or, in case of Youtube / NetFlix, we need to recommend videos, which could be potentially watched / viewed by the end user. ( more time you spend on youtube, its an indicator to advertisers that, the audience is engaged with this platform and it makes sense for the advertisers to spend on Youtube. more time you spend on netflix, its an indicator that, Netflix is providing content suited to your taste, so, the end user is maintained with the perpetual subscription. ) So, ultimately, there are bunch of videos/ items to be recommended to user. Recommendation algorithm picks the best video to be shown to the user

Ranking

In this, for the end user, some kind of ranking to be done on content which is being recommended to the end user. So, what's the first video, what is the next best video to engage the user, what video after that, so, there is already set of videos which is stored/created. Ranking algorithms picks the best set of videos and ranks them and the videos are recommended in the order of rank of that video.

How does Tensorflow play a key role in solving machine learning problems ?

Machine Learning Tech Stories — Tue, 15 Jun 2021 17:33:45 +0000

( This is a placeholder for my learnings in the context of Tensorflow , updated almost daily, until I complete the course)

Tensorflow is a tool created by Google, which was opensourced, inorder to provide capabilities to solve machine learning problems, for researchers, developers, to not just toy with ML problems, but also produce solutions.

Machine learning vs Traditional Programming

Traditional programming -- We have data coming from diff data sources and we have rules, as to how to manipulate / query the data , to arrive at some answers.

Machine learning -- Its a new paradigm. Given the data and answers, the machine figures out the rules.

courtesy: Laurence Moroney

Neural Networks

Neural Networks is basically a set of functions, which can learn patterns from data.

Learning continues....

What are the practical aspects to Data Science ?

Machine Learning Tech Stories — Tue, 15 Jun 2021 17:29:10 +0000

( This is a placeholder for my learnings in the context of 'Practical Data Science Specialization', updated almost daily, until I complete the course )

What is Data Science ?

It is an intersection between various domains/ toolsets, to solve problems dealing with data, namely:

Artificial Intelligence
Machine Learning
Deep Learning
Having domain knowledge of the business
Having knowledge about mathematics behind techniques to deal with data
Statistics
Visualization - to deal with data visualization
Programming - to program with python, numpy etc

courtesy; [Andrew Ng]

Doing Machine Learning Projects or Data Science projects in Laptop vs Cloud

When we do our projects in our local laptop, we are actually limited by the resources provided by that laptop. The maximum amount of memory, processing power and whether it is CPU or GPU etc. All this matters in how efficient you can run your machine learning project in all phases of the project, for example:

Data ingestion
Data exploration
Data analysis
Data visualization

All the above steps are key steps which are prior to forming up the machine learning model, which will be used to make some inference.
So, each of the above steps need to be done, in efficient manner and Cloud provides resources at scale and on need basis, you can increase the amount of storage/RAM/processing power. ( which brings me to the question - suppose there is some project which is undergoing some kind of training in the cloud, and I , as a ML engineer/researcher realize that, it is happening very slow ( yes, it depends on the mathematical equation/parameters which is fitting the data ), but, it is slow. How do I , move this running project into another machine with greater capacity, without downtime for my training. For example, already as part of my old machine RAM, some kind of computations would have been and stored. How would these computations be pushed to the new machine ? Basically, how does scale work, while machine learning training happens, or even for that matter, machine learning inference matters - i don't have answers, but, the parallel I am looking at is, from software projects, where we have application servers, which are mapped to kubernetes pods and you can instantaneouly scale the pods, so, effectively, it becomes a distributed machine learning problem. So, the training / inference happens over a distributed system. So, the data required for training / inference gets distributed across systems. So, when a new machine is added which brings in the scalability, effectively , we are giving some portion of existing data to this new machine to process , whether it is training phase / inference phase - I am still curious as to how this happens - I have heard of tools like KubeFlow, need to explore them )

Obviously companies like Google, Microsoft, OpenAI, definitely use distributed machine learning. For example, when they say that, there are 1.75 trillion parameters for a model, which is able to infer something. Ofcourse the architecture would involve either a giant machine with huge processing power or RAM OR scores of distributed systems which participate in the overall processing.

Data slicing or Data transformations in parallel

While doing our machine learning projects, we might come across scenarios, where we need to slice data (slicing & dicing) meaning that, we want to reduce complex set of data, into small meaningful or focused set of columns . For example, in the initial dataset, we might have 100's of columns. But, we might be interested, in only few columns , so we can limit the data. we can also rename columns, to our needs.

General steps in any data science process.

We deal with massive data sets
We need to extract relevant features from this dataset
We then to gain knowledge/insight from these set of relevant features

courtesy: aws/deep-learning.ai

Advantages of doing data science projects in the cloud

With our laptop, we are limited by the hardware, sometimes, training the model might consume whole of our RAM and even CPU might get hogged. In this case, if it were cloud, we could have switched from cpu to gpu compute instance and also chosen a sizeable RAM to continue our task.

courtesy: aws/deep-learning.ai

What is the machine learning workflow, to be worked on ?

courtesy: aws/deep-learning.ai

As we see above, in the first phase - which is Ingest & Analyze,

Initially, we need to ingest the data, this will be done using Amazon S3,
The data exploration will be done using SQL queries, using Amazon Athena
We need to perform statistical bias detection on the input data, which will be done using Amazon Sagemaker Clarify (at this point, I am clueless about what is bias detection and why do we need them ? )
AWS Glue will be used to catalogue the data

In the next phase - Prepare & Transform

We need to extract relevant features from the input data set and this involves feature engineering, which will be done using SageMaker Datawrangler, Processing Jobs
The features which are extracted, needs to be stored, for which we will use SageMaker FeatureStore

In the next phase - Train & Tune

We need to now , go into the phase of model development. Using Autopilot, we will be having set of model candidates which will be trained on the data, which will be then chosen upon for the best candidate, with best score/accuracy.
Trainer & Debugger could be used to detect, for improving the model accuracy.
Hyperparameter Tuning - why this is required?

In the next phase - Deploy & Manage

Here is when, the model has been built, now its ready for use. So, we need to deploy the model .
We will be creating automated pipeline, so that, once the model is built, it is automatically deployed as well.
SageMaker endpoints will serve the model, which can be used for inference.
BatchTransform and Pipelines will also be used in this stage.

Popular Machine learning tasks

courtesy: aws/deeplearning.ai

Supervised - In this, the machine learns with examples - Classification could involve categorizing sentiment of text into positive, neutral, negative. Regression could involve predicting a continuous value, given set of parameters. For example, predicting house price value.
Unsupervised - This will involve determining patterns and clustering / grouping the datapoints
Imageprocessing / CV - In this we need to determine whether an image contains a dog/cat. OR, we need , for self driving cars, to differentiate between speed signs and trees.
NLP/ NLU - In this, we need to do sentiment analysis / machine translation / transfer learning / question-answering.

Multi-class classification for sentimental analysis of product reviews.

We have set of product reviews, for example, from amazon.com.
For each product review, we will have to classify the sentiment, whether it is positive/negative and other classes.

courtesy: aws/deep-learning.ai

We need to do training, this is a supervised machine learning problem, so we need to provide labels as shown below.

courtesy: aws/deep-learning.ai

Learning continues....

What all goes into the landscape of AWS Machine Learning ?

Machine Learning Tech Stories — Tue, 15 Jun 2021 17:22:50 +0000

( This is a placeholder for my learnings in the context of 'AWS Machine Learning' , updated almost daily, until I complete the course)

What are the different toolsets available in Amazon WebServices, which cater to machine learning based applications ?

Currently, what I know is Amazon Sagemaker , using which you can create a model.

Learning continues....

How essential is Mathematics to Machine learning ?

Machine Learning Tech Stories — Tue, 15 Jun 2021 17:18:52 +0000

( This is a placeholder for my learnings in the context of Mathematics for Machine Learning , updated almost daily, until I complete the course)

How does Mathematics play a very important role in Machine Learning ?

Mathematics topics such as Probability, Statistics, Linear Algebra, Calculus - are the basis for Machine Learning.

Linear Algebra Motivation

Solving Simultaneous linear equations

Suppose we have 2 variables - apples, bananas.
And we want to determine the prices of apples, bananas - given set of incidents.

First incident - it was - 2a + 3b = 13
Second incident - it was - 3a + 4b = 27

We go to market and buy apples and bananas and we want to solve the above equations, to determine the price of a, price of b.

The above problem can be formulated in terms of matrices and vectors.

| 2 3 | |a| = |13|
| 3 4 | |b| |27|

We have a 2x2 matrix, multiplied with a column vector, producing another column vector.

Optimization problem of fitting data with an equation with fitting parameters

Suppose we have an histogram of heights of people in a given region. Histogram means that, we have range of heights in x-axis of our graph and in y-axis, we have count of people who fall in that range. Now, with good amount of data, we have plotted an histogram. Now, we want to fit an equation or a line to this data. Basically, we want to find the optimal value of the parameters governing that particular equation/line which will fit the data.
The reason for fitting the line/ deriving the equation is that, we want to determine how the heights are distributed across people. Basically we also want to answer some questions, like: what is the average height of people in this region ? If we pick a random person from this region, what is the most likely height for this person ? We want to understand the distribution of the data. We have lot of data points, but, we want to understand the pattern, something like - there are very few people who are extremely tall. Most of the population in this region tend to have this height etc.

Another example to understand histogram is, distribution of night prices for airbnb houses , courtesy: (Yan Holtz) https://www.data-to-viz.com/graph/histogram.html

As I learnt from Yan Holtz's blog , the beauty of histogram is to understand the distribution of data primarily and that is evident visually from the following image, courtesy: Yan Holtz

We can see multiple types of distribution:
a. Skewed
b. Normal
c. Uniform - almost all ranges have equal amount of values.
d. Comb
e. edge peak
f. bimodal

With neural networks/ machine learning, there are 2 things to consider about fitting equation;
a. What is the mathematical equation which would fit this data?
b. How to best fit this equation - in the sense, what are the optimal values of the parameters of the equation, such that , this line representing the equation fits the data, in the best way possible.

Vectors in Linear Algebra

As we saw in the previous example of histogram, we want to fit an equation to the data representing the heights of people.
Then, let's consider a Gaussian distribution or a Normal distribution ( Normal means that, area under the curve is 1 ). In this equation, mu represents the center height of the curve. And sigma represents the width of the curve from the center. And it is represented by an equation shown below:

courtesy: imperial college, london - coursera

Learning continues....

Why is MLOps crucial in the domain of Machine Learning Engineering for Production ?

Machine Learning Tech Stories — Mon, 14 Jun 2021 19:07:34 +0000

( This is a placeholder for my learning in the space of MLOps, updated almost daily, until I complete the course )

Intro

Generally, a machine learning engineer / modeler would start their project in a Jupyter notebook environment to create a model.
Machine learning does not end with creating a good model. ( with some good accuracy )
But, the focus is to put the model into production.
How do you deploy a machine learning model into production, so that, it can be used for making predictions, given some input data ?

What is MLOps ?

MLOps is a discipline of building and maintaining production machine learning applications, with the processes and tools.

Ponderings

Some of the ponderings/ questions we need to answer is:

What goes into the lifecycle of a production machine learning application ?
How will a model be deployed ?
How will a model be updated in production, as and when it is tuned by the modelers, to improve accuracy ?
What goes into testing the model ? Do we some integration tests, which run on machine learning models, just as we have them for normal software projects ?

Project Lifecycle of a production machine learning application

courtesy: Andrew Ng

What goes into the process of MLOps ?

courtesy: Andrew Ng

Data drift

The machine learning model which you have developed in your lab would be trained on some data. Once you deploy this model into production, you might encounter several issues and one of them is data drift. What this means is that, the distribution of data which was used , during the time of training, will be very different from the distribution of data during inference. So, accordingly the model needs to be retrained, taking into account the data drift.

Importance of 'Production machine learning' knowledge

You might be working on a product which is performing some kind of inference in production. And this product operates continuously in production. So, the knowledge of modern software practices is very essential to maintain your product.

Model creation vs Model deployment & maintenance

The scenario of model creation will be way different from the scenario of model deployment. For example, during model deployment, there might be a case that, you deploy on smartphones. And the requirement from end users could be that, the data on their device, should never leave that device for privacy. Then the model deployment/maintenance should be such a way that, model on the smartphone/device is continuously updated/refreshed, whenever the model is updated/tuned.

The objective is to have a very good sense of entire lifecycle of a machine learning project.

Reality of machine learning projects

Let's assume that, our usecase is this. We have a manufacturing factory which makes smartphones. The job of machine learning system is to take pictures of the manufacturing assembly line containing the smartphone. And figure out, whether a smartphone is defective or not. If the smartphone is defective, then the inspection module will remove the smartphone from the assembly.

It starts with training the model. The final step is to put the model into a prediction server, by setting up API interfaces.
These API interfaces of prediction server could be hosted in the cloud. OR, we could have the edge deployment itself, where the API interfaces of prediction server is available within the factory itself, without relying on internet connection.

Within the lab/jupyter notebook, we might be able to train the model and check its accuracy, it might be very accurate on the testing dataset. But, in practice, in production deployment, based on the real world data, the model may not perform very well due to various reasons. One of the reason is data drift. In lab, we might be working with training dataset, where the lighting is good enough. But, for example, in a manufacturing factory, which could be a edge deployment, the lighting conditions might not be so good that, there might be some darker images, sent to the prediction server. And the prediction server might not do well on these kinds of images.

What about the amount of code involved in machine learning ? OR the amount of aspects to deal with, in machine learning projects ? We might assume that, we should be responsible just for the model code or just the algorithm. But, in reality, the ecosystem consists of several moving parts which enables the prediction. The entire infrastructure required for machine learning, will require various components for :

Configuration
Data collection
Data verification
Feature extraction
Serving infrastructure
ML model code
Monitoring infrastructure
Analysis tools
Process management tools
Machine resource management tools.

So, ML code is just 5-10% of the overall ecosystem.

courtesy: Andrew Ng.

Learning continues....

Why do we need to think of Explainable AI ?

Machine Learning Tech Stories — Sun, 13 Jun 2021 11:23:30 +0000

(This is a placeholder for my learnings in the space of Explainable AI, updated almost daily, until I complete the course)

Introduction

Most of the times, AI algorithms might give us good results, probably accurate results, as to what products to be shown to customers while they browse the catalogue.
Or whether a particular customer is not genuine, in the incident of fraud detection scenarios.
Or it could also predict house price accurately, given certain factors.

But, can AI explain, why did it reach to a particular conclusion ?
AI remains as a blackbox for the humans. And humans cannot reason enough, as to how AI algorithm was able to arrive at a particular output, given a particular input.

Explainable AI is the key to solve such problems.

What is XAI ?

XAI is a type of AI that gives insights, as to the reasons behind the outcomes of AI.

My ponderings

Some of the questions which I have got are as follows:
1) Do we really need models to be explainable ?
2) If models cannot explain why it has reached a particular conclusion, what are the consequences?
3) Given a set of domains, where machine learning plays key role, what would be the case, if models cannot be explained ?

For example, in the domain of finance, if a customer incident is classified as a fraud transaction, how did it arrive at that conclusion ? Or, in the case of a medical diagnosis, why did a model arrive at a conclusion that, a patient is diagnosed with a brain tumour. Because, day by day, machines/systems are going to be trusted with their decisions. Sometimes, blindly trusting machines might work, but, given certain domains, blindly trusting will not work.
4) What should a machine learning engineer be aware of, if they need to make their models explainable ?

With / Without xplainable AI (famous picture related to this topic):

courtesy: Aki Ohashi

What are some companies who provide their solutions related to XAI ?

Know the how & why behind your AI solutions by fiddler.ai fiddler.ai

Related resources:

( Throughout the course of learning about explainable AI, I do collect some interesting online resources, books, youtube videos so that, we get a full and deeper picture in this field )

Learning continues....

How can we build Modern Java apps on AWS ?

Machine Learning Tech Stories — Thu, 10 Jun 2021 12:19:48 +0000

( This is going to be placeholder, as I learn more about how to build modern apps on AWS using various amazon webservices, such as : Amazon Cognito, Lambda, Stepfunctions, S3 for storage, Cloudwatch for monitoring, Raytracing for distributed tracing etc , updated almost daily, until I complete the course)

When we want to build modern Java apps on AWS, what do we mean by that ?

Modern Java app on AWS - workflow

Firstly, we are going to have an end 2 end project running on AWS.
By end 2 end, in the sense, there's going to be user authentication/ authorization using Amazon Cognito
Amazon S3 will be used to store some data
Amazon API Gateway will be used to design APIs and perform traffic monitoring, providing security to the APIs.
Amazon Lambda will be used to retrieve data from S3, when a GET API is invoked. And for adding data to S3, a POST API will be used. But for posting data, there will be a validation performed and then data will be posted to S3. Since we have stepwise approach to add data, we would leverage Amazon Step Functions to do that.
Amazon CloudWatch will be used for monitoring. Amazon RayTracing will be used for distributed tracing purposes.

Overall that's the end2end project for building modern Java apps on AWS.

So, we will be leveraging the serverless capabilities of Amazon Lambda, along with the authentication services of Amazon Cognito.

Amazon Cognito

User Authentication & Authorization workflow with Amazon Cognito & API Gateway

Within Amazon Cognito, there is a user pool maintained. For example, you could maintain data like: useremail and password.
And then use this capability to authenticate the users.
The next thing is Amazon Cognito can be integrated with 3rd party identity providers for authentication purposes. For example, OpenIDConnect, Social Sign-in providers. This is based on OAuth2 technology.
For example, once user signs in with Facebook, a token will be generated and sent to the user. The user will exchange this token with Amazon Cognito and Amazon Cognito will validate that token with the identity provider. Once validated, Amazon Cognito will provide a JWT token to the user/client.
Client can then provide this JWT token to API Gateway, in order to access an API.
API Gateway will then, validate this JWT token with Cognito, against this user/client. And once validated, will allow the user/client to access the backend API, which is proxied by APIGateway.

The above flow is called Federation with Cognito user pools, wherein the identity is authenticated via identity providers.

Federated Identities

Apart from this , Cognito also has another feature called Federated identities. And with this, basically, once identity is authenticated with identity provider, that is used to assign IAM role, with which AWS services can be accessed, including API Gateway.

Unauthenticated Identities

There's another small feature within Amazon Cognito, which is unauthenticated identities, which is used, where users don't want to share their identities, but still are allowed to access AWS services, but the scope would be limited.

HostedUI

Amazon Cognito also has a nice functionality called: hostedUI - using which you can signup and signin users, if you are not going for 3rd party identity providers.
So, you can define the redirect URI, once the user is signed in.
For example, you can have a webpage within your website, which can be treated as your callback html file.

Amazon Lambda

Amazon Lambda is powered by a virtualization technology called Firecracker. Using this, microvm's are created, which is used for execution of code within lambda.

Learning continues....

Principles of economics

Machine Learning Tech Stories — Wed, 09 Jun 2021 13:10:03 +0000

( This is going to be a placeholder for things I learn in the context of 'Principles of economics' , updated almost daily, until I complete the course)

What it takes to think like an economist?

The fundamental idea in economics is scarcity.
Any entity - individual, corporation, government - has access to scarce resources and they need to decide how and where to allocate them.

People make choices with scarce resources, and they interact with other people ( in market ) when they make these choices.

Choices, Scarcity, Interaction.

What choices do people make with the scarce resource and how they interact with the world ? - This is the mantra of economics.

Suppose you are a developer - you have a limited resource - time / money. How do you allocate that and where do you allocate that ?

Suppose you are a corporation - you have a limited resource - time / money / people working for you. How do you allocate these and where do you allocate these and when do you allocate these ?

Suppose you are a government - you have limited resources - time, money, natural resources, people. How do you allocate these and where do you allocate these and when do you allocate these ?

When we examine the behavior of governments, what we are dealing with is: macroeconomics.
When we deal with individual firms / individuals - what we are dealing with is: microeconomics.

And there are opportunity costs - it is the cost of what you give up, if you pick up something among various choices. For example, if you can do some coding which will take 1 hour. And you can also watch Netflix webseries which will take 1 hour. You have chosen to watch NetFlix, then you have paid opportunity cost of not coding 1 hour. You could as well have done coding and completed some task, for example.

What's happening in the world economy ?
What's happening in economy in your country ?

Did you know that downtime of Fastly affects economy in a way ? Fastly outage can affect sales of Shopify or any other ecommerce business based on Fastly.

In a way, all things are related.

Production Possibilities Curve

Production Possibilities Curve is a depiction of increasing opportunity costs for goods/products from a sector, when workers from that sector move to other sector, producing more goods/products of the other sector.

courtesy: Stanford University

For example, if we consider 2 sectors - movies, computers. We initially have skilled workers in computer sector. When we move these workers gradually to movies sector, slowly there is a decline in the output of no. of computers being produced. So, the involvement of workers in movies sector is an opportunity cost in the computers sector. More of something leads to less of something else.

And the opportunity costs are increasing , in the sense, that the rate with which the productivity reduces with computer, with the increase in productivity of movies. The curve represents the production possibilities of goods/services representing these sectors.

courtesy: Stanford University

Possibilities and Impossibilities -

The curve also represents the possibilities and impossibilities, for example, we cannot produce 30000 computers and simultaneously produce 1000 movies. The curve and the area under the curve represent the possibilities of production.

Efficient/ Inefficient -

The curve also allows to understand efficient production / inefficient production. If production of goods/services in both the sectors happen according to the boundaries of the curve, then, there's efficient productivity. But, if the production of the products lies within the curve, then there's inefficient productivity. For example, there's an efficient possibility of producing 5000 computers and 200 movies. But if the number of movies produced are less than 200, for example, 150, then it means that there's inefficient productivity of the movies.

Evolution of PPC -

As the economy evolves, the old PPC is replaced with new PPC, when the possible no. of goods/services produced increases, which lead to economic growth. When we produce more, we earn more, the standard of living improves etc.

courtesy: Stanford University

Investment vs Consumption

Products such as computers lead to investment, you can use them to produce something, in the future.
Products such as movies are consumption products, because you can just consume them.

Real Gross Domestic Product

A given country consists of producing various goods/services across sectors. And there is a quantifiable number which represents the total no. of goods/services produced by the country, which represents the economic state of the country, which is GDP. The real GDP is GDP, where inflation is not accounted for.
So, over the years, the GDP as a number can be plotted against the year. And we can see that GDP curve increasing to top right.
Sometimes , there might be dips in the GDP curve, which indicates the dip in productivity, which could be caused, by various factors, for example - recession.

No. of employed workers curve

And also, there is another curve which represents the total no. of employed workers in an economy, which can be plotted against the year. So, more number of workers, more productivity.
Sometimes, again, there could be dips in no. of employed workers, due to recession. (covid pandemic - black swan).

Real GDP per worker curve

Since we have GDP and we also have no. of employed workers per year, so, we could plot another curve, which would represent real GDP , per worker.

(Opportunity costs in cases of good/services with concentration of workers in sectors , mean that, when workers embrace the opportunity in one sector (for eg: movies), they simultaneously embrace the opportunity cost in another sector (for eg: computers), that means same no. of workers, same set of opportunities, but, when workers move across sectors, this incurs opportunity cost of productivity in one sector)

GDP studies across countries

There could also be an evaluation of GDP growth across countries, there could be dips/ peaks / rise - so, what were the factors which led to these characteristics of the curve - that could be studied in context of economics.

Observing and Explaining the Economy

There are lots of datasets related to economy. One of them is income distribution.
The question is:

In a country like United States, How much do the top 10% earn ( on an average ) and bottom 10% earn ( on an average ) and how has this distribution evolved over the years ?

Sometimes, change in data representation helps. For example, absolute values might not help, but, a slighly diff way of looking at things, for example: percentage change in income.

What about percentage change ?

In different time periods, different growth rates are there for both the groups. So, specifically, what happened in the 80s and 90s, what led to the steep growth of top 10% ? What were the economic policies which led to that or , what opportunities opened up during that time ?

There's an economic theory that, during the same time period, relative returns from higher education, meant that , higher the education/ skills, more returns. So, that's one possibility to see the widening of income distribution around 80s and 90s.

# Positive economics and Normative economics

Positive economics deals with explaining why something occurs, using economic theory.

Normative economics deals with suggestions/ recommendations of economic policies.

Supply and demand model

This is one of the most important and basic models in economics.

It can explain the chaotic nature of markets like auction markets.

You can have any kind of markets. Market of goods/services. For example, bike market, labour market, stock market, ( we can also treat internet as a marketplace, where the goods can be data centers provided by major cloud providers / services provided by diff software companies and many other entities )

Demand is relationship between price of a good and the quantity being demanded. Economists like to understand, how the price of a good affects its demand. Higher the price probably lowers the demand. It describes behaviors of consumers.

# Movement along the Demand Curve

This is where we see the impact of prices on the demand. So, as price increases, what happens to demand and if price decreases, what happens to demand. You go up the curve, when the demand is less and price is high.

Shift in demand curve

This can happen, where prices stay the same, but demand either increases/ decreases depending on other factors. For example, price of roller blades might decreases drastically, people might demand fewer bicycles because of that. OR, depending on the weather or environment concerns, the demand for bicycles might go up , so there's shift in the demand curve.

Supply

Supply is a relationship between price of a good and quantity supplied.
We also have supply curve, which is upward sloping.
It describes behaviors of firms.

Market equilibrium

This is where quantity being demanded equals quantity being supplied. And the price at which this occurs, is the equilibrium price. And the quantity is equilibrium quantity.

What if there's an increase in demand ?

That means, from the equilibrium, we want to shift and create a new demand curve, hitting the supply curve at some intersection point, which is the new equilibrium, new price which is increased and quantity also increased. So, basically prices have increased, as a result of demand increase, to maintain the new equilibrium.

Elasticity

Resources

Awesome Economics collection on Github

So, what do economists do ?
They look at data and try to interpret data.
Once data is there, they plot the data, to explain.

Causation: one event brings about another event
Correlation: one event closely occur alongside another event, but does not necessarily imply.

Learning continues....

What's the key mantra for machine learning ?

Machine Learning Tech Stories — Mon, 07 Jun 2021 11:56:12 +0000

Have lots of data
Have the data labelled
Let the machine figure out the relationships between the data and the labels
The output of above exercise is the machine learning model, which can be used to predict/classify.

Courtesy: @lmoroney