DEV Community: Divyessh Maheshwari

Catch Up with AI

Divyessh Maheshwari — Wed, 19 Mar 2025 23:27:34 +0000

Photo by charlesdeluvio on Unsplash

It has been over 10 months since I last posted, so now it's time to catch up. I used to write about technical concepts from a fundamental point of view so that people with non-technical backgrounds could understand them. This led to a series of 11 articles on topics like full stack development, machine learning and artificial intelligence, cloud computing, and some other fundamental topics like blockchain and version control. They can still be accessed on my Medium/Dev.to profile.

Now, I plan on starting to write more about my experiences that I have had over the past few years being a student and professional in the tech industry and express my personal views on ongoing events in the industry. Let me start with a little background about myself. I am currently a student enrolled in an undergraduate degree in computer science, and I have been working in tech as an independent contractor for close to 5 years now. I am an AWS certified Solution Architect, and I specialize in developing data-intensive systems along with a few other fields. Enough about me, now let’s dive into what’s going on lately.

AI Hype and History

AI (or more specifically GenAI) is by far one of the hottest topics in Silicon Valley. The funding for startups involving the use of AI has been astronomically high. According to Scale Capital, while the global venture funding for startups saw an increase of just 3% from 2023 to 2024, the funding for AI startups saw an increase of 80%.

In my case, the implications of GenAI are more directly seen in two popular scenarios:

In college, by the students (including me) to complete assingments.

In the industry, by startups trying to add GenAI to their existing products.

Now AI itself is as old as computers, and ML advancements have been taking place for a very long time. AI-based virtual assistants like Amazon Alexa, or Apple’s Siri have been around for over a decade now. Email spam filters, which most of us use, have also been using ML since the early 2000s. So what’s causing this hype now? I think two major factors are: Accessibility of GenAI and Scalability of GenAI.

Before 2017, GenAI was not scalable since it took a lot of computation power and was not parallelized but after researchers at Google introduced the transformer architecture in 2017, the GenAI became much faster to train and hence much more scalable across fields.

Building the GenAI model still takes a lot of computation, but now it does not take as much time. This led to further enhancements like the introduction of the BERT and GPT series in the following years.

But still, it was not very accessible to the general masses until the ChatGPT model went viral in 2022. Since for most of the masses, the concept of a fundamentally intelligent model was very unexpected, this started the hype around what people nowadays call AI.

ChatGPT was and is an excellent model, but people tend to forget that it is essentially an autocomplete tool with the capabilities of a search engine.

Current Economic Stance

Circling back to the industry implications, most startups started adding GenAI-based features in their products to catch up with the public’s demand. Even though many of these features are very useful, I think the majority are unnecessary additions. Now, I could be completely wrong about this since at one time, e-commerce was also considered unnecessary by many, and today it is one of the leading industries. But even with e-commerce (ecom), the market hyped it too much, and countless people created their personal stores hoping to generate high revenues who ended up spending high advertising bills with negligible online sales. We will table the discussion of the social impacts of ecom for some other post.

I on many occasions forecast the hype of AI to have similar trends to that of Blockchain (Cryptos or NFTs) in 2021. According to Pitchbook, Over $30B of venture funding was deployed in crypto startups in 2021. That bubble burst in Q2 of 2022 with more than an 80% dip in funding compared to Q1 of the same year. Again, this does not mean that Web3 or Blockchain is a bad technology, or that it is a scam. All I am trying to say is, that it was overhyped and hence many many engineers hired for those startups on astronomically high packages during the hype were laid off in the following years. I think a similar trend will be seen for GenAI startups. Right now, AI startups — hiring engineers — that are trying to fit in AI where it is not really needed will soon realize how the liquidity is not high enough for them to exit.

College Life

I have been wanting to write on this topic for a very long. This topic even deserves its own article. The college experience is so complex. The networking and social aspects, the classes and academic aspects, the exposure and opportunities, the obstacles and challenges. But focusing this post more towards AI, let’s talk about the academic aspect.

I consider college to be a great place since you not only learn from your classes but also develop very strong and helpful networks among your peers. But lately, the learning part of the classes has been negligible for most of the students around me in computer science. The majority can get solutions for their assignments with minimal effort using AI, and most of them are essentially not understanding anything. I am a strong advocate of the ideology that “if you can find answers in your classes, you can find answers in your jobs.” But that is only valid as long as one can understand what is happening. If you find the answers to your assignments without giving any thought to what the problem asks for and how the answer answers it, it probably in my opinion is not going to benefit anyone. And when faced with a challenge that requires an actual understanding of the tech concepts, merely finding answers might not be the best skill. It has to be accompanied by the skill of knowing how to use those answers to solve problems.

So, when students who barely own any fundamental technical knowledge go out into the industry, they will struggle as they are neither employable nor coachable. This is what I have been noticing as well — there are plenty of recruiters saying “there is no talent available to hire” and plenty of graduates saying “there are no jobs available.” I think just developing some fundamental technical skills will go a long way in the future since the other competing professionals will lack that.

Final Notes

I do not mean to look down upon the AI industry, on the contrary, I see great potential in it. I think developing relevant skills in this sector can go a long way, but if this hype continues the bubble will keep getting bigger and bigger. If and when the bubble bursts, the industry might find itself in a strong recession. I do not reflect anyone else’s opinion but mine (with some conviction though).

I hope you enjoyed the read. Please share your opinion on this even if it does not match mine.

Feel free to connect with me on Linkedin.

Sources

CourseBot - Coze AI Bot Challenge

Divyessh Maheshwari — Wed, 17 Apr 2024 19:42:21 +0000

This is a submission for the Coze AI Bot Challenge: Bot Innovator.

What I Built

I present CourseBot, a skillful course recommender designed to find the most recent and popular free courses on various topics. CourseBot utilizes online platforms such as YouTube and Open edX to curate a selection of high-quality courses tailored to the user's interests.

Demo

You can experience CourseBot in action at the Coze Bot Store. Below are some screenshots showcasing CourseBot's chat interface:

Your Configuration

Here's a glimpse of CourseBot's configuration, showcasing its prompts and skills:

Journey

My journey crafting this chatbot began with ensuring a clear understanding of user needs and preferences. With a well-defined thought and utility in hand, I defined the persona.

Process of customising and launching the bot was made very easy by the Coze AI Bot Store, whose intuitive interface streamlined the creation and deployment process. Their platform was very user-friendly and enabled me to convert the idea into bot quickly. Its array of plugins and tools provided a seamless experience, allowing us to focus on perfecting our bot's functionality and prompt.

Thank you for considering CourseBot for the Coze AI Bot Challenge!

Git and GitHub — The Version Control System

Divyessh Maheshwari — Sun, 02 Apr 2023 11:04:11 +0000

Git and GitHub — The Version Control System

Have you ever wondered how multiple teams collaborate on a project? Or what happens if we want to check the history of a file in our codebase? Or how are different versions of an application managed?

In this article, we are going to understand what is a version control system, and how is it used to manage codebases for different projects. We are also going to look at what are Git and GitHub, and how are they useful for teams collaborating on a single codebase.

Introduction

Version Control System is a very essential part of any project with code. Whether you are creating a simple Calendar Mobile App or building a Real-Time Web Dashboard, you need to manage and track the changes taking place within your programs. We can take the example of the bank balance — it is not just sufficient to have your final account balance, but it is also important to have a history of transactions for proper accounting. This record of transactions also helps you to identify fraud or file for wrong transfers. Similarly, having a history of a file in the source code helps in determining the point at which an error (or a bug) occurred and who was responsible for those changes.

Git today is the most popular version control system used by developers across the world. And GitHub is one of the most popular platforms that provide Remote Hosting Services for Git. Git can be installed in our systems, and we can use it locally to manage the versions of our project, while GitHub provides a web interface to manage these Git repositories (source code) including some additional features.

Photo by Devon Janse van Rensburg on Unsplash

Terminologies

Let us get familiar with some of the techy terms that we are going to use in the rest of the article.

Repositories (Repo)

A git repository refers to the source code, along with its history of changes and metadata. This is stored within a folder named “.git/” in the root(top-most folder) of your project directory.

Commits

The changes to the files can be saved in batches, and committing these changes refers to the process of saving the set of changes to the repository history. These different batches can be called commits.

Branches

We can choose to create multiple snapshots of the source code at different times and start working on those snapshots individually. These snapshots are called branches, and there is always one primary branch for one repository where generally the latest and most stable code is stored. To understand it better think of 2 developers collaborating on two different features of the same app. They can create two different branches, work on their individual features in the respective branch, and then merge those branches into the primary(main) branch.

Push and Pull

When it comes to remote repositories, we can use Push and Pull commands to sync our changes with the remote branch. For example, when we have made some changes, we can use the push command to send our changes to the remote repository, while if some other member of the team has made a change and pushed it to the remote repo, we can use the pull command to get those changes on our local system.

Photo by Glenn Carstens-Peters on Unsplash

Remote Repository

One can ask “ Why do we need a remote repo like Github, and why can't we just collaborate locally with Git itself?”

One of the most important reasons for using cloud/remote repos is Collaboration with team members. Suppose, I am living in India, and my team member with whom I am working on a project lives in France, then we will not be able to share our local repositories. We will need a cloud-based or remote repo, which we can share and access.

The second reason to use a remote repository is the Durability of the source code. Imagine I am working on an application for months, and I am using Git locally for my version control. It has been going great, but one day, my laptop gets crushed by a gain rock (or maybe just gets infected by a virus), and all my files are gone. Now, what will I do? If I had maintained my repo on a remote system, the chances of my codebase being gone are very very low.

Photo by krakenimages on Unsplash

Examples

Let us have a look at some examples of where we can use version control to make our lives easier.

Mobile Application Versions

I am sure that at some point in time, you have to update your mobile apps by going to PlayStore or App Store or setting up an automatic update. Whenever an app releases a new version of an application that might not be compatible with older versions of Android and IOS, it maintains an old version that was compatible with these older OS versions. This history is maintained and tracked using the version control systems.

Open Source Contributions

Most of the open-source software and packages are hosted on GitHub, where thousands of different can contribute to some part of the project. To manage such a huge scale of collaborators, it is necessary to maintain a proper track of changes made by different contributors.

Error Management

Suppose, you and your team are working on a web application that trades stocks in real-time using a strategy. And when you pull the latest changes made by your teammates in the morning, and test them, you notice, that your strategy that used to yield a 17% return is now showing 98% losses. You can go to the history in Git/Github, and check which change made by which one of your teammates caused this excellent transition. And if needed, you can revert to a previous commit to undo these changes.

Conclusion

Version Control is one of the most essential skills that a developer needs throughout his career. It is largely used in deployment, integration, and distribution. Almost every single organization that has anything to do with code is using some kind of version control and enjoys the several advantages that come with it.

I hope you enjoyed reading the post. To follow up with the new articles, follow me on Dev.to, or subscribe to my Newsletter on my official blog Thinkfeed.

Image processing and Computer Vision

Divyessh Maheshwari — Sat, 27 Aug 2022 18:40:23 +0000

Have you ever wondered how your mobile uses facial recognition to unlock itself? Or how different deep learning models are used to detect anomalies in an image? Or how do the Instagram filters work in order to manipulate the image it receives?

In this post, we are going to understand how image processing and computer vision work, and how they are used with deep learning in order to create innovative and complex solutions for many day-to-day problems. In order to get a better understanding of this article, I recommend you to read my article on Artificial intelligence and Machine Learning.

Introduction

Image processing, as the word suggests, is applying different algorithms and techniques to manipulate or modify the image for making it suitable for the task and use case. Almost every one of us has used image processing for one task or another, for example, when we use portrait mode to click a selfie, we are using image processing to blur the background.

Computer Vision, on other hand, is one of the applications of Artificial Intelligence and used algorithms and techniques to identify patterns within the image data. To relate it with image processing, we can once again take the example of using portrait mode. While image processing is used for blurring the background, artificial intelligence is used to identify or recognize the background, and the whole solution developed by the combination of the two will be a Computer Vision Solution.

The answer to the question “How do our mobile know which objects to blur and which objects to not blur?” is Artificial Intelligence or Deep Learning. The answer to the question “How do our mobile blur the objects?” is Image Processing. And the answer to “How does our mobile implement portrait mode?” is Computer Vision.

Venn Diagram to represent correlation among AI, Image processing and Computer Vision

Image processing is very commonly used to pre-process the image before implementing computer vision algorithms on it. As the applications of Artificial Intelligence are growing, the fields of Computer Vision and Image Processing are progressing alongside. This nature is very intuitive as when the use cases of something diversify, the pre-processing techniques also get in demand.

Image Processing

Now imagine you are given a task to create an Instagram filter that recognizes the user(human), and makes everything in the background blurry as well as makes the background of the image darker. For now, let us assume, that we already have an AI model that detects the user from the image, and gives out its coordinates. Now how can we approach this problem?

Image Structure

While developing AI solutions like this, we need to remember one thing, most of the time, while using computer vision, the models are not trained on the color images, but they are first converted to grayscale(black and white) images, and then used. That is because color images are much more complex to process and relatively take a lot more time to train the model. Let us understand the reason behind it.

Images are stored in the form of a matrix (2-D arrays/vectors) when it comes to processing it. In the case of the color images, each element of that matrix contains 3 numbers, each representing the concentration of Red, Green, and Blue colors from a scale of 0 to 255. On the other hand, each element of grayscale images contains only a single number ranging from 0 to 225 where 0 represents black and 225 represents white. These elements are called pixels and these numbers are called color channels. So, in order to process three color channels a lot more computation power and time are required, and since that can be easily avoided by using grayscale images, most of the models present today, takes only grayscale images as input.

Image Processing Algorithms and Functions

Coming back to our problem, our first step is going to be to convert the input image from color to grayscale. We can do that by using a pre-built algorithm/technique and sending it as input to our AI model. The model will process this image and return the coordinates of the detected user, and we can store them for our future reference. Now, we create our blurring function, and the function to increase the darkness.

There are many popular algorithms we use to blur our background, one of the most famous ones being Gaussian Blur, you can read more about it on this Wikipedia. Once we have selected an algorithm of our choice and created a function, we need to use it on the image. But how can we leave the detected user, out of our processing? We will use something called a mask. Making covers or replaces the entire image, and prevents it from being processed, and once the rest of the image is processed, we can remove the mask.

Once we have our blurred image, we move towards the darkening of the background, for that, we will again use the same mask, and darken the image, by decreasing the concentration of each color channel of all the pixels. After this, we can remove the makes, and output our freshly processed image.

During this segment, all the strategies or techniques we discussed are part of image processing. Computer Vision also includes Artificial Intelligence which was used for creating the model itself that detected the user from the image. The development of the model again includes a lot of image processing and the use of deep learning algorithms.

Conclusion

Computer vision is not only limited to using image processing along with AI but can also include signal processing. This is one of the most growing fields with numerous applications being implemented and used as I write this article. For innovative minds, there is nothing that can restrict them from finding more and more applications for CVs. Especially now, when the application has diversified in the fields of Healthcare, Space(Astronomical) Research, Crime Control, and so on.

I hope you enjoyed reading the post. To follow up with the new articles, follow me on Dev.to, or subscribe to my Newsletter on my official blog Thinkfeed.

Web 3.0 and Decentralization

Divyessh Maheshwari — Tue, 05 Jul 2022 08:07:58 +0000

Have you ever wondered how does the blockchain works? or what are the tokens or cryptocurrencies on a fundamental level? or why NFTs are non-fungible?

In this post, we are going to explore what really is Web 3.0, and also understand some basics of decentralization and blockchains. We will also take a look at what smart contracts and tokens are, and at last, we are also going to learn how are these cryptocurrencies implemented and what really makes the NFTs authentic.

Introduction

In order to understand these concepts of Web3we first need to know what is Web 2.0, and how it is any different. Web 2.0 is the World Wide Web based on the concepts of social media, where the user can create content, post it online, and engage with other user-generated content. But the upcoming issue was that they did not own this content or the revenue being generated by it. The company that provided the platform for sharing the content has the maximum ownership of the revenue generated by that content. This led to the centralization of the data and traffic influence.

Centralization

Let us say I start my own social media application. I create a great user interface, and a strong and secure backend interface as well. So if any of you join my social media application, and start posting content on it, it will be stored in a database I set up, and only I have the access to. This means that I have the maximum control over your data, and hence I also own the benefits and revenue I generate from this. Not only this, but I am also in control of making any decisions for your data. For example, I can decide that I no longer want to store your data in a relational database and migrate it to a NoSQL database, and you as a user will not be able to restrict this decision. This is known as centralization.

Centralisation or centralization (see spelling differences) is the process by which the activities of an organisation, particularly those regarding planning and decision-making, framing strategy and policies become concentrated within a particular geographical location group. This moves the important decision-making and planning powers within the center of the organisation. ~ Wikipedia

We can also say this as when one node or a group of nodes are in control of governance in a system, the system is centralized. Nodes are just systems on a network. Like in the social media application example above, all the systems of users using the application as well as the main server are nodes on the network.

Decentralization

Decentralization is when no node or a particular group of nodes has a central control or governance over all the other nodes. This does not necessarily mean that no node has any control over the network, rather it means that all nodes have equal control and access to the network, and changes made on one node affect the whole system. For example, if our social media application above was decentralized, the complete data would have been stored on each and every user's system, and whenever one user made a change to this data, maybe by uploading a pic, the same change would have taken place on all the nodes. Because of this method of ensuring the decentralization of the network, also called consensus , the decentralized systems require a lot of computing power and resources.

Photo by Shubham Dhage on Unsplash

Web 3.0

Web 3.0, unlike Web 2.0 — which generally has a centralized system, has a decentralized distributed system. That means that all the nodes on the system in Web 3.0 have equal control and access. One of the key features of Web 3.0 is that it implements Smart Contracts and Token using the blockchain mechanism. Let us understand what the terms I just used mean.

Blockchain

Blockchain, in simple words, is a type of distributed database, which means that data is not stored on a centralized server, and rather is stored as a distributed system. We will not be discussing distributed systems in this post, but I will surely create a post on the topic in the future. The data in the blockchain is stored in groups called blocks. Once the capacity of a block is filled, they are linked to the previous blocks which form a chain of these blocks called a blockchain.

The significance of this blockchain mechanism is that the data in these are immutable which makes the record stored in these authentic.

Blockchain Networks

Now in order to give equal control and access to this blockchain, there needs to be a copy of this blockchain on all the nodes, and these nodes together form the blockchain network. The major thing to ensure is that all blockchains have the same data and no one manipulates adding fake data to the chain. To ensure this, the algorithms developed are called blockchain consensus, and they make these networks almost impossible to corrupt.

Smart Contracts

Smart contracts can be understood as programs stored on any blockchain network which get executed automatically whenever the terms present in the contract are met. For example, I can create a smart contract that helps me store a variable and get this variable. I can deploy it on any blockchain of my choice, and depending on the chain I choose, I will have to pay a gas fee for making changes to this blockchain. Once I do that, the contract will be registered on the chain, and anyone with access to this can update or read this variable.

Tokens

Tokens are assets developed on existing blockchain networks that have a limited total supply on the chain. These are often confused as being the same as cryptocurrencies, but even though they share many similarities, they differ as the cryptocurrencies are native to the blockchain, which means that each cryptocurrency has its own blockchain network. At the same time, the token is created on an existing blockchain network. For example, I can declare a token named “DToken” and limit its supply to 50 tokens, and publish it on the blockchain network of my choice. Then a record on the blockchain has been created that states that I own 50 “DTokens”. Now, initially, Tokens do not have a monetary value, but for the sake of this example, let us say that I manage to create a demand for my token in the market, and it gets valued at 6 USD per token, so now I can trade this token for different items like I can purchase two 3$ coffees and pay them using my token, given that the person wants to trade my token at that price. Cryptocurrencies work in a very similar way.

Web 3.0 is the revolutionized web because of incorporating the concepts of Tokens and Blockchain Networks. This helps add a feature of ownership to any data present on the web. The consensus of these networks also helps ensure the authenticity of this data, promoting decentralization to a great extent. This has helped develop many new concepts such as NFTs and DeFi(Decentralized Finance).

Photo by Andrey Metelev on Unsplash

NFTs (Non-Fungible Token)

NFTs are tokens that follow the ERC 721 standard. Unlike other tokens, NFTs are Non-Fungible which means each NFT is unique and cannot be copied or mutated. Just like any other token, we can create digital assets as NFTs, and trade them for other tokens or cryptocurrencies.

Let us see how this works and how NFTs are different from other tokens and cryptocurrencies.

So, I create a digital asset, let us say a small image of size 1 kb. Now if I publish this asset as a token, and limit the supply to 5 tokens, each token is going to be identical to the other, meaning each token will be equivalent to the other. On the other hand, if I mint this asset as an NFT, and I mint 5 of these NFTs, each of them will have a unique token id, which can be used to track its ownership details. If I trade an NFT in the future, any person with access to blockchain data can track the original owner of that particular NFT back to me. This is what makes these Non Fungible.

Conclusion

Web 3.0 and Decentralization are getting more and more popular every day. Big giants like Meta, Google, and Apple are getting involved in the development of Metaverse, which is a super application of this Web 3.0. This has the potential to lead to a completely digital world, where all the transactions and assets will be stored on blockchain networks. It can digitalize the whole financial system of the world, and it is already progressing in that direction very quickly.

I hope you enjoyed this article, to stay updated subscribe to my newsletter on my official blog site: Thinkfeed.

Unsupervised Machine Learning

Divyessh Maheshwari — Thu, 09 Jun 2022 06:43:48 +0000

Have you ever wondered how different things can be segmented through machine learning? Or how do recommendation models like Amazon detect similarities among different purchases? Or how can we process natural language for spam filters or chatbots?

In this post, we are going to understand what unsupervised learning is and how it works on a mathematical level. We will take a look at different types of unsupervised learning and analyze their implementations. We will also go through different algorithms that are commonly used as well. There are no programming pre-requisites for this, although to better understand the content in this post, I would recommend you to go through my previous articles on Machine Learning and AI.

Introduction

In supervised learning, the data available to us is generally well defined and labeled, which means we know what different features (input variables) are present, and we can find correlations among them to determine the outcome (response). But in the case of unsupervised learning, the data we receive is generally not well labeled and hence consists of more randomness or makes it difficult for us to predict or determine the response.

Let us understand what we mean when by not well-defined data. As referenced in the previous post on supervised learning, the sales data of a product has many defined factors like the number of impressions (views), traffic distribution (where the customer comes from), and so on.

Now if I have to develop a model that detects the sentiment of a tweet and classifies that into one of the three categories — positive, negative, and neutral, we will require a lot of Twitter data (tweets) and their sentiment, which might be divided manually or through a different program. To use this data and train a model with it, we need to use Unsupervised Learning, because we don't have any defined data, as the different tweets would contain a lot of randomnesses, and there are no labeled factors as well, we will just need to extract information from the tweets and find a pattern among it.

On basis of different use cases, we can classify Unsupervised Learning into two types, association, and clustering.

Association

Association rule learning is a rule-based machine learning method for discovering interesting relations between variables in large databases. ~ Wikipedia

Association algorithms can also be called mining algorithms. In mining algorithms, we analyze the variables and find their correlation with each other. As a very popular example of this, you can think of a database consisting of sales in an online grocery store. So, that database contains many invoices or transactions, and those transactions would contain the number of different items that were purchased and their quantity. Now what we will try to do is, we will try to find relations among different products, so that we can create a recommendation system. Let us say that a common pattern was observed: people who buy Onions and Bread also buy Cheese. So next time, when a person buys onion and bread together, we can give them the recommendation to buy cheese as well.

We find these patterns by performing some mathematical calculations on the data and using the processed data to find the probability of different items that can be purchased after adding certain items to the cart. Let us take a very brief look at how this is done.

Let us say we are going to build a recommendation system for groceries and there are just 5 products that we are selling-Milk, Egg, Flour, Cheese, and Onions.

Now let us create a small data of transactions as well.

+----+------------------+--------------------------+
| | inovice_number | items_purchased |
|----+------------------+--------------------------|
| 0 | 1 | Milk, Egg, Floor |
| 1 | 2 | Milk, Cheese, Onions |
| 2 | 3 | Egg, Floor |
| 3 | 4 | Cheese, Onions |
| 4 | 5 | Egg, Milk, Floor, Cheese |
+----+------------------+--------------------------+

We would also need to learn about a few terms such as support, confidence, and lift, to understand the math behind it.

Support: frequency of items appearing per transaction

So for milk, the support would be 3/5 or 0.6 since milk appears in 3 transactions out of 5. Similarly, support for milk+cheese would be 2/5 or 0.4.

Confidence: frequency of the relation/rule per transaction

So for milk and cheese, if we try to find confidence for milk -> cheese, it would come out as support of milk and cheese divided by the support of milk.

Confidence(milk->cheese) = support(milk+cheese)/support(milk)
                         = 0.4/0.6 = 0.67

Confidence(cheese->onions) = support(cheese+onions)/support(cheese)
                         = 0.4/0.6 = 0.67

Confidence gives us a good idea of how often people buy a product, after previously buying a product.

Lift: Define the strength of the rule/relation

So for milk and cheese, if we try to find the lift for milk -> cheese, it would come out as support of milk and cheese divided by the support of milk into support of cheese.

Lift(milk->cheese) = support(milk+cheese)/(support(milk)*support(cheese))
                         = 0.4/(0.6*0.6) = 1.11

Now the way we determine the strength from the lift is as follows.

lift > 1 : products are related ( greater the lift, stronger the relation )

lift < 1 : products are negatively related, meaning people dont like to buy these together

lift = 1 : products are unrelated

So when we build the recommendation system, we compute lifts for different products and on the basis of their strength, recommend the products to the users.

I can understand if that was way too much maths, but it gives you the core idea of how the association algorithms work.

One of the most popular association algorithms is Apriori which is presently used worldwide to build recommendation engines and sales analysis.

Clustering

Clustering is the task of dividing the population or data points into a number of groups such that data points in the same groups are more similar to other data points in the same group than those in other groups. ~ AnalyticsVidhya

In these algorithms, we try to identify similarities among the data given to us on basis of many available factors, and then group that data into clusters, where each cluster contains data that has more similarity to the data from the same cluster than other clusters.

Some of its common use cases are recommendation systems, customer segmentation, and anomaly detection. For example, if we again have to create a recommendation system for the online grocery store, and we have a lot of customer data, which includes customer details, and their purchases, we can group these customers together into clusters on basis of their purchase patterns. And once, we have clusters of customers with similar patterns, we can now recommend customers from the same clusters of products that we found other customers in the cluster liked or purchased.

This is generally less complex in comparison to the classification algorithms we studied in supervised learning. Although, many people find it hard to distinguish the basic concepts of the classification and clustering algorithms. So, the major difference among these is, that the classification algorithm being a supervised learning algorithm requires well-labeled training data, and uses different algorithms to find a pattern among the variables of training data, and use those to classify, on the other hand, clustering does not require any training data since it uses statistical algorithms or models to find similarities in the data, and divide them into clusters.

In general, when applying clustering algorithms, we experiment with the number of clusters we want the data to be divided in, and the value that best suits us is used for test cases, and other uses later. For example, the same set of data can be divided into two clusters, as well as three clusters as shown in the figure. Although the three clusters arrangement would be preferable to us since it clusters the data into more accurate and better clusters.

The common algorithms used in clustering are K Means Clustering, BIRCH Algorithm, and Gaussian Mixture Model.

Conclusion

Unsupervised Learning in general is based more on the statistical implementation to find patterns and relations while supervised learning uses the pre-defined algorithms created on the basis of the training data. On basis of our use case, we can use either of these techniques and optimize them to give us more accurate results more efficiently.

I hope you enjoyed reading the post. To follow up with the new articles, do subscribe to the newsletter on my official blog: Thinkfeed.

Supervised Machine Learning

Divyessh Maheshwari — Mon, 16 May 2022 15:17:04 +0000

Have you ever wondered how the defined data gets processed and predictions are drawn? or how we classify the data or understand the patterns from a dataset?

In this post, we will learn about supervised machine learning, and how it works on a mathematical level. We will explore different types of supervised learning and try to analyze their practical implementation. As usual, you do not need any programming experience to understand this, although I really recommend you to have a look at one of the previous blog posts “Artificial Intelligence and Machine Learning” which can be used to better understand the concepts I explain here.

Introduction

In machine learning, we use data and analyze it to determine a pattern among it and use that to draw predictions or classifications. When this data is well defined and labeled, which means when we understand what each of the input parameters represents, and we use particular algorithms to train our model with it, it is classified as Supervised Learning.

For example, if I have a dataset of different real estate properties, which contains many different parameters including the location, plot area, number of bedrooms, bathrooms, and stories, and some other factors as well as the price of the house. I can use Supervised Learning to create an ML model that when provided with these parameters, can predict the price of the property. So, once my model would be prepared, if I would provide it with the location of the house, the area, as well as the other details as input, then it can output me the price of that house.

On the basis of the use case, we can divide Supervised Learning into two types, regressions, and classification.

Regressions

“In statistical modeling, regression analysis is a set of statistical processes for estimating the relationships between a dependent variable (often called the ‘outcome’ or ‘response’ variable) and one or more independent variables (often called ‘predictors’, ‘covariates’, ‘explanatory variables’ or ‘features’).~https://en.wikipedia.org/wiki/Regression_analysis”

In the definition above, you can correlate the independent variables as our input parameters ( more commonly called features ), and the dependent variables as our output ( what we want to predict ).

So in the regression, the model tries to find patterns or a relation between the features and the outcome. Just like in our example, the area of the house would have a strong relation ( very correlated ) with the price of the house, since as the area increases, the price of the house also increases. Whereas the features like latitude and longitude may not show a very direct strong relation.

The model trains on the data we provide to it and using that, it comes up with particular weights for each feature, and the higher the weight, the higher the correlation between the feature and outcome. Then it can develop some sort of algorithm with those weights that can predict near accurate or accurate outcomes.

The accuracy of the regressions is generally tested through loss functions that calculate the difference between the predictions of the model and the real outcomes of test data. Depending on the choice of our matrics, calculate the loss.

If you are interested in learning about how the models are evaluated in more depth, then do comment on this post, and I will write an article explaining several kinds of evaluation metrics.

The most common use cases of the regression models include predicting sales of a product, market research, forecasting financial data, and so on.

Classifications

“In statistics, classification is the problem of identifying which of a set of categories (sub-populations) an observation (or observations) belongs to. ~https://en.wikipedia.org/wiki/Statistical_classification”

As the name suggests, in classification algorithms, the model tries to predict which category the input will belong to. One of the most typical use cases for classification is the email spam filter since we use the email data, which has many defined features like number of words, domain name, number of images, number of links, and many more, and we try to predict if the email is spam or not using those features.

In Classification algorithms, the model tries to divide each of the features in the data into different clusters or sections and tries to find a pattern in the clusters with the outcome. Like, in our email spam detection, the number of links can be divided in a way like, if the number of links is greater than 6 then most of the time, the email was spam, while if the number of links was less than 4, then most of the times the email was not spammed ( or ham ). So we divided the “number of links” feature into three clusters (categories ), less than 4, greater than 6, and the remainder ( {4, 5, 6} links ), and determined a correlation with our classification.

When such clustering is developed for all features, all of those are again assigned with weights and are further used to predict the outcome when tested.

The accuracy of the model or the evaluation of the classification model can again be evaluated using many different matrics, the most popular one being precision and recall, where we see how many classifications were predicted correctly, while how many were predicted incorrectly, and determine the score on basis of that.

The common use cases of classification on the other hand include classifying plant species, image detections, recommendation models, and many more.

Neural Networks ( Additional Topic )

Now this topic has a little bit more advanced concept, but I just want to give you a feel of what we are going to be learning, so feel free to skip this if you want, or else be ready to have some fun going deep in statistics.

Neural Networks are a part of deep learning, where the algorithm consists of forming a neuron-like pattern, where several layers are created in order to process the features and since there is the presence of multiple layers, there is also the presence of multiple weights.

Now imagine, each and every feature, going through so much processing to determine a pattern that is not only limited to the individual correlation of features with the outcome but processing all the features together and reaching an algorithm so complex that the model becomes almost as accurate as humans. The image below should give you a small idea of how this is implemented, and we will be studying these in great detail in the upcoming articles.

Conclusion

Regression and Precision in themselves consist of many different algorithms and types, and I will surely be making an article dedicated to exploring different types of regressions and classification algorithms.

I hope you enjoyed reading the post. To follow up with the new articles, do subscribe to the newsletter on my official blog: Thinkfeed.

Serverless Architecture

Divyessh Maheshwari — Thu, 05 May 2022 05:58:00 +0000

Have you ever wondered how the AWS Lambda functions or Cloud Functions by Google or other Functions as a Service work? or is serverless architecture really serverless? or how a third-party service manages your complete backend service code in containers?

In this post, we are going to understand the serverless architecture and how it actually works. We are also going to explore the answer to the question, is serverless architecture really serverless? and we are also going to explore how this architecture is provided in handy to us by different cloud providers like AWS, GCP, and many others. We will also take a look at what is FaaS (Function as a Service) and how we can actually implement a FaaS using any of these cloud providers.

To have a better understanding of this article, I really recommend you to read one of my previous articles on “Basics of Web App Architecture” and if you want to know more about the cloud, check out “Cloud Computing”. In case you already have an understanding of how the web works and what are some different commonly used web architectures, feel free to move ahead.

Introduction

Serverless architecture is one of the modern forms of web architecture where we deploy our application that incorporates the third-party services that manage the server configurations, load balancing, and other server-side tasks.

The distribution of the services that need to be provided in the backend is somewhat similar to the microservice architecture. Just like in microservice architecture, the backend is distributed among different services, each of them being used for different tasks, in serverless architecture (especially in FaaS), the backend is divided into different tasks, and each of these tasks can be implemented as a separate and individual function. These functions are then served through an API over different endpoints and can be accessed by the frontend.

This is what a basic application built over serverless architecture would look like.

Function as a Service (FaaS)

FaaS is the most popular and newer implementation of serverless architecture. In this implementation, we create multiple functions to perform each task which is an independent part of the backend server logic or service.

Let us consider the case of AWS Lambda functions to understand this.

“AWS Lambda is a serverless, event-driven compute service that lets you run code for virtually any type of application or backend service without provisioning or managing servers.” ~ https://aws.amazon.com/lambda/

The lambda functions do not require you to manage their deployment or load balancing yourself, and rather the infrastructure of AWS manages that for you. These functions are deployed as containers on the server, where each server can have multiple function instances running on it depending upon the user traffic and computation power required by the functions.

These are very easy to build and AWS allows you to integrate many other services like DynamoDB ( AWS NoSQL Database ), S3 (Storage Service), SQS (Queuing System), SNS (Notification System), and so on.

Lambda functions are not only easy to build and deploy but they are also very cost effective since you are charged on the basis of computation time used by the function instances, which is only accounted for when the function is called. AWS even provides you with about 1 million free requests and 3.2 million seconds of compute time per month.

Use Cases and Limitations

Now let us have a look at some different cases where we can actually use the serverless functions and how it will be beneficial for us, or what can be some of the limitations.

Data Processing or Collection

Lambda functions can be great for the lightweight processing of data for an application since they generally do not take much compute time and it will be very easy to store the data using other services. You can create a very small function to just store the form data of a web application submitted by a user to DynamoDB (Database), and later use that data for self-analysis.

Backend Logic (CLient Heavy Application)

Since for the client-heavy applications, the backend is very lightweight and has high latency, the backend or server side logic is very easy to be implemented in the lambda functions and since you do not have to manage your deployment, it becomes somewhat similar to Backend as a Service (BaaS), which basically are platforms that provide you with backend without you requiring to manage it.

You will not need to know concepts like the CI/CD pipelines or other DevOps-related concepts to scale or deploy your logic.

The client heavy applications might include a personal blog, a designing application, or any other application where a good part of logic is implemented over frontend.

Limitations

When a lambda function is left idle for some time, they are required to go through a cold start which means it takes time to start and respond. So if a quick response is needed, and there can be an expected low traffic period, then lambda functions are not very suitable. Also when the applications have a heavy backend, which would require a lot of computation power, then we should ideally not use lambda functions, since they are not efficient in those use cases.

For example, you should not use Lambda functions for MLOps ( Deployment of ML Models ), since they can require very high computation and have high latency.

Conclusion

Serverless architecture is a great framework to be implemented in your applications. With its many advantages, it also has some limitations, and hence as a best practice, you should not depend solely on lambda functions for your backend, and instead develop a hybrid backend, with the lightweight functions getting implemented by the serverless functions. This will not only reduce loads of you for management of the server logic but also let you focus more on the main product or service that you intend to provide.

Now that you understand the basic function of serverless architectures, we can move on to understanding more details of cloud computing and architecture. We can explore other forms of architecture as well as understand how each of these cloud functionality works individually.

To support my articles, you can check out my official blog at Thinkfeed and subscribe to the newsletter over there so that you never miss a post and keep yourself updated. Enjoy Reading!

This article was originally published on https://thinkfeed.divyesshm.com/post/serverless-architecture

Basics of Web App Architecture

Divyessh Maheshwari — Mon, 18 Apr 2022 08:35:22 +0000

Have you ever wondered how the data flow takes place in all the web applications? or how the distributed systems works? or how the large applications with hundreds of services are managed individually?

In present days, even the minimal viable products or proofs of work can also have a combination of many different services and use several different integrations. Building a web application from scratch that includes all those services also comes with an important question of which architecture to use. First, let us understand how web applications work and how the client and server interact. During this journey, we are going to understand how we can build a basic application that has an authentication service, a storage service, a database, and a basic UI.

It is recommended to check out the “ backend development ” post and the “ application programming interface ” post in order to have better clarity of the concepts getting covered in the following post.

Introduction

Our app will be an image-sharing platform, where one person can only share a single image on their profile, and he can change that image at any time, and to view others' profiles or to edit their profile, they would also need to log in or signup to the application.

So the components of our application will include an authentication and authorization service which can log in the user, a database service where the user data like login credentials and the path of the image they upload can be stored, a storage service where the image itself uploaded by the user can be stored and of course a user interface for the user to do all that.

Now to do all that the most basic data flow can be shown in the image.

The UI requests the webserver for different services, and accordingly, the webserver queries the database or requests the storage and returns a response to the UI back.

Now let us learn what web app architectures are there and see how this data flow changes as we use different architectures.

Web App Architecture

Web application architecture is a blueprint of simultaneous interactions between components, databases, middleware systems, user interfaces, and servers in an application. It can also be described as the layout that logically defines the connection between the server and client-side for a better web experience. ~www.simform.com

Before mapping different methods of client-server interactions, let us also take a look at the general data flow through different layers in web architecture.

Data Flow through Layers of Web Architecture

There are 4 layers through which the data transfer generally takes place.

Presentation Layer: It consists of the frontend ( The UI Components ) of the application. It is generally built with HTML, CSS, and JS frameworks, and it enables the client to communicate with the interface.

Business Layer: It consists of the business logic, validations, and utilities of the application. It is responsible for routing, error handling, data handling, and other backend processes.

Data Access Layer: It works as an interface between the business logic and the database so that the business logic does not have direct access to the database. It handles and manages the data stored in the database and is responsible for all the CRUD requests.

Database Layer ( Data Service Layer ): It secures the data stored in the database by the user as well as the server by providing an additional option of protection for the data.

Different Web Architectures

We can divide the different architectures on the basis of their data flow patterns in frontend and backend respectively. We are not going to discuss all these architectures in complete detail right now, rather I am going to provide brief info about each of them, and we will discuss these architectures in detail in the dedicated future posts.

On the basis of different data flow patterns in the frontend, the websites, in general, can be classified into Legacy HTML apps ( Traditional ), Widget Web Apps, and Single Page Applications (SPA).

Legacy HTML Web Apps

These are the most traditional form of web architecture. In the applications using this architecture, the client or browser sends a request to the server, that consists of the web page construction logic, and receives a complete HTML page in response. To update the page, the user is required to reload the page which again sends a request to the server, and generates a new HTML response again from scratch.

This architecture is generally considered good for static web pages so that user does not need to reload again and again.

If we use this to build the frontend of our Image Sharing Application, then the data flow would look like this.

Widget Web Apps

In this type of web architecture, the client has different components called widgets, which receive data by sending queries to the web services and displaying the data without reloading the entire page. These are very complex to build but have certain advantages like being dynamic, mobile-friendly, and very responsive.

Single Page Applications

These are one of the most modern forms of architecture that are widely used throughout the industry for all purposes. We can build both dynamic as well as static programs in this architecture and both have very good efficiency.

In this type of architecture, only a single web page is downloaded at once and the JavaScript used in the page constantly communicates with the server.

Most of the modern and popular JavaScript and TypeScript frameworks like Vue, React, and Angular are used to build the SPAs.

On the basis of the backend server configuration and data transfer patterns, the web app architectures can be classified as Monolithic, Microservices, and Serverless.

Monolithic

This is again the most traditional architecture used. In the applications using this type of architecture, the complete backend business logic and database logic are combined into a single large codebase. These applications initially are easy to build and simple to test, but they are very hard to scale, since you can only scale them up vertically, and the flexibility and agility in the code are highly reduced.

Microservices

This is a more modern solution to the problems that were raised in a monolithic architecture. These applications have a distributed backend, where each service has its own separate codebase and have a great scope of scalability and agility. Also since one service is not affected by another, it increases the flexibility of the application to work on updates without compromising other services.

Serverless Architecture

“Serverless architecture is an approach to software design that allows developers to build and run services without having to manage the underlying infrastructure. Developers can write and deploy code, while a cloud provider provisions servers to run their applications, databases, and storage systems at any scale.” ~ www.datadoghq.com

This architecture is very highly scalable since all the services are created as separate entities as well as the number of instances of the entities can be modified as per the traffic.

For example, FaaS ( Function as a Service ) is one of the most popular examples of serverless architecture. So if we have to create a small data processing API, we would just need to create an AWS Lambda function, connect it with the API Gateway, and that's all. It will be deployed without any excessive costs or management required.

Following is an AWS-specific example.

Conclusion

Now that we have a basic understanding of what web app architectures are, and the different types we can use to build our application, we can study the specific architectures in detail. For instance, the future posts will consist of a detailed explanation of serverless architecture and some other topics which would require you to know these basics as prerequisites.

I hope you enjoyed reading the post. This was originally published at my official blog thinkfeed.divyesshm.com. Do visit there and check out the articles. I will be posting some of the posts exclusively on that site, so do follow up and subscribe to the newsletter.

Application Programming Interface

Divyessh Maheshwari — Thu, 27 Jan 2022 19:58:39 +0000

Have you ever wondered how can the frontend and the backend of an application communicate with each other? Or how do different apps extract data of other apps and websites like YouTube or Twitter?

All the applications that consist of a separate backend and frontend, have to use APIs ( Application Programming Interface ) to communicate with each other. The data transfer, logic processing, and other triggers take place through APIs only. If you want to know more about the backend of applications, you can check out this post by clicking here.

Today in this post, I am going to talk about APIs, and how they are used for different purposes and in different architectures. So let us begin.

Suppose you are a data scientist or a machine learning engineer, and you want to create a program to perform analysis over some youtube videos and channels, for example, you might want to predict the number of likes a video will get in a given time duration depending upon the number of subscribers of channel and data collected from previously posted videos. So in order to do that you will need a lot of data on the videos, as well as different channels.

Now in order to get so much data in bulk, you will be required to use the YouTube API, which will provide you a direct medium or bridge to all this data, and then you can simply extract it as per your use.

On the other hand, if you were not given access to this API, you would have done the same task manually or by creating a scrapper, and either of those methods would have been very inefficient and could have given trouble many times.

Types Of APIs

There are four different types of APIs depending upon their usage and accessibility. Let us study these in brief.

Open/Public APIs

These are APIs that can be accessed after getting a subscription or registration by developers. They have a low restriction and are generally used to access public data. Twitter API is one of the examples of public API since it enables users to get access to some public data which can then be used for different purposes such as educational projects, analysis, trends studies, etc.

Partner APIs

These are similar to Open APIs, although these are more restricted and are shared with limited partners after getting subscribed and approved by respective organizations.

Internal/Private APIs

These are not exposed publicly and are used internally among the system for different purposes like communication within the applications or system. They can also act as mediums for the frontend to communicate with the backend.

Composite APIs

These are generally a combination of multiple API endpoints and can be for the public as well as for internal use.

API Architectures

APIs also have a set of rules, structure, and format which defines their usage and operations. These are called its architectures or protocols. There are majorly three different protocols that we are going to discuss in this post.

Remote Procedure Calls (RPC)

“In distributed computing, a remote procedure call is when a computer program causes a procedure to execute in a different address space, which is coded as if it were a normal procedure call, without the programmer explicitly coding the details for the remote interaction. ~ Wikipedia”

These were one of the oldest API calls and are generally used less in present-day applications.

Simple Object Access Protocol/Service-oriented architecture (SOAP/SOA)

“SOAP is a messaging protocol specification for exchanging structured information in the implementation of web services in computer networks. ~Wikipedia”

These were structured API protocols but were very rigid and complex, and hence are not used often by emerging systems, but still used in many legacy applications.

Representational State Transfer (REST)

This is presently the most widely used architecture for APIs. It is primarily used in microservice architectures. It is highly flexible, cacheable, fast, efficient, and stateless architecture.

“REST has been employed throughout the software industry and is a widely accepted set of guidelines for creating stateless, reliable web APIs. A web API that obeys the REST constraints is informally described as RESTful. RESTful web APIs are typically loosely based on HTTP methods to access resources via URL-encoded parameters and the use of JSON or XML to transmit data. ~Wikipedia”

So, now let us also understand what is JSON or XML.

Data Interchange Format

Data is transferred on the web in majorly two forms: JSON or XML. This can be data sent as request payload as well as response.

JSON (JavaScript Object Notation)

“JSON is an open standard file format and data interchange format that uses human-readable text to store and transmit data objects consisting of attribute–value pairs and arrays. ~Wikipedia”

JSON is language independent and allows to sharing of key-value pairs, arrays, or objects during web calls. It is readable and accessible as JSON objects and doesn’t need any further. So for example, if I have created an API that takes 2 numbers as input, and returns their sum as output. Then following can be examples of input and output.

XML (Extensible Markup Language)

“Extensible Markup Language is a markup language and file format for storing, transmitting, and reconstructing arbitrary data. It defines a set of rules for encoding documents in a format that is both human-readable and machine-readable. ~Wikipedia”

XML data needs to be parsed before using and a similar example as above can be represented in XML as follows.

Conclusion

Now that we have a better understanding of APIs work, we can understand how these are created or implemented in different computer architectures. So to know more about the topic, just follow up with the post and stay tuned. I will keep adding different posts related to computer science concepts from a very fundamental point of view.

Cloud Computing

Divyessh Maheshwari — Mon, 03 Jan 2022 08:06:43 +0000

Have you ever wondered where are your emails stored? or how you can effortlessly switch mobiles and all your backed up data is transferred so smoothly? or what does it really mean by backed up data itself?

Even though you may not know it, but almost all of us are using the cloud in one way or another. If you have a Google or an Apple account, use Gmail, backup WhatsApp data, you are already on the cloud. Cloud Computing is such an essential part of our lives, that today without the cloud, we cannot make a search on Google, or shop on Amazon, or even use Skype.

Today, in this post, I am going to talk about cloud computing, and how is it being used by developers and engineers. So let us start.

“Cloud computing is the on-demand availability of computer system resources, especially data storage and computing power, without direct active management by the user. ~ Wikipedia”

Cloud Providers such as Amazon Web Services (AWS), Google Cloud Platform (GCP), Microsoft Azure Cloud, and many others, provide many different resources and services such as storage, computing, database, etc to users on an as-needed based. For example, suppose you have a Google Account, then you will already be registered on the Google One platform, which provides you with 15 GB of free space. This space is consumed by the mails we receive, the images and videos we back up, the media stored in the drive, the app data of different apps we use, and many others. Google also provides a cloud identity service, which we use whenever we sign in on a platform with Google. And not only Google, but Microsoft (Yahoo), and many other cloud providers, also provide similar services.

I hope you can imagine the scope of services that are provided to us by the cloud. These services are not limited to just general purpose use, these are provided to companies and organizations that use the cloud for various purposes including computing, database, storage, analytics, security, identity, and so many others.

Photo by Ian Battaglia on Unsplash

Cloud Services in Industries

Today, cloud computing is being used in almost every enterprise of any scale in every industry. Especially, it has a lot of advantages to small enterprises since, its setup is very quick, easy, and cost-effective. Without cloud computing, if you want to create and host your website, you will need to purchase your own physical server, use its storage, create and manage a database, manage the loads, and do a lot more work. But using cloud services, depending on the type of industry and use case, we don’t need any physical setup. We just need to register on a cloud-providing platforms, such as AWS, GCP, or Azure, and purchase different services as our need, like storage, database, computing, etc. and deploy our application. That’s all you need to do.

Let us now have a look at how the cloud is being utilized in different industries.

Education

Educational Institutes have adopted many cloud solutions to provide various services to both, the students and faculty. They store their huge student and faculty records on the cloud for complete safety and security. They use cloud platforms in order to stream live videos, conduct virtual meetings, and even upload assignments. There has been significant growth in cloud usage since the COVID 19 because due to lockdowns, physical classes were no longer being conducted. Hence the migration to the cloud for most of the tasks became very essential.

Finance

Since, in finance industries, there are a lot of data that needs to be stored or processed, or analyzed. Storing so much data on one local server can be very risky since, in case it crashes, there will be a huge loss faced by the organization, and if backups are created, it will be very costly to manage them. Hence using the cloud in order to store these data, was a very beneficial option as it provides very high security to your data. Another part was implementing artificial intelligence in capital markets. These AI programs could need very specific server configurations to be optimized and needed to be secure and fast. So hosting these models on the cloud was another great advantage for the industry.

Distributors and Retail Stores

Most of the distributors or retailers already use the cloud as accounting software and stock management systems. There can be a possibility that they can use CRM(Customer relationship management) software or probably eCommerce platforms. Many suppliers have recently started implementing affiliate marketing to their products, and it can be very beneficial for them if they implement these systems overcloud.

Healthcare

The healthcare industry puts requires a lot of patient safety and privacy. Many cloud providers provide HIPAA compliance to ensure proper treatment of patient information. Diagnosis Systems are also popularly being implemented in cloud. These implement artificial intelligence and predictive analytics to make medical diagnoses and recommend treatments.

IT Industries

Obviously, there is a huge implementation of the cloud in IT industries, as it is the backbone of many enterprises. Most IT solutions are cloud-based, such as all modern SaaS(Software as a Service) implementations are cloud-based. The outbreak of serverless architecture is also becoming a popular cloud feature. There can be an endless list of cloud implementations in IT industry.

Cloud Providers

Now let us have a brief introduction of the three major cloud providers: Amazon Web Services, Google Cloud, and Microsoft Azure.

Amazon Web Services

“ Amazon Web Services, Inc. ( AWS ) is a subsidiary of Amazon providing on-demand cloud computing platforms and APIs to individuals, companies, and governments, on a metered pay-as-you-go basis. These cloud computing web services provide a variety of basic abstract technical infrastructure and distributed computing building blocks and tools.~ Wikipedia”

AWS today has a market share in the cloud vendors of about 32% which is the highest among the vendors, and more than the combined market share of GCP and Azure.

It conducts many events throughout the year. You can view and attend these events over here.

Google Cloud

“ Google Cloud Platform ( GCP ), offered by Google, is a suite of cloud computing services that runs on the same infrastructure that Google uses internally for its end-user products, such as Google Search, Gmail, Google Drive, and YouTube. Alongside a set of management tools, it provides a series of modular cloud services including computing, data storage, data analytics and machine learning.~Wikipedia”

GCP has a market share of around 7% and has shown a growth rate of more than 45% in 2021. It also conducts a few events which you can view over here.

Microsoft Azure

“Microsoft Azure , often referred to as Azure, is a cloud computing service operated by Microsoft for application management via Microsoft-managed data centers. It provides software as a service (SaaS), platform as a service (PaaS) and infrastructure as a service (IaaS) and supports many different programming languages, tools, and frameworks, including both Microsoft-specific and third-party software and systems.~Wikipedia”

Azure has a market share of around 19% and has shown the highest growth rate of around 50% in the year 2021.

Azure conducts many events, conferences, and webinars. You can register or view them here.

Conclusion

Cloud is not a new conception, it has been active for a long time, although, it has been diversifying in a lot of industries and can be very beneficial for certain tasks. Like in case you want to experiment with starting your own website or SaaS, or maybe want to try to implement some IT solutions in your industry, it can be a great starting point.

To know more about the topic, just follow up with the post, and I will keep adding different posts related to computer science concepts, and try to keep them from a very fundamental point of view.

I hope that you enjoyed reading this post. If you would want me to create a post on any different topic, let me know in the comments section.

Artificial Intelligence and Machine Learning

Divyessh Maheshwari — Wed, 22 Dec 2021 05:18:51 +0000

Have you ever wondered how does the google voice assistant recognizes what we speak and convert it to text? Or how does Amazon gives us different and mostly accurate recommendations? Or maybe how gmail knows which email is a potential spam?

Many applications that we see around us uses Machine Learning or Data Science or Artificial Intelligence to provide us with different services. From Google Translate to Self Driving Cars, all the applications and software that tries to predict, classify or maybe even analyse some data uses Artificial Intelligence and Machine Learning.

In this post we are going to discuss what really is machine learning or artificial intelligence and how it actually works on the ground level. So let us begin by understanding what is Artificial Intelligence.

Artificial Intelligence

Artificial Intelligence until now has no exact definition and constantly gets redefined. But to understand it’s basis, we can define AI as something that is both autonomous and adaptive, which means, the program should be able to perform tasks by itself without requiring constant guidance as well as keep improving itself by learning from previous experiences.

For Example, in spam filter of Gmail, it detects the most of the spam emails without requiring us to guide it or reprogram it, and when we mark an email as spam, it automatically reinforces or updates its program in order to more efficiently and accurately detect spam mails in future. This is a typical example of AI.

Also, a very similar patter is followed in most of the recommendation systems, like the one in Amazon and Google News. From our previous activities, they determine our interests and then show products or contents related to it.

Now if you are a Benedict Cumberbatch fan, you might have seen the movie “The Imitation Game” which also holds a 8.0 IMDB Rating, where Benedict Cumberbatch has starred as Alan Turning (Do check the footnotes for more info).

Alan Mathison Turing

Alan Turing was one of the greatest mathematicians, who is widely considered to be the father of theoretical computer science and artificial intelligence. He was the first person to theoretically and mathematically explore the possibilities and concepts of Artificial Intelligence. He published the paper named “Computing Machinery and Intelligence”. It was the first one to introduce the concept of AI and the Turing Test, which can be stated as the first serious proposal in the philosophy of AI.

“The ACM A. M. Turing Award is an annual prize given by the Association for Computing Machinery (ACM) for contributions ‘of lasting and major technical importance to the computer field’. It is generally recognized as the highest distinction in computer science and is known as or often referred to as the ‘Nobel Prize of Computing’. ~ Wikipedia”

John McCarthy

John McCarthy, also known as the Father of AI, has given some of the most important contributions in the field of AI. He developer the first programming language for AI programming LISP, that is still widely used in most of the AI based applications.

He was awarded many honours for his works, including ACM Turing Award, Kyoto Prize, National Medal of Science and Benjamin Franklin Medal.

Machine Learning

There is a very small difference between the term Machine Learning (ML) and Artificial Intelligence(AI). Now in some cases there is a possibility that AI might not be adaptive and only autonomous.

For example in a chatbot which works on a large set of human defined rules. Let us understand this by thinking that it is a health test chatbot, which can test our mental health. So, we define some set of questions and possible answers in chatbot, like a question can be “Do you have fever?” and the possible keywords in the answer can be “yes”, “no”, “a little” etc. Now if you observe carefully, this chatbot is autonomous, which means that it will ask questions by itself, collect answer by itself and also determine a result by itself, but it is not adaptive which means that it will not improve by itself.

This chatbot will be consider as an AI program but not as a ML program, because adaptability is one of the key requirements of Machine Learning.

Today ML has a variety of algorithms, which are used for majorly two purposes : Classification and Prediction, and we can categorise different ML applications into three categories : Supervised Learning, Unsupervised Learning and Reinforcement Learning.

The Basic Idea behind ML

Even the most random data points filled with chaos contains a patter on symmetry in them. In the real world, for most of our practical purposes, the classification or prediction of anything can be broken down into a set of objective and subjective categories, where the objective ones can be used to detect patterns in data.

Image for the email spam filter

For Example, let us bring back our email spam filter. Now to know if an email is a spam or not, there are several factors (subjective and objective), but if we start picking the objective ones, they would be : number of words, country of origin, number of links, number of some common spam words and a few more. Now, we can create a dataset of emails, and there objective factors, and label them as spam or not spam. Once we have built a dataset of a few thousand emails, we can create a ML model to find patterns and classify the mails. Fortunately, most of the algorithms are nowadays prebuilt, and we can just configure them as per our need. Like in this case, I am going to use a algorithm named logistics regression. You can read more about it over here.).

It is often used for binary classifications. Now since we would like to train our model (feed it data so that it can recognise patters), we would also like to keep some data for testing our models performance after training. So, we randomly split the data in an appropriate ratio, like in this case, we can reserve 20% data for testing. Once we have trained the model, and tested it, we will have our metrics that determine accuracy and efficiency of model.

Now it is alright if the above paragraph was not completely clear to you, since it does contain some technical details, but I hope you have got a basic idea for how exactly the ML models work. Although, if you want a more technical or detailed explanation for spam detection using logistic regressions, do checkout this post.

Conclusion

Now that we have a basic understanding of artificial intelligence and machine learning, we can understand about how different algorithms work and how they can be optimised using different functions. So, to know more about the topic, just follow up with the post, and I will keep adding different posts related to computer science concepts, from a very fundamental point of view.

I hope that you enjoyed reading this post. If you would want me to create a post on any different topic, let me know in the comments section.

The Imitation Game : Wikipedia / IMDB