DEV Community: Prazwal Ks

RealTime Detection

Prazwal Ks — Fri, 06 May 2022 16:19:34 +0000

Real-time detection uses streaming data by comparing previously seen data points to the last data point to determine if your latest one is an anomaly. This operation generates a model using the data points you send, and determines if the target (current) point is an anomaly. By calling the service with each new data point you generate, you can monitor your data as it's created.

Real-time detection example
Consider a scenario in the carbonated beverage industry where real-time anomaly detection may be useful. The carbon dioxide added to soft drinks during the bottling or canning process needs to stay in a specific temperature range.

Bottling systems use a device known as a carbo-cooler to achieve the refrigeration of the product for this process. If the temperature goes too low, the product will freeze in the carbo-cooler. If the temperature is too warm, the carbon dioxide will not adhere properly. Either situation results in a product batch that cannot be sold to customers.

This carbonated beverage scenario is an example of where you could use streaming detection for real-time decision making. It could be tied into an application that controls the bottling line equipment. You may use it to feed displays that depict the system temperatures for the quality control station. A service technician may also use it to identify equipment failure potential and servicing needs.

You can use the Anomaly Detector service to create a monitoring application configured with the above criteria to perform real-time temperature monitoring. You can perform anomaly detection using both streaming and batch detection techniques. Streaming detection is most useful for monitoring critical storage requirements that must be acted on immediately. Sensors will monitor the temperature inside the compartment and send these readings to your application or an event hub on Azure. Anomaly Detector will evaluate the streaming data points and determine if a point is an anomaly.

Text Analytics

Prazwal Ks — Wed, 20 Apr 2022 16:27:38 +0000

Text Analytics Techniques
Text analytics is a process where an artificial intelligence (AI) algorithm, running on a computer, evaluates these same attributes in text, to determine specific insights. A person will typically rely on their own experiences and knowledge to achieve the insights. A computer must be provided with similar knowledge to be able to perform the task. There are some commonly used techniques that can be used to build software to analyze text, including:

Statistical analysis of terms used in the text. For example, removing common "stop words" (words like "the" or "a", which reveal little semantic information about the text), and performing frequency analysis of the remaining words (counting how often each word appears) can provide clues about the main subject of the text.
Extending frequency analysis to multi-term phrases, commonly known as N-grams (a two-word phrase is a bi-gram, a three-word phrase is a tri-gram, and so on).
Applying stemming or lemmatization algorithms to normalize words before counting them - for example, so that words like "power", "powered", and "powerful" are interpreted as being the same word.
Applying linguistic structure rules to analyze sentences - for example, breaking down sentences into tree-like structures such as a noun phrase, which itself contains nouns, verbs, adjectives, and so on.
Encoding words or terms as numeric features that can be used to train a machine learning model. For example, to classify a text document based on the terms it contains. This technique is often used to perform sentiment analysis, in which a document is classified as positive or negative.
Creating vectorized models that capture semantic relationships between words by assigning them to locations in n-dimensional space. This modeling technique might, for example, assign values to the words "flower" and "plant" that locate them close to one another, while "skateboard" might be given a value that positions it much further away.
While these techniques can be used to great effect, programming them can be complex. In Microsoft Azure, the Language cognitive service can help simplify application development by using pre-trained models that can:

Determine the language of a document or text (for example, French or English).
Perform sentiment analysis on text to determine a positive or negative sentiment.
Extract key phrases from text that might indicate its main talking points.
Identify and categorize entities in the text. Entities can be people, places, organizations, or even everyday items such as dates, times, quantities, and so on.
In this module, you'll explore some of these capabilities and gain an understanding of how you might apply them to applications such as:

A social media feed analyzer to detect sentiment around a political campaign or a product in market.
A document search application that extracts key phrases to help summarize the main subject matter of documents in a catalog.
A tool to extract brand information or company names from documents or other text for identification purposes.
These examples are just a small sample of the many areas that the Language service can help with text analytics.

Regression

Prazwal Ks — Tue, 19 Apr 2022 15:01:08 +0000

Regression is a form of machine learning that is used to predict a numeric label based on an item's features. For example, an automobile sales company might use the characteristics of a car (such as engine size, number of seats, mileage, and so on) to predict its likely selling price. In this case, the characteristics of the car are the features, and the selling price is the label.

Regression is an example of a supervised machine learning technique in which you train a model using data that includes both the features and known values for the label, so that the model learns to fit the feature combinations to the label. Then, after training has been completed, you can use the trained model to predict labels for new items for which the label is unknown.

DataBricks,Spark Summary

Prazwal Ks — Thu, 14 Apr 2022 14:32:59 +0000

Databricks was founded by the creators of Apache Spark, Delta Lake, and MLflow.

Over 2000 global companies use the Databricks platform across big data & machine learning lifecycle.

Databricks Vision is to Accelerate innovation by unifying data science, data engineering, and business.

Databricks offers
1.Databricks Workspace - Interactive Data Science & Collaboration
2.Databricks Workflows - Production Jobs & Workflow Automation
Databricks Runtime
3.Databricks I/O (DBIO) - Optimized Data Access Layer
4.Databricks Serverless - Fully Managed Auto-Tuning Platform
5.Databricks Enterprise Security (DBES) - End-To-End Security & Compliance

Apache Spark
Spark is a unified processing engine that can analyze big data using SQL, machine learning, graph processing, or real-time stream analysis

Spark Engine
At its core is the Spark Engine.
The DataFrames API provides an abstraction above Resilient Distributed Datasets (RDDs) while simultaneously improving performance 5-20x over traditional RDDs with its Catalyst Optimizer.
Spark ML provides high quality and finely tuned machine learning algorithms for processing big data.
The Graph processing API gives us an easily approachable API for modeling pairwise relationships between people, objects, or nodes in a network.
The Streaming APIs give us End-to-End Fault Tolerance, with Exactly-Once semantics, and the possibility for sub-millisecond latency.
And it all works together seamlessly!

Azure Data Bricks

Prazwal Ks — Thu, 14 Apr 2022 14:26:36 +0000

Azure Databricks is a fully managed, cloud-based Big Data and Machine Learning platform, which empowers developers to accelerate AI and innovation by simplifying the process of building enterprise-grade production data applications. Built as a joint effort by the team that started Apache Spark and Microsoft, Azure Databricks provides data science and engineering teams with a single platform for Big Data processing and Machine Learning.

By combining the power of Databricks, an end-to-end, managed Apache Spark platform optimized for the cloud, with the enterprise scale and security of Microsoft's Azure platform, Azure Databricks makes it simple to run large-scale Spark workloads.

Optimized environment
To address the problems seen on other Big Data platforms, Azure Databricks was optimized from the ground up, with a focus on performance and cost-efficiency in the cloud. The Databricks Runtime adds several key capabilities to Apache Spark workloads that can increase performance and reduce costs by as much as 10-100x when running on Azure, including:

High-speed connectors to Azure storage services, such as Azure Blob Store and Azure Data Lake
Auto-scaling and auto-termination of Spark clusters to minimize costs
Caching
Indexing
Advanced query optimization
By providing an optimized, easy to provision and configure environment, Azure Databricks gives developers a performant, cost-effective platform that enables them to spend more time building applications, and less time focused on managing clusters and infrastructure.

Reading data from images

Prazwal Ks — Wed, 13 Apr 2022 02:20:57 +0000

Suppose you are given thousands of images and asked to transfer the text on the images to a computer database. The scanned images have text organized in different formats and contain multiple languages. What are some ways you could complete the project in a reasonable time frame and make sure the data is entered with a high degree of accuracy?

Companies around the world are tackling similar scenarios every day. Without AI services, it would be challenging to complete the project, especially if it were to change in scale.

Using AI services, we can treat this project as a computer vision scenario and apply Optical Character Recognition (OCR). OCR allows you to extract text from images, such as photos of street signs and products, as well as from documents—invoices, bills, financial reports, articles, and more.

To build an automated AI solution, you need to train machine learning models to cover many use cases. Azure's Computer Vision service is a Cognitive Service that gives access to advanced algorithms for processing images and returns data to secure storage.

The Computer Vision service offers two APIs that you can use to read text.

OCR API:
Use this API to read small to medium volumes of text from images.
The API can read text in multiple languages.
Results are returned immediately from a single function call.

Read API:
Use this API to read small to large volumes of text from images and PDF documents.
This API uses a newer model than the OCR API, resulting in greater accuracy.
The Read API can read printed text in multiple languages, and handwritten text in English.

The initial function call returns an asynchronous operation ID, which must be used in a subsequent call to retrieve the results.
The computer vision service offers the OCR API and Read API.

You can access both technologies via the REST API or a client library. In the next few units, we'll show you how to call the REST API and return a JSON response. Then for the exercise, you'll use a client library to return objects that abstract the JSON response.

ML Model

Prazwal Ks — Fri, 08 Apr 2022 17:37:45 +0000

The model is the core component of machine learning, and ultimately what we are trying to build. A model might estimate how old a person is from a photo, predict what you might like to see on social media, or decide where a robotic arm should move to. In our scenario, we want to build a model that can estimate the best boot size for a dog based on their harness size.

Models can be built in lots of ways. For example, a traditional model that simulates how an airplane flies is built by people, using knowledge of physics and engineering. Machine learning models are special—rather than being edited by people so that they work well, machine learning models are shaped by data they learn from experience.

How to think about models
A model can be thought of as a function that accepts data as an input and produces an output. More specifically, a model uses input data to estimate something else. For example, in our scenario, we want to build a model that is given a harness size, and estimates boot size

Note that harness size and dog boot size are data, they are not part of the model. Harness size is our input, Dog boot size is the output.

Models are often simple code
Models are often not meaningfully different from simple functions you are already familiar with. Like other code, they contain logic and parameters. For example, the logic might be “multiply the harness size by parameter_1”.

If parameter_1 here was 2.5, our model would multiply harness size by 2.5 and return the result:

Selecting a model
There are many model types, some simple and some complex.

Like all code, simpler models are often the most reliable and easy to understand, whilst complex models can potentially perform impressive feats. Which kind of model should be chosen depends on your goal. For example, medical scientists often work with models that are relatively simple because they are reliable and intuitive. By contrast, AI-based robots typically rely on very complex models.

The first step in machine learning is selecting the kind of model that you would like to use. This means we are choosing a model based on its internal logic. For example, we might select a two-parameter model to estimate dog boot size from harness size.

Notice how we have selected a model based on how it works logically, but not based on its parameter values. In fact, at this point the parameters have not yet been set to any particular value.

Parameters are discovered during training
Parameter values are not selected by the human designer. Instead, parameter values are set to an initial guess, and then adjusted during an automated learning process called training.

Given our selection of a two-parameter model (above), we now provide random guesses for our parameters.

These random parameters will mean the model isn’t good at estimating boot size, so we perform training. During training, these parameters are automatically changed to two new values that give better results.

Diagram showing a model with 1.5 and 4 as the parameters.

Exactly how this process works is something we will progressively explain throughout your learning journey.

NumPy and Pandas

Prazwal Ks — Tue, 05 Apr 2022 15:53:50 +0000

Data scientists can use various tools and techniques to explore, visualize, and manipulate data. One of the most common ways in which data scientists work with data is to use the Python language and some specific packages for data processing.

NumPy
NumPy is a Python library that gives functionality comparable to mathematical tools such as MATLAB and R. While NumPy significantly simplifies the user experience, it also offers comprehensive mathematical functions.

Pandas
Pandas is an extremely popular Python library for data analysis and manipulation. Pandas is like excel for Python - providing easy-to-use functionality for data tables.

Explore data in a Jupyter notebook
Jupyter notebooks are a popular way of running basic scripts using your web browser. Typically, these notebooks are a single webpage, broken up into text sections and code sections that are executed on the server rather than your local machine. This means you can get started quickly without needing to install Python or other tools.

Testing hypotheses
Data exploration and analysis is typically an iterative process, in which the data scientist takes a sample of data and performs the following kinds of task to analyze it and test hypotheses:

Clean data to handle errors, missing values, and other issues.
Apply statistical techniques to better understand the data, and how the sample might be expected to represent the real-world population of data, allowing for random variation.
Visualize data to determine relationships between variables, and in the case of a machine learning project, identify features that are potentially predictive of the label.
Revise the hypothesis and repeat the process.

Ai

Prazwal Ks — Sat, 02 Apr 2022 18:36:31 +0000

Artificial Intelligence is the creation of software that imitates human behaviors and capabilities.
The Key elements of AI are:

1.Machine learning - is the way we "teach" a computer model to make prediction and draw conclusions from data.

2.Anomaly detection - The capability to automatically detect errors or unusual activity in a system.

3.Computer vision - The capability of software to interpret the world visually through cameras, video, and images.

4.Natural language processing - The capability for a computer to interpret written or spoken language, and respond in kind.

5.Conversational AI - The capability of a software "agent" to participate in a conversation.

1.Machine Learning:-
One common question for everyone is how do machines learn?
The answer is, from data. In today's world, we create huge volumes of data as we go about our everyday lives.

From the text messages, emails, and social media posts we send to the photographs and videos we take on our phones, we generate massive amounts of information. More data still is created by millions of sensors in our homes, cars, cities, public transport infrastructure, and factories.

Data scientists can use all of that data to train machine learning models that can make predictions and inferences based on the relationships they find in the data.

Machine Learning is the foundation for most AI solutions.

2.Anamoly Detection:-
anomaly detection - a machine learning based technique that analyzes data over time and identifies unusual changes.

Imagine you're creating a software system to monitor credit card transactions and detect unusual usage patterns that might indicate fraud. Or an application that tracks activity in an automated production line and identifies failures. Or a racing car telemetry system that uses sensors to proactively warn engineers about potential mechanical failures before they happen.
These kinds of scenario can be addressed by using anomaly detection

An anomaly detection model is trained to understand expected fluctuations in the telemetry measurements over time.
If a measurement occurs outside of the normal expected range, the model reports an anomaly that can be used to alert the race engineer to call the driver in for a pit stop to fix the issue before it forces retirement from the race

3.Computer vision solutions are based on machine learning models that can be applied to visual input from cameras, videos, or images.

For Example, Object detection machine learning models are trained to classify individual objects within an image, and identify their location with a bounding box.

Face detection is a specialized form of object detection that locates human faces in an image. This can be combined with classification and facial geometry analysis techniques to infer details such as age and emotional state; and even recognize individuals based on their facial features.

4.Natural language processing (NLP) is the area of AI that deals with creating software that understands written and spoken language.
NLP enables you to create software that can:
•Analyze and interpret text in documents, email messages, and other sources.
•Interpret spoken language, and synthesize speech responses.
•Automatically translate spoken or written phrases between languages.
•Interpret commands and determine appropriate actions.

For example, Starship Commander, is a virtual reality (VR) game from Human Interact, that takes place in a science fiction world. The game uses natural language processing to enable players to control the narrative and interact with in-game characters and starship systems.

5.Conversational AI is the term used to describe solutions where AI agents participate in conversations with humans. Most commonly, conversational AI solutions use bots to manage dialogs with users. These dialogs can take place through web site interfaces, email, social media platforms, messaging systems, phone calls, and other channels.

Bots can be the basis of AI solutions for:
•Customer support for products or services.
•Reservation systems for restaurants, airlines, cinemas, and
other appointment based businesses.
•Health care consultations and self-diagnosis.
•Home automation and personal digital assistants.