DEV Community: alphaneuron.ai

NATURAL LANGUAGE PROCESSING

alphaneuron.ai — Mon, 22 Mar 2021 18:17:48 +0000

Natural language processing, which is also known as NLP can be easily explained by the figure present below.

So basically it means that as we speak it creates certain meaningful sentence or we can say data by which our brain is able to understand what others are speaking and even what we are speaking, similarly NLP help the artificial brain to create a meaningful data out of the bunch of data it has received.

Let's first point out the basic procedure of almost every NLP or sentence-related intelligence experiment.

A human talks to the machine(input, it may be in audio form or may be in text form)
The machine captures the audio
Audio to text conversion takes place(if the input is in audio form)
Processing of the text’s data
Data to audio conversion takes place(after predicting the required output)
The machine responds to the human by playing the audio file

so let's relate this a little bit to our day-to-day life, The various examples of NLP that we are observing are the chatbots, The auto-corrects, Personal assistant applications such as OK Google, Siri, Cortana, and Alexa.

Now let's jump into the code/fun part i.e how a machine understands the data.
for example:

He is a boy
she is a girl

name	gender	human
Ram(boy)	boy(1)	1
sita(girl)	girl(0)	1

Basically, each word In a sentence that the machine obtained is converted in word vectors that mean into some numeric values and these numeric values play different roles according to their presence in a sentence,e.g.boy and girl plays a different role under gender category but they play the same role under the human category.

Now let's have a look at how these word vectors are formed.we are going to use the NLTK library.it has several features which you can use by first installing all of them:
import nltk nltk.download('all')

now you can easily convert your paragraph into sentences or even words to preprocess your text more specifically.
e.g.
for converting paragraph to sentences
nltk.sent_tokenize(paragraph)
for converting sentences to words
nltk.word_tokenize('sentences')

Here are some Computer algorithms are used to apply grammatical rules to a group of words and derive meaning from them.

Lemmatization: It entails reducing the various inflected forms of a word into a single form for easy analysis.
Morphological segmentation: It involves dividing words into individual units called morphemes.
Part-of-speech tagging: It involves identifying the part of speech for every word.
Sentence breaking: It involves placing sentence boundaries on a large piece of text.
Stemming: It involves cutting the inflected words to their root form.

There are many more like word2vec , bagofwords ,Transformers , encoder & decoders and some most powerful like BERT etc.

But mostly between stemming and lemmatization lemmatization is preferred because it includes calculation based on log values

Now let's see a simple example of Natural entity recognition using NLP
importing the libraries
import nltk from nltk.tokenize import word_tokenize from nltk.tag import pos_tag

converting each category of entity into a respective vector

from bs4 import BeautifulSoup import requests import re def url_to_string(url): res = requests.get(url) html = res.text soup = BeautifulSoup(html, 'html5lib') for script in soup(["script", "style", 'aside']): script.extract() return " ".join(re.split(r'[\n\t]+', soup.get_text())) ny_bb = url_to_string('https://en.wikipedia.org/wiki/Wikipedia') article = nlp(ny_bb)

now marking the entities given using some predefined styling

So, That's a brief on what is NLP and how it can be used in every text-related data, and how to preprocess those types of data.
For more preprocessing related stuff do checkout my GitHub profile:https://github.com/Ashishkumarpanda

Do give your reviews on this,
Thank you for reading :)

A NOTE TO REMEMBER

alphaneuron.ai — Mon, 02 Nov 2020 02:41:33 +0000

Hello Everyone

Hope you all are doing well in this lockdown. From the title, it might be confusing what note you should remember, I started exploring the field of data i.e. ML, DS, DL, etc. & actually it is pretty cool when you find a future prediction as to the output of your code.

So what is the first thing that comes to our mind when we hear about this field??

so we think it's all about learning a number of algorithms, then 2 to 4 libraries of python for cleaning the data, and then it's done!!!!!

so the most important thing I am going to discuss here is the backbone of this field i.e "Data"

The number of algorithms is fixed into three categories

supervised(you know the past relation(labeled data))
unsupervised(no past relation is known to you, you form a different group out of them)
reinforcement(you get rewarded with success and vice versa)

After getting familiar with these you now try to learn how to implement them on data to predict future outcomes.
Basically, we have two types of data:

structured data
unstructured data

structured data means no data cleaning part(the different terms like visualization, wrangling you have heard)..you just import it and then train_test_split and fit the model.

Now let's get our hand's dirty with the unstructured data because that's what I learned in these months, we will always face the unstructured data.

so basically I am going to use the following libraries for this purpose:
Step1:-importing the libraries:

NumPy - import numpy as np(for data preprocessing)
pandas - import pandas as pd(for data cleaning)
matplotlib - import matplotlib.pyplot as plt(for data visualization)
seaborn - import seaborn as sns(for data visualization)

Matplotlib is a python library used to create 2D graphs and plots by using python scripts. But I think if you're handling a larger dataset with very much non-linearity seaborn should be your major weapon

step2:-where and how to use different plots of seaborn:

plot name	where to use	how to use
heatmap	basically used to know the overall information and relation between the data	sns.heatmap(data)
barplot	when we are comparing between two categories	sns.barplot(value1,value2)
countplot	same as barplot but use to know the occurrence of a label	sns.countplot(value,data)
distplot	used to get the distribution of data	sns.distplot(data)
box-plot	shows the distribution of quantitative data in a way that facilitates comparisons between variables or across levels of a categorical variable	sns.boxplot(data)

Now after you have completely visualized the data provided and understand the relation between different parameters provided to you, you're ready to clean your data.

However, we found these problems while handling the unstructured data:

categorical columns
null values(Nan)
biased column
no values(blank)
outliers
starting from the end the outliers are like the Cardamom to your biriyani, They're the ones which will cause less accuracy of your model
how to solve??

1)univariate method:-This method looks for data points with extreme values on one variable.
2)multivariate method:-Here we look for unusual combinations on all the variables
3)Minkowski error: This method reduces the contribution of potential outliers in the training process
blank values i.e. missing values sometimes you will see some data is missing in some columns but the output depends on that data so you have to fill that place accordingly with the maximum frequency of the data, some times the average of the data
data.fillna(value)-when you put a fixed value
data.fillna(method = bfill\ffill)-backward/forward filling
data.fillna(data.mean())-average value
now what is a biased column:- suppose for a prediction you have a gender column in data which is required for prediction, but the male: female ratio is 95:5, this called a biased column, so try to keep values appropriately else the model will predict according to a single value.
The traditional method to deal with null value is to drop them
data.dropna(), but if it is required for your prediction instead of dropping it try to fill this place by replacing with another value as mentioned in no values case
Last but not least how to handle the categorical columns
a)creating dummies:
Easy to use and fast way to handle categorical column values.(ps: not useful for many categories)
pd.get_dummies(Data)
b)When the categorical variables are ordinal(labeled), the easiest approach is to replace each label(not useful for nominal)
data.replace(man,0,inplace=True)
data.replace(woman,1,inplace=True)
c) one hot encoding:-applicable for a lesser number categories i.e. convert the data in 1 or 0
from sklearn.compose import ColumnTransformer
ColumnTransformer([('encoder', OneHotEncoder(), [no.of categories])], remainder='passthrough')
data = np.array(columnTransformer.fit_transform(data), dtype = np.str)
LabelEncoder:- the most useful part to convert any number of categories into different numerical values
from sklearn.preprocessing import LabelEncoder
LabelEncoder().fit_transform(data)

That's all. Hope this will help you a lot in data preprocessing and in ml term, we call feature engineering

for examples you can check my githup repo :-https://github.com/Ashishkumarpanda

just a beginner do comment any other methods if I missed something.Thank you :)
`

HACKTOBER FEST 2020

alphaneuron.ai — Sat, 17 Oct 2020 07:20:44 +0000

What I Learned From Hacktoberfest

In the beginning i am not so aware about opensource but after this great event i was able to submit more than 4 pull requests as well as learn how to create issues solve issues make a pull request and also how to create and edit a beautiful file generally known as readme.md using markdown for my repositories

The 4 projects for which i completed this ,i will only share about them.
1)A cancer type predicting model using Artificial neural network
repo link:-https://github.com/Ashishkumarpanda/Cancer-Type-detection
2)i have started with a basic hello world program using python programming language
repo link:-https://github.com/Ashishkumarpanda/Hello-World
3)I have worked on some cool projects like creating a pencil sketch of a provided image using opencv and python.In which it will be helpful if anyone give some suggestions.
repo link:-https://github.com/Ashishkumarpanda/Photo-Sketch-maker
4)After gaining some experience i have contributed to Devincepts deep learning repository and i was very much happy when recieved the mail that the pr was merged from them.
repo link:-https://github.com/Ashishkumarpanda/Contribution-program

so that's all about my first hacktoberfest experience.
happy coding:)

ML FOR DEVELOPER

alphaneuron.ai — Sun, 23 Aug 2020 13:58:10 +0000

Hello Everyone,This is Ashish Kumar ,I have just started Machine learning and the most interesting thing is that it helps us in predicting results,like we can get an approximate result of a situation on the basis of it's past occurrence.

Basically as a beginner i can describe Machine Learning in the following three magical words
1 TRAIN
2 TEST
3 PREDICT

which means to train the given data and from that predict the future outcomes.So this is the basic thing about ML,now what an android developer can do with it????

The most amazing fact is that the applications that our generation is using in our day-to-day life like Netflix,Tinder etc. are one of the top machine learning applications.
So how can We do it??
Step:1

ML Kit brings Google’s machine learning expertise to mobile developers in a powerful and easy-to-use package. Make your iOS and Android apps more engaging, personalized, and helpful with solutions that are optimized to run on device.

now using these packages and dependencies we can easily create a ml app for our phone.This includes some cool stuffs like text recognition,Smart reply,Face detection etc...isn't it cool??

So here is a picture of my first machine learning app using Firebase Ml kit of Smart reply.It will automatically generate text as per your text.it has some basic text features like hi,hello,how are you? etc..and some commonly used emojis.

So this is all abou the ground level now how about creating something cool like object detecting app.just think no need to remeber a person's name just open the app scan the face and it will detect it's name:))

So for this we have Tensorflow lite,in simple words we have convert our data into .tflite file,like the.sfb we did in case of ar app.then the further stages are as i have told
1)first collect data for training.
2)Transfer them into required image
3)Again train model with new images now make sure you're
running it on a suitable device
4)our data is trained now so attach .tflite file to it and
run your app
And one more thing
What is Tensorflow??
TensorFlow is an end-to-end open source platform for machine learning. It has a comprehensive, flexible ecosystem of tools, libraries and community resources that lets researchers push the state-of-the-art in ML and developers easily build and deploy ML powered applications

so this image shows how cool it is..Hope this article will be helpful soon i will share with you something like this,till then
Happy coding:))))

AR BASICS FOR ANDROID STUDIO

alphaneuron.ai — Wed, 29 Jul 2020 14:11:35 +0000

Hello Everyone,
This post is going to be an interesting one because this is how i created my own pokemon(pikachu).What if we can create an app that can show us our favourite character,so it can be done by a simple addition of augment reality to our android app.

WHAT IS AUGMENT REALITY?

Augmented reality (AR) is an interactive experience of a real-world environment where the objects that reside in the real world are enhanced by computer-generated perceptual information, sometimes across multiple sensory modalities, including visual, auditory, haptic, somatosensory and olfactory. AR can be defined as a system that fulfills three basic features: a combination of real and virtual worlds, real-time interaction, and accurate 3D registration of virtual and real objects ~~(WIKIPEDIA)

in simple words pikachu in pokemon go,you can create it:)

so As an android learner how to implement this in our app,

REQUIRMENTS

1)install your app in a device that supports google arcore

2)uses-permission android:name="android.permission.CAMERA"
add this in a tag for asking the user to allow camera
permission.

3)Now after this add the following dependencies in your (app-
level)build.gradle file
implementation "com.google.ar.sceneform.ux:sceneform-ux:1.5.0"(use the supporting version or the errors will blow your mind) and
dependencies { classpath 'com.google.ar.sceneform:plugin:1.5.0'}(in project level build.gradle file)

this will add sceneform sdk to your project and also allows to implement different 3D assests to your project.you can download them from the internet or you can follow this link
:-https://poly.google.com
remember to download the .obj file that is your actual model

4)now create a sampledata directory and add your assets there.

5)Add a fragment to your main layout.xml file
fragment android:name="com.google.ar.sceneform.ux.ArFragment"

android:id="@+id/ux_fragment"

android:layout_width="match_parent"

android:layout_height="match_parent"
i.e. the sceneform fragment.

Another important thing the sdk requires Api level-27 to run this or more so keep this in mind while creating the project(if your phone has inbuilt ar you're lucky my friend)

Next, we are using the ModelRenderable class to build our model. With the help of setSource method, we load our model from the .sfb file. This file was generated when we imported the assets. thenAccept method receives the model once it is built. We set the loaded model to our lampPostRenderable.

After all this thing you need the java code to run your first ar app.So here i am going to discuss about a few terms that i found confusing while implementing the code:

a)Scene: This is the place where all your 3D objects will be
rendered. This scene is hosted by the AR Fragment which
we included in the layout. An anchor node is attached
to this screen which acts as the root of the tree and
all the other objects are rendered as its objects.

b)HitResult: This is an imaginary line (or a ray) coming from
infinity which gives the point of intersection of
itself with a real-world object.

c)Anchor: An anchor is a fixed location and orientation in the
real world. It can be understood as the x,y,z coordinate in
the 3D space. You can get an anchor’s post information from
it. Pose is the position and orientation of the object in the
scene. This is used to transform the object’s local coordinate
space into real-world coordinate space.

d)** AnchorNode** This is the node that automatically positions
itself in the world. This is the first node that gets set when
the plane is detected.

e)TransformableNode: It is a node that can be interacted with.
It can be moved around, scaled rotated and much more. In this
example, we can scale the lamp and rotate it. Hence the name
Transformable.

         i hope now you can jump into the code https://github.com/Ashishkumarpanda/AR/blob/master/MainActivity.java)

IT consits of the following parts:

1)build the model
2)add your model to scene

hope this article will be helpful:)

HOW TO START APP DEVELOPMENT

alphaneuron.ai — Sun, 26 Jul 2020 11:59:13 +0000

Hello Everyone,
This post is for those people out there just like me who get excited when they see real time working projects,but don't know how to make one of them and get disappointed.

so here i am ,discussing about my best friend during this lockdown mr.android .Ever thought of not writing your assignments but still submit it by changing just the font of your friend's assignment,well you can build an app for that.This is how android development attracts me towards itself(just kidding:p).

What you need to become a android developer

1)Knowledge about a good programming java,and yes it's java(because i love java,so it is best for me){ps:-there are c++,c#,kotlin,python too.}

WHY JAVA

a) Java is secure (no threat to security because nothing gets
executed outside the JVM)
WHAT IS JVM?????
{JVM (optimised for android): a virtual machine makes life of a
developer /programmer easier. And android uses dalvik, a VM
optimized to suit android needs.}

b) Object oriented paradigms.
c) Rich set of core features (java’s core features are
complete and vast. Also, they’re regularly updated and
maintained by oracle).

d) Frameworks & Classes: outside the core library, java has
many frameworks and classes for features like networking,
threading, IO operations and thus, programmers can
leverage these qualities in their apps.
e) Open source nature of java and android.

so after having a clear concept on the programming language you can jump into the ide.So here comes the best friend of the hero(jAVA)
2) ANDROID STUDIO

Android Studio by Google is by far the most used and popular
IDE (Integrated Design Environment) for Android app
development. It was launched in 2015 and replaced Eclipse for
building Android apps.

Since Android Studio is the official Android IDE by Google, it has the best tools for faster and feature-rich app development, testing, debugging, code development & editing, and deploying. Moreover, it is updated regularly with new & advanced features.

The best part is that the apps that you will build you can also test them on your android devices..isn't it different:)?

so here is my very first app not so good ui but it can store your real time info. and can update them....
https://github.com/Ashishkumarpanda/FRIENDLY-MEDICO

AS we are going to create some cool stuffs like the apps we are using in our phones so the most important part is a

3)**Database**!!

we all heard about SQLite...but in todays world who stores data offline?SO here comes the very helpful person

4)FIREBASE
Firebase is built on Google infrastructure and scales automatically, for even the largest apps. One platform, with products that work better together Firebase products work great individually but share data and insights, so they work even better together

you can't imagine in the beggining what you can create with the help of firebase.starting from face detection to chat apps and so on.

Now you all are ready to go to become an android developer

       last but not least

Building Universal apps

Even if you’re building an app only for Android OS, you must know that Android devices come in a variety of configurations and sizes. So, your app needs to be able to adapt to all types of screens.

In other words, your app must be responsive and designed to work across all Android devices, screen sizes, languages, and OS versions. To achieve this, you can consider using separate layouts for different devices and ensure consistent density across all the devices.

Hope this article will be helpful

Happy coding :)