DEV Community: Mabwa Charles

The intersection between AI and RPA

Mabwa Charles — Tue, 21 Mar 2023 22:26:47 +0000

The rise of artificial intelligence (AI) and robotic process automation (RPA) is transforming the way businesses operate. Although AI and RPA are distinct technologies, they share many similarities and often intersect in their use cases. In this article, we'll explore the intersection between RPA and AI, their differences, and how they can work together to create more efficient and effective workflows.

RPA and AI: What's the difference?
RPA and AI are two distinct technologies with different functions. RPA is a software tool that automates repetitive, rule-based tasks such as data entry and form filling. On the other hand, AI refers to a set of technologies that enable machines to simulate human intelligence, including natural language processing, machine learning, and computer vision.
RPA is a rules-based automation technology that is designed to replace human activities with computerized ones, while AI is a technology that can learn from data and make predictions based on that learning. RPA can handle simple, repetitive tasks while AI can handle complex, decision-based tasks.

How do RPA and AI intersect?
The intersection between RPA and AI occurs when AI is used to enhance the capabilities of RPA systems. By using AI, RPA systems can become more intelligent and capable of handling more complex tasks. The following are some examples of how RPA and AI intersect:

Natural Language Processing (NLP)
NLP is a branch of AI that enables machines to understand human language. By using NLP, RPA systems can understand and interpret unstructured data, such as emails, documents, and chat transcripts. This means that RPA systems can perform tasks such as data extraction and classification, sentiment analysis, and entity recognition.

Machine Learning (ML)
ML is another branch of AI that enables machines to learn from data and make predictions based on that learning. By using ML, RPA systems can improve their decision-making capabilities and become more efficient over time. For example, an RPA system that uses ML can automatically identify patterns in data and adjust its processes accordingly.

Computer Vision (CV)
CV is a technology that enables machines to interpret visual information. By using CV, RPA systems can automate tasks that require visual recognition, such as identifying and extracting data from images and videos. For example, an RPA system that uses CV can automatically extract information from scanned documents.

Benefits of combining RPA and AI
The combination of RPA and AI can provide many benefits for businesses, including:

Increased efficiency
By automating repetitive, rule-based tasks with RPA and using AI to handle complex tasks, businesses can increase their efficiency and reduce the time required to complete tasks.
Improved accuracy
RPA and AI systems can work together to reduce errors and improve accuracy in tasks such as data entry and data processing.

Better decision-making
By using AI to analyze data and provide insights, businesses can make better-informed decisions.

Cost savings
By automating tasks with RPA and reducing the need for human intervention, businesses can save on labor costs.

Conclusion

RPA and AI are two different technologies that are often used together to automate business processes. By combining RPA and AI, businesses can increase efficiency, improve accuracy, make better-informed decisions, and save on costs.
Join our community forum to learn more about RPA and AI intersections. https://forum.electroneek.com

How to Optimize Robotic Process Automation for Maximum Efficiency

Mabwa Charles — Fri, 24 Feb 2023 19:47:05 +0000

Robotic Process Automation (RPA) is a powerful tool for companies looking to streamline their processes and increase the efficiency of their operations. With the right implementation, RPA can help businesses save time, money, and resources by automating mundane and repetitive tasks. However, ensuring that RPA is optimized for maximum efficiency is important. Here are a few tips on how to optimize RPA for maximum efficiency.

Before implementing RPA, it is important to define the goals you want to achieve. This will help you determine which processes should be automated and how they should be automated. It is also important to consider the cost-benefit ratio of automation. You may want to concentrate on low or high priority tasks or a mix of both;

Numerous RPA platforms are available, so it is important to make an informed decision when choosing the right one for your organization. Different RPA platforms offer different features and capabilities, so it’s a good idea to research the options available and select the one that best meets your requirements

Automating the right processes can lead to more efficient operations and save your organization time, money and resources. When you automate the wrong processes, you will face inefficiencies, so it is important to first identify which processes are suitable for automation, and then schedule a project that will help improve your company’s efficiency.

Once you have fully implemented your RPA platform, testing and monitoring the automation is important to ensure it is running as expected. This will help you identify any issues and make sure the automation is running at its optimal level.

Adapt and Improve: As technology evolves, it is important to adapt and improve the automation process. This will ensure that the automation remains relevant and continues to be effective.

By following these tips, your company can optimize your RPA for maximum efficiency. This will help you save time, money, and resources while improving the quality of your operations.

Take a look at ElectroNeek to fulfill all your automation requirements

You can also join our community forum HERE

Web Scrapping with ElectroNeek

Mabwa Charles — Fri, 27 Jan 2023 16:55:57 +0000

Robotic Process Automation (RPA) is a technology that has revolutionized how businesses automate their processes. RPA has made it easier and faster to automate mundane and repetitive tasks, freeing employees to focus on more meaningful work. One of the most popular RPA tools is web scraping, which is used to extract data from websites. Web scraping is an effective way to quickly gather data from multiple sources, allowing businesses to make better decisions.

Web scraping is the process of using software to extract data from websites. It works by using a web crawler to identify and extract specific pieces of data from a website. The data is then stored in a structured format, such as a CSV or JSON file, which can be used for further analysis.

Before we continue, take a breather and Sign Up for Exclusive Content, Exclusive Posts from our Forum by navigating to. https://forum.electroneek.com

Let’s continue.

Web scraping can be used for a variety of purposes, such as collecting product prices, monitoring competitor activity, or gathering customer feedback. It can also be used to automate tedious tasks, such as filling out forms or downloading images.
When using RPA for web scraping, businesses can choose from a variety of tools, such as web crawlers, API integrations, and screen scraping. Web crawlers are automated programs that can be used to extract data from websites. API integrations allow businesses to connect to external data sources and extract data from them. Screen scraping is a technique used to extract data from graphical user interfaces, such as web forms.

Web Scraping is no longer a complicated task. With ElectroNeek, you can automate the scraping process from any website. The ElectroNeek platform can easily scrape data from any website, no matter how large the data is.

To Scrape Data with the ElectroNeek Platform, you can either use:

The Scrape Structured Data Activity on our platform to Scrape data from any website and save this to an Excel table.
The CSS and Xpath Selectors to scrape your data from any website.

Deep dive into the entire process of scraping data with ElectroNeek by following these questions in our community forum.

Post 1:
How do I scrape data from all pages

Post 2:
Create data tables from Web Scraping activities?

Let us know your thoughts in the comment section below.

Happy Automation!!!

How to Build Bots Via APIs with ElectroNeek

Mabwa Charles — Thu, 01 Dec 2022 19:17:56 +0000

What Are Bots?

Bots are intelligent software programs designed to optimize business processes. These programs may be designed to perform one or more automated tasks or even simulate human interaction with customers and users. Bots can be used to automate repetitive tasks performed by humans

What is Automation?

Automation is the technique of performing tasks and processes with minimal human assistance. The objective of automation is to achieve a greater level of productivity and quality in human work. Automation has evolved from a mere job to a critical business tool that allows you to do more with less. The art of automation applies to most sectors. With automation, you set up the system to do predictable things automatically, enabling people to focus on more important activities like creating new ideas or building relationships with customers.

You can use automation to automate your organization's tasks, from automating your email activities to your Excel activities. If you would like to learn how you can automate your email activities, you can navigate to this article that I did recently.

There are many types of bots for businesses. For example, bots can be built to automate any repetitive tasks in your business. Your bot is going to be used to save you time and revenue while improving your ROI, so think about what services and tasks you want the bot to perform for your business.

Building your first bot on the ElectroNeek platform is simple. There are multiple ways to build a bot on our platform. Learn how you can build bots via APIs using ElectroNeek here:
https://forum.electroneek.com/t/how-to-build-bots-via-api/902

Don’t be left out; join our community here: https://forum.electroneek.com

Happy Automation!!

Email Automation

Mabwa Charles — Fri, 25 Nov 2022 17:56:05 +0000

With more people having access to the internet, there has been a gradual increase in the number of emails sent out each year. An estimated 300 billion emails are sent out each day, with this number projected to increase by 2025.

Emails are the primary form of communication used by businesses worldwide and are a crucial part of any business organization. Using email automation, you can automate your repetitive email tasks, including opening emails, reading and processing them, sorting them, extracting data from them, working on them, and responding to them accordingly.
Email automation lets you increase employee productivity by allowing them to concentrate on tasks that are important to them and stop wasting too much time reading and sending emails.

ElectroNeek offers integrated email activities that are ready to be used for automation across various platforms, including Microsoft Outlook, Microsoft 365, Google, Yahoo, iCloud, Yandex, and others. By selecting the service’s name, users can choose the default settings for any email server on the list with ease.

Find out how you can automate your email activities using ElectroNeek:
https://forum.electroneek.com/t/email-automation/1102

You can also be part of the community by navigating to:
https://forum.electroneek.com/signup

Happy Automation!!!

What is an Orchestrator

Mabwa Charles — Fri, 18 Nov 2022 13:00:26 +0000

Orchestrators are high-level software applications that orchestrate the work of robots. They enable end users to deploy robots, build sequences, control the status of the robots, monitor their health, manage their functioning and results, analyze reports, and perform maintenance on them. One can deploy one or many robots at once from a single location. Users can set a specific schedule for the bots, generate webhook links to trigger those bots in third-party apps, and more. Learn about the ElectroNeek SaaS Orchestrator here.

Advantages of using an Orchestrator

1. Bot Management
Scheduling unattended bots allow you to focus on more Return On Investment driven tasks while bots execute the automation. You don’t need to worry about whether your bots are scheduled correctly or not; automate it and enjoy your business, so you can do what you love doing best.

2. Integration

Orchestrator lets you easily integrate with your favorite business software and third-party applications for the most effective automation.

3. Credential Management

When the robot is executed, the Orchestrator stores and retrieves credentials, including user IDs and passwords.

4. Scalability

Bot orchestration eliminates frequent problems that frequently obstruct automation scaling by enabling customers to control all robots in a single control room.

5. Accessibility

Orchestrator is available in the cloud and accessible from a range of devices, including mobile ones, unless it is deployed on-premises.

6. Compliance

In order to maintain compliance and security, Orchestrator records and logs every activity performed by robots as well as by people using robots.

Get to deep dive into a hands-on experience with the ElectroNeek Orchestrator here:

https://forum.electroneek.com/t/electroneek-saas-orchestrator-product-overview/738

Thanks, and Happy Automation!!

Excel Automation with RPA

Mabwa Charles — Sun, 13 Nov 2022 18:01:51 +0000

Excel automation is a technique for using a program to carry out tasks automatically without any manual assistance. Excel automation generally involves building code to interact with Excel and automatically perform tasks. Before we continue with the piece, please read more on Excel Automation Here.

You can use robotic process automation, or RPA, to incorporate Excel tasks into automated processes.

All of the processes that Excel can complete on its own can be accomplished by robotic process automation. It is capable of reading and writing all file types that Excel supports. RPA can also effectively analyze data and produce in-depth reports at the same time. RPA can then be installed by businesses on their computers to function like Excel software.

So what can Excel automation do for you?

Excel automation can perform a variety of tasks for users to enhance their experience while using an Excel workbook. It simplifies complicated tasks and enables businesses to quickly and precisely repeat procedures to save time, money, and effort.

It gives users a user-friendly experience by taking care of all the manual and repetitive tasks so they can concentrate on the importance of the data in the spreadsheets.
Excel Automation can help businesses to make informed decisions
Integrating spreadsheets and their data into any enterprise system requires automating Excel processes. Spreadsheets are now just another source that can be integrated with more advanced enterprise ones thanks to automation.
Excel Automation makes the transfer of data between spreadsheets risk-free.
Users can securely share data in the formats .xls, .xlsx, .csv and .xlsm (Excel files that specifically support .vba macros). The data can be compressed, encrypted, and transferred to others via Excel automation, regardless of where they are physically located.

We have a comprehensive guide on how you can automate your excel activities on our forum, navigate to: https://forum.electroneek.com/t/first-excel-steps/1017

You can also be part of our community by registering an account here: https://forum.electroneek.com

What is Optical Character Recognition (OCR)

Mabwa Charles — Thu, 10 Nov 2022 17:57:07 +0000

Optical character recognition (OCR) is sometimes referred to as "text recognition." It is the extraction of data into machine-encoded text from scanned documents, images, and PDFs. The original material can be accessed and edited by using OCR software, which isolates letters on the image, turns them into words, and then turns the words into sentences.

OCR is extensively used as a form of data entry from printed records. This technology allows people to digitize data for increased productivity.

Using OCR, ElectroNeek can recognize and read information from document sources that range from accounts, invoices, contracts, tables, and other financial documents.

You can do a deep dive into OCR automation with ElectroNeek here:

https://forum.electroneek.com/t/electroneek-ocr-automation-to-word-file-transferring-ocr-content-to-a-word-file/763

Also, don't forget to post a comment or a question.

You can also join our forum at: https://forum.electroneek.com

Scrape YouTube comments for Sentimental Analysis using SerpApi

Mabwa Charles — Wed, 08 Dec 2021 13:38:58 +0000

Introduction

Content creation revolves around authenticity in delivering required knowledge that not only informs a learner and the versed audience but also helps them grow in skill which results in the application of the knowledge acquired. From YouTube feedback of the content created based on the views and comments one can be able to determine the quality of the video based on the sentiments of views. Let us get to it.

Prerequisite

Python and the relevant libraries
Jupyter Notebook, VScode and PyCharm editors depending on your preference.
Google Cloud Platform

Setting Environment

We will need the textblob, wordcloud libraries for the sentimental analysis. The pandas, numpy and re libraries will help in cleaning, restructuring and summarizing statistics of the data. We will use the matplotlib library for visualization of our data.

# Import libraries
from textblob import TextBlob
from wordcloud import WordCloud
import pandas as pd
import numpy as np
import re
import matplotlib.pyplot as plt
plt.style.use('fivethirtyeight')

Data

In an attempt to analyze the sentiments of the SQL content in YouTube, we picked three videos with a high number of comments from Edureka, freeCodeCamp and Programming with Mosh YouTube Channels. The three have been extracted from the YouTube data extracted by the SerpApi Tool.

From the Google Cloud Platform acquire a YouTube API key which will be inserted in the below comment extraction tool:

# Scrape Comments for SQL Using Python Through The Youtube Data API
api_key = "your Api key here" # Insert your Api key here.

from googleapiclient.discovery import build
youtube = build('youtube', 'v3', developerKey=api_key)

import pandas as pd

box = [['Name', 'Comment', 'Time', 'Likes', 'Reply Count']]

#Eudereka, Programming with Mosh, freeCodeCamp
code_lang = [
    {"id":"HXV3zeQKqGY"},
    {"id":"7S_tz1z_5bA"},
    {"id":"zbMHLJ0dY4w"}
]

# sql_vids = pd.DataFrame([])

for id_code in code_lang:
    def scrape_comments_with_replies():
        data = youtube.commentThreads().list(part='snippet', videoId=id_code['id'], maxResults='100', textFormat="plainText").execute()

        for i in data["items"]:
            name = i["snippet"]['topLevelComment']["snippet"]["authorDisplayName"]
            comment = i["snippet"]['topLevelComment']["snippet"]["textDisplay"]
            published_at = i["snippet"]['topLevelComment']["snippet"]['publishedAt']
            likes = i["snippet"]['topLevelComment']["snippet"]['likeCount']
            replies = i["snippet"]['totalReplyCount']

            box.append([name, comment, published_at, likes, replies])

            totalReplyCount = i["snippet"]['totalReplyCount']

            if totalReplyCount > 0:

                parent = i["snippet"]['topLevelComment']["id"]

                data2 = youtube.comments().list(part='snippet', maxResults='100', parentId=parent,
                                            textFormat="plainText").execute()

                for i in data2["items"]:
                    name = i["snippet"]["authorDisplayName"]
                    comment = i["snippet"]["textDisplay"]
                    published_at = i["snippet"]['publishedAt']
                    likes = i["snippet"]['likeCount']
                    replies = ""

                    box.append([name, comment, published_at, likes, replies])

        while ("nextPageToken" in data):

            data = youtube.commentThreads().list(part='snippet', videoId=id_code['id'], pageToken=data["nextPageToken"],
                                             maxResults='100', textFormat="plainText").execute()

            for i in data["items"]:
                name = i["snippet"]['topLevelComment']["snippet"]["authorDisplayName"]
                comment = i["snippet"]['topLevelComment']["snippet"]["textDisplay"]
                published_at = i["snippet"]['topLevelComment']["snippet"]['publishedAt']
                likes = i["snippet"]['topLevelComment']["snippet"]['likeCount']
                replies = i["snippet"]['totalReplyCount']

                box.append([name, comment, published_at, likes, replies])

                totalReplyCount = i["snippet"]['totalReplyCount']

                if totalReplyCount > 0:

                    parent = i["snippet"]['topLevelComment']["id"]

                    data2 = youtube.comments().list(part='snippet', maxResults='100', parentId=parent,
                                                    textFormat="plainText").execute()

                    for i in data2["items"]:
                        name = i["snippet"]["authorDisplayName"]
                        comment = i["snippet"]["textDisplay"]
                        published_at = i["snippet"]['publishedAt']
                        likes = i["snippet"]['likeCount']
                        replies = ''

                        box.append([name, comment, published_at, likes, replies])

        df = pd.DataFrame({'Name': [i[0] for i in box], 'Comment': [i[1] for i in box], 'Time': [i[2] for i in box],
                       'Likes': [i[3] for i in box], 'Reply Count': [i[4] for i in box]})

        sql_vids = pd.DataFrame([])

        sql_vids = sql_vids.append(df, ignore_index = True)

        sql_vids.to_csv('youtube-comments.csv', index=False, header=False)

    scrape_comments_with_replies()

The data is then stored in the CSV file from whence our analysis begins.

Import the data

Depending on the location of your data file import it using the pandas function:

# import data:
data = pd.read_csv(r'youtube-comments.csv',
                 lineterminator='\n')

data.head()

The data head gives you the first five rows of the data:

Clean the Comments

To carry out the sentimental analysis, we will now remove the punctuation marks in the Comment strings by the function:

# Clean the text:
# Function
def cleanTxt(text):
  text = re.sub(r'[^\w]', ' ', text)

  return text

data['Comment'] = data['Comment'].apply(cleanTxt)

Sentimental Analysis

Polarity and Subjectivity

In starting with the analysis we will create the new columns namely Polarity and Subjectivity and acquire the very values of each comment. Polarity ranges from -1 to 1 and measures how positive or negative a comment is. It simply means emotions expressed in a sentence. Subjectivity expresses some personal feelings, views, or beliefs. A subjective sentence may not express any sentiment.

To find the two, use the defined function using the textblob library:

# get subjectivity
def getSubjectivity(text):
  return TextBlob(text).sentiment.subjectivity

# get polarity
def getPolarity(text):
  return TextBlob(text).sentiment.polarity

#Columns
data['Subjectivity'] = data['Comment'].apply(getSubjectivity)
data['Polarity'] = data['Comment'].apply(getPolarity)

data

The two will be added to the data as follows:

WordCloud

In an attempt to view the most used words in the comment we will use the Word Cloud library to view. It is from these words that we get the Polarity of a comment. To view this:

# WordCloud
allWords = ' '.join( [cmts for cmts in data['Comment']])
wordCloud = WordCloud(width = 500, height = 300, random_state = 21, max_font_size = 119).generate(allWords)

plt.imshow(wordCloud, interpolation= 'bilinear')
plt.axis('off')
plt.show

As noted from the above word cloud the most used words are Thank, video, SQL, course, link, zip file, tutorial, database, download. Most of these words are relatable to the content and suggest the in-depth skills that the learners desire to grow using the SQL tool. Thank being the dominant word suggests gratitude and high positive response.

Is this true?

The answer can be found by the code below:

First, compute the Analysis:

# function to compute analysis
def getAnalysis(score):
  if score < 0 :
    return 'Negative'
  elif score == 0:
    return 'Neutral'
  else:
    return 'Positive'

data['Analysis'] = data['Polarity'].apply(getAnalysis)

Compute the percentages of the analysis:

# % Percentages:
pcomments = data[data.Analysis == 'Positive']
pcomments = pcomments['Comment']

print('Positive: ' +str(round((pcomments.shape[0]/data.shape[0])*100, 1))+ '%')

ncomments = data[data.Analysis == 'Negative']
ncomments = ncomments['Comment']

print('Negative: ' +str(round((ncomments.shape[0]/data.shape[0])*100, 1))+ '%')

nucomments = data[data.Analysis == 'Neutral']
nucomments = nucomments['Comment']

print('Nuetral: ' +str(round((nucomments.shape[0]/data.shape[0])*100, 1))+ '%')

Output

Positive: 46.0%
Negative: 9.3%
Nuetral: 44.7%

From the comments, most learners perceive the use of the SQL content from Edureka, freeCodecamp and Programming with Mosh as very useful. 46.0% agree that the content is useful and 9.3% do not agree. From this anyone desiring to learn SQL is advised to subscribe to the content of these three bloggers. The statistics suggest that the bloggers give in-depth knowledge of the SQL tool that is beneficial to a beginner in this field.

To graph the above do:

# Value Count
data['Analysis'].value_counts

# Plot
plt.title('Sentiment Analysis')
plt.xlabel('Sentiment')
plt.ylabel('Counts')
data['Analysis'].value_counts().plot(kind= 'bar')
plt.show()

Conclusion

From the three platforms; Edureka, freeCodeCamp and Programming with Mosh and SQL, as a data tool, content is very useful. These are not the only bloggers with such content hence from a learner desiring to perform a preferential analysis, one can find the above tool in this GitHub repo.

You can also contribute to our user forum here: https://forum.serpapi.com/

If you'd like to contribute either by sharing suggestions, recommendations and any questions, feel free to drop a comment in the comment section or reach out to me via Twitter at @mabwa, or @serp_api.

Join us on Reddit | Twitter | YouTube.

YouTube Views Analysis and Prediction using SerpApi: Ep 2

Mabwa Charles — Wed, 08 Dec 2021 09:46:14 +0000

From the data extracted using YouTube Search API, it can only be useful if used to deduce conclusions from which decision can be made. This is the whole idea behind machine learning. The data extracted from the SerpApi YouTube Data Extraction contained YouTube video content for the data manipulating tools.

From one end, a learner would desire to know what type of tools are commonly studied and searched. This would suggest the particular tool with many views is what the market demands. This insight can be deduced from the number of videos extracted and the number of views in each video. On the other hand for a blogger, who is already skilled in specific data tools and desires to venture into creating content for learners. He would desire to know the tools which have not been well taught based on the number of views registered.

The blogger also desires to know what is the scale of growth and start earning from the content created. This will be deduced by training the data with a regression model.

Prerequisites

We will be using VS Code in this analysis and Python 3 interpreter. One might as well use Jupyter Notebook or Pycharm IDE depending on your preference. Mid-level knowledge of scikit-learn is required as well as the models used in machine learning. We will also be using the linear regression model.

Setting Environment

Import the required libraries as follows:

from datetime import timedelta
from dateutil import parser
import seaborn as sns
import pandas as pd
import numpy as np
import warnings
import re

warnings.filterwarnings("ignore")

If the above libraries have not yet been installed, you can do so using pip install or the conda install depending on your preference.

Import Data

From the YouTube SerpApi tool, the data extracted was stored in a CSV format. To import it, we will use the read.csv from the pandas library as follows:

# Import csv file:
data = pd.read_csv(r'Data/code_languages.csv')

Cleaning data and Feature engineering

The raw data extracted presents us with quality information which we will require to restructure before committing it to a model. Below are the steps taken:

From the title, variable extract the tools that were searched using the YouTube SearchApi tool.
From the published_date extract the numeric figures.
Modify the length of each video extracted to be in seconds. This answers the question “how long is the video?”
Finally, replace the missing values in the channel.verified with “False”. Assuming they are not verified.

To do the above follow:

# Extract tool name from title:
# Names of tools:
tools = ['SQL', 'MySQL','Excel','Tableau', 'Azure', 'AWS','R']
pat = '|'.join(r"\b{}\b".format(x) for x in tools)

data['tool'] = data['title'].str.extract('('+ pat + ')', expand=False, flags=re.I)

# Extract the age of  video; time past since video was posted:
data['duration_from_posted'] = data.published_date.str.extract('(\d+)')

# Length of the video in seconds
data['length'] = data['length'].str.strip()
data['length'] = pd.to_datetime(data['length'],format='%H:%M:%S',errors='coerce').fillna(pd.to_datetime(data['length'], format='%M:%S', errors='coerce'))

time = pd.to_datetime('1900-01-01 00:00:00', format =  "%Y-%m-%d %H:%M:%S")
time_delta = (data['length']  - time).astype('timedelta64[s]')
data['length'] = time_delta

# Replace NA in channel.verified assuming missing means False
data[ 'channel.verified'] = data[ 'channel.verified'].replace(np.nan, "False")

From these restructured data we will need the title of the tool, the age of the videos, the views gained overtime and the status of the channel verification:

# Required data:
cols = ['tool', 'duration_from_posted', 'length', 'channel.verified', 'views']
req_data = data[cols]

Correlation Analysis

We would like to know what are the factors that affect the number of views. Numerically, the length of the video may have an implication on the views a content gets. Also, time posted has an effect on the eventual views a video can get. As initially proved the more a video stays on the YouTube platform the more likely it is to be viewed.To what extent, though, does the length of a video and the duration since posted affect the views gained on content? This can be answered by a heat-map generated by:

# Correlation heatmap
num_data = req_data[['duration_from_posted', 'length', 'views']]
cols = ['duration_from_posted', 'views', 'length']

# Convert all columns to numeric
num_data[cols] = num_data[cols].apply(lambda x: pd.to_numeric(x, errors='coerce'))

# Drop all missing values
num_data = num_data.dropna()

# View heatmap
sns.heatmap(num_data.corr(),  annot=True)

Output

The length has a positive effect on the views. This suggests that preferentially many learners prefer in-depth tutoring for tools in data analytics, data science and data engineering. A blogger venturing in this field should therefore have excellent knowledge in this field to be able to explain in broader lengths the concepts required.

How old a video content does not guarantee more views on the YouTube platform. Though weak in effect, this can be deliberate against the hypothesis of common knowledge that suggests the older the video the more the views. This, however, can only be consistent if the content in the video produced in time past is needed today; i.e. the relevance of the video content.

Regression Model

# Divide data into attributes and labels
X = num_data.iloc[:, 0:2].values
y = num_data.iloc[:, 2].values #Views

# Divide the data into training and testing sets
from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=0)

# Training a linear regression model
from sklearn import linear_model
from sklearn.metrics import mean_squared_error, r2_score

# Build model
model = linear_model.LinearRegression()
model.fit(X_train, y_train)

# Apply trained model to predict logS from the training and test set
y_pred_train = model.predict(X_train)

print('Coefficients:', model.coef_)
print('Intercept:', model.intercept_)
print('Mean squared error (MSE): %.2f'
      % mean_squared_error(y_train, y_pred_train))
print('Coefficient of determination (R^2): %.2f'
      % r2_score(y_train, y_pred_train))

In formulating the model we used the length of a video and the duration since posted as independent factors and the number of views as the dependent factor. 20% of our data was used as a test case. The train data was subjected to a linear regression model.

Test Data

Fitting the model in the test data:

#  apply the trained model to make predictions on the test set
y_pred_test = model.predict(X_test)

# Evaluation
print('Coefficients:', model.coef_)
print('Intercept:', model.intercept_)
print('Mean squared error (MSE): %.2f'
      % mean_squared_error(y_test, y_pred_test))
print('Coefficient of determination (R^2): %.2f'
      % r2_score(y_test, y_pred_test))

The regression equation obtained with an acceptable 82.234% model accuracy is as follows:

# Regression equation
yintercept = '%.2f' % model.intercept_
dp = '%.2f dp' % model.coef_[0]
lt = '%.4f lt' % model.coef_[1]

print('Views = ', yintercept,  '+', dp, '+', lt )

Output:

The equation unlike the correlation effect suggests that both the length of the video and the age on the video content have an incremental effect on views.The analysis can be differentiated from the type of content posted based on categories of the kind of content created. It would therefore be wise for the platform to include channel categories for such an analysis to take place.

Conclusion

It is therefore advised that any vlogger desiring to produce data tools content should:

Start producing the video content now. This will give them the advantage of views over time.
Produce detailed content for the tools teaching all viewers. This helps the vlogger attain Youtube viewing time that will enable them to monetize their content faster.

Ending

The complete code can be found in this Github repo.

You can also contribute to our user forum here: https://forum.serpapi.com/

Yours,
Mabwa, and the rest of the SerpApi Team.

Join us on Reddit | Twitter | [YouTube]

SerpApi YouTube Data Extraction Tool: Ep 1

Mabwa Charles — Wed, 08 Dec 2021 08:40:57 +0000

Introduction

YouTube is an American online video sharing and social media platform owned by Google. With many people shifting to digital online broadcasting, the platform has grown exponentially. YouTube data has become a major part of the analysis in machine learning and data analytics.

Using SerpApi we will extract YouTube data and query it for analysis.

Prerequisites

The product is easy to use and flexible to tailor across multiple YouTube content depending on the field of interest. However, one will require a mid-level knowledge and understanding of:

Python
VS code and/or Jupyter Notebook

Tool

To have access to the general tool, go to the YouTube Search API. The below tool has been tailored to extract video results based on searches of desired facets of the data analytics, data science and data engineers tools:

import pandas as pd
from serpapi import GoogleSearch
import json

api_key =  "serp_api_key"
engine_search = "youtube"

#data analysis tools:
code_langs = [
    {"name":"engine", "query":"sql"},
    {"name":"engine", "query":"excel"},
    {"name":"engine", "query":"tableau"},
    {"name":"engine", "query":" microsoft azure"},
    {"name":"engine", "query":"amazon web services"},
    {"name":"engine", "query":"r programming"}
    #{"name":"engine", "query":"name_of_product"}   
]

data_vids = pd.DataFrame([])

for lang in code_langs:
    params = {
        lang['name']:engine_search, 
        "search_query": lang['query'],
        "api_key": api_key
    }

    search = GoogleSearch(params)
    results = search.get_dict()
    playlist_results = results['video_results']
    data_vids = data_vids.append(pd.json_normalize(playlist_results), ignore_index = True)
    data_vids.to_csv('code_languages.csv')

The tool uses pandas and SerpApi libraries. One is required to have a basic familiarity with the two. In case you are beginning to use the product and are new to python, it is possible that you should first install the required libraries using conda install or pip install.

The code_langs is a tailored list of parameters to be looped over and desired video results appended in the data frame data_vids. For example:

{"name":"engine", "query":" microsoft azure"},

Using for loop the tool will return microsoft azure content from YouTube. This will be the case for all the queried lists in the code_langs until all the desired videos are appended.

The data is then stored in a CSV file, find the CSV file here

Data

The data extracted has 13 columns. The first 10 are the essential columns with variables that could trigger an analysis.

position_on_page    title   link    published_date  views   length  description extensions  channel.name    channel.link    channel.verified    channel.thumbnail   thumbnail.static    thumbnail.rich

These variables include:

a. position on the page: This is the position of the specific video upon the search of a data analytics, data science or engineering tool.

b. title: The title of the video. This tells what the video is all about.

c. published_date: The date the video was uploaded on YouTube.

d. views: The number of people who have watched the video.

e. length:This tells the duration it takes to watch the video.

f. channel.name: The channel from which a video was retrieved.

To further know how to extract channel details, visit Channel Results

Check out Part 2 of this article where we handle the Youtube views analysis and Prediction.

Ending:

The complete code to the tool can be found in this Github Repo

YouTube Search API documentation: https://serpapi.com/youtube-search-api

You can also contribute to our user forum here: https://forum.serpapi.com/

Yours,
Mabwa, and the rest of the SerpApi Team.

Join us on Reddit | Twitter | YouTube

Walmart and eBay Electronic Brand Analysis using SerpAPI

Mabwa Charles — Tue, 07 Dec 2021 18:15:17 +0000

In a bit to compare Walmart and eBay. It is noted that eBay is better than Walmart in selling Electronics of all brands. And reviews are one of the best factors to consider in stocking products on a selling platform. Let's deep dive into how to use SerpApi to extract, visualize, analyse and draw conclusions about this data.

Requirements:

Python 3.x.x
VS Code/Jupyter Notebook

Import Libraries

import re
import json
import time
import pandas as pd
import plotly.express as px
from serpapi import GoogleSearch
import warnings
warnings.filterwarnings("ignore", 'This pattern has match groups')

Let’s get started.

The complete code for the tutorial can be found Here

Data Extraction

We will extract the data of electronic brands from the Walmart and the eBay market platforms using the respective APIs. To achieve this, on a trial basis. First, we will need to sign up for the SerpApi services.

Once you have signed up, navigate to the Ebay Search API which has a variety of options of structured data to extract. A SerpApi, api_key will be generated and attached to every code required to help you extract data.

Use Case

We will go with organic data which is the raw structured data for electronic data. We will attempt to analyze Samsung, Hisense, Sony and TCL products in the eBay and Walmart platforms.

This is the data we are required to extract.

Let's dig in:

Once you are in the Ebay Search API, click on the eBay Search API drop-down menu then click on the organic results. This will lead you to a code block that contains the code for the data extract we desire:

From the above, we will develop our Walmart and eBay data extraction tools.

Tools

Ebay Tool

api_key =  "serp_api_key"
engine_search = "ebay"

# Ebay products:
products = [
    {"name":"engine", "query":"sony"},
    {"name":"engine", "query":"samsung"},
    {"name":"engine", "query":"tcl"},
    {"name":"engine", "query":"lg"},
    {"name":"engine", "query":"hisense"}
    #{"name":"engine", "query":"name_of_product"}   
]

ebay_data = pd.DataFrame([])

for product in products:
    params = {
        product['name']:engine_search, 
        "_nkw": product['query'],
        "api_key": api_key
    }

    search = GoogleSearch(params)
    results = search.get_dict()
    organic_results = results['organic_results']
    ebay_data = ebay_data.append(pd.json_normalize(organic_results), ignore_index = True)

The products list contain multiple parameter dictionaries that we would love to loop in to acquire the specified electronic brands of interest. All the data will be appended in the empty data frame initialized:ebay_data.

Using for loop we perform the scraping using the SerpApi query. The organic results contain a .json data of all the brand products in the Ebay platform. We transform the json data into a data frame using the json_normalize function from the pandas package and append for each parameter value.

The same is applied to the Walmart data:

Walmart Tool

api_key =  "serp_api_key"
engine_search = "walmart"

# Walmart products:
products = [
    {"name":"engine", "query":"sony"},
    {"name":"engine", "query":"samsung"},
    {"name":"engine", "query":"tcl"},
    {"name":"engine", "query":"lg"},
    {"name":"engine", "query":"hisense"}   
]

walmart_data = pd.DataFrame([])

for product in products:
    params = {
        product['name']:engine_search, 
        "query": product['query'],
        "api_key": api_key
    }

    search = GoogleSearch(params)
    results = search.get_dict()
    organic_results = results['organic_results']
    walmart_data = walmart_data.append(pd.json_normalize(organic_results), ignore_index = True)

Data Cleaning

In cleaning data:

From the product titles, extract the brand names for easy manipulation.

# List of brands
L = ['Samsung', 'Hisense','TCL','Sony']
pat = '|'.join(r"\b{}\b".format(x) for x in L)

# Extract the brand names, making all characters small for merging
ebay_data['brand'] = ebay_data['title'].str.findall(pat, flags=re.I).str.join(' ')
ebay_data = ebay_data.apply(lambda x: x.astype(str).str.lower())
ebay_data['brand'] = ebay_data['brand'].str.replace(r'\b(\w+)(\s+\1)+\b', r'\1', regex=True)

In the eBay data, from the extensions variable, extract the number of items so far sold.

#From the items_sold, extract the minimum integer value of the brand items sold as of the date of data extraction

# Define the function to remove the punctuation
ebay_data['extensions'] = ebay_data['extensions'].str.replace('[^\w\s]','',  regex=True)
ebay_data['extensions'] = ebay_data.extensions.str.extract('(\d+)')

Another advantage of the data extracted from eBay and the Walmart search API has few missing variables, thus dropping the missing row data leaves us with sufficient data for analysis.

# Drop all the missing values: this will/may not help in our analysis.
# Drop missing values
ebay_data.dropna()

# Drop title variable
ebay_data.drop('title', inplace=True, axis=1)

Replicate the same steps for the Walmart data. With this, our data is ready for analysis. Find the above step by step processes on the .py file in this GitHub repository.

Visualization

We will analyze the markets with visuals while answering the common business analysts questions that tend to have an impact on investment.

What is the best platform for selling electronic brands?
Code:

#Brand Analysis
#Aggregate the number of products for each brand and market

ebay_brands = ebay_data[['brand', 'market']].reset_index()
walmart_brands = walmart_data[['brand', 'market']].reset_index()
market_brands = pd.concat([ebay_brands, walmart_brands], ignore_index=True)
count_series = market_brands.groupby(['brand', 'market']).size()
count_series = count_series.to_frame(name = 'size').reset_index()
count_series['brand'] = count_series['brand'].replace(r'^\s*$', 'lg', regex=True)
count_series.drop(count_series.tail(1).index,inplace=True)

To answer the above one will first extract the brand and market variables from the Walmart and eBay data. We will then summarize the count of each brand in each market then plot the groupings. Below are the results:

Output:

Across all brands, electronic products are acquired by consumers more on eBay than Walmart. In analytics, this is what we would call descriptive analysis. But why is this so?

2. Why are there more brands selling on eBay?
In diagnostic analytics, we seek to know why consumers buying preference is from eBay rather than Walmart. Normally, we know that the extra cost to acquire a product is a put off on any consumer. As a factor, has shippingcost-influenced the customers to buy more from eBay?

The answer to this is from the below visual produced by the code:

#Shipping cost effect on the sales of a brand in Ebay?
#Group the data by shipping status and the number of sales:

ebay_data['shipping'] = ebay_data['shipping'].replace("nan", "with shipping cost", regex=True)
ebay_data['shipping'] = ebay_data['shipping'].str.replace('[^\w\s]','',  regex=True)

ebay_data.loc[ebay_data.shipping.str.contains('(\d+)')  == True, 'shipping'] = 'with shipping cost'

# Required data
shipped_sales = ebay_data[['brand', 'shipping', 'extensions']].reset_index()
shipped_sales['extensions'] = pd.to_numeric(shipped_sales['extensions'])

# Aggregate the data by the grouping
shipped_sales = shipped_sales.groupby(['shipping','brand'])['extensions'].sum().reset_index()
shipped_sales["extensions"] = pd.to_numeric(shipped_sales ["extensions"])
shipped_sales['brand'] = shipped_sales['brand'].replace(r'^\s*$', 'lg', regex=True)
shipped_sales = shipped_sales.drop(labels=[5,7,13], axis=0)

# Visual
fig = px.bar(shipped_sales, x="brand", y="extensions", color="shipping")
fig.show()

Output:

We see from the above output that free shipping on eBay has influenced buyers to acquire electronic products irrespective of the brand. This is but one of many factors that consumers focus on while meeting their needs. If one can acquire a product at the least cost possible, that is an option one would take, and eBay seems to offer such an option.

3. What are some of the factors a company should consider when determining a platform on which to stock a product?

In understanding the above, perform a correlation analysis. The below are the results of the analysis:

The analysis was carried on eBay data.

It is clear that reviews have a strong positive items_sold. Thus, one of the factors that a company can consider while making the decision to increase their stock on a selling platform is the number of reviews on that particular product, generally. Whether the reviews are positive or negative is another concern. This can be determined by sentimental analysis. This is still possible as the SerpApi gives us an option to scrape sentiments on products and thereafter perform the required analysis.

Below is the code to reproduce the correlation plot:

#Correlation in Ebay sales
#Extract data:

cor = ebay_data[['price.extracted', 'extensions', 'reviews']].reset_index()

#Drop data whose values are not known
cor_data = cor_data[~cor.reviews.str.contains("nan")]
col = cor.columns.drop('index')
cor[col] = cor [col].apply(pd.to_numeric, errors='coerce')

import matplotlib.pyplot as plt
import numpy as np
import seaborn as sns
plt.figure(figsize=(16, 6))
mask = np.triu(np.ones_like(cor.corr(), dtype=np.bool))
heatmap = sns.heatmap(cor.corr(), mask=mask, vmin=-1, vmax=1, annot=True, cmap='BrBG')

Here is a scatterplot showing the relationship between reviews against items sold:

The cost of a product, irrespective of the brand has an implication on the number of products sold. The variation differs from individuals across economical brackets.

We also have seen that number of reviews have a positive effect on sales. Is there a relationship between prices and review? See the diagram below;

Diagram:

An increase in prices has a significant yet slight increase in the reviews. This is consistent as consumers may desire quality for their expense.

Ending:

The complete code for the tutorial can be found Here

Please view the Walmart Search API documentation: https://serpapi.com/walmart-search-api

Ebay Search API documentation: https://serpapi.com/ebay-search-api

You can also contribute to our user forum here: https://forum.serpapi.com/

If you'd like to contribute either by sharing suggestions, recommendations and any questions, feel free to drop a comment in the comment section or reach out via Twitter at @mabwa, or @serp_api.

Yours,

Mabwa, and the rest of the SerpApi Team.

Join us on Reddit | Twitter | YouTube