DEV Community: Lorena

How hyperautomation will transform business operations

Lorena — Fri, 11 Feb 2022 16:02:51 +0000

Are you trying to convince your manager to invest (more) in automation? Or are you the manager who went a bit over budget with automation tools and you're wondering if it was worth it? Or maybe you're just exploring the vast space of workflow automation tools and wondering: "Is this the future of real work, or is it just fantasy?"

In this post we’ll share with you the key facts you need to know about hyperautomation: what it is, why it is important, what are examples of hyperautomation tools, how businesses can use them, and how hyperautomation is predicted to evolve in the next few years.

What is hyperautomation?
How are businesses leveraging hyperautomation?
What is the future of hyperautomation?
1. Orchestrated automation processes
2. Automation marketplaces
3. Vendor-agnostic hyperautomation
4. Infrastructure automation
Start automating!

What is hyperautomation?

Gartner defines hyperautomation as "a business-driven, disciplined approach that organizations use to rapidly identify, vet and automate as many business and IT processes as possible. Hyperautomation involves the orchestrated use of multiple technologies, tools or platforms."

Examples of hyperautomation tools are:

no-code/low-code application platforms (N/LCAP)
workflow automation tools (WAT)
robotic process automation (RPA)
Artificial Intelligence (AI) and Machine Learning (ML)
chatbots and conversational agents

The hyperautomation-enabling software market has been rising in the past two years and is expected to reach $596bn in 2022.

How is hyperautomation different from automation?

Automation refers to the accomplishment of a specific task without manual or human intervention. For example, you can use a no-code workflow that creates tickets from form submissions automatically, instead of doing this manually. Automation is well-suited for repetitive, boring, regular, rule-based, software-based, and time-consuming tasks at small scale.

Hyperautomation refers to the combination and connection of several automated workflows, thus creating an orchestrated automation. This orchestration feature takes automation to the hyper level, allowing businesses to scale individual processes. Taking the example above a step further, you can add this workflow alongside an ML model or service that detects the sentiment of user reviews, a chatbot that assists customers, an application that processes text from invoices, and a database synchronization to keep the information up-to-date -- all these forming a hyperautomated business.

Why do we need hyperautomation?

The examples above highlight the two main benefits of hyperautomation: increased productivity and seamless scaling of business operations. Without automation orchestration, business departments risk working out of sync, thus impacting the overall progress and costs of the organization.

Gartner points out that hyperautomation is driven by two forces: operational excellence and digital acceleration. Operational excellence is reflected in profits (businesses being able to deliver faster or cheaper), whereas digital acceleration is reflected in adoption (attracting more customers at a faster pace).

To get to that point, organizations can go two ways. Traditionally, they can increase productivity by hiring department-specific people or IT-skilled engineers who can set up automations. IT teams become fusion teams, where employees with different skills can directly contribute to the automation processes needed in their department.

In short, hyperautomation helps organizations to save costs, increase efficiency, and overall improve their business model. On an individual level, employees whose tedious tasks are automated have more time to focus on meaningful and creative work, which in turn increases their job satisfaction.

How are businesses leveraging hyperautomation?

According to a Gartner study, businesses have on average 4 automation processes. This number seems low even for small businesses, considering how many individual tasks are on the to-do lists of employees in every department. However, 80% of senior business executives say they will spend more on digital initiatives in 2022, aiming to accelerate their business (65%) and go to market faster (71%).

Businesses in all industries can leverage the power of hyperautomation. For example:

E-Commerce can automate almost the entire journey, from announcing product launches, sending and analyzing emails, issuing invoices, tracking packages, running inventories, and notifying customers.
IT can automate DevOps and SecOps use cases like contributions to a repository, critical incident response, or vulnerability disclosure.

What is the future of hyperautomation?

Gartner foresees four trends in hyperautomation in the next few years.

1. Orchestrated automation processes

"By 2024 diffuse (siloed) approach to hyperautomation initiatives will drive up initiative specific total cost of ownership by 40-fold, making adaptive governance a differentiating factor in financial performance."

Think of how many different apps and services you use in your daily job, and how many more your colleagues from other departments use as well. In fact, 78% of business professionals use three tools from different categories to accomplish their daily tasks, which is not really practical or efficient.
In the next few years, businesses will try to turn these disparate (disconnected) automations into orchestrated (connected) hyperautomation workflows. For example, you can have one workflow that synchronizes data between the Sales Pipedrive and Marketing Hubspot.

2. Automation marketplaces

"By 2024, growth of “automation marketplaces” will propel 80% of the large enterprises to pivot to principles of composability to minimize operational interdependencies and maximize value of hyperautomation initiatives."

Businesses will change the way they deliver their digital products, replacing packaged applications (like individual projects or products) with composed applications (in the style of catalogs or markets). Think of these "automation marketplaces" as curated, interactive exhibitions.

For example, in a workflow automation marketplace, you would not only see a list of integrations, but also sort them by industry or function, try out automation templates, and learn from supportive content––maybe even tailored to your personal role.

3. Vendor-agnostic hyperautomation

"By 2024, the lack of standardization and uniformity in vendor pricing structures will continue driving 40% of clients to increase hyperautomation vendor-agnostic business capabilities."

It's hard to find the one tool that ticks all the boxes: affordable price, powerful functionality, intuitive UI, blazing speed. Commonly, businesses are willing to compromise on some features for the sake of simplicity (keep their processes on one platform––at the risk of vendor lock-in.

Hyperautomation tools can diminish this risk, since they make it possible to interconnect apps services. As a consequence, in the future businesses will move away from a commitment-based model to a consumption-based model, preferring to combine features of different tools until it's a match for their use case.

4. Infrastructure automation

"By 2024, 40% of organizations will use managed service provider hyperautomation offerings to fill infrastructure operations gaps fortifying a foundation for TCO and scaled automation."

The COVID-19 pandemic has forced many companies to go into remote work mode. But with employees working around the world, it's challenging to ensure smooth connectivity, data security, incident response management, timely decision-making, and ongoing support and maintenance.

To create a solid infrastructure for these processes, businesses will rely on hyperautomation tools. For example, you can build no-code workflows for automatic incident response or database monitoring.

Start automating!

Going back to the questions in the introduction, we hope the information in this post helps you make a compelling case for automation in your workplace, rest assured that your investment in automation tools is worth every penny, and dare to explore the hyperautomation space.

One more thing: 3% of hyperautomation professionals characterize their organization as having a high impact on hyperautomation governance. Are you going to be among them?

Building a dockerized ETL pipeline for streaming tweets in Slack

Lorena — Fri, 04 Feb 2022 10:19:01 +0000

One of the projects in my Data Science Bootcamp was about creating a database of tweets, along with their sentiment score, and post positive tweets in a Slack channel. This pipeline had to be orchestrated with Docker Compose. The pipeline looks like this:

In this post, I'll show you how I set up each step.

0. Prerequisites & tech stack

Here's an overview of the apps, services, and libraries I used in this project:

Apps & Databases	Python libraries
Twitter	`tweepy` & `vader`
Slack	`slackclient`
MongoDB	`pymongo`
PostgreSQL	`psycopg2-binary` & `sqlalchemy`
Docker-Compose	-

1. Collecting tweets

To collect tweets, I used the Twitter API along with the tweepy library.

First, I created an app on Twitter and got my credentials (API key and Access Token). Then, I wrote the Python code for streaming live tweets, using tweepy with my Twitter credentials. I chose to stream the hashtag #OnThisDay (thought it would be interesting to get a daily notification of what happened years ago) and collected the tweet text and user handle.

from tweepy import OAuthHandler, Stream, API
from tweepy.streaming import StreamListener
tweet = {
    'username': t['user']['screen_name'],
    'text': t['text'],
}

stream_listener = StreamListener()
stream = tweepy.Stream(auth=api.auth, listener=stream_listener)
stream.filter(track=['OnThisDay'])

2. Storing tweets in MongoDB

After collecting the tweets, I had to store them in MongoDB, a non-relational (NoSQL) database that stores data in JSON-like documents. Since the tweet data is collected as key-value pairs (JSON format), MongoDB is a good way to store this information.

First, I had to create a MongoDB instance, set up a cluster, and create a database and a collection within it:

Create a MongoDB account
Set up a cluster: cloud.mongodb.com > Clusters > Create New Cluster
Create a database: Cluster > Collections > Create Database
Create a collection: Cluster > Collections > Database > Create Collection
Create a field: Collection > Insert document > Type the field text below _id
Allow access to the database: Project > Security > Network Access > IP Access List > Add your IP address.
Connect to the database from your terminal:\ mongo "mongodb+srv://YourClusterName.mongodb.net/YourDatabaseName" --username YourUsername

Second, I wrote the Python code for storing tweets in MongoDB using the pymongo library.

import pymongo

client = pymongo.MongoClient(host='mongo_container', port=27018)
db = client.tweets_db

def warning_log(tweet):
    logging.critical('\n\nTWEET: ' + tweet['username'] + 'just tweeted: ' + tweet['text'])
    db.collections.onthisday.insert_one(tweet)

The host mongo_container is one of the Docker containers, explained in section 5.

3. Performing ETL job

The ETL (Extract, Transform, Load) job involves three actions: extracting tweets from MongoDB, analyzing their sentiment, and storing them into a new Postgres database. Here is the Python code for the ETL job.

3.1. Extracting tweets from MongoDB

To extract the tweet texts from MongoDB, I used again the pymongo library.

def extract_tweets():
    tweets  = list(db.onthisday.find())
    if tweets:
        t = random.choice(tweets)
        logging.critical("Random tweet: "+t["text"])
        return t

3.2. Transforming tweets with sentiment scores

To analyze the sentiment of the tweets, I used the VADER library , which returns (among others) a compound sentiment score.

from vaderSentiment.vaderSentiment import SentimentIntensityAnalyzer

def transform_tweets(tweet):

    tweet_text = tweet['text'].replace("\'","")

    sia = SentimentIntensityAnalyzer()
    tweet_sia = sia.polarity_scores(tweet_text)['compound']

    return tweet_sia

3.3. Loading tweets into PostgreSQL

To load the tweets with their sentiment scores into a Postgres database, first you need––well, a Postgres database. I installed Postgres, then created a database and a table for tweets right from the terminal:

Connect to Postgres: psql
Create a database: createdb twitter
Go into the created database: psql twitter
Create columns in the database: CREATE TABLE tweets (text varchar(280), score numeric(4,3));

Then, I wrote the Python code for inserting tweets into the tweets table, using the sqlalchemy library.

def load_tweets(tweet, sentiment):

    insert_query = """
    INSERT INTO tweets VALUES ('{tweet["text"]}', {tweet_sia});
    """

    engine.execute(insert_query)
    logging.critical(f'Tweet {tweet["text"]} loaded into Postgres.')

4. Extracting tweets from Postgres

After having a database of tweets and their sentiment score in place, I had to select and extract some tweets, that would be sent to Slack.

query = pg.execute(
    "SELECT text FROM tweets ORDER BY sentiment DESC LIMIT 1")
msg = str(list(query))
output = f'NEW TWEET! {user} just tweeted: {msg} \nSentiment score: {blob_score}'

5. Posting tweets with a Slackbot

The last step in the pipeline is posting tweets in a Slack channel. To do this, first I created a Slackbot.

Then, I wrote the Python code for posting tweets in a Slack channel, including the code from the previous step:

import time
import slack
from sqlalchemy import create_engine
import config

engine = config.PG_ENGINE
webhook_url = config.WEBHOOK_SLACK


while True:
    logging.critical("\n\nPositive tweet:\n")
    query = pg.execute(
        "SELECT text FROM tweets ORDER BY sentiment DESC LIMIT 1")
    msg = str(list(query))
    logging.critical(msg + "\n")
    output = f'NEW TWEET! {user} just tweeted: {msg} \nSentiment score: {blob_score}'
    data = {'text': output}
    requests.post(url=webhook_url, json=data)

    time.sleep(30)

And 🎉 –– here's the tweet that was posted in Slack:

Tweet posted by Slackbot

6. Creating the Docker Compose pipeline

The final touch of this project is orchestration. The individual Python scripts for each step work when you run them manually, but the goal is to run this pipeline from beginnning to end with only one command. This is where Docker Compose comes in.

Compose is a tool for defining and running multi-container Docker applications. With Compose, you use a YAML file to configure your application’s services. Then, with a single command, you create and start all the services from your configuration.

Each of the five previous steps (or the rectangles in my messy schema) represents a Docker container, so in my docker_compose.yml file I had five containers (services): tweet_container, postgres_container, mongo_container, etl_container, and slackbot_container.

For the two database containers, I used Docker images, since they didn't depend on custom code stored in my project folders. For the other three containers, I referefenced the respective code location (build) and their dependencies (depends_on) (for example, the tweet_collector depends on postgres and mongo, since the tweets are stored in these databases).

I also used Docker volumes to keep the data when the containers are stopped (data persistence).

version: '3'

services:
  tweet_container:
    build: tweet_collector/
    depends_on:
      - postgres_container
      - mongo_container
    volumes:
      - ./tweet_collector/:/app

  postgres_container:
    build: postgresdb
    image: postgres:13.0
    ports:
      - 5555:5432
    environment:
      - POSTGRES_USER=your_user
      - POSTGRES_PASSWORD=your_password

  mongo_container:
    build: mongodb
    image: mongo
    ports:
      - 27018:27018
    volumes:
      - ./mongodb:/app

  etl_container:
    build: etl_job/
    depends_on:
      - postgres_container
      - mongo_container
    volumes:
      - ./etl_job/:/app

  slackbot_container:
    build: slackbot/
    depends_on:
      - mongo_container
      - postgres_container
    volumes:
      - ./slackbot/:/app

Finally, here some of the CLI commands I used for managing the Docker containers (you can find more in their docs):

docker images to list all the used images (postgres and mongo)
docker ps -a to list all my containers
docker -v to mount volumes
docker build to build an image from a Docker file
docker run to run the containers

And that was it: my very first dockerized ETL pipeline –– a week's work and a few hours writing packed in a 6-minute blog post.

How to create animated scatterplots with seaborn and imageio

Lorena — Sun, 23 Jan 2022 18:20:02 +0000

In this quick tutorial, I'll show you how to create an animated scatterplot using the libraries matplotlib or seaborn and imageio. Here's the GitHub repo of this project.

The scatterplot illustrates the relationship between life expectancy and fertility rate of world's countries from 1960 to 2015, based on the Gapminder data set:

	country	year	population	life_expectancy	fertility_rate	continent
0	Afghanistan	1800	3280000.0	28.21	7.0	Asia

The animated scatterplot is basically made of several overlapping static plots. Here's how to create the animation step-by-step:

1. Create static scatterplots for each year in the data set.

The scatterplots depict life_expectancy on the x axis and fertility_rate on the y axis. To make the plots even more insightful, the size of the points illustrates the population number and their color illustrates the continent.

import matplotlib.pyplot as plt 
%matplotlib inline
import seaborn as sns

sns.scatterplot(x='life_expectancy',
                y='fertility_rate',
                hue='continent',
                size='population',
                sizes=(10, 1000),
                legend=False,
                data=gapminder_df.loc[gapminder_df['year']==year],
                alpha=0.7,
                palette='Set2')

plt.title(f'{year}', loc='center', fontsize=20, color='black', fontweight='bold')

plt.xlabel('Life expectancy')
plt.ylabel('Fertility rate')

2. Export the scatterplot images to a designated folder.
You need to save all the individual scatterplots, so that you can overlap the images in the next step.

import imageio
import os

images = []
folder='/path/to/folder/images'

if not os.path.exists(folder):
    os.mkdir(folder)

3. Join the individual images in chronological order.

filename = f'lifeexp_{year}.png'
plt.savefig(os.path.join(folder,filename))
images.append(imageio.imread(os.path.join(folder,filename)))

4. Export the scatterplots sequence as a gif.
The fps (frames per second) parameter sets the speed of the animation.

imageio.mimsave(os.path.join(folder,'scatterplot.gif'), images, fps=20)

Now, putting everything together, here's the full code and the animated scatterplot:

import pandas as pd
import matplotlib.pyplot as plt 
%matplotlib inline
import seaborn as sns
import imageio
import os

images = []
folder='/home/lorena/Documents/bootcamp/W1/images'

if not os.path.exists(folder):
    os.mkdir(folder)

for year in range(1960, 2016):

    plt.axis((0, 100, 0, 10))

    sns.scatterplot(x='life_expectancy',
                    y='fertility_rate',
                    hue='continent',
                    size='population',
                    sizes=(10, 1000),
                    legend=False,
                    data=gapminder_df.loc[gapminder_df['year']==year],
                    alpha=0.7,
                    palette='Set2')

    plt.title(f'{year}', loc='center', fontsize=20, color='black', fontweight='bold')
    #plt.title(f'inspired by Hans Rosling', loc='right', fontsize=10, color='grey', style='italic', pad=-20)

    #plt.legend(bbox_to_anchor=(0.74, 0.85), loc='center')

    plt.xlabel('Life expectancy')
    plt.ylabel('Fertility rate')
    #plt.annotate({country}, )

    filename = f'lifeexp_{year}.png'

    plt.savefig(os.path.join(folder,filename))

    images.append(imageio.imread(os.path.join(folder,filename)))

    plt.figure()

imageio.mimsave(os.path.join(folder,'scatterplot.gif'), images, fps=20)

5 ways to keep your skills fresh after finishing a coding bootcamp

Lorena — Sun, 28 Nov 2021 09:37:25 +0000

One year ago at this time, I was nervously making last-minute changes to slides for my final project of a Data Science Bootcamp.

Today, I work as a technical writer at a startup that is developing a low-code workflow automation tool. Though in this role I don't use my data science and Python skills on a daily basis, I still apply them occasionally in data analyses and personal projects.

In this post, I'll share with you five tips for maintaining and even developing your coding skills after you're done with formal education.

1. Improve your school projects

Bootcamps are fast-paced. So much so that you might barely complete some projects before the deadline and if you somehow do, they still won't be perfect. There will always be things left to improve, and you should take the time to do them.

One way to improve your projects and coding skills is to try new models and libraries. For example, if you did classification with logistic regression, try also with random forest; if you used Tensorflow, now try Keras; if you scraped a website with BeautifulSoup, now do it with Scrapy. You get the point.

2. Work on new projects

Even with all the mandatory projects you'll need to complete in the bootcamp, you'll probably get a lot of ideas for others. After the bootcamp is the time to explore them!

Ideally, work on real-life projects or some that have business value for the field you're targeting. There are many data sets available for marketing, finance, medicine, and other fields. Find a relevant data set and apply different models to derive insights from raw numbers. For more ideas, check out the Kaggle data sets and competitions.

3. Code regularly

What you do every day matters. Small things add up: actions turn into habits turn into skills. That's why it's important to code regularly. It doesn't have to be a complex project, even a 15-minute coding session or a short exercise counts.

For example, you can block one hour every Saturday to practice algorithms and data structures on LeetCode, Codewars, or HackerRank. You'll not only sharpen your coding skills, but also get a confidence boost as you progress through levels and get badges.

4. Keep on learning

In data science, machine learning, and AI, research and applications are advancing fast! New papers, models, libraries, and business applications are coming out almost every day. That's why it's important to keep up with the news and advances in the field.

There are many resources for this. You can read blogs (like Towards Data Science and Data Science Central) and papers, watch videos (for NLP enthusiasts I recommend the YouTube channel AI Coffee Break with Letitia), listen to podcasts, and take online courses.

5. Document your learnings

I find the best way to learn something is by teaching it. Explaining something to others helps you structure the learned information and identify problems or issues that are unclear.

To document your learnings, you can create a blog on Medium or dev, where you write about your projects. If you're into web development, you can even build your own blog (I made mine with Jekyll).

With these five tips in mind, keep on coding and learning!

6 features of tasks that can be automated 📑

Lorena — Sat, 20 Nov 2021 07:52:57 +0000

You're working long hours now, six in the morning to six in the afternoon. Sometimes even eight in the afternoon, six days a week. Sometimes seven days a week. It's a long hustle but it keeps you busy.

Busy but unfulfilled, because many of the things you do are plain boring, repetitive, unengaging, and could probably be done (better) by a machine. If you've found your way to this article, it means you've had it with manual work and you're ready to start automating at least part of it. Congratulations, welcome to the future of work!

There are plenty of things in our daily lives that surely would rather be automated than half-heartedly accomplished by a bored human who, mind you, might even make mistakes. Workflow automation platforms like n8n enable you to automate even complex tasks with no code (or a bit of JavaScript, if you insist).

But how do you decide where to begin? In this post, I present to you six features of tasks that can (and should) be automated.

1. Repetitive tasks

Download this from here, upload it there, write this, click that. Repeat 10 times a day, 50 times a year, until the end of time. Or until you discover automation, because tasks like this shouldn't be accomplished manually anymore.

For example, getting your daily news, checking the weather, creating backups of your work (this is also a reminder to do it!), or cross-posting articles on different channels are all activities that can be automated with no code.

2. Boring tasks

"My favorite part of my job is copy-pasting data from one file into a spreadsheet!", said no one ever. More likely, you wish you could be doing anything but that. If a task is unengaging, it doesn't involve decision-making, higher-order thinking, creativity, or "the human touch", then it's a good candidate for automation.

For example, a common activity in sales is collecting information about companies (like the number of employees, industry, and location) from the website of a business event, in order to create contacts or leads in a CRM. Instead of manually copy-pasting data, you can create a workflow that does web-scraping, data transfer, and even email validation for you.

3. Frequent, regular tasks

It's the end of yet another month and you need to calculate yet another budget for your business, run an inventory on your products and orders. You need to set a reminder and block a full day to get this job done–a day when you could be doing more exciting work or even take a holiday.

If the task has to be done at the same time or interval and it involves the same sequence of steps, then it could be automated. In fact, we've already built a workflow for the use case mentioned above.

4. Rule-based tasks

Most automation-friendly tasks are rule-based, meaning they follow a logical sequence of steps in the form of "if A, then B, else C". This is the kind of low-level decision-making that can be established by a human and, if you know that the process won't change, delegated to a machine.

For example, filtering sales orders based on their value was a boring task that our friend Nathan had to do for his team, before we taught him how to automate it. In the same way, you could automate your e-commerce business, for example by filtering positive and negative reviews or issuing invoices.

5. Software-based tasks

Did you know that U.S. users had on average 20 apps installed on their mobile? And that organizations worldwide were using on average 80 software as a service (SaaS) applications? Now think of all the tasks that you're doing daily and how many of those involve transferring or synchronizing data between different apps, without human input.

For example, if you need to sync data between your CRM and a database, you can create a no-code workflow for that and let the two systems communicate with each other. Don't act as an intermediary machine, you're just making it awkward.

6. Time-consuming tasks

I don't know about you, but for me, the most annoying thing about boring tasks is that they are time-consuming. Fine, life and work are not always rainbows and butterflies, sparkling with creativity and meaningful activities. I can do some brainless tasks for a while if needed, but when they start taking up hours of my precious time–that's where I draw the line and reach for n8n.

For example, in one of my previous roles, I was responsible for creating reports, which involved aggregating data from different sources (Google Analytics, BigQuery, Salesforce, Postgres), calculating some custom metrics, and sending the results to management or clients. This kind of reporting could take up to two hours, every month/quarter/year–or only a few minutes once to set up a workflow in n8n.

In this post, you've learned how to identify tasks that can be automated. To sum up, automatable tasks are repetitive, boring, regular, rule-based, software-based, and time-consuming. Keep this in mind next time you're working on something, and try to automate it!

Detecting emotions from speech with neural networks in Python

Lorena — Fri, 12 Nov 2021 12:11:44 +0000

During a data science bootcamp, I built a machine learning model that detects emotions from speech (pre-recorded files and live-recorded voices). The code is available on my GitHub.

This has been one of the most challenging projects I've worked on, but also the most exciting. In this post, I'll walk you through my project: from planning adn choosing a data set to building machine learning models and evaluating their performance.

Project planning

First and foremost, I designed a project plan, after having a brief look at the data set. From my work experience and the assignments completed in the past three months, I've learned that this step is crucial for the success of a coding project. Planning helps me (and the team) organize my ideas, break down the big project into smaller tasks, identify issues, and track the progress -- and not despair at the amount of work to be done in a short time.

For this purpose, I created a simple Kanban board directly in the GitHub repository of my project, so that I have the code and tasks in one place.

Project board in GitHub

To create a project board linked to a repository in GitHub:

In your desired repository, click on the tab Projects, then on Create project.
Enter the Project board name.
(Optional) Enter a Description of the project and select a Project template.
Click on Create project.

Data set

I used the RAVDESS data set, which contains 1440 audio files. These are voice recordings of 24 actors (12 male, 12 female) who say two sentences in two different intensities (normal and strong) with eight intonations that express different emotions: calm, happy, sad, angry, fearful, surprised, disgusted, and neutral. There are 192 recordings for each emotion, except for neutral, which doesn't have recordings in strong intensity.

To sum up, the original RAVDESS data set includes:

1440 recordings
24 speakers
12 male, 12 female
2 sentences
2 intensities
8 intonations / emotions
192 recordings for 7 emotions
96 recordings for 1 emotion

RAVDESS data set distribution

Oversampling

The data set was imbalanced, so I used the RandomOversample method to create new features for the neutral class.

def oversample(X, y):
    X = joblib.load("speech_emotion_recognition/features/X.joblib")
    y = joblib.load("speech_emotion_recognition/features/y.joblib")
    print(Counter(y)) 

    oversample = RandomOverSampler(sampling_strategy="minority")
    X_over, y_over = oversample.fit_resample(X, y)

    X_over_save, y_over_save = "X_over.joblib", "y_over.joblib"
    joblib.dump(X_over, os.path.join("speech_emotion_recognition/features/", X_over_save))
    joblib.dump(y_over, os.path.join("speech_emotion_recognition/features/", y_over_save))

Oversampling added 96 new datapoints, so in the end I had 1536 audio files to work with.

Another imbalance was gender-related: there were slightly more recordings by males and in normal intensity. I didn't deal with this imbalance because it wasn't significant to my project, since I only wanted to predict the emotion. However, it would be interesting to explore in the future.

Feature extraction

There are many features that can be extracted from audio files, but I decided to work with the Mel Frequency Cepstral Coefficient (MFCC).

Mel-frequency cepstrum (MFC) is a representation of the short-term power spectrum of a sound, based on a linear cosine transform of a log power spectrum on a nonlinear mel scale of frequency. Mel-frequency cepstral coefficients (MFCCs) are coefficients that collectively make up an MFC.

The difference between the cepstrum and the mel-frequency cepstrum is that in the MFC, the frequency bands are equally spaced on the mel scale, which approximates the human auditory system's response more closely than the linearly-spaced frequency bands used in the normal spectrum. This frequency warping can allow for better representation of sound, for example, in audio compression.

source

To extract the MFCC from the audio files, I used the Python library librosa:

def extract_features(path, save_dir):
    feature_list = []

    start_time = time.time()
    for dir, _, files in os.walk(path):
        for file in files:
            y_lib, sample_rate = librosa.load(
                os.path.join(dir, file), res_type="kaiser_fast"
            )
            mfccs = np.mean(
                librosa.feature.mfcc(y=y_lib, sr=sample_rate, n_mfcc=40).T, axis=0
            )

            file = int(file[7:8]) - 1
            arr = mfccs, file
            feature_list.append(arr)

    print("Data loaded in %s seconds." % (time.time() - start_time))

    X, y = zip(*feature_list)
    X, y = np.asarray(X), np.asarray(y)
    print(X.shape, y.shape)

    X_save, y_save = "X.joblib", "y.joblib"
    joblib.dump(X, os.path.join(save_dir, X_save))
    joblib.dump(y, os.path.join(save_dir, y_save))

    return "Preprocessing completed."

The visual representation of MFCC looks like this:

MFCC plot

Machine Learning Models

I trained three different neural networks models on the MFCC and emotion labels:

Multi-Layer Perceptron (MLP)

def mlp_classifier(X, y):
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42
)

mlp_model = MLPClassifier(
    hidden_layer_sizes=(100,),
    solver="adam",
    alpha=0.001,
    shuffle=True,
    verbose=True,
    momentum=0.8,
)

Convolutional Neural Network (CNN)

def cnn_model(X, y):

X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42
)

x_traincnn = np.expand_dims(X_train, axis=2)
x_testcnn = np.expand_dims(X_test, axis=2)

model = Sequential()
model.add(Conv1D(16, 5, padding="same", input_shape=(40, 1)))
model.add(Activation("relu"))
model.add(Conv1D(8, 5, padding="same"))
model.add(Activation("relu"))
model.add(
    Conv1D(
        8,
        5,
        padding="same",
    )
)
model.add(Activation("relu"))
model.add(BatchNormalization())
model.add(Activation("relu"))
model.add(Flatten())
model.add(Dense(8))
model.add(Activation("softmax"))

model.compile(
    loss="categorical_crossentropy",
    optimizer="adam", 
    metrics=["accuracy"],
)

cnn_history = model.fit(
    x_traincnn,
    y_train,
    batch_size=50, 
    epochs=100,
    validation_data=(x_testcnn, y_test),
)

Long Short-Term Memory (LSTM)

def lstm_model(X, y):
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42
)
X_train_lstm = np.expand_dims(X_train, axis=2)
X_test_lstm = np.expand_dims(X_test, axis=2)

lstm_model = Sequential()
lstm_model.add(LSTM(64, input_shape=(40, 1), return_sequences=True))
lstm_model.add(LSTM(32))
lstm_model.add(Dense(32, activation="relu"))
lstm_model.add(Dropout(0.1))
lstm_model.add(Dense(8, activation="softmax"))

lstm_model.compile(
    optimizer="adam", loss="categorical_crossentropy", metrics=["accuracy"]
)

lstm_model.summary()
lstm_history = lstm_model.fit(X_train_lstm, y_train, batch_size=32, epochs=100)

After several iterations of tweaking the hyperparameters, I found that generally the models performed better with low learning rates (0.001), adam optimizer, and less layers. All models overfit (they couldn't generalize on unseen data), but this seems to be a common issue in neural networks and on audio data.

As expected, MLP had the lowest accuracy, since it's a very basic model (a simple feed-forward artificial neural network). CNN and LSTM had similar train accuracy (80%), but CNN performed better on test data (60%) than LSTM (51%). To give you some context, state-of-the-art models for speech classification have an accuracy of 70-80%, so I was quite happy with my CNN model accuracy.

Accuracy of different ML models

It was particularly interesting to look at the actual vs. predicted emotions, to see what emotions were misclassified. From the correlations matrices of CNN and LSTM, I noticed that both models misclassified emotions that sound similar or are ambiguous (even for humans), like sad-calm or angry-happy.

Confusion matrix of LSTM

Confusion matrix of CNN

Predictions

The exciting part was to make predictions on new data, more specifically on movie sound clips and my own voice in real-time. To record my voice, I used the Python library sounddevice:

import soundfile as sf
import sounddevice as sd
from scipy.io.wavfile import write


def record_voice():
    fs = 44100  # Sample rate
    seconds = 3  # Duration of recording
    # sd.default.device = "Built-in Audio"  # Speakers full name here

    print("Say something:")
    myrecording = sd.rec(int(seconds * fs), samplerate=fs, channels=2)
    sd.wait()  # Wait until recording is finished
    write("speech_emotion_recognition/recordings/myvoice.wav", fs, myrecording)
    print("Voice recording saved.")

I then tested the CNN and LSTM models on pre- and live-recorded audio files:

def make_predictions(file):
    cnn_model = keras.models.load_model(
        "speech_emotion_recognition/models/cnn_model.h5"
    )
    lstm_model = keras.models.load_model(
        "speech_emotion_recognition/models/lstm_model.h5"
    )
    prediction_data, prediction_sr = librosa.load(
        file,
        res_type="kaiser_fast",
        duration=3,
        sr=22050,
        offset=0.5,
    )

    mfccs = np.mean(
        librosa.feature.mfcc(y=prediction_data, sr=prediction_sr, n_mfcc=40).T, axis=0
    )
    x = np.expand_dims(mfccs, axis=1)
    x = np.expand_dims(x, axis=0)
    predictions = lstm_model.predict_classes(x)

    emotions_dict = {
        "0": "neutral",
        "1": "calm",
        "2": "happy",
        "3": "sad",
        "4": "angry",
        "5": "fearful",
        "6": "disgusted",
        "7": "surprised",
    }

    for key, value in emotions_dict.items():
        if int(key) == predictions:
            label = value

    print("This voice sounds", predictions, label)

Both models identified the correct or plausible emotion from recorded speech!

Next steps

It was super exciting to work on this project and I'm already thinking of improving and extending it in some ways:

Try other models (not necessarily neural networks).
Extract other audio features to see if they are better predictors than the MFCC.
Train on larger data sets, since 1500 files and only 200 samples per emotion is not enough.
Train on natural data, i.e. on recordings of people speaking in unstaged situations, so that the emotional speech sounds more realistic.
Train on more diverse data, i.e. on recordings of people of different cultures and languages. This is important because the expression of emotions varies across cultures and is influenced also by individual experiences.
Combine speech with facial expressions and text (speech-to-text) for multimodal sentiment analysis.

Create a toxic language detector for Telegram in 4 steps 🤬

Lorena — Tue, 21 Sep 2021 08:06:38 +0000

When was the last time you talked to someone online, be it friends, coworkers, or even strangers? Nowadays, you most likely do it every day. Online communication platforms like Telegram, Reddit, or Discord have made it possible for people from all over the world to connect and share their thoughts on pretty much any topic, instantly. This can be an enriching experience for users, but these platforms can also foster toxicity like cyberbullying, threats, and insults, forcing some users offline and silencing their voices.

One solution to this problem comes from Jigsaw and Google's Counter Abuse Technology team, who developed Perspective API: a free API that uses machine learning to identify toxic language in English, Spanish, French, German, Portuguese, Italian, and Russian. Toxic language is defined here as"a rude, disrespectful, or unreasonable comment that is likely to make someone leave a discussion".

Perspective API in action

In practice, Perspective scores a phrase based on the perceived impact the text may have in a conversation. The phrase can be analyzed on different attributes: flirtation, identity attack, insult, profanity, sexually explicit, threat, and (severe) toxicity. Keep in mind though that machine learning models can only be as good as the data they're trained on. This means that they may misclassify as toxic some innocent comments (and vice versa), so the flagged comments should be reviewed by a human eye.

Perspective API has been implemented by several major publishers and platforms like Reddit, The New York Times, and DISQUS, helping them moderate online comments. At n8n, we communicate with our 16,000+ community members in the Discourse forum, on Discord, Twitter, and even via Telegram for Spanish speakers. We value open, inclusive, and respectful communication and want to ensure that everyone has a positive experience in the n8n community – and beyond.

To this end, we used the Perspective API to build the Google Perspective node, which allows you to integrate toxic language detection in your workflows.

Workflow for detecting toxic language in Telegram messages

To give you an idea of how you can use the Google Perspective node, we created a workflow that detects toxic language in messages sent in a Telegram chat and replies with a warning message.

Workflow for detecting toxic language in Telegram

Telegram Trigger node starts the workflow when a new message is sent in a Telegram chat.
Google Perspective node analyzes the text of the message and returns a probability value between 0 and 1 of how likely it is that the content is toxic.
IF node filters messages with a toxic probability value above 0.7.
Telegram node sends a message in the chat with the text "I don't tolerate toxic language" if the probability value is above 0.7.
NoOp node takes no action if the probability value is below 0.7. This node is optional and serves only to show that the workflow can be extended in this direction.

Now let's see how to configure each node step by step. If you haven't built an n8n workflow yet, you might want to take a look at our quickstart guide or take the beginner's course. This will help you understand the configuration of the nodes used in this workflow.

1. Get new messages from Telegram

First of all, you need to create a Telegram bot and get credentials. Start a chat with Botfather in your Telegram account and follow the instructions to create your bot and get credentials. Make sure you add your newly created bot to the channel you want to monitor.

Then, open the Telegram Trigger node and add your Credentials Name and Access Token in Telegram API.

In the Updates field select: message, edited_message, channel_post, and edited_channel_post. These update options will trigger the workflow when a text message is posted.

To test if the bot works well so far, execute the Trigger node and send a message to the Telegram channel. We tested this workflow with the message "You're a stupid bot! I hate you!" (we swear it's just for testing purposes, we actually think bots are pretty cool and smart). The Telegram Trigger node should output the following result:

Configuration of the Telegram Trigger node

2. Analyze the toxicity of the message

In the second step, the incoming message from Telegram has to be analyzed with Perspective. In the Google Perspective node configure the following parameters:

Operation: Analyze ContentThis operation analyzes the incoming text message.
Text: {{$json["message"]["text"]}} This expression selects the incoming Telegram message to be analyzed.

In the section Attributes to Analyze you can add one or more attributes supported by Perspective that you want to be detected in the incoming message. If you don't add any attribute, all will be returned by default. For this example, the node is configured to detect profanities and identity attacks in the text, so two attributes are added with the properties:

Attribute Name: Profanity
Score Threshold: 0.00 This value sets the score above which to return results. The score is a value between 0 and 1 representing the probability that the text is toxic; it doesn't reflect the intensity (how toxic the text is). For example, if you set the Score Threshold at 0.5, then only messages that are 50% likely to be toxic are returned. If no value is set, at zero all scores are returned. You can read more in this article about what the scores mean.

In the section Options, you can select the Language of the text input. This option is useful if you want to monitor only a specific language. If unspecified, the node will auto-detect the language. In our example, we select the Language English.

Now if you execute the Google Perspective node, the output should look like this:

Configuration of the Google Perspective node

3. Filter toxic messages

In the third step, the toxic messages with a probability higher that 0.7 have to be filtered out. For this, you need to set up an IF node with the following parameters:

Value 1: {{$json["attributeScores"]["PROFANITY"]["summaryScore"]["value"]}} This expression selects the score value of the respective attribute.
Operation: Larger
Value 2: 0.7 This is the value we want to compare the score with.

If you execute the IF node now, it outputs the following results:

Configuration of the IF node

The message "You're a stupid bot! I hate you!" scored 0.92 for profanity and 0.62 for identity attack, which means it has downright strong toxic language on these attributes.

4. Send a warning message to Telegram

The final step is taking action against the toxic message. A mild action would be to just reply to the message in the Telegram channel warning the user that "We don't tolerate toxic language here!". To do this, configure the Telegram node with the following parameters:

Resource: Message
Operation: Send Message
Chat ID: {{$node["Telegram Trigger"].json["message"]["chat"]["id"]}}
Text: I don't tolerate toxic language!
Add Field > Reply to Message ID: {{$node["Telegram Trigger"].json["message"]["message_id"]}}

Configuration of the Telegram node

Now the bully will be publicly admonished in Telegram (once again, sorry, bot, you're really cool):

Response to a toxic message in Telegram

What's next?

In this post, you've learned about the challenge and importance of monitoring toxic language in online communities and how you can build a no-code Telegram bot for this purpose. The use case in this tutorial is fairly simplistic, but this kind of toxic language detector can be implemented in various platforms at scale.

For example, you could tweak this workflow and connect the Google Perspective node to Discord, Discourse, or DISQUS to detect toxic language in online communities and forums, or even to Gmail to filter out toxic emails. You can take different actions to toxic messages, for example forwarding them to a moderator, storing them in a database, flagging or banning the user depending on their message scores.

Here's what you can do next:

Try this workflow yourself: install n8n or sign up for a free 30-day trial on n8n.cloud ☁️
Discover more workflows using the Telegram (Trigger) node ⚙️
Join the discussion in the n8n community forum 🗣️
Read more workflow tutorials 💡

How to get started with CRM automation (with 3 no-code workflow ideas) 👥

Lorena — Tue, 14 Sep 2021 07:45:48 +0000

If you run a business, sell a product, or offer services, you know how important it is to nurture the relationship with your customers and gain new ones. Whether you operate alone or with a sales team, you probably also know how difficult it can be to keep track of your leads, customers, and orders.

We're here to show you that you can optimize your sales and customer workflows with two keywords: CRM and automation. Read on to learn what exactly a CRM is, why and when you should use a CRM for your business, and how to automate three common CRM sales workflows in only a few clicks.

Why and when you should use a CRM
3 steps to CRM automation for the sales funnel
- Choose the right CRM for your use case
- Decide what you will automate
- Build workflows
  - Capture leads from Typeform submissions
  - Send reminders after meetings with prospects
  - Process newly created deals based on their stage, value, and priority
What's next?

A customer relationship management (CRM) tool is pretty self-explanatory: it helps you manage the relationships with your customers. This means that it stores information on your customers (such as name, title, company, role), the relation they have with your company (e.g., lead, opportunity, deal), and the status of the relationship (e.g., closes, open, waiting), and the monetary value of closed deals (like quote and price).

The CRM software market is forecast to grow to $43.5 bn in 2024, proving the increasing popularity and necessity of CRM tools for companies.

Global CRM software market growth

Most commonly, CRMs are used by sales teams to track their sales activities and milestones in the sales funnel, from lead to deal.

Why and when you should use a CRM

To better understand the value of a CRM, let's take a business example.

Say you have a creative business specializing in professional corporate photography. Your visitors can book shootings on your website (which offers plenty of opportunities for automation), but you also actively network to find new opportunities. At a business event you've attended, you've met Marketing and Branding employees from different companies (contacts). Of these, a couple have expressed interest in your professional photo shootings for their team, qualifying them as leads for your business. This means they could become your customers, which is awesome! Now you need to follow up with them, make them an offer they can't refuse, and eventually close the deal.

The sales and marketing funnel

How can you and your sales team keep track of all these steps, for each lead, while also making sure that you nurture the relationship with your (potential) customers and organize the logistics with your team of photographers? If you think a digital calendar, a paper agenda, and a spreadsheet could do the job, you wouldn't be completely wrong. Sure they can help you organize your time and contacts, but provide limited features, are error-prone and even unmanageable in the long run.

Here are 3 common challenges that sales-oriented teams face in their organization:

Information is spread in different sources.\ You contact a customer via email, your salesperson talks to them on the phone, then you write down key information on a post-it and your colleague inserts the order details in a spreadsheet. If anyone asks something about that specific customer, you'd need to sift through emails and notes and ask several people who've come in contact with the customer--a highly inefficient process.
Information is duplicated or missing completely.\ If your salespeople have back-to-back calls and meetings with leads and customers, inevitably they might forget to pass on some information, or even assume that a colleague has taken care of that. At the other end, two salespeople might contact the same lead because they don't have an overview of their assignments. This is how meetings get overlooked, clients get annoyed, and you don't get orders.
Salespeople are unmotivated and exhausted by repetitive tasks.\ It's no secret that sales is a fast-paced and high-pressure field. But the role of a salesperson can become particularly challenging if they are often tasked with repetitive work (like sending the same email to different leads) or their work is inefficiently organized. These problems lead to a decrease in performance and invariably in sales.

A CRM solves the five problems listed above, by providing you with a feature-rich integrated system that can save your business up to 10 hours of work every week. In short, you should use a CRM if you sell a product, provide a service, or deal with customers or clients in any way.

If you are convinced by the advantages of a CRM, but reticent about the costs, note that the cost of not automating is higher than the investment in a CRM, which ultimately increases the productivity and value of your sales team.

3 steps to CRM automation for the sales funnel

Now that you've seen the advantages of using a CRM, it's time to start implementing it.

1. Choose the right CRM for your use case

On a first look, you might get overwhelmed by all the CRM providers available on the market. There are options for different team sizes, departments, and budgets. To help you get an overview of their features, we compiled a list of 10 of the most popular CRMs, which also come with n8n integrations that allow you to perform common CRUD (create, read, update, delete) operations on your saved contacts, companies, deals, and more.

CRM integrations on n8n

Agile CRM is an all-in-one CRM software for marketing, sales, and service. With the Agile CRM node you can manage company, contact, and deal details in your workflows.
Copper is a CRM integration for Google Workspace and is best suited for small and medium-sized businesses. The n8n nodes Copper and Copper Trigger provide the basic CRUD operations for companies, customer sources, leads, opportunities, persons, projects, tasks, and users.
HubSpot's CRM platform provides tools for social media marketing, sales, content management, and customer service. With the HubSpot node and HubSpot Trigger node you can manage contacts, contact lists, companies, deals, forms, and tickets.
Intercom is a conversational relationship platform which allows businesses to communicate with prospective and existing customers within their app, on their website, through social media, or via email. The Intercom node lets you manage companies, leads, and users from the CRM.
Keap offers an e-mail marketing and sales platform for small businesses, including products to manage and optimize the customer lifecycle, customer relationship management, marketing automation, lead capture, and e-commerce. The Keap node and Keap Trigger node allow you to manage companies, contacts, contact notes and tags, ecommerce orders and products, emails, and files.
Pipedrive is a cloud-based sales software company that aims to improve the productivity of businesses through the use of their software. You can use the Pipedrive node and Pipedrive Trigger node to manage activities, deals, deal products, files, leads, notes, organizations, persons, and products.
Salesforce is the leading vendor in the CRM market worldwide. Salesforce provides customer relationship management service and also sells a complementary suite of enterprise applications focused on customer service, marketing automation, analytics, and application development. The Salesforce node allows you to manage over 10 different fields, such as contacts, leads, opportunities, flows, and tasks.
Salesmate is a cloud-based CRM solution that caters to small and midsize businesses across various industries. Key features include contact management, sales pipeline management, email marketing and internal chat and phone integration. The Salesmate node lets you manage information about activities, companies, and deals.
Zoho CRM is an online Sales CRM software that manages sales, marketing and support. The Zoho CRM node allows you to manage accounts, contacts, deals, invoices, leads, products, purchase orders, quotes, sales orders, and vendors. With the Zoho CRM node you can perform CRUD operations on deals, invoices, leads, quotes, and many more.
Freshworks is a cloud-based CRM that helps businesses manage their interactions with their customers and leads. The Freshworks CRM nodeprovides basic operations for managing sales activities, tasks, deals, and more.

2. Decide what you will automate

After you've picked a CRM and explored its functionalities, you should define what you want to automate. Think of the tasks involved in every step of the sales funnel and ask yourself these questions:

Is the task repetitive?
Is the task time-consuming?
Do you need to perform the task often and regularly?
Does the task have a high value?

If you answered yes to these questions, then your task is most probably a case for automation. Once you've identified the pain points in your current manual workflows, you can start defining and designing automated workflows.

3. Build workflows

Before starting to create workflows for the tasks identified in the previous step, ask yourself one more question: do you need to automate tasks that take place only within the CRM or also between the CRM and other apps or services?

For the first case, note that some of the CRMs listed above offer built-in automation functionality for simple workflows. For the second case, you can take advantage of the n8n nodes, which allow you to connect your CRM to 200+ apps or services.

To help you get started, we've created 3 workflows with HubSpot and Pipedrive for automation at every step of the sales journey. Of course, you can replace the HubSpot and Pipedrive nodes with another CRM of your choice.

Capture leads from Typeform submissions

Typeforms are a presentable and efficient way of capturing leads and feedback from your customers. For example, you can embed a typeform on your website where visitors can request a quote for your services, or one which asks them to submit their contact details in order to download gated content.

This workflow is triggered when a typeform is submitted, then it saves the sender's information into HubSpot as a new contact.

Workflow for capturing leads from Typeform submissions

Send reminders after meetings with prospects

We mentioned that a common problem within sales teams is synchronization and information transfer since salespeople might forget to note down details from their conversations with leads, jumping from one call to another.

This workflow is triggered when a client meeting is scheduled via Calendly. Then, an activity is automatically created in Pipedrive, to keep track of the lead cycle. Fifteen minutes after the end of the meeting, a message is sent to the responsible salesperson in Slack, reminding them to write down their notes and insights from the meeting with the lead.

Workflow for sending reminders after Calendly meetings

Process newly created deals based on their stage, value, and priority

You're reaching the bottom of the sales funnel and getting deals--good for you! From here, there are several tasks you can automate to speed up the sales process.

This workflow is triggered when a new deal is created in HubSpot. Then, it processes the deal based on its type and stage.

The first branching follows three cases:

If the deal is closed and won, a message is sent in a Slack channel, so that the whole team can celebrate the success.
If a presentation has been scheduled for the deal, then a Google Slides presentation template is created.
If the deal is closed and lost, the deal's details are added to an Airtable table. From here, you can analyze the data to get insights into what and why certain deals don't get closed.

The second branching follows two cases:

If the deal is for a new business and has a value above 500, a high-priority ticket assigned to an experienced team member is created in HubSpot
If the deal is for an existing business and has a value below 500, a low-priority ticket is created.

Workflow for processing new leads created in HubSpot

Apart from Typeform, you can also use the Eventbrite Trigger nodeto capture the contact information of people who registered for an event, or the SurveyMonkey Trigger node to save the responses of a survey.

What's next?

In this post, you've learned about the advantages of CRM tools, when and why you should use a CRM, and what workflows you can automate with different CRMs. You've seen how automating different processes in the sales funnel can increase your productivity and minimize the time between the first contact and a closed deal.

Try these workflows yourself: install n8n or sign up for a free 30-day trial on n8n.cloud ☁️
Discover more workflows using CRM nodes ⚙️
Join the discussion in the n8n community forum 🗣️
Read more posts about workflow ideas 💡

6 e-commerce workflows to power up your Shopify store 🛒

Lorena — Fri, 03 Sep 2021 09:08:04 +0000

The online shopping trend has been driven by increasing digitalization in the past years, and the COVID-19 outbreak has only fueled e-commerce growth. Consumers around the world are turning to online stores for pretty much all product categories, due to in-person restrictions or contamination concerns.

In 2020, retail e-commerce sales worldwide amounted to $4.28 trillion and in 2021, over 2.14 billion people worldwide are expected to buy goods and services online.

But as online shoppers have grown in numbers, so have digital shop owners. Thanks to e-commerce platforms, almost anyone can set up an online shop with only a few clicks. However, running even a small digital business also involves some manual, repetitive tasks that might add up and steal too much of your precious time. Luckily, these kinds of tasks can be automated.

In this post, we'll have a look at the most popular e-commerce platforms and how to automate common e-commerce workflows in n8n.

The e-commerce platforms leading the global market

Many e-commerce software platforms have emerged in the last years, but the most popular ones, leading the market, are WooCommerce and Shopify.

Market share of e-commerce software platforms worldwide in 2021

Shopify is a paid e-commerce platform that offers templates for quickly designing your online shop. With Shopify, you don't have to worry too much about technicalities and instead focus on selling your products.

WooCommerce is the second most popular e-commerce software platform as of April 2021, owning over 23% of the market share. WooCommerce is actually an open-source WordPress plugin, making it the go-to choice for smaller and cost-conscious shop owners.

Why and when you should automate your online shop

Regardless of the platform you choose or the products you sell, there might come a time when managing your orders becomes too time-consuming and you'll probably find yourself in one of these two common situations:

Your business is growing and you don't have the bandwidth anymore to manage all the orders. You might consider hiring one or two people to help you out, but be aware that additional employees could mean additional responsibilities for you.
Your online shop is a side hustle and you don't have much time to take care of it besides your main job. You neither want to compromise your career nor give up your hobby business.

If you identify yourself with one of these cases, you should consider automating at least part of your work. Workflow automation might sound intimidating or a skill reserved for the tech-savvy ones -- but don't be intimidated.

No-code tools with a visual user interface like n8n make automation feel like childsplay. You can easily combine WooCommerce and Shopify integrations with other apps or services to automate common workflows in your digital store.

Workflow automation ideas for Shopify

n8n offers four nodes for Shopify and WooCommerce: Shopify node, Shopify Trigger node, WooCommerce node, and WooCommerce Trigger node. They provide various operations for managing orders, products, customers, and carts. Read our docs to learn how to configure the nodes, then you can start configuring various parameters to build workflows for your online store.

The Shopify and WooCommerce nodes open up many possibilities for automation, helping you win back time and focus on things that matter. Here are six ideas of workflows you can automate for your Shopify store (and adapt for WooCommerce):

Promote your new products on social media

A survey on social commerce revealed that 43% of users research products online via social networks and 28% discover brands via ads on social media. This goes to show how important it is to have a presence on social media and regularly share and promote your products.

To help you automate your social media activity, we created a workflow that is triggered when you create a product on your shop and automatically shares the news on your Twitter account and a Telegram channel with the message "Hey there, my design is now on a new product ✨ Visit my [shop_name] shop to get this cool product_name 🛍️ [shop_link]".

Workflow for social media promotion

You can make the message even more appealing to your audience by adding a Bannerbear node that automatically creates template images for your new product announcements.

Update customer and order details in Zoho CRM

Once the first orders start to come in, it's time to neatly track the orders and nurture the relationship with your customers, ideally in a customer relationship management (CRM) system.

The first branch of this workflow saves customer orders from Shopify to the Zoho CRM and Trello. In the Zoho node, you can select the option Create or Update, which creates a new contact if a contact with a matching last name and email address exists. This way, you don't have to worry about duplicate contacts.

Workflow for saving order details from Shopify to Trello and Zoho CRM

Of course, you can replace the Zoho node with another CRM, for example, Salesforce or Agile CRM.

Create invoices for new orders

A not-so-fun part of being a shop owner is paperwork like invoices. Manually writing the details of each order for each customer is not only tedious but also error-prone. So why not automate this task?

The second branch of this workflow automatically generates invoices with Harvest when an order is created in Shopify. Then, it creates a Trello card with the order information and the invoice attached.

Workflow for creating invoices in Harvest from Shopify orders

Offer coupons and discounts to high-order customers

In 2020, the leading reasons why internet users around the world added a product to their online basket and purchased the item were free delivery, coupons or discounts, and reviews from other customers.

Though free delivery is up to your pricing model, you can (and should) invest in the latter two incentives. For example, you can offer discounts to high-order customers.

The third branch of this workflow is triggered when a new order is created and checks if the order value is above 100 (or any value you set) -- if it is, it sends an email to the customer with a 10% discount coupon for their next order, otherwise, it sends them an email thanking them for their order.

Workflow for filtering high-value customers from Shopify

For extra productivity, you can combine this and the previous two workflows into one super-workflow:

Workflow for processing new Shopify orders

Request customers to write a review after they have received their order

In the previous workflow, we've mentioned product reviews and the third reason why customers buy products online. In recent years, it has become increasingly important to the consumer to read up on a product, business, or service before spending any money. In 2020, reviews were the third top reason that convinced shoppers to purchase a product online. This year, nearly 70% of online shoppers typically read between one and six customer reviews before making a purchasing decision.

This proves the value of investing in customer experience and incentivizing shoppers to write reviews about your products. To incentivize them, you can tweak this workflow to be triggered when an order is marked as fulfilled by selecting the Topic: Order Fulfilled in the Shopify Trigger node. Then, an email should be sent to the customer, asking them to write a review about their product.

Run sales inventories and reports in Google Sheets

When your online shop generates a steady flow of orders, it's necessary to keep an inventory and track your growth regularly. For small businesses, even a Google Sheet can be enough for keeping records.

This workflow is scheduled to run every week, when it gets all your Shopify orders, calculates their sales value, and stores the data in Google Sheets for you to evaluate. Additionally, it can send a message to a Slack channel (if you work together with a small team) or Telegram to inform you about your weekly sales.

Workflow for running inventories of Shopify orders

What's next?

In this post, we talked about the growing popularity of e-commerce and possibilities for automation. Whether as a full-time business or a fun side hustle, if you want to sell goods online -- from art prints to clothing and cosmetic products -- e-commerce platforms like Shopify and WooCommerce enable you to set up an online shop with only a few clicks.

Now that you've learned how to automate six common workflows in your online shop to save precious time, here's what you can do next:

Try these workflows yourself: install n8n or sign up for a free 30-day trial on n8n.cloud ☁️
Discover more workflows using the Shopify (Trigger) and WooCommerce (Trigger) nodes on the n8n workflows page -- and feel free to share your workflows as well ⚙️
Join the discussion in the n8n community forum 🗣️
Discover more automation use cases 💡

Web-scraping IMDb with R

Lorena — Mon, 28 Jun 2021 10:45:52 +0000

You don't need to be a data scientist to work with data. In fact, you most probably use a lot of data every day in both your professional and personal life. For example, price comparison sites, movie rankings, sales leads, and trending topics are all examples of data that is available online in some sort of table form.

Why web-scraping?

If you want to collect or analyse this data, copy-pasting each record is most likely out of the question. This manual task is not only time-consuming, but also expensive and prone to error – not to mention it's also downright boring.

Here's where automatic web-scraping can help! Web scraping is a method of automatically gathering data from websites in a structured manner and storing it into a local database or spreadsheet.

There are many no-code tools for web scraping, like browser plug-ins (e.g. Webscraper) and software (e.g. Parsehub). However, if you need more advanced scraping settings and have basic coding skills, I recommend the Python libraries Beautiful Soup or Selenium, and the R package rvest. The latter is the one I used for scraping IMDb and you can find the commented code on my GitHub.

Before I proceed to the fun part, note that the legality of web scraping is not clearly defined around the world, so you should check the website's terms of use before scraping it!

Now let's dive in. I wanted my data analysis to answer three questions:

What are the most successful movies released in 2018?
What genres do the popular movies belong to?
What is the duration of the most popular movies?

Top popular movies

I used IMDb as a reference, because it contains all the information I need. On the website I selected the movies released between 01.01.-31.12.2018, sorted by popularity, and limited my search to the first page, so the top 50 movies.

library(rvest)
url <- "https://www.imdb.com/search/title?year=2018"
imdb <- read_html(url)
head(imdb)

The top 5 popular movies in 2018 were:

Aquaman
Green Book
Bohemian Rhapsody
Spider-Man: Into the Spider-Verse
Avengers: Infinity War

Top movie genres

Scraping the genre tags of each movie is pretty straightforward:

genre_data_html <- html_nodes(imdb, ".genre")
genre_data <- html_text(genre_data_html)
head(genre_data)

However, this returns a list of genres for each movie, because the movies are labeled with multiple genres. The text data needs to be cleaned a bit:

#remove the \n in front of the genres
genre_data <- gsub("\n", "", genre_data)

#remove the spaces between genres
genre_data <- gsub(" ", "", genre_data)

Now, another tricky thing is that the genres of each movie are enumerated alphabetically, not in order of importance. To simplify my work, I selected only the first genre:

#display only the first genre in the list
genre_data <- gsub(",.*", "", genre_data)

At a glance, I noticed that 3 out 5 are action-hero movies, so I visualized closer at the genre distribution:

#plot the number of movies by genre
library(ggplot2)
ggplot(imdb_df, aes(x=genre_data)) +
geom_bar(color="purple", fill="green", alpha=0.3) +
ggtitle("Number of movies by genre") +
xlab("Genre") + ylab("Number of movies")

My initial observation was confirmed: Action and Drama are the most popular genres, followed by Biography. I guess most people enjoy, on one hand, movies that transport them into wild worlds and simulate experiences out of the ordinary, and on the other hand, movies that depict dramatic life stories and relate to some extent to their real life.

Top movies duration

Next, I analyzed the distribution of movie duration:

#plot the movies by runtime
barplot(table(imdb_df$Runtime))
hist(imdb_df$Runtime)
ggplot(imdb_df, aes(x=runtime_data)) +
geom_histogram(color="purple", fill="green", alpha=0.3) +
ggtitle("Distribution of movie runtimes") +
xlab("Minutes") + ylab("Number of movies")

The plot shows that most popular movies last on average 104 minutes (median 117 minutes). The longest movie is Avengers: Infinity War (149 minutes) and the shortest movie (excluding TV-shows) is A.I. Rising (85 minutes). From the histogram it is clear that the bars on the left represent the TV-shows (under 60 minutes).

I also analyzed the runtime distribution by genre. First, I aggregated the movies by genre:

#group movies by genre
library(dplyr)
genre_cat = group_by(imdb_df, Genre)
genre_runtime = summarize(genre_cat, Minutes=mean(Runtime))
plot(genre_runtime)

Then, I visualized the average duration for each genre:

counts = table(genre_runtime$Genre, genre_runtime$Minutes)
ggplot(data=genre_runtime, aes(x=Genre, y=Minutes)) +
geom_bar(stat = "identity", color="purple", fill="green", alpha=0.3) +
ggtitle("Mean movie duration by genre")

I found that among genres Biographies are longest (on average 127 minutes) and Crimes are shortest (on average 85 minutes). This is not entirely surprising, since I think that, first, it is quite a challenge to pack a lifetime in a biographical movie, and second, there's only so much nerve-wrecking tension a person can take following a crime. However, I was expecting the average duration of Animations to be shorter than 110 minutes, because they are produced mainly for children, who have a short attention span and low patience to sit through a two-hour movie. But then again, we are talking about the most popular movies of last year on IMDb, which means that adults made up the large audience.

Conclusion

This is a simple web scraping project that can reveal a lot of information about people's movie preferences. It would be interesting to also analyze the total gross and see which movies and genres have sold best in 2018. Now you could try to scrape and analyze this information with your preferred tool and let me know what you found out!

5 tasks you can automate with the new Notion API ⚡

Lorena — Wed, 09 Jun 2021 15:03:56 +0000

If you're into productivity and organisation hacks, you've probably heard of (and even got to love) Notion, the all-in-one workspace app that allows you to take notes, create databases, manage projects, and schedule tasks–all with highly customisable designs.

At n8n, we've been using Notion since day one for the internal organisation: from meeting notes and onboarding checklists to content calendars and product research. Imagine our excitement when we heard that Notion launched their API (beta), thus opening new possibilities of using the app in a more personalised way!

Notion integrations

Our developers got to work right away and we launched two of the most awaited nodes: Notion node and Notion Trigger node.

The Notion node has five basic operations that allow you to manage blocks, get and query databases, create and update records in a database, create and search for pages, and get users who are part of your workspace. The Notion Trigger node allows you to check at regular intervals when a page is added to the database, then trigger a workflow.

Now you can easily connect your tools to Notion to sync data and boost your productivity. We're super excited to automate some of our workflows, and even more so to see what other creative ideas the n8n community comes up with. In this article, we'll present to you 5 Notion workflows you can automate in n8n:

Add candidates' profile assessment to Notion before an interview
Check to-do's in Notion and notify the assignee in Slack
Send notifications about new Notion notes to Mattermost
Add positive feedback messages to a compliments table in Notion
Add memorable articles to your Notion reading list

Add candidates' profile assessment to Notion before an interview

Scheduling interviews has become easier thanks to apps like Calendly, but evaluating candidates' skills and personality still requires a human touch and good psychological understanding. Humantic AI can complement recruiters' evaluation, by generating psychometric assessments (including DISC and the Big Five personality traits) from candidates' résumés.

For this use case, we created a workflow that is triggered when an interview is scheduled via Calendly. The Humantic AI node retrieves the LinkedIn profile of the candidate from Calendly, creates their psychometric assessment, then the Notion node inserts this information into a dedicated page.

By automating this process, you don't have to worry about your applicant database being up to date, and instead, you have more time to prepare for the meeting.

Check to-do's in Notion and notify the assignee in Slack

One of the most useful applications of Notion for teams is keeping to-do lists where you can assign tasks to specific people. A limitation is that the in-app or email notifications can easily be overseen and are out of context. To overcome this issue, you can use this workflow that regularly checks your to-do list in Notion and notifies the person in Slack when a new task is assigned to them.

Of course, you can replace the Slack node with another messaging app of your choice. We prefer Mattermost and have built several workflows around it, which substantially improve our communication and productivity.

Send notifications about new Notion notes to Mattermost

Another practical use of Notion pages is for taking notes in meetings and keeping them well-organised by topic or meeting type. After the meeting, you might first want to go get a coffee before taking care of the action items just discussed. But as it often happens, even a short break can interrupt your working flow, so you may easily forget what you needed to do or who you had to follow up with once you're back at your desk.

Luckily, you can automate your chores away! This workflow is triggered whenever new meeting notes are added in Notion. If the property field in the notes mentions the Marketing team, a message about the new notes will be sent to the Marketing channel in Mattermost, so all team members are up to date. Now you can enjoy your long-awaited coffee break knowing that the main action items will be taken care of.

Add positive feedback messages to a compliments table in Notion

We, humans, are prone to negativity bias, meaning that we remember negative experiences more (strongly) than positive ones. For example, in a business context, when getting feedback from customers on your product or service, you are more likely to keep in mind that one negative review over the other tens of praise messages. This bias can impact your perception of your work and the team spirit, so it's important to highlight the good news.

For this use case, we built a workflow that analyses the sentiment of feedback messages submitted via Typeform: messages with negative sentiment are added to a Trello board, while the ones with positive sentiment are added to a compliments table in Notion, then shared in a Slack channel. Looking once in a while at the impact you have on your customers is a wonderful way to keep your team motivated and inspired!

Add memorable articles to your Notion reading list

"The modern day reading list includes more than just books.", writes Notion in their Reading List template, and we couldn't agree more. Millions of blog posts, news articles, and research papers are published every day, serving you information on how to solve problems, improve various skills, make informed decisions--or are just food for thought. You'll most probably stumble on some articles worth sharing or saving for a later (re-)read.

To help you manage your reading list, we designed a workflow that automatically adds important articles to a Notion page from Discord. When you type in Discord the slash command /[URL], with the URL of the article you want to save, the workflow extracts the article title and adds the linked title to the reading list in your Notion page. When all is done, you get a confirmation message on Discord: "The link was added to Notion." Now you can focus only on reading and taking notes.

Next Steps

So, which workflow will you try first? On our blog, you can find more ideas of workflows that you can automate in the workplace and in your personal life, which you can now tweak to integrate Notion. Share your ideas with us on Twitter or in the community forum — we're curious to see what you build! And if you want to get the latest content on automation with n8n, subscribe to our newsletter 💌

6 findings from analysing Oscars speeches with Python

Lorena — Sun, 06 Jun 2021 11:36:44 +0000

On the occasion of the 93rd Oscars Award Ceremony, I was curious to do some text mining on the acceptance speeches. Specifically, I analysed the speeches of the Best Directors between 1941 and 2019. I used a dataset from Kaggle and added missing data for 2017, 2018, and 2019 directly from the Academy Awards Acceptance Speech database.

In total, 74 Best Directors have been awarded and almost all of them gave acceptance speeches, which I analysed with Python and the NLTK library. You can find the Jupyter notebook here. Let's see what the words reveal about the Best Directors and the Oscars!

1. Average speech length

The speech of a Best Director has 104 words on average, but speeches range widely from 8 to 267 words.

Here's how to calculate the number of words in a text:

directing["words"] = directing['Speech_clean'].str.split().str.len()

2. Longest & shortest speeches

The longest speech runs at 267 words and was given by Mel Gibson at the 68th Academy Awards in 1995 for his film Braveheart. This guy had a looot of people to thank to and seems to have used up all his words for saying pretty much nothing.

The shortest speech was summed up in 8 words by Delbert Mann at the 28th Academy Awards in 1955 for his film Marty. I really like his efficient "I came. I won. I thanked." structured speech:

Thank you. Thank you very much. Appreciate it.

Here's how to find the longest and shortest text in a dataframe with pandas:

directing.sort_values(by="words")

3. Lexical richness

Lexical richness is a measure of how many unique words are used in the text. Lexical richness is calculated as the total number of unique words divided by the total number of words. The higher the score, the richer the vocabulary–and vice-versa. Here's to calculate lexical richness for each speech in the dataframe with Python:

def lexical_richness(text):
    return round(len(set(str(text))) / len(str(text)), 3)
directing["lex_rich"] = [lexical_richness(directing["Speech_clean"][i]) for i in range(len(directing))]

The speech with the highest lexical richness (0.408) is Delbert Mann's, the director of Marty, awarded in 1955. This means that 40.8% of the words he used are distinct.

At the other end, the speech with the lowest lexical richness (0.034) is Mel Gibson's, the director of Braveheart, awarded in 1995. This means that 3.4% of the words he used are distinct.

4. Longest words

The longest words used in directors' speeches have 15 words: administrations, cinematographer, and czechoslovakian.

Here's how to select the longest words in a text:

long_words = [w for w in all_speeches_tokenized if len(w) > 14]
sorted(long_words)

5. Most common words

The top 10 most common words in all acceptance speeches are: thank (201 occurrences), much (56), like (50), people (48), want (42), would (30), movie (26), film (26), say (24), and many (22).

Interestingly, out of these 10 words, 3 are nouns (referring to people and film/movie), 2 express large quantities (much and many), and 5 are verbs that express personal feelings (want, like) or actions (say, thank). It's also worth noting that the word thank has a significantly higher frequency than the following common words, which is however understandable.

Here's how to find the frequency distribution of words in a text with NLTK:

FreqDist(all_speeches_tokenized).most_common(10)

6. "Thank" to...

Ok, winners thank a lot, but who do they thank to? It turns out... to you, but also to the Pacific Command of the United States, Mr. harry Cohn, Marlon, the producers, and each one of them.

Here's see the location of a word in context with NLTK:

Text(word_tokenize(all_speeches)).concordance('thank')

That's all, folks! Could/should I have analysed anything else? Let me know what you think in the comments below ⬇️

DEV Community: Lorena

How hyperautomation will transform business operations

Table of Contents

What is hyperautomation?

How is hyperautomation different from automation?

Why do we need hyperautomation?

How are businesses leveraging hyperautomation?

What is the future of hyperautomation?

1. Orchestrated automation processes

2. Automation marketplaces

3. Vendor-agnostic hyperautomation

4. Infrastructure automation

Start automating!

Building a dockerized ETL pipeline for streaming tweets in Slack

0. Prerequisites & tech stack

1. Collecting tweets

2. Storing tweets in MongoDB

3. Performing ETL job

3.1. Extracting tweets from MongoDB

3.2. Transforming tweets with sentiment scores

3.3. Loading tweets into PostgreSQL

4. Extracting tweets from Postgres

5. Posting tweets with a Slackbot

6. Creating the Docker Compose pipeline

How to create animated scatterplots with seaborn and imageio

5 ways to keep your skills fresh after finishing a coding bootcamp

1. Improve your school projects

2. Work on new projects

3. Code regularly

4. Keep on learning

5. Document your learnings

6 features of tasks that can be automated 📑

1. Repetitive tasks

2. Boring tasks

3. Frequent, regular tasks

4. Rule-based tasks

5. Software-based tasks

6. Time-consuming tasks

Detecting emotions from speech with neural networks in Python

Project planning

Data set

Oversampling

Feature extraction

Machine Learning Models

Predictions

Next steps

Create a toxic language detector for Telegram in 4 steps 🤬

Workflow for detecting toxic language in Telegram messages

1. Get new messages from Telegram

2. Analyze the toxicity of the message

3. Filter toxic messages

4. Send a warning message to Telegram

What's next?

How to get started with CRM automation (with 3 no-code workflow ideas) 👥

Why and when you should use a CRM

3 steps to CRM automation for the sales funnel

1. Choose the right CRM for your use case

2. Decide what you will automate

3. Build workflows

Capture leads from Typeform submissions

Send reminders after meetings with prospects

Process newly created deals based on their stage, value, and priority

What's next?

6 e-commerce workflows to power up your Shopify store 🛒

The e-commerce platforms leading the global market

Why and when you should automate your online shop

Workflow automation ideas for Shopify

Promote your new products on social media

Update customer and order details in Zoho CRM

Create invoices for new orders

Offer coupons and discounts to high-order customers

Request customers to write a review after they have received their order

Run sales inventories and reports in Google Sheets

What's next?

Web-scraping IMDb with R

Why web-scraping?

Top popular movies

Top movie genres

Top movies duration

Conclusion