DEV Community: Davide Gazzè

An overview of the Process Mining utilizing Python

Davide Gazzè — Tue, 07 Mar 2023 11:03:35 +0000

Process mining is a data mining technique to obtain knowledge utilizing the event data of the process itself. It is a valuable tool for businesses wanting to improve their business processes. More in detail, process mining is a type of data mining that analyzes business processes to improve process efficiency and effectiveness by revealing hidden insights and patterns in process data. Process mining begins with data collection from multiple sources, such as event logs from IT systems or BPMN process models. From the data, it is possible to generate a process map that depicts the flow of activities and their interactions. After creating the process map, it is possible to use numerous techniques to examine the data. Process discovery, for example, uses data to develop a process model without prior knowledge of the process. Conformance checking compares the actual process to a pre-defined process model to find variations and non-compliance. Process cycle time and resource consumption can be optimized through performance analysis. These identify regions with long processing cycle times or unused resources. You could use the insight to improve performance, resulting in cost savings, higher revenue, and increased customer satisfaction. You can integrate process mining into an organization’s workflow with the help of numerous Python modules to compute some metrics. Some of them are:

Process performance metrics such as process cycle time, throughput time, and waiting time
Process compliance metrics such as conformance, fitness, and simplicity
Resource utilization metrics such as resource workload and resource efficiency
Business performance metrics such as revenue, cost, and customer satisfaction
Quality metrics such as error rates and rework rates

Continue on An overview of the Process Mining utilizing Python

Language Detection with Fastlangid

Davide Gazzè — Sun, 05 Mar 2023 18:28:49 +0000

In this short post, I will show you an example of the module Spacy fastlangid. This module can detect the language of a text.

The module is based on the model lid.176.ftz based on fastText.

Usage

To use it, import the library and use it. Now, let's see an example:

from fastlangid.langid import LID
langid = LID()

Now, we have the langid object, and we can pass it to the predict:

result = langid.predict('This is a test')
print(f"The detected language is {result}")

The output is:

The detected language is en

For more details see this

From python function to latex description using latexify_py

Davide Gazzè — Fri, 03 Mar 2023 07:51:34 +0000

Hi everyone,
this is a small post about a simple, but an interesting project that generates LaTeX math descriptions from Python functions.
As usual, writing code is also an enjoyable activity, however, in the end, writing documentation is needed. Unfortunately, writing just text is not always sufficient, and some diagram is always useful. If you want, you can use PlantUML (more details are available here,
and here).
However, also mathematical functions are worthy of proper documentation. For these, normally, in academics, and industry, you can use Latex, read
here for an introduction. However, writing mathematical functions is time-consuming, the best would write the code, and use software that writes the latex formula for you.
This project is the goal of latexify_py.
The official notebook is available on Google Colab.

Installation

The installation of the package is simple:

# Install
pip install latexify-py==0.2.0

To ensure the correctness type:

import math
import latexify

latexify.__version__

the output will be:

0.2.0

One of the official examples is:

@latexify.function(reduce_assignments=True)
def solve(a, b, c):
  return (-b + math.sqrt(b**2 - 4*a*c)) / (2*a)

solve

the output is:

Moreover, some math symbols, like alpha, beta, gamma, and delta, are converted automatically like:

@latexify.function(use_math_symbols=True)
def f0(alpha, beta, gamma, omega, delta):
  return alpha * beta + gamma - omega * delta

f0

the output is:

The complete list is available here.
For example, we will implement the Collatz conjecture.
The conjecture concerns the following algorithm:

Take a positive integer n.
If n = 1, the algorithm ends.
If n is even, divide by two; otherwise multiply by 3 and add 1.

if you iterate the function, the result will converge to 1.

The code is the following:

@latexify.function(reduce_assignments=True)
def collatz(value):
  if value % 2 == 0:
    value = value / 2
  else:
    value = 3 * value + 1 

collatz

The output is:

very impressive, is it?

Summary

In this post, we see some practical examples of the package latexify_py. You can see the results in this Google Colab Notebook.
Finally, this project is currently hosted on https://github.com/google but is not officially supported by Google.
I hope this project will be expanded with some features, like the support of for.

Running Apache Airflow via Docker Compose

Davide Gazzè — Wed, 01 Mar 2023 09:09:37 +0000

Hi everyone,
in this post, I will summarize how to run Apache Airflow using Docker-compose.

Apache Airflow is a platform to programmatically author, schedule, and monitor workflows.
It is simple to start with it because:

Apache Airflow has a nice UI
Apache Airflow has a programmatic way to create workflows
Apache Airflow is very important in the community, and that means that you have a lot of courses and posts for starting
Apache Airflow is simple compared to Apache Nifi (see my articles here and here)

For your first steps, in Apache Airflow worlds, you can use the development as explained here. In this post, you will use SQLite as the database for running the tutorial.

However, if you want to use this fantastic tool, you can learn how to directly from the Apache Airflow website.

To summarize, the post will tell to you to:

select a Database backend like Mysql/MariaDB or Postgres
use the LocalExecutor for the local machine of Kubernetes executor or the Celery executor in a multi-node setup
setting Stackdriver Logging, Elasticsearch, or Amazon CloudWatch for saving the logs

The above information is just the starting point for a robust orchestrator in production.

As usual, it is not always simple to set up everything in your local machine for testing a production environment.
For this reason, Docker, and in particular docker-compose can help us.

As described here, you can test a production environment on your local machine.

The proposed configuration

An Airflow installation is composed of the following components:

scheduler for triggering scheduled workflow:
executor for running the tasks
webserver for managing, inspecting, triggering and debugging the DAGs and tasks
folder with all DAG files
metadata database for saving the states of the scheduler, executors and webserver.

The image below represents the Airflow architecture:

The proposed configuration is not a run production-ready Docker Compose Airflow installation. It is just a quick-start docker-compose to get your hands dirty with Airflow.

Of course, you have to install it on your laptop:

Docker Community Edition (CE)
Docker Compose v1.29.1 and newer

It is recommended to reserve at least 4GB (better 8GB) of memory for Docker.

The docker-compose.yaml file

The community file is available here, you can download it using the command:

curl -LfO 'https://airflow.apache.org/docs/apache-airflow/2.3.4/docker-compose.yaml'

The docker-compose file is formed by:

airflow-scheduler: The scheduler monitors all DAGs
airflow-webserver: The AIRFLOW webserver (available at the URL: http://localhost:8080)
airflow-worker: The executor of each task in a DAG
airflow-init: The initialization service
postgres: The database
redis: The broker that forwards messages from scheduler to worker
flower: The optional application that monitors the environment (you can start using: docker-compose --profile flower up)

The docker-compose set three volumes:

dags: the folder where you can put your DAG
logs: the folder that contains logs from task execution and scheduler
plugins: the folder where you can put your custom plugins

all of these volumes are persisted onto your local machine. So it is simple to perform some tests locally and then move to another machine without losing anything.
In any case, the folders are created in the same folder where the file docker-compose.yaml is present.
If you want, you can change the local folders in lines 63 - 65:

  volumes:
    - /tmp/airflow/dags:/opt/airflow/dags
    - /tmp/airflow/logs:/opt/airflow/logs
    - /tmp/airflow/plugins:/opt/airflow/plugins

if you do not want the initial dags, you should set false line 59:

    AIRFLOW__CORE__LOAD_EXAMPLES: 'true'

Another little improvement is to set the name of the PostgreSQL container adding the following in line 77:

    container_name: db

In this way, you can refer to the PostgreSQL database with the name db.
Otherwise, you could change line 83 to set the Postgres data folder.

Moreover, you can change some other basic configurations:

AIRFLOW_IMAGE_NAME: The Docker image name used to run Airflow (Default: apache/airflow:2.3.4)
AIRFLOW_UID: The user ID in Airflow containers (Default: 50000)
_AIRFLOW_WWW_USER_USERNAME: The username for the administrator account (Default: airflow)
_AIRFLOW_WWW_USER_PASSWORD: The password for the administrator account (Default: airflow)
_PIP_ADDITIONAL_REQUIREMENTS: The additional PIP requirements to add when starting all containers (Default: )

Toward the starting

Before starting everything, you have to set the following:

Create the Airflow folders:

mkdir -p ./dags ./logs ./plugins ./postgres-db
Set the Airflow user:

echo -e "AIRFLOW_UID=$(id -u)" > .env
Initialize the database:

docker-compose up airflow-init

In particular, after the last step, you will see the following output:

Attaching to airflow-init_1
....
airflow-init_1  | DB: postgresql+psycopg2://airflow:***@postgres/airflow
airflow-init_1  | Performing upgrade with database postgresql+psycopg2://airflow:***@postgres/airflow
airflow-init_1  | [2022-09-10 07:47:18,664] {db.py:1466} INFO - Creating tables
airflow-init_1  | INFO  [alembic.runtime.migration] Context impl PostgresqlImpl.
....
airflow-init_1  | Upgrades done
....
airflow-init_1  |   FutureWarning,
airflow-init_1  | 2.3.4
airflow-init_1 exited with code 0

Start Airflow typing:

docker-compose up -d

If everything goes well, the output of the docker ps is the following:

CONTAINER ID   IMAGE                  COMMAND                  CREATED        STATUS                    PORTS                    NAMES
5508c60831d4   apache/airflow:2.3.4   "/usr/bin/dumb-init …"   12 hours ago   Up 40 seconds (healthy)   0.0.0.0:8080->8080/tcp   resources_airflow-webserver_1
37f71f65f758   apache/airflow:2.3.4   "/usr/bin/dumb-init …"   12 hours ago   Up 40 seconds (healthy)   8080/tcp                 resources_airflow-worker_1
44c2588958cb   apache/airflow:2.3.4   "/usr/bin/dumb-init …"   12 hours ago   Up 40 seconds (healthy)   8080/tcp                 resources_airflow-scheduler_1
cc939447d676   apache/airflow:2.3.4   "/usr/bin/dumb-init …"   12 hours ago   Up 40 seconds (healthy)   8080/tcp                 resources_airflow-triggerer_1
d36e8e849ff8   redis:latest           "docker-entrypoint.s…"   12 hours ago   Up 40 seconds (healthy)   6379/tcp                 resources_redis_1
9ba46b104c7a   postgres:13            "docker-entrypoint.s…"   12 hours ago   Up 41 seconds (healthy)   5432/tcp                 resources_postgres_1

Some useful commands

Some useful commands are the following:

Run airflow commands

To run an airflow command type:

docker-compose run airflow-worker airflow info

otherwise, you can download the wrapper script (only for MacOS or Linux):

curl -LfO 'https://airflow.apache.org/docs/apache-airflow/2.3.4/airflow.sh'
chmod +x airflow.sh

finally, run:

./airflow.sh info

See Graphical User Interface

The Airflow GUI is available http://localhost:8080 with:

Username: airflow
Password: airflow

Shutting down Airflow and clean

For shutting down everything, you can type:

docker-compose down

If you also want to delete the volumes type:

docker-compose down --volumes --rmi all

Final recommendations

It is funny the final recommendations that the Apache Airflow community tells here:

DO NOT attempt to customize images and the Docker Compose if you do not know exactly what you are doing, do not know Docker Compose, or are not prepared to debug and resolve problems on your own.
....
Even if many users think of Docker Compose as “ready to use”, it is really a developer tool ...
It is extremely easy to make mistakes that lead to difficult-to-diagnose problems and if you are not ready to spend your own time on learning and diagnosing and resolving those problems on your own do not follow this path. You have been warned.
...
DO NOT expect the Docker Compose below will be enough to run production-ready Docker Compose Airflow installation using it. This is truly quick-start docker-compose for you to get Airflow up and running locally and get your hands dirty with Airflow.

In any case, you can see the Helm Chart for Apache Airflow for more information on how to install Airflow over Kubernetes.

The rise of Amazon Web Service

As is usually the case, there where there is a configuration problem. Companies see a sales opportunity. And it happened here, too. AWS provides Amazon Managed Workflows for Apache Airflow (MWAA)
MWAA is a managed orchestration service for Apache Airflow to create end-to-end data pipelines in the cloud at scale.
Managed Workflows is optimized to use Airflow and Python to create workflows without taking care the scalability, availability, and security.

Summary

In this post, we start to see the importance of setup a production environment for Apache Airflow.
After a short introduction, we see a simple docker-compose implementation to start with Apache Airflow local.
Finally, we shortly see the Amazon Managed Workflows for Apache Airflow and an all-in-one solution for the developer.

That's all for this post, in the next, I will go deeper inside the production problem of Apache Airflow, and
we will analyze the helm and MWAA.

Original published on [Davide Gazzè's Medium]((https://medium.com/p/bcbb19f30cd6)

Print Color Text with Python

Davide Gazzè — Wed, 01 Mar 2023 08:56:37 +0000

Hi everyone,

one of the most interesting things when I started using a computer, a Commodore 64, was always the ability to write colored text. It may sound childish, and indeed I was a child, but at the time the ability to display colors at my leisure was always something that filled me with joy. Of course, age and technological advancement have led me to consider different aspects of computing cool. But sometimes coming back as a child gives a little help to move forward. Therefore, in this post, the possibility of printing colored text in python will be discussed.

Introduction
On Linux, it is possible to print colored text using some `ANSI escape sequences [1], for example, if you open your terminal, and type:

echo -e "\e[31mColored Text\e[0m"

and you will see the following:

Write color text with Bash

The text is now Red. Entering in details, we can analyze the pattern:
echo -e "\e[COLORmSample Text\e[0m"
this string is formed by different parameters:

-e: Enable interpretation of backslash escapes
\e[: Start the color modifications
COLORm: Represents the color code, plus m at the end
\e[0m: End color modifications

Return to our example echo -e "\e[31mColored Text\e[0m", you can see that we use the code 31 to display the Red text. There are different complete articles on the Internet, I suggest you read [1] and [4]. Just to summarize, there are codes for color, code to have Bold, Underlined, and also Blinking Characters. Moreover, you can add more colors modification separated by ;:

echo -e "\e[COLOR1;COLOR2mText\e[0m"

From Bash to Python
One fantastic thing is that you can reuse all the concepts that you learned until now also in python. Python supports color modifiers:

print(f"\033[31mRed Text")

In practice, the \e became \033. Ok, that's great, but writing the color modifications reduces the code's legibility. The best solution could be to use and high-level package which solves it.
This is the goal of the package colorama.

Introduction to colorama
Colorama is a package able to create the ANSI escape character sequences to colorize your output. Colorama is multi-platform it works under Linux, Mac, and Windows. The colorama's demo scripts print some colored text using ANSI sequences. The installation is simple, just type:

pip install colorama

If your environment is Windows, you have to init in the following way:

from colorama import just_fix_windows_console just_fix_windows_console()

after that, you can use it.

Using Colorama
For starting to use Colorama, let's see the code for printing Red text:

`
import colorama
from colorama import Fore

print(Fore.RED + 'This text is red in color')
`

the output is:

The first example with colorama

Continue Reading on Syntax Error