DEV Community: Jesus Blazquez

Machine Learning: The Crystal Ball of Data

Jesus Blazquez — Sun, 16 Jun 2024 19:09:23 +0000

This is a submission for DEV Computer Science Challenge v24.06.12: One Byte Explainer.

Explainer

Machine learning is the crystal ball of data. It sifts through vast amounts of information, learning and predicting future trends, much like nature evolves patterns over time but in record time, allowing us to find relationships in data that are beyond our comprehension.

Additional Context

Machine learning is like a superpower for uncovering hidden patterns and relationships in data, almost like magic. It can detect subtle trends and connections that human minds often overlook, enabling breakthroughs in fields like healthcare, finance, and environmental science. By continuously learning and adapting, machine learning models provide insights that drive innovation and decision-making, transforming how we understand and interact with the world.
Just as the Fibonacci sequence and the golden ratio or Benford numbers are human discoveries that show the magic implicit in nature, what new natural mechanisms can we discover with the powerful tool of machine learning?

Dividir Dataframe en CSV s con OS y Pandas

Jesus Blazquez — Thu, 22 Dec 2022 08:48:45 +0000

Para este ejercicio se ha utlizado un ejemplo de datos de RRHH de la Comunidad de Madrid, descargado del siguiente enlace «https://datos.madrid.es/egob/catalogo/300584-12-rrhh_efectivos_ult_periodo.csv«

Importamos las librerias:

import os
import pandas as pd

con os.mkdir creamos el directorio:

os.mkdir('data/Datos_distrito')

Listamos la carpeta de origen de datos:

os.listdir('data')
['Datos_distrito', 'rrhh202210.csv']

Creamos dos DF, uno con la lectura del CSV de origen de datos y otro con la seleccion de datos del Distrito Centro:

# Load the data into a DataFrame
datos_df = pd.read_csv('data/rrhh202210.csv')

# Select only data for the year 2002
DistritoCentro = DistritoCentro = datos_df[datos_df.DENOMINACION_AREA == 'Distrito de Centro                                                              ']
DistritoCentro

Debido a que los datos del campo DENOMINACION_AREA tienen un espacio posterior, debemos contar con ese espacio para que el comando anterior funcione correctamente:

Para solventar el problema de esta columna y eliminar el espacio en el string, ejecutamos el siguiente comando (este comando elimina los espacios anteriores y posteriores de una columna):

datos_df['DENOMINACION_AREA'] = datos_df['DENOMINACION_AREA'].str.strip()
datos_df

Ahora ya podemos ejecutar el comando anterior, sin el espacio en el string

DistritoCentro = datos_df[datos_df.DENOMINACION_AREA == 'Distrito de Centro']
DistritoCentro

exportamos los datos a un CSV:

# Write the new DataFrame to a CSV file
DistritoCentro.to_csv('data/Datos_distrito/DistritoCentro.csv')

Para no realizar esta tarea repetitiva, distrito por distrito, podemos crear un loop que recorra los denominaciones de area y genere los ficheros.

Listamos la columna «DENOMINACION_AREA» con el parametro «unique», para que solo nos muestre nombres unicos:

datos_df['DENOMINACION_AREA'].unique()

Probamos toda esta informacion en un loop:

for DENOMINACION_AREA in datos_df['DENOMINACION_AREA'].unique():
   filename='data/Datos_distrito/Distrito' +'_'+ str(DENOMINACION_AREA) + '.csv'
   print(filename)

Todo parece funcionar de forma correcta, pero antes de crear el loop y dividir la informacion por distritos, debemos realizar algunas modificaciones en los datos, ya que contienen valores nulos y mas espacios en blanco que arruinarian nuestra salida, por ello comenzaremos de nuevo con todo el origen de datos.

Importamos los datos:

datos_df_raw = pd.read_csv('data/rrhh202210.csv', on_bad_lines='skip', sep=";")
datos_df_raw

Eliminamos los valores nulos de todo el DF:

datos_df=datos_df_raw.dropna()
datos_df

Borramos todos los espacios delante y detras de todos los string, para ello creamos una funcion que haga un loop y recorra todos los string, eliminando los espacios delante y detras:

def space_remover(dataframe):

    # iterating over the columns
    for i in dataframe.columns:

        # checking datatype of each columns
        if dataframe[i].dtype == 'object':

            # applying strip function on column
            dataframe[i] = dataframe[i].map(str.strip)
        else:

            # if condn. is False then it will do nothing.
            pass

# applying whitespace_remover function on dataframe
space_remover(datos_df)

# printing dataframe
datos_df

A continuacion remplazamos todos los espacios en banco por _, en todo el DF, dado que es necesario para que el ultimo paso se ejecute de forma correcta (los espacios en blanco causan muchos problemas con la libreria Pandas, siempre es mejor eliminarlos):

datos_df=datos_df.replace({" ": "_"}, regex=True)
datos_df

añadimos el resto de pasos para crear los ficheros con la informacion por distritos:

for DENOMINACION_AREA in datos_df['DENOMINACION_AREA'].unique():

    # Select data for the distrito
    datos_area = datos_df[datos_df.DENOMINACION_AREA == DENOMINACION_AREA]

    # Write the new DataFrame to a CSV file
    filename = 'data/Datos_distrito/Distrito' + str(DENOMINACION_AREA) + '.csv'
    datos_area.to_csv(filename)

como podemos ver se han creado lo ficheros CSV:

Y comprobamos que se ha realizado la division de forma correcta:

How create a new field based on other numerical fields in a Pandas Dataframe

Jesus Blazquez — Thu, 22 Dec 2022 08:31:39 +0000

To create a new field based on other numerical fields in a pandas DataFrame, you can use the pandas.DataFrame.assign() method. This method allows you to specify a new column name and the values for the column, which can be derived from one or more existing columns in the DataFrame.

For example, suppose you have a DataFrame with columns A, B, and C, and you want to create a new column called D that is equal to the sum of columns A and B. You can use the following code to do that:

import pandas as pd

df = pd.DataFrame({'A': [1, 2, 3], 'B': [4, 5, 6], 'C': [7, 8, 9]})

df = df.assign(D = df['A'] + df['B'])

print(df)

This code will create a new column D in the DataFrame with the values from columns A and B added together. The resulting DataFrame will look like this:

   A  B  C  D
0  1  4  7  5
1  2  5  8  7
2  3  6  9  9

Alternatively, you could also use the pandas.DataFrame.apply() method to apply a custom function to each row of the DataFrame, and create the new column based on the output of that function. For example, the following code would accomplish the same task as the code above:

import pandas as pd

df = pd.DataFrame({'A': [1, 2, 3], 'B': [4, 5, 6], 'C': [7, 8, 9]})

def sum_columns(row):
    return row['A'] + row['B']

df = df.assign(D = df.apply(sum_columns, axis=1))

print(df)

This code will also create a new column D in the DataFrame with the values from columns A and B added together. The resulting DataFrame will be the same as the one shown above.

SBC ORACLE ACME PACKET HA IN AWS

Jesus Blazquez — Fri, 01 Jul 2022 22:58:21 +0000

To configure the HA utility of the Acme Packet SBCs in AWS, we must make use of the script pre-installed in the OS of our SBC, we do not need to make any changes to this script, but if we want to consult it we can access it by entering the SHELL of the SBC, writing our password, which we previously had to define for shell access:

Once inside we can consult the script:

Analyzing the script, we can see that it uses a variable to check the status of the SBC and assign which interfaces are active at that moment, so that AWS will assign the IPs to that machine, we can see in the AWS console that the IPs assigned change from one machine to another in the cluster, depending on whether one or the other is the active node.

The prerequisites for this to work are to have DHCP activated in the SBC (we will assign a static IP from the AWS console), for this we must not configure IPs in Bootparam:

and we must confirm that we have the AWS DNS server IP in the SBC /etc/resolv.conf file (accessed from the shell):

Once these points have been confirmed, for this change to be possible we need to assign a specific IAM role and a specific IAM Policy to the 2 EC2 of the node, where we need to define the variables that AWS will use to know which of the 2 cluster machines it should assign to. the IP's.
Once the IAM Role is created, we must paste this JSON code in the IAM Policy, so that AWS can work with the script developed by Oracle for HA in AWS.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Action": [
                "ec2:DescribeAddresses",
                "ec2:DescribeNetworkInterfaces",
                "ec2:DescribeNetworkInterfaceAttribute",
                "ec2:DescribeInstanceAttribute",
                "ec2:DescribeSubnets",
                "ec2:AssignPrivateIpAddresses",
                "ec2:AssociateAddress",
                "ec2:DescribeSecurityGroups",
                "ec2:DescribeInstanceStatus"
            ],
            "Resource": "*",
            "Effect": "Allow"
        }
    ]
}

We save this information and we can now test if by changing the active/passive node, all our additional IPs as well as the elastic IPs must balance between the different nodes, always being assigned to the active node at that moment.
Also comment that at the time of assigning the additional IP's as well as the elastic IP's we must make sure that the node in which we are assigning the IP's is active.

Enjoy

How to deploy multiweb wordpress SSL certified (Let's Encrypt) in lightsail with docker-compose in 5 minutes

Jesus Blazquez — Fri, 01 Jul 2022 16:41:13 +0000

Deploy Centos7 in lightsail, attach static IP to your server and open this flows inside you Networking server config:

Now you have Lightsail done

Berfore to deploy the solution you need redirect your domain to the server static IP.

Access by console to your new server and install docker and docker-compose:

Install docker:

yum update && yum install -y yum-utils device-mapper-persistent-data lvm2 &&
yum-config-manager  --add-repo https://download.docker.com/linux/centos/docker-ce.repo &&
yum install docker-ce docker-ce-cli containerd.io &&
systemctl enable docker &&
systemctl start docker

Install docker-compose:

yum install curl &&
curl -L "https://github.com/docker/compose/releases/download/1.23.2/docker-compose-$(uname -s)-$(uname -m)" -o /usr/local/bin/docker-compose &&
chmod +x /usr/local/bin/docker-compose &&
ln -s /usr/local/bin/docker-compose /usr/bin/docker-compose

Now make a copy of my Github repository https://github.com/jblazquezmartin/docker_multiweb_https, with the following command:

git clone https://github.com/jblazquezmartin/docker_multiweb_https

Create a network in docker for nginx-proxy:

docker network create nginx-proxy

Run the following command inside the "nginx-proxy" directory, to launch the web server:

docker-compose up -d

verify that it was raised correctly with the following command:

docker-compose ps

Now you must adapt the code downloaded from Github inside the «websites» directory, so that it adapts to the web(s) you want to deploy, use «websites/example.es/docker-compose.yml» as an example, you will have to adapt it to your domain:

Keep in mind that an automatic registration request for your certificate is sent to Let's Encrypt, for which you will need to fill in the "LETSENCRYPT_HOST" and "LETSENCRYPT_EMAIL" fields correctly, as well as having your domain redirected to the external IP assigned in Lightsail.

Run again, inside your web directory, where you modified "docker-compose.yml", the command:

docker-compose up -d

verify that it was raised correctly with the following command:

docker-compose ps

Apache and MariaBD will be up.

Now access your domain and magic!!, you have your wordpress installed with the Let's Encrypt SSL certificate configured:

Enjoy!!