DEV Community: Lucas M. Ríos

How to configure multiple environments with Ansible Automation Platform?

Lucas M. Ríos — Sun, 02 Apr 2023 20:28:48 +0000

🦾 Introduction to environment configuration with Ansible Automation Platform

🔗Related content

You can find repo related in:

🐱‍🏍GitHub

You can connect with me in:

🧬LinkedIn

Resume 🧾

🚀 Ansible is a powerful configuration management tool that can be used for provisioning infrastructure.

🛠️ Provisioning refers to the process of configuring and setting up remote systems or infrastructure.

📝 Ansible uses Playbooks, which are written in YAML format, to define the desired state of the infrastructure.

📦 Ansible provides many built-in modules for performing provisioning tasks, such as installing packages, configuring network settings, managing users and groups, and more.

Glosary

AAP: Ansible Automation Platform

Introduction

Supose you have an app and need deploy it in three or more enviornment with specific configuration.

Imagine deploy manually all application included configuration is easy and manageble, but in a small organization will have a minimum of ten applications this is tedious but managable configurate in each server.

Now think in twenty applications this is simply unimaginable manage fast and apply configuration correctly.

In my last time, I have worked a lot with Ansible using AAP and solved many problems in deploying and updating applications.

Other times we don't make mistakes just simply the server doesn't have infrastructure settings appropriately.

What we need to fix all of that?

We will use a function called renderize.

I want belive you have this step.
We need use good practices of development and understand holisticly all system. Included code, data bases, server, settings...
Create a repo with app. I calling app to code.
Supose this is called as "repo-app".
Create a repo with configuration values by each environment.
Supose this is called as "repo-config".
Create a repo with ansible code configuration.
Supose this is called as "repo-ansible".

What we need make in repo-app?

In files where exists configuration specific of each environment like database of develop, qa and master; or values to configure Auth0.
Files can has any format, i.e: json, yaml, xml, etc.

We will write files where:

# appsetings.json
{
  "API": {
    "Endpoint": "**Endpoint-develop**"
  },
  "reCaptcha": {
    "SiteKey": "**SiteKey-develop**",
    "SecretKey": "**SecretKey-develop**",
    "Score": **Score-develop**,
  },  
  "Logging": {
    "LogLevel": {
      "Default": "Information",
      "Microsoft.AspNetCore": "Warning"
    }
  },
  "AllowedHosts": "*"
}

Insted set values set { { NAME_VARIABLE } }, key squares need are join:

# appsetings.json
{
  "API": {
    "Endpoint": "{ { Client.Endpoint } }"
  },
  "reCaptcha": {
    "SiteKey": "{ { Client.SiteKey } }",
    "SecretKey": "{ { Client.SecretKey } }",
    "Score": { { Client.Score } },
  },  
  "Logging": {
    "LogLevel": {
      "Default": "Information",
      "Microsoft.AspNetCore": "Warning"
    }
  },
  "AllowedHosts": "*"
}

What we need make in repo-config?

WARNING: This repo is special, so stay focus. We don't make merge or promote between branchs. We work with each branch as independent.

In my case I use a file in format yaml to set values of configurations.

Here we set values and this values will appear in server where will be app and values where we configurate with { { NAME_VARIABLE } }:

---
# appsetings.yaml
Client:
  Endpoint: Endpoint-develop
  SiteKey: SiteKey-develop
  SecretKey: SecretKey-develop
  Score: Score-develop

In qa:
Insted set values set { { NAME_VARIABLE } }:

---
# appsetings.yaml
Client:
  Endpoint: Endpoint-qa
  SiteKey: SiteKey-qa
  SecretKey: SecretKey-qa
  Score: Score-qa

In master:

---
# appsetings.yaml
Client:
  Endpoint: Endpoint-master
  SiteKey: SiteKey-master
  SecretKey: SecretKey-master
  Score: Score-master

Here you can put some secrets encrypted with vault and later this will be renderize with AAP credentials.

And other file with origin and destination of files to renderize:

---
# templates.yaml
templates:
  - origin: { { local_path } }/appsettings.json
    dest: { { dest_path } }/appsettings.json

And add many origin and dest for files to set values in your server.

What we need make in repo-ansible?

Usually we will have a repo with only a branch that will be used.

The tasks we need use to renderize app are:
Note: Imagine we use group vars.

---
# renderize.yaml
- name: Download app
  git:
    repo: "https://{ { repos.app.url } }"
    dest: "{ { repos.app.dir } }"
    version: "{ { repos.app.branch } }"
    force: no
  delegate_to: localhost
  run_once: true

- name: Download config
  git:
    repo: "https://{ { repos.config.url } }"
    dest: "{ { repos.config.dir } }"
    version: "{ { repos.config.branch } }"
    force: no
  delegate_to: localhost
  run_once: true

# TO WINDOWS
- name: Renderize templates in windows
  win_template:
    src: "{ { repos.app.dir } }/{ { item.origin } }"
    dest: "{ { item.dest } }"
  loop: "{ { templates } }"

# TO LINUX
- name: Renderize templates in linux
  template:
    src: "{ { repos.app.dir } }/{ { item.origin } }"
    dest: "{ { item.dest } }"
  loop: "{ { templates } }"

Say thanks, give like and share if this has been of help/interest 😁🖖

Provisioning Infrastructure with Ansible

Lucas M. Ríos — Sun, 26 Mar 2023 23:00:19 +0000

🦾 Introduction to provision with Ansible

🔗Related content

You can find repo related in:

🐱‍🏍GitHub

You can connect with me in:

🧬LinkedIn

Resume 🧾

🚀 Ansible is a powerful configuration management tool that can be used for provisioning infrastructure.

🛠️ Provisioning refers to the process of configuring and setting up remote systems or infrastructure.

📝 Ansible uses Playbooks, which are written in YAML format, to define the desired state of the infrastructure.

📦 Ansible provides many built-in modules for performing provisioning tasks, such as installing packages, configuring network settings, managing users and groups, and more.

💻 To provision infrastructure with Ansible, you'll need to:

Install Ansible
Define your infrastructure using Playbooks
Create an inventory file
Execute the playbook using the ansible-playbook command
🖥️ Ansible can also be used for provisioning Windows systems, but this requires some additional setup and configuration, such as configuring the WinRM service on the target Windows system and specifying the WinRM connection details in the inventory file.

Overall, provisioning with Ansible provides a way to automate the process of configuring and setting up infrastructure, reducing the risk of human error and increasing efficiency. 🤖

What is provisining in Ansible?

In Ansible, provisioning refers to the process of configuring and setting up remote systems or infrastructure. Ansible is a configuration management tool that uses a declarative language to describe the desired state of the infrastructure. This means that instead of specifying individual tasks to be performed, you describe the desired end state of the system, and Ansible figures out how to get there.

Provisioning in Ansible involves defining the configuration of the remote systems or infrastructure in a playbook, which is a YAML file that lists the tasks to be performed. These tasks can include installing packages, configuring network settings, setting up users and groups, and more. Ansible then connects to the remote systems over SSH and executes the tasks in the playbook.

Ansible provides a number of built-in modules that can be used to perform provisioning tasks, such as the apt module for managing packages on Ubuntu systems, the yum module for managing packages on Red Hat-based systems, and the user module for managing users and groups.

Overall, provisioning with Ansible provides a way to automate the process of configuring and setting up infrastructure, reducing the risk of human error and increasing efficiency.

How I can provision infrastructure with Ansible?

To provision infrastructure with Ansible, you'll need to follow these basic steps:

Install Ansible on your local machine or on a server that will be used to run Ansible commands.
Define the desired state of your infrastructure using Ansible Playbooks, which are written in YAML format. Playbooks contain a list of tasks that will be executed on the remote systems.
Create an inventory file that lists the hostnames or IP addresses of the systems you want to provision, as well as any necessary connection details such as SSH keys or usernames/passwords.
Execute the playbook using the ansible-playbook command, specifying the inventory file and playbook file.

Here's an example of what a simple Ansible playbook might look like:

---
- name: Provision a web server
  hosts: webservers
  become: true

  tasks:
  - name: Install Apache web server
    apt:
      name: apache2
      state: present

This playbook will provision a web server by installing the Apache web server package using the apt module. The hosts parameter specifies the target systems, and become: true tells Ansible to use sudo to execute the tasks as a privileged user.

To execute this playbook, you would run the following command:

ansible-playbook -i inv.yaml playbook.yml

where inv.yaml is the name of your inventory file, and playbook.yml is the name of your playbook file.

Ansible provides many built-in modules that can be used to perform various provisioning tasks, such as installing packages, configuring network settings, managing users and groups, and more. You can also create your own custom modules if needed.

Look we talk to provision in systems with base Unix.

And in Windows?

Ansible can also be used for provisioning Windows systems. However, provisioning Windows systems with Ansible requires some additional setup and configuration compared to provisioning Linux systems. Here are the basic steps to provision a Windows system with Ansible:

Install Ansible on a Linux system or a Windows system with the Windows Subsystem for Linux (WSL) enabled. Ansible cannot be installed directly on a Windows system.
Configure the WinRM service on the target Windows system to allow remote management. WinRM is Microsoft's implementation of the WS-Management protocol, which allows remote management of Windows systems.
Create an inventory file that lists the target Windows system and specifies the connection details, including the WinRM username and password.
Write a playbook that contains tasks to configure the Windows system as desired. Ansible provides several built-in modules for configuring Windows systems, including modules for managing users, groups, and Windows features.
Execute the playbook using the ansible-playbook command, specifying the inventory file and playbook file.

Here's an example of what a simple Ansible playbook for Windows might look like:

---
- name: Provision a Windows server
  hosts: windows
  gather_facts: no
  become: yes

  tasks:
  - name: Install IIS web server
    win_feature:
      name: Web-Server
      state: present

This playbook installs the IIS web server feature on a Windows system using the win_feature module. The hosts parameter specifies the target Windows system, and become: yes tells Ansible to execute the tasks as an administrator.

To execute this playbook, you would run the following command:

ansible-playbook -i inv.yaml playbook.yml --extra-vars "ansible_user=<winrm_username> ansible_password=<winrm_password> ansible_connection=winrm ansible_winrm_transport=basic ansible_winrm_server_cert_validation=ignore"

where inv.yaml is the name of your inventory file, playbook.yml is the name of your playbook file, and winrm_username and winrm_password are the username and password for the WinRM connection. The --extra-vars parameter is used to pass the WinRM connection details to Ansible. The ansible_winrm_server_cert_validation=ignore option is used to disable certificate validation, which is often necessary in testing or development environments.

Overall, provisioning Windows systems with Ansible requires some additional setup and configuration compared to provisioning Linux systems, but Ansible provides many built-in modules that can be used to configure and manage Windows systems.

Say thanks, give like and share if this has been of help/interest 😁🖖

Getting Started with Ansible: An Introduction to Automation, Configuration Management, and Orchestration

Lucas M. Ríos — Sat, 18 Mar 2023 21:40:44 +0000

🔐 Introduction to Ansible

🔗Related content

You can find repo related in:

🐱‍🏍GitHub

You can connect with me in:

🧬LinkedIn

Resume 🧾

📜 Nothing to resume,it's simple.

Introduction

🤖 Ansible is an open-source automation tool that simplifies complex tasks, speeds up workflows, and increases productivity. It allows users to automate configuration management, application deployment, and task automation.

🔑 Ansible is agentless, which means it does not require any software or agent to be installed on the remote machines that it manages. Instead, it uses SSH or WinRM to connect to remote hosts.

📝 Ansible uses a simple and human-readable syntax called YAML (Yet Another Markup Language), which makes it easy for users to create and manage their automation scripts. Ansible playbooks are written in YAML and are a set of instructions that tell Ansible what to do.

Uses

💻 Ansible supports a wide range of platforms, including Linux, Unix, macOS, and Windows. It also integrates with a variety of cloud providers such as AWS, Azure, and Google Cloud Platform.

♻️ Ansible's idempotent nature allows it to be run multiple times without changing the end result. This makes it easy to test and re-run automation scripts without worrying about undesired changes or side effects.

🛠️ Ansible is highly extensible and can be integrated with other tools and services. For example, it can be integrated with Git, Jenkins, and many other DevOps tools to create a powerful and streamlined automation pipeline.

Examples

Here are some examples of what Ansible can do:

Automate the installation and configuration of software packages on multiple servers
Deploy web applications and services to multiple servers simultaneously
Manage network devices such as switches and routers Provision virtual machines in a cloud environment
Configure and manage container orchestration platforms like Kubernetes

Overall, Ansible is a powerful and flexible automation tool that can help organizations automate their IT workflows and increase their efficiency. Whether you're a developer, system administrator, or IT professional, Ansible can help you streamline your automation processes and achieve your automation goals.

Practical case

Let's say you have a web application that you want to deploy to multiple servers simultaneously. You want to automate the process of deploying the application to the servers and make sure that the servers are configured correctly before the deployment. You also want to ensure that the application is deployed consistently across all servers, even if you need to re-run the deployment process.

To accomplish this, you can use Ansible to write a playbook that automates the deployment and configuration process. Here's an example of what the playbook might look like:

---
- name: Deploy web application
  hosts: webservers
  become: yes

  vars:
    app_name: myapp
    app_version: 1.0.0
    app_port: 8080

  tasks:
    - name: Install dependencies
      apt:
        name: ["git", "python3-pip", "nginx"]
        state: present

    - name: Clone repository
      git:
        repo: https://github.com/myusername/myapp.git
        dest: /opt/{{ app_name }}-{{ app_version }}

    - name: Install Python dependencies
      pip:
        requirements: /opt/{{ app_name }}-{{ app_version }}/requirements.txt

    - name: Configure Nginx
      template:
        src: nginx.conf.j2
        dest: /etc/nginx/sites-available/{{ app_name }}
      notify:
        - Reload Nginx

    - name: Enable site
      file:
        src: /etc/nginx/sites-available/{{ app_name }}
        dest: /etc/nginx/sites-enabled/{{ app_name }}
        state: link
      notify:
        - Reload Nginx

    - name: Start application service
      systemd:
        name: {{ app_name }}
        state: started
        enabled: yes
        daemon_reload: yes

  handlers:
    - name: Reload Nginx
      systemd:
        name: nginx
        state: reloaded

...

In this example, the playbook specifies that it will be run on a group of hosts called "webservers". It then defines a set of tasks that will be executed on each of these hosts.

The first task installs the necessary dependencies on the hosts using the apt module. The next task clones the application's Git repository to the hosts using the git module. The third task installs the Python dependencies of the application using the pip module.

The fourth task configures Nginx by generating a template file based on a Jinja2 template and copying it to the appropriate location using the template module. It then notifies the Reload Nginx handler, which is defined later in the playbook.

The fifth task enables the Nginx site by creating a symbolic link to the site's configuration file using the file module. It also notifies the Reload Nginx handler.

The sixth and final task starts the application service using the systemd module. It also ensures that the service is enabled to start automatically on boot and notifies daemon_reload to reload the systemd configuration.

The playbook also defines a handler called Reload Nginx, which reloads the Nginx configuration using the systemd module.

By running this playbook with Ansible, you can automate the deployment and configuration of your web application across multiple servers with ease.

Say thanks, give like and share if this has been of help/interest 😁🖖

A Beginner's Guide to Vault: Understanding Secrets, Credentials, and Secure Data Management

Lucas M. Ríos — Sat, 11 Mar 2023 02:29:43 +0000

🔐 Introduction to Credential and Secrets

🔗Related content

You can find repo related in:

🐱‍🏍GitHub

You can connect with me in:

🧬LinkedIn

Resume 🧾

📜 Vault: Definition and explanation of Vault as a tool for securely storing and managing sensitive data.
🤫 Secret: Explanation of what secrets are and their importance in security.
🔑 Credential: Explanation of credentials as a type of secret used for authentication and authorization.
🏆 Importance: Discussion of the importance of secrets and credentials in security.
🚀 Use Cases: Examples of use cases for Vault, including managing application secrets, secure access management, configuration management, dynamic secrets, and encryption/tokenization.
💻 Examples: Additional examples of how Vault can be used in practical applications.
📄 Command Line: Explanation of how to create a Vault from a file or string in command line.

What is Vault? 📜

In Ansible, a "vault" is a feature that allows you to encrypt sensitive data such as passwords, API keys, and other credentials used in your playbooks and roles.

When you create a vault, you use a password to encrypt the data, and only someone who has the password can access the data. This helps ensure that sensitive information is not visible to unauthorized users who may have access to your Ansible code.

To create a vault in Ansible, you use the ansible-vault command line tool. You can create a new encrypted file or edit an existing one with the edit subcommand, and you can also encrypt and decrypt files with the encrypt and decrypt subcommands.

Once you have created a vault, you can use the include_vars module or the vars_prompt module to prompt the user for the password and then use the vault lookup plugin to access the encrypted variables in your playbook or role. The vault lookup plugin will automatically decrypt the data using the password provided.

Using a vault in Ansible is a best practice for handling sensitive data and can help you secure your automation workflows.

What is a secret? 🤫

In the context of computing and security, a "secret" refers to any piece of sensitive or confidential information that should be kept private and protected from unauthorized access. Secrets can take many forms, such as passwords, encryption keys, API keys, access tokens, private certificates, and other types of credentials that are used to authenticate users or systems and grant access to protected resources.

Keeping secrets secure is critical for protecting sensitive data and preventing unauthorized access and data breaches. This is why secrets are often stored in secure locations, such as encrypted databases, hardware security modules, or cloud-based key management services, and are accessed using strict authentication and authorization controls.

Many software tools and frameworks provide built-in mechanisms for storing and managing secrets securely, such as Ansible Vault, HashiCorp Vault, Kubernetes Secrets, and AWS Secrets Manager, to name a few. These tools allow you to store, manage, and distribute secrets across different systems and environments securely, using encryption, access controls, and other security features to ensure that sensitive information is protected from unauthorized access.

What is a credential? 🔑

In the context of computing and security, a "credential" refers to any piece of information that is used to authenticate and authorize a user, system, or application to access a resource or perform an action. Credentials can take many forms, including usernames, passwords, access keys, API tokens, certificates, and other types of secrets.

Credentials are used to establish trust between different systems and to ensure that only authorized entities can access protected resources. For example, when you log in to a website, you typically provide a username and password as credentials, which are verified by the website's authentication system to determine if you are authorized to access your account. Similarly, when you use an API to access a cloud service, you may need to provide an access key and secret as credentials to authenticate your request and authorize access to the service.

Credentials are a critical aspect of security in computing, and their management is essential to ensure the confidentiality, integrity, and availability of sensitive data and systems. Credential management involves storing credentials securely, using encryption, access controls, and other security mechanisms to protect them from unauthorized access, and rotating them periodically to minimize the risk of compromise. Many tools and frameworks provide built-in support for credential management, such as password managers, key management services, and identity and access management (IAM) systems.

What is difference between a credential and secret? 🏆

While the terms "credential" and "secret" are often used interchangeably, there is a subtle difference between them.

A "credential" refers to any piece of information that is used to authenticate and authorize a user, system, or application to access a resource or perform an action. Credentials can take many forms, including usernames, passwords, access keys, API tokens, certificates, and other types of secrets.

On the other hand, a "secret" specifically refers to any piece of sensitive or confidential information that should be kept private and protected from unauthorized access. Secrets can take many forms, such as passwords, encryption keys, API keys, access tokens, private certificates, and other types of credentials used to authenticate users or systems and grant access to protected resources.

In essence, all secrets are credentials, but not all credentials are secrets. Credentials include both public and private information that is used for authentication and authorization, whereas secrets specifically refer to private information that must be kept confidential to prevent unauthorized access.

For example, a username is a credential but not necessarily a secret, since it may be publicly known or easily guessed. A password, on the other hand, is both a credential and a secret, since it must be kept private to ensure secure authentication.

Which is more important? 🏅

Both credentials and secrets are important in the context of computing and security, and their management is essential to ensure the confidentiality, integrity, and availability of sensitive data and systems.

Credentials are critical because they are used to establish trust between different systems and to ensure that only authorized entities can access protected resources. Strong credentials, such as complex passwords and unique access keys, can help prevent unauthorized access and reduce the risk of data breaches and other security incidents.

Secrets are also critical because they are used to protect sensitive information and prevent it from falling into the wrong hands. Effective secret management involves storing secrets securely, using encryption, access controls, and other security mechanisms to protect them from unauthorized access, and rotating them periodically to minimize the risk of compromise.

In summary, both credentials and secrets are important, and their management is essential to ensure the security of computing systems and data. Organizations must adopt best practices for managing credentials and secrets, including strong authentication and authorization mechanisms, secure storage, and regular rotation to minimize the risk of compromise.

Where I can use vault? 💻

A vault is a secure storage location used to store and manage sensitive data such as credentials, secrets, and other types of confidential information. Vault tools can be used in a variety of contexts where secure storage and management of secrets are required, including:

Configuration management: Vault can be used with tools like Ansible and Terraform to store sensitive configuration data such as API keys, passwords, and access tokens.

Application development: Vault can be used to store sensitive data such as encryption keys, certificates, and database credentials required by applications during runtime.

Cloud computing: Vault can be used with cloud service providers such as AWS, Google Cloud Platform, and Microsoft Azure to store secrets required by cloud applications, services, and APIs.

DevOps: Vault can be used to manage secrets required by DevOps tools like Jenkins, GitLab, and CircleCI.

Cybersecurity: Vault can be used to securely store and manage sensitive data required by cybersecurity tools such as firewalls, intrusion detection systems, and threat intelligence platforms.

In summary, Vault can be used in a variety of contexts where secure storage and management of secrets are required, including configuration management, application development, cloud computing, DevOps, and cybersecurity.

Use cases 🚀

Vault is a tool for securely storing and managing sensitive data such as credentials, secrets, and other types of confidential information. Some common use cases of Vault include:

Managing application secrets: Applications often require sensitive information such as API keys, database credentials, and encryption keys to function. Vault can be used to store and manage these secrets, providing a centralized and secure way to manage application secrets.

Secure access management: Vault can be used to manage access to resources such as SSH keys, TLS certificates, and cloud service access keys. By storing these credentials in Vault and using Vault's access controls and policies, administrators can manage and audit access to these resources more easily.

Configuration management: Vault can be used with tools such as Ansible and Terraform to store sensitive configuration data, such as secrets required by infrastructure-as-code scripts. This makes it easier to manage and maintain these scripts while keeping sensitive data secure.

Dynamic secrets: Vault can generate dynamic secrets that are short-lived and tied to specific applications or services. This can help reduce the risk of stolen or leaked secrets, as these dynamic secrets expire after a specified period of time.

Encryption and tokenization: Vault can be used to encrypt data at rest and in transit, as well as to tokenize sensitive data. This can help protect sensitive data and make it easier to manage and store securely.

In summary, Vault can be used in a wide range of use cases that involve managing sensitive data securely, including managing application secrets, access management, configuration management, dynamic secrets, and encryption/tokenization.

How can create a vault from a file and by a string in command line? 📄

To create a vault in command line, you can use the ansible-vault command.

To create a vault from a file, you can use the following command:

ansible-vault create <filename>

This will create a new encrypted vault file with the specified filename. You will be prompted to enter a password that will be used to encrypt and decrypt the file.

To create a vault from a string, you can use the following command:

ansible-vault encrypt_string <string>

This will encrypt the specified string and output it in a format that can be included in a playbook or inventory file. You will be prompted to enter a password that will be used to encrypt and decrypt the string.

For example, to encrypt the string "mysecret" and output it to the console, you can run the following command:

ansible-vault encrypt_string mysecret

This will output the encrypted string in a format like this:

!vault |
          $ANSIBLE_VAULT;1.1;AES256
          66326266373333633033646434333231353062313661646637626637346464353534653937623566
          3833336333323064373632653664623131653966623361320a306132383933343734656433313439
          63386334333737363862393461396334613836613132303265333865303265373465343561643130
          6430316166633437360a623366613632663238666565353832306233393064313964633062393238
          3235

This encrypted string can then be included in a playbook or inventory file using the !vault tag, and decrypted using the same password used to encrypt it.

Say thanks, give like and share if this has been of help/interest 😁🖖

Introduction with examples to Emu8086

Lucas M. Ríos — Fri, 03 Mar 2023 23:29:05 +0000

☢️ Introduction with examples to Emu8086

🔗Related content

You can find repo related in:

🐱‍🏍GitHub

You can connect with me in:

🧬LinkedIn

Resume 🧾

Here we will have a brief introduction with many examples of scripts to Emu8086.

[WARNING ⚠️] This is a low level of programming, so I recommend you be so inspired. 😁

What is Emu8086?

EMU8086 is an emulator that allows you to run and test programs written in assembly language on a Windows computer. It is a software tool that can emulate the hardware and software of a microprocessor, specifically the Intel 8086 microprocessor, which was used in the original IBM PC.

The emulator can be used to write, debug and test programs for the 8086 architecture. It provides an environment that is similar to the actual hardware on which the program would run, allowing programmers to develop code that is optimized for the specific hardware platform.

EMU8086 is popular among students and hobbyists who are interested in learning assembly language programming, as well as in reverse engineering and debugging software. It also includes a built-in assembler, which allows users to write and assemble programs directly within the emulator.

What is assembly?

Assembly language is a low-level programming language that is used to write programs that are executed by a computer's microprocessor. It is a symbolic representation of the machine language code that can be executed by the CPU (Central Processing Unit) of a computer.

Assembly language programs are typically used to control hardware directly, as they can be used to manipulate the individual bits and bytes of a computer's memory and CPU registers. Unlike high-level programming languages such as Java or Python, which are easier to read and write but require an interpreter or compiler to run, assembly language programs are written using mnemonic codes that correspond to specific CPU instructions.

Writing programs in assembly language can be challenging, as programmers must have a good understanding of the underlying hardware and be able to write code that is both efficient and concise. However, assembly language can be useful in situations where high performance or low-level control of hardware is required, such as in embedded systems, operating systems, or device drivers.

Examples

Okay, now we will see some examples. If you want prove this programs you can download Emu8086 remember this is only for Windows systems.

1️⃣Simple example of move data

; Perform a program that exchanges the contents
; of two decimal values defined in data1 and data2

.data
data1 db 11
data2 db 22

.code

EXCHANGE:
    MOV AL, data1
    MOV AH, data2
    MOV data1, AH
    MOV data2, AL

END:
    HLT

2️⃣Simple(no really) example of multiply two numbers interactively

; Make the multiplication of two numbers entered by keyboard,
; to store the result in the result variable
; print the result on the screen

INCLUDE 'EMU8086.INC'

org 100h

.data

Result db ?

.code
    INIT_VAR:
        MOV Result, 0
    GET_VALUES:
        CALL CLEAR_SCREEN
        CALL PTHIS
        DB "SET A MULTIPLIER: ",0
        CALL SCAN_NUM
        MOV BL, CL 
        CALL CLEAR_SCREEN
        CALL PTHIS
        DB "SET A MULTIPLICATOR: ",0   
        CALL SCAN_NUM
        MOV AL, CL

    MULTIPLY:
        ADD Result, BL
        DEC AL        
        JZ SHW_RESULT
        JMP MULTIPLY 

    SHW_RESULT: 
        CALL CLEAR_SCREEN
        CALL PTHIS
        DB "RESULT IS: ", 0       
        MOV AL, Result
        CALL PRINT_NUM

END:
    DEFINE_PTHIS
    DEFINE_SCAN_NUM
    DEFINE_PRINT_NUM 
    DEFINE_PRINT_NUM_UNS  
    DEFINE_CLEAR_SCREEN
    HLT

3️⃣Super simple(defintively no really) example of a calculator

; Perform a program that works as a calculator of 2 whole numbers,
; and allow the 4 basic operations. The keyboard must be entered by the
; numbers and the operation to be performed. Then print the result obtained.
INCLUDE "EMU8086.INC"

ORG 0x100h

.DATA
OPTION DB 0
VALUE1 DB 0
VALUE2 DB 0
RESULT DB 0

.CODE
START:
    GET_OPERATION:
        CALL PTHIS
        DB "SET A OPTION. 1-SUM, 2-SUBSTRACTION, 3-MULT, 4-DIVISION: ",0
        MOV CX, 0
        CALL SCAN_NUM
        MOV OPTION, CL

    GET_NUMS:
        CALL CLEAR_SCREEN
        CALL PTHIS
        DB "SET VALUE 1: ",0
        MOV CX, 0
        CALL SCAN_NUM
        MOV VALUE1, CL

        CALL CLEAR_SCREEN
        CALL PTHIS
        DB "SET VALUE 2: ",0 
        MOV CX, 0
        CALL SCAN_NUM
        MOV VALUE2, CL        

    SELECT_OPT:
        MOV AL, OPTION
        SUB AX, 1
        JZ SUM

        MOV AL, OPTION
        SUB AX, 2
        JZ RESTA

        MOV AL, OPTION
        SUB AX, 3
        JZ MULT

        MOV AL, OPTION
        SUB AX, 4
        JZ DIVISION

        JMP NO_VALID

    NO_VALID:
        CALL CLEAR_SCREEN
        CALL PTHIS
        DB "OPTION NO VALID",0
        JMP END

    SUM:
        MOV AX, 0
        MOV AL, VALUE1
        ADD AL, VALUE2
        MOV RESULT, AL
        CALL CLEAR_SCREEN
        CALL PTHIS
        DB "RESULT DE SUM: ",0
        MOV AX, 0
        MOV AL, RESULT
        CALL PRINT_NUM
        JMP END         

    RESTA:
        MOV AX, 0
        MOV AL, VALUE1
        SUB AL, VALUE2
        MOV RESULT, AL
        CALL CLEAR_SCREEN
        CALL PTHIS
        DB "RESULT DE RESTA: ",0 
        MOV AX, 0
        MOV AL, RESULT
        CALL PRINT_NUM
        JMP END

    MULT:
        MOV AX, 0
        MOV BX, 0
        MOV AL, VALUE1
        MOV BL, VALUE2
        IMUL BL
        MOV RESULT, AL
        CALL CLEAR_SCREEN
        CALL PTHIS
        DB "RESULT DE MULT: ",0 
        MOV AX, 0
        MOV AL, RESULT
        CALL PRINT_NUM
        JMP END

    DIVISION:
        MOV AX, 0
        MOV AL, VALUE1
        MOV BL, VALUE2
        IDIV BL
        MOV RESULT, AL
        CALL CLEAR_SCREEN
        CALL PTHIS
        DB "RESULT DE DIVISION: ",0
        MOV AX, 0
        MOV AL, RESULT
        CALL PRINT_NUM
        JMP END

END:
    HLT
    DEENDE_PTHIS
    DEENDE_PRINT_NUM
    DEENDE_PRINT_NUM_UNS
    DEENDE_SCAN_NUM  
    DEENDE_CLEAR_SCREEN

You can get more exercises in repo

Say thanks, give like and share if this has been of help/interest 😁🖖

How upload data in format JSON to Datastore from GCP?

Lucas M. Ríos — Sat, 25 Feb 2023 00:39:59 +0000

⛵ Simple way to upload data in format JSON to Datastore from GCP.

🔗Related content

You can find repo related in:

🐱‍🏍GitHub

You can connect with me in:

🧬LinkedIn

Resume 🧾

All this blog is using shell.

We will upload data to Datastore we can made this with Cloud SDK or directly from shell in GCP.

We will create a script in json_to_datastore.py to upload data from restaurants.json to Datastore.

If you want test this script, you can get example data in my repo that is above.

For more information and potential of this, click here.

1st - Create a program python ✍️

I use following command:

touch json_to_datastore.py

2nd - Write code to do transforms 🧵

Open IDE(I use vi) and put the following script

# Only import libraries
from google.cloud import datastore
import sys
import json
import os

def to_unicode(string):
    return unicode(string, "utf-8")

def get_values_json(jsondata):
    # Getting values of object root
    restaurant_id = jsondata.get("restaurant_id")
    name = jsondata.get("name")
    cuisine = jsondata.get("cuisine")
    borough = jsondata.get("borough")
    address = jsondata.get("address", {})
    # Getting values of object address
    zipcode = address.get("zipcode", '')
    street = address.get("street", '')
    building = address.get("building", '')
    coord = address.get("coord", [0, 0] )
    longitude = coord[0]
    latitude = coord[1]
    return restaurant_id, name, cuisine,borough, zipcode, street, building, longitude, latitude

def set_values_entity(restaurants, restaurant_id, name, cuisine,borough, zipcode, street, building, longitude, latitude):
    # Assign encoded values to key
    restaurants['id'] = to_unicode(restaurant_id)
    restaurants['name'] = to_unicode(name)
    restaurants['cuisine'] = to_unicode(cuisine)
    restaurants['borough'] = to_unicode(borough)
    restaurants['zipcode'] = to_unicode(zipcode)
    restaurants['street'] = to_unicode(street)
    restaurants['building'] = to_unicode(building)
    restaurants['longitude'] = longitude
    restaurants['latitude'] = latitude
    return restaurants

def convert():
    BASE_DIR = os.path.dirname(os.path.abspath(__file__))
    readFile = os.path.join(BASE_DIR, "restaurants.json")
    # Instantiates a client
    datastore_client = datastore.Client()
    # The kind for the new entity
    kind = 'restaurants'
    print ('\n' + 'Convert to Datastore - Please Wait' + '\n')

    # Set lines where will be row data
    lines = []
    # Open file restaurants.json
    with open(readFile, "r") as f:
        lines = f.readlines()

    # By each line of file we transform data and put it in entity in Datastore
    for line in lines:
        jsondata = json.loads(line)
        # Get values from json
        restaurant_id, name, cuisine,borough, zipcode, street, building, longitude, latitude = get_values_json(jsondata)
        # The Cloud Datastore key for the new entity
        rest_key = datastore_client.key(kind)
        # Prepares the new entity
        restaurants = datastore.Entity(key=rest_key)
        # Set values to entity
        restaurants = set_values_entity(restaurants, restaurant_id, name, cuisine,borough, zipcode, street, building, longitude, latitude)
        # Saves the entity
        datastore_client.put(restaurants)

    print ('\n' + 'Saved Rows to csv' + '\n')

if __name__ == '__main__':
    convert()

3rd - Run script 🙈

We need restaurants.json is in same path of script.

We run script with following command:

python3 json_to_datastore.py

UALAAAAA! 🎩🪄
You now have data in your instance of Datastore...

4th - Say thanks, give like and share if this has been of help/interest 😁🖖

How create cluster in Dataproc and install Jupyter?

Lucas M. Ríos — Thu, 16 Feb 2023 20:07:16 +0000

💠 Simple way to start a cluster in Dataproc from GCP.

🔗Related content

You can find repo related in:

🐱‍🏍GitHub

You can connect with me in:

🧬LinkedIn

Resume 🧾

All this blog is using shell.

We will start a cluster in Dataproc where we need first have SDK's Google and have instanced a bucket in Cloud Storage.

For more information and potential of this, click here.

Pre-Conditions:

You need have installed SDK. You can do that following next steps Note: I set alias to use SDK like msdk.
You need have instanced a bucket. You can do that following next steps Note: I set a bucket called xlmriosx_bucket.

1st - Set a name to cluster ✍️

We assign to a variable the name that will have cluster.

I use following command:

CLUSTER_NAME="xlmriosx_cluster"

2nd - Create bucket 💠

Using SDK installed with alias like msdk I use following command to create a cluster:

With alias:

msdk gcloud beta dataproc clusters create $CLUSTER_NAME \
    --optional-components=ANACONDA,JUPYTER \
    --image-version=$IMAGE_VERSION \
    --enable-component-gateway \
    --region=$REGION \
    --num-workers=$NUM_WORKERS \
    --master-machine-type=$MASTER_MACHINE_TYPE \
    --worker-machine-type=$WORKER_MACHINE_TYPE \
    --bucket=$BUCKET_NAME \
    --tags=$TAGS

Where:
$CLUSTER_NAME -> Is name that will have your cluster. In this case xlmriosx_cluster.

$IMAGE_VERSION -> Is version of image that will we use. Ex.: 1.3

$REGION -> Region where you want stay your storage bucket. Ex.: us-central1

$NUM_WORKERS -> Is amount of workers or nodes that will process data. Ex.: 2

$MASTER_MACHINE_TYPE -> Is type of machine that will use our master machine. Ex.: n1-standard-1

$WORKER_MACHINE_TYPE -> Is type of machine that will use our workers machine. Ex.: n1-standard-1

$BUCKET_NAME -> Is name of our cluster where will use like Source and Destination. Ex.: xlmriosx_bucket

$TAGS -> Is a tag to know about what is the process that make cluster. Ex.: datascience

3rd - Verify that your bucket was created ✅

We will list clusters to verify that was created:

gcloud beta dataproc clusters list --region=$REGION

OUTPUT:
| NAME | WORKER_COUNT | PREEMPTIBLE_WORKER_COUNT | STATUS | ZONE | SCHEDULED_DELETE |
| --- | --- | --- | --- | --- | --- |
| xlmriosx_cluster | 3 | - | - | RUNNING | us-central1-c |

4th - Open the Jupyter notebook in your local browser 🕸

See Viewing and Accessing Component Gateway URLs to click Component Gateway links on the Cloud Console to open the Jupyter notebook and JupyterLab UIs running on the cluster's master node in your local browser.

4th - Open the Jupyter notebook in your local browser 🕸

5th - Say thanks, give like and share if this has been of help/interest 😁🖖

How create and upload content in Cloud Storage GCP?

Lucas M. Ríos — Thu, 09 Feb 2023 21:46:44 +0000

🛢 Simple way to create a bucket in Cloud Storage from GCP.

🔗Related content

You can find repo related in:

🐱‍🏍GitHub

You can connect with me in:

🧬LinkedIn

Resume 🧾

All this blog is using shell.

Using SDK's Google we will use tool to manage Cloud Storage. In this blog we will create a storage and upload content in it.

For more information and potential of this, click here.

Pre-Conditions:

You need have installed SDK. You can do that following next steps Note: I set alias to use SDK like msdk. ---

1st - Set a name to bucket ✍️

We assign to a variable the name that will have bucket.

I use following command:

BUCKET_NAME="xlmriosx_bucket"

2nd - Create bucket ✅

Using SDK installed with alias like msdk I use following command to create a bucket:

With alias:

mdsk gsutil mb -p $PROJECT_NAME -c $BUCKET_TYPE -l $REGION gs://$BUCKET_NAME

Without alias:

docker run --rm --volumes-from gcloud-config gcr.io/google.com/cloudsdktool/cloud-sdk gsutil mb -p $PROJECT_NAME -c $BUCKET_TYPE -l $REGION gs://$BUCKET_NAME

Where:
$PROJECT_NAME -> Is name of the project where you wnat instance a bucket.

$BUCKET_TYPE -> Is type of storage you want, like standard, nearline, coldline, archive.

$REGION -> Region where you want stay your storage bucket.

OUTPUT:

Creating gs://xlmriosx_bucket/...

3rd - Verify that your bucket was created ✅

We will list buckets to verify that our bucket was created:

With alias:

docker run -ti --name gcloud-config gcr.io/google.com/cloudsdktool/cloud-sdk gcloud auth login

Without alias:

msdk gsutil ls gs://$BUCKET_NAME

4th - Upload content in bucket ⬆️

We can copy content to bucket with the following command:

msdk gsutil cp $DATA gs://$BUCKET_NAME

OUTPUT:

Copying file:///$DATA
[Content-Type=X]...
- [1 files][    Y.0 B/    Y.0 B]
Operation completed over 1 objects/Y.0 B.

Where:
$DATA -> Directory or file that we want upload in the bucket.

5th - Say thanks, give like and share if this has been of help/interest 😁🖖

How install Cloud SDK Docker image from GCP?

Lucas M. Ríos — Tue, 31 Jan 2023 23:57:19 +0000

🧰 Simple way to install SDK from GCP.

🔗Related content

You can find video related in:

📺YouTube

You can find repo related in:

🐱‍🏍GitHub

You can connect with me in:

🧬LinkedIn

Resume 🧾

I will install Docker image to use it as a SDK in Console GCP.

For more information and potential of this, click here.

1st - Pull Docker image ⬇

To use the image of the latest Cloud SDK release we will need pull image from repository in console.

I use following command:

CONSOLE:

docker pull gcr.io/google.com/cloudsdktool/cloud-sdk:latest

OUTPUT:

latest: Pulling from google.com/cloudsdktool/cloud-sdk
50e431f79093: Pull complete
6005b060ba7d: Pull complete
c49059196e30: Pull complete
Digest: sha256:fd9985597827057effdb04fd1b07db9463f4a00ac24684cd3726c05e146eafa1
Status: Downloaded newer image for gcr.io/google.com/cloudsdktool/cloud-sdk:latest
gcr.io/google.com/cloudsdktool/cloud-sdk:latest

2nd - Verifying installation ✅

Verify the installation (if you've pulled the latest version) by running the following command:

CONSOLE:

docker run gcr.io/google.com/cloudsdktool/cloud-sdk:latest gcloud version

OUTPUT:

Google Cloud SDK 284.0.0
alpha 2020.03.06
app-engine-go
app-engine-java 1.9.78
app-engine-python 1.9.88
app-engine-python-extras 1.9.88
beta 2020.03.06
bigtable
bq 2.0.54
cbt
cloud-datastore-emulator 2.1.0
core 2020.03.06
datalab 20190610
gsutil 4.48
kubectl 2020.03.06
pubsub-emulator 0.1.0

3rd - Authenticate with the gcloud 👤

We need authenticate to use gcloud tools. We can do that by the following command:

CONSOLE:

docker run -ti --name gcloud-config gcr.io/google.com/cloudsdktool/cloud-sdk gcloud auth login

Note: Once we've authenticated successfully, credentials are preserved in the volume of the gcloud-config container.

OUTPUT:

Go to the following link in your browser:

    https://accounts.google.com/o/oauth2/auth?...


Enter verification code: ********

You are now logged in as [xlmriosx@gmail.com].
Your current project is [xlmriosx].  You can change this setting by running:
  $ gcloud config set project PROJECT_ID

4th - Creating an alias 🔤

For simplicty create an alias by running:

CONSOLE:

alias csdk='docker run --rm --volumes-from gcloud-config -v ~/:/home/shared gcr.io/google.com/cloudsdktool/cloud-sdk'

You can change if you want. In my case I prefer 'csdk' like a nice alias.

5th - List your projects 📄

We can list projects by two forms a large and a short.

Large:
CONSOLE:

docker run --rm --volumes-from gcloud-config gcr.io/google.com/cloudsdktool/cloud-sdk gcloud projects list

Short:
CONSOLE:

csdk gcloud projects list

OUTPUT:

PROJECT_ID              NAME              PROJECT_NUMBER
xlmriosx                xlmriosx          783709003945

6th - Prove change of a project 🔀

We can change name of projects by two ways a large and a short.

Large:
CONSOLE:

docker run --rm --volumes-from gcloud-config gcr.io/google.com/cloudsdktool/cloud-sdk gcloud config set project $PROJECT_NAME

Short:
CONSOLE:

csdk gcloud config set project $PROJECT_NAME

OUTPUT:

Updated property [core/project].

7th - Say thanks, give like and share if this has been of help/interest 😁🖖

How working/install Pig with Notebooks?

Lucas M. Ríos — Fri, 27 Jan 2023 00:55:32 +0000

🐷📝 Basic commands to work with Pig in Notebooks

🔗Related content

You can find post related in:

📀Google Colab

You can find repo related in:

🐱‍🏍GitHub

You can connect with me in:

🧬LinkedIn

Resume 🧾

I will install Hadoop with Pig program and will use a library of Python to write a job that answer the question, how many row exists by each rating?

First I install Hadoop using same commands that I have used before but without put a number of step.

Install Hadoop 🐘

I use following command but you can change to get current last version:

!wget https://downloads.apache.org/hadoop/common/hadoop-3.3.4/hadoop-3.3.4.tar.gz

You would can get other version if you need in: https://downloads.apache.org/hadoop/common/ and later replace it in the before command.

Unzip and copy 🔓

I use following command:

!tar -xzvf hadoop-3.3.4.tar.gz && cp -r hadoop-3.3.4/ /usr/local/

Set up Hadoop's Java ☕

I use following command:

#To find the default Java path and add export in hadoop-env.sh
JAVA_HOME = !readlink -f /usr/bin/java | sed "s:bin/java::"
java_home_text = JAVA_HOME[0]
java_home_text_command = f"$ {JAVA_HOME[0]} "
!echo export JAVA_HOME=$java_home_text >>/usr/local/hadoop-3.3.4/etc/hadoop/hadoop-env.sh

Set Hadoop home variables 🏡

I use following command:

# Set environment variables
import os
os.environ['HADOOP_HOME']="/usr/local/hadoop-3.3.4"
os.environ['JAVA_HOME']=java_home_text

1st - Install Pig 🐷

I use following command but you can change to get current last version:

!wget https://downloads.apache.org/pig/pig-0.17.0/pig-0.17.0.tar.gz

You would can get other version if you need in: https://downloads.apache.org/pig/ and later replace it in the before command.

2nd - Unzip and copy 🔓

I use following command:

!tar -xzvf pig-0.17.0.tar.gz

3rd - Set Pig home variables 🏡

I use following command:

# Set environment variables
import os
os.environ['PIG_HOME']="/content/pig-0.17.0"
os.environ['PIG_CLASSPATH']="/usr/local/hadoop-3.3.1/conf"
os.environ["PATH"] += os.pathsep + "/content/pig-0.17.0/bin"

We can validate installation with command:

!pig -version

4th - Create a folder with HDFS 🌎📂

I use following command:

!/usr/local/hadoop-3.3.4/bin/hadoop fs -mkdir file:///content/data_pig

4.1 - Remove folder with HDFS ♻

Maybe, later you need remove it. To do that you must apply following command:

!/usr/local/hadoop-3.3.4/bin/hadoop fs -rm -r file:///content/data_pig

5th - Getting a dataset to anlyze with Pig 💾

I use a dataset from grouplens. You can get other in:
http://files.grouplens.org/datasets/

This time I use movieslens and you can download it using:

!wget http://files.grouplens.org/datasets/movielens/ml-100k.zip

To use data extract files. I extract files in path later of -d in command:

!unzip "/content/ml-100k.zip" -d "file:///content/data_pig"

For list them:

!/usr/local/hadoop-3.3.4/bin/hadoop fs -ls /content/data_pig/ml-100k

6th - Creating process to use Pig with Pig Syntax 🐖

To create job in Pig, you must see structure of dataset to configure jobs.
In this case we print dataset with following command:

!head /content/data_pig/ml-100k/u.data

I can get following information of dataset:

First column reference to userID.
Second column reference to movieID.
Third column reference to rating.
Fourth column reference to timestamp.

# Create pig script
%%writefile id.pig
/* id.pig */

student = LOAD 'file:///content/data_pig/ml-100k/u.data' USING PigStorage(' ')
   as (userId:int, movieId:int, rating:int, timestamp:int);

student_order = ORDER student BY rating DESC;

Dump student_order;

7th - Running the process 🙈

Here we run the process specifing some parameters:

Pig file program is id.pig
Dataset is in file:///content/data_pig/ml-100k/u.data

When run process, maybe take a few minutes...

You can run script with:

!pig -x local id.pig

But we run script and save results in a file .txt:

!pig -x local id.pig > results.txt

8th - Advancing in the logic of the scripts 😎

Now we will advance in logic of the script to get answer to next questions:

What are the oldest 5 star movies?
What are the worst movies?

8.1 - Find oldest 5 star movies start ⭐

%%writefile fiveStarMovies.pig

ratings = LOAD 'file:///content/data_pig/ml-100k/u.data'
    AS (userID:int, movieID:int, rating:int, ratingTime:int);
metadata = LOAD 'file:///content/data_pig/ml-100k/u.item' USING PigStorage('|')
    AS (movieID:int, movieTitle:chararray, releaseDate:chararray, videoRealese:chararray, imdblink:chararray);

nameLookup = FOREACH metadata GENERATE movieID, movieTitle, ToUnixTime(ToDate(releaseDate, 'dd-MMM-yyyy')) AS releaseTime;

ratingsByMovie = GROUP ratings BY movieID;

avgRatings = FOREACH ratingsByMovie GENERATE group as movieID, AVG(ratings.rating) as avgRating;

fiveStarMovies = FILTER avgRatings BY avgRating > 4.0;

fiveStarsWithData = JOIN fiveStarMovies BY movieID, nameLookup BY movieID;

oldestFiveStarMovies = ORDER fiveStarsWithData BY nameLookup::releaseTime;

DUMP oldestFiveStarMovies;

Run script and save results in a file .txt:

!pig -x local fiveStarMovies.pig > fiveStarMovies.txt

8.2 - Find most rated bad movies ⭐

%%writefile BadPopularMovies.pig

ratings = LOAD 'file:///content/data_pig/ml-100k/u.data'
  AS (userID:int, movieID:int, rating:int, ratingTime:int);
metadata = LOAD 'file:///content/data_pig/ml-100k/u.item' USING PigStorage('|')
    AS (movieID:int, movieTitle:chararray, releaseDate:chararray, videoRealese:chararray, imdblink:chararray);

nameLookup = FOREACH metadata GENERATE movieID, movieTitle;

groupedRating = GROUP ratings by movieID;

avgRatings = FOREACH groupedRating GENERATE group as movieID, AVG(ratings.rating) as avgRating, COUNT(ratings.rating) AS numRatings;  

badMovies = FILTER avgRatings BY avgRating < 2.0;

namedBadMovies = JOIN badMovies BY movieID, nameLookup BY movieID;

results = FOREACH namedBadMovies GENERATE nameLookup::movieTitle as movieName,
          badMovies::avgRating as avgRating, badMovies::numRatings as numRatings;

finalResults = ORDER results BY numRatings DESC;

DUMP finalResults;

Run script and save results in a file .txt:

!pig -x local BadPopularMovies.pig > BadPopularMovies.txt

9th - Say thanks, give like and share if this has been of help/interest 😁🖖

How install Jupyter?

Lucas M. Ríos — Thu, 19 Jan 2023 23:58:50 +0000

🌰 Simple step to install and use Jupyter in windows and Linux.

🔗Related content

You can find video related in:

📺YouTube

You can find repo related in:

🐱‍🏍GitHub

You can connect with me in:

🧬LinkedIn

Resume 🧾

I will install Jupyter in local machine with some simple steps.

Pre-Conditions:

Have Python

1st - Clone repository ⬇

We will need clone repository in our local machine.

I use following commando:

git clone https://github.com/xlmriosx/data-projects.git

2nd - Go to path where is script 🏄‍♂️

If your are in console and ran it the before command you can go to the path with following command:

cd data-projects
cd how-install-notebook

3rd - Run script 🙈

If you use Windows run the following script:
powershell.exe base-notebook.ps1

If you use Linux run the following script:
sh base-notebook.sh

4th - Say thanks, give like and share if this has been of help/interest 😁🖖

How working/install Spark with Notebooks?

Lucas M. Ríos — Mon, 16 Jan 2023 02:05:59 +0000

🌟📝 Basic commands to work with Spark in Notebooks like a Standalone cluster

🔗Related content

You can find post related in:

📀Google Collab

You can find video related in:

📺YouTube

You can find repo related in:

🐱‍🏍GitHub

You can connect with me in:

🧬LinkedIn

Resume 🧾

I will install Spark program and will use a library of Python to write a job that answer the question, how many row exists by each rating?

Before start we setup environment to run Spark Standalone Cluster.

1st - Mount Google Drive 🚠

We will mount Google Drive to can use it files.

I use following script:

from google.colab import drive
drive.mount('/content/gdrive')

2nd - Install Spark 🎇

Later got a Colab notebook up, to get Spark running you have to run the following script (I apologize for how ugly it is).

I use following script:

%%bash
apt-get install openjdk-8-jdk-headless -qq > /dev/null 
if [[ ! -d spark-3.3.1-bin-hadoop3.tgz ]]; then
  echo "Spark hasn't been installed, Downloading and installing!"
  wget -q https://downloads.apache.org/spark/spark-3.3.1/spark-3.3.1-bin-hadoop3.tgz
  tar xf spark-3.3.1-bin-hadoop3.tgz
  rm -f spark-3.3.1-bin-hadoop3.tgz

fi
pip install -q findspark

You would can get other version if you need in: https://downloads.apache.org/spark/ and later replace it in the before command.

3rd - Setting environment variables 🌐

I use following command:

import os
os.environ["JAVA_HOME"] = "/usr/lib/jvm/java-8-openjdk-amd64"  
os.environ["SPARK_HOME"] = "/content/spark-3.3.1-bin-hadoop3"
import findspark
findspark.init()

4th - Configuring a SparkSession 🚪

I use following command:

from pyspark.sql import SparkSession
spark = SparkSession.builder \
    .master("local[*]") \ # set up like master using all(*) threads
    .appName("BLOG_XLMRIOSX") \ # a generic name
    .getOrCreate()
sc = spark.sparkContext
sc

Later of execute this we can use many data strcuture to manage data, like RDD and Dataframe(Spark and Pandas).
Exists one more but only exists in Scala and it is Dataset.

5th - Getting a dataset to anlyze with Spark 💾

I use a dataset from grouplens. You can get other in:
http://files.grouplens.org/datasets/

This time I use movieslens and you can download it using:

!wget http://files.grouplens.org/datasets/movielens/ml-100k.zip

To use data extract files. I extract files in path later of -d in command:

!unzip "/content/ml-100k.zip" -d "/content/ml-100k_folder"

6th - Configuring data to analyze 💿

We create a variable called data where put path where data is like:

data = '/content/ml-100k_folder/ml-100k/u.data'

Later define a variable called df_spark where put information of data:

df_spark = spark.read.csv(data, inferSchema=True, header=True)

We can inspect type of variable df_spark like:

print(type(df_spark))

We can inspect data frame of variable df_spark like:

df_spark.show()

We can see format is incorrect so we will fix that where we configure format of data by following way:

df_spark = spark.read.csv(data, inferSchema=True, header=False, sep="\t")

9th - Making a query 🙈

To make this we need know format of data, so I infer the following structure:

First column reference to userID.
Second column reference to movieID.
Third column reference to rating.
Fourth column reference to timestamp.

I will answer the question, how many movies by each rating exists...

9th - Making a query with SQL syntax ❔🛢

First, create a table with dataframe.

df_spark.createOrReplaceTempView("table")

Second, can make query to answer question.

sql = spark.sql("SELECT _c2, COUNT(*) FROM table GROUP BY _c2")

To see results:

sql.show()

9th - Making a query with Dataframe ❔📄

It's so easy make this query with dataframe.

dataframe.groupBy("_c2").count().show()

9th - Making a query with RDD ❔🧊

First, we transform dataframe to rdd type.

rdd = df_spark.rdd

Second, make query with RDD functions.

rdd\
.groupBy(lambda x: x[2])\
.mapValues(lambda values: len(set(values)))\
.collect()